News: 1594019714

Good creative thinking, I know it's only Monday morning but have one on me -->

I think they needed something a wee bit stronger after that.

Where is the whisk(e)y icon?

"and accidentally deleted the Sybase master device file while it was running."

I did something similar on a DB/2 for OS/2 installation on a customer site ironically whilst doing a roll-forward recovery after a database corruption. Some quick thinking got me out of a hole - connected to another of the customer's sites and copied the relevant file off that server, as the builds were identical. The connection... Lan Netview Managment Utilities to remote server at 9600baud

There are much better ways of recovering a deleted open file, than crashing the system and hoping fsck recovers it. I did it the other day on Linux when I deleted an open log file, it wasn't very important but I got it back anyway. I believe even on Solaris the file handle will be under /proc//, and a quick google shows the fsdb command will help in this situation.

Assuming this is a warning from the gods about impending doom, I'm going to have a play with debugfs to see how this is done (hopefully before I need to use it in anger!)

This is assuming that a) there is such a thing as Google, when you do this b) you think about looking in /prod/ etc.

In the middle of the night, in a time before Google or other major search engines, you were left to your own devices and what you could remember from reading the f'ing manual.

I was thinking the same thing. However, if you need 45 minutes to figure all that out it's risky. Anything can happen in that time causing more permanent damage. If you know how to do it on top of your head it surely is the better option, but if it's going to take time to figure out it quickly becomes scary...

At least it was an old enough system to be not running ZFS. No fsck there, ZFS is too perfect to need it...

I got called out to help a "production down" problem during an upgrade. I was trying to help over the phone, and not being able to see what was going on was a major problem.

The instructions were clear.

1 Delete the following files config1,config2 etc

2 Recreate the system

3 Enter the config data when asked.

What could go wrong?

I got called at step 3. "Where is the config info we have to enter?"

"It is in config1"

"You mean the file we just deleted?"

They could not recover the file from the backups - because they were not authorised.

We eventually found the data because someone has copied all of the config files into one place for education.

We changed the instructions from "delete..." to "rename... " and added "step 0 - print out...".

Surely step 1 was rename and also copy off box?

It is mildly worrying how many things are resolved thanks to someone having a copy of the data sat in their personal filesystem.

At one point, I had a minion sheepishly come up to me (while I was talking a new minion through their first day of the job, entertainingly/ironically) and sheepishly announce that they'd deleted something which Really Shouldn't Have Been Deleted.

And while there were daily backups, the data was being constantly updated, so would require a fair amount of work to rebuild from the "last known good" state.

Thankfully, someone had just that very morning cloned the data for testing purposes, so we were able to use that to restore 90% of missing data, leaving my very subdued minion with just an hour or two of hard graft to finish the cleanup.

Fun, fun fun! Thankfully, it didn't scare the new minion off :)

The drama is palpable in this one, likely because I’ve been in a few (differing technologies each time) similar situations myself.

Great save!!!

And the relief can be sensed even this far, geographically and temporally from the incident.There but for the grace of... I suspect many readers will be thinking

Which reminds me, I was supposed to do a backup run yesterday

"There but for the grace of God go I."

I managed to forget which drives I had my OS installed on for my home server and happily nuked the raid array whilst finishing up adding all 12, 600gb drives I've bought (they're getting pricy secondhand now, not impressed).

Thankfully, it was just windows 2012 r2 and I hadn't finished playing with it yet enough ot put anything sensitive.

So it's now running Ubuntu server and couldn't be happier with only cli commands.

You can do just as many daft things with Ubuntu, but there is much less cllicking involved!!

On the one hand, you've got the technician that knows the system, makes a mistake, analyzes the situation correctly, finds the loophole and re-establishes functionality without any major hiccup. Hair-raising to be sure, heavy implications for failure, but in the end his in-depth knowledge allowed him to gracefully recover from the error.

Then, on the other hand, you've got the blithering idiot that knows just enough to make himself dangerous, has no idea of the consequences of his actions, and will be totally incapable of recovering anything.

I know who I'd prefer working with.

Good article.

"overconfident DBA"

The first requirement of a DBA is paranoia.

Though this is true, I have been at the end of a paranoid rant from the DBA at a certain (no longer exists) insurance company. In the course of this he accidentally gave away that his paranoia was due to the company having provided totally inadequate disk space and backup capability.

The next day I moved my car insurance to another company.

A friend of mine who was doing his PhD and programming PCs in Prolog to do something to do with analysing people's understanding of skin diseases (shades of The Singing Detective) took it on himself to 'tidy up' a sparcstation 2 that was about to replace something ancient we'd been using as a fileserver and host for some early experiments in website design (this is 1990 or so). For some reason he decided (as root) to delete /dev as it seemed to be full of lots of useless empty files. Not a good idea.The machine was connected to a network and the console was running the SunOS 4.1 GUI with a terminal open. I did not know much more than my friend, but I did have my own Sun 3/60 and I'd been on a short course for scientists who had to deal with new-fangled workstation things. I have forgotten how I did it, but armed with my trusty SERC 'how to be a unix system admin' manual that came with the two day course I'd done, I managed to retrieve everything. In the land of the blind the one-eyed man is king.

'how to be a unix system admin'

Manual?

Why, when we got our SPARCstations, we could only *dream* of a manual. We were lucky to get all the pieces, and the mouse pad.

Data General, ca. 1990. The decision had been made that the in-house MV machines and their in-house written CAD system were an expensive dead end for the engineering staff, so Suns and Viewlogic, it was. Got to name my own system, was my own sysadminnand it was visible from The Internet, because -- no nasties. Mr Morris and his worm, Canter and Siegel were all far in the future. Learn UNIX or sink, and learn, we did. Among other things, we found Usenet, and comp.os.minix

Thanks for the memories!

Otherwise known as Microsoft SQL Server from 1993.

Remember when it was delivered on two or three 5 1/4 diskettes for some flavour of OS/2...

Once, many decades ago, I was sysadmin for my company's three Sun-3 workstations (one diskless node, and two disked nodes). One morning, fairly early before my brain had fully booted, I decided to clean up the cruft in /tmp (it had got pretty full and the OS was complaining on occasion). So I loggedin as root and typed the immortal command "rm -rf / tmp". I then reaslised about the significant space in the command and hit "control-C" PDQ, but not quite Q enough to stop half of /user from having been deleted. I subsequently 'fessed to my manager and spent the rest of morning rebuilding the workstation.

Way back in the 90's I did POS support for a couple of clients in Pretoria.

One difficult client had all the bells and whistles - shiny new Novell 3.12 plus a couple of DOS workstations and a Windows 3.1 workstation for himself. And 120Mb tape drive.

Laughably small in today's terms of Giga- and Terabytes. Anyway.

He made a backup for the day, put it in the safe with the other backup tapes, locked the safe and went home for the weekend.

Come Monday morning we received a frantic call from him - ne'er-do-wells happened during the weekend, they took the file server, workstations and safe (including the backup tapes). So he had nothing to fall back on, and he was due for a SARS (income tax) revenue. Ouch.

That happened to a mate a few years ago, also South African (although he was in Fulham at the time). Scrotes broke into his flat and took his backup disks as well as his laptop.

"A master device file"?

It's six of the best and off to PC reeducation camp for you.

" Did you really say... "A master device file"? "

Out of interest, what does MBR stand for nowadays? Main boot record?

I am cursed with having to work with people who, when a filesystem goes full, go gzip happy.

Java file? Ohh, that's big. I'll gzip it. Queue me looking around trying to see what was done because now the billing system is falling over with a Java error.

Log file? Ohh, that's big. I'll gzip it. Log rotation breaks.

LOG file? Ohh, there's a few. I'll gzip the lot. Who needs LDAP anyway?

Pesky files under /usr/lib64? I have people who can deal with that. Who needs a running system anyway.

Not as destructive as deleting, but still.....

Eons ago, we had the monitoring system reporting Oracle log file systems to be partly full.

And operators zipping the 2 files in there.

And Oracle DB not being happy about it :)

LOL, Oracle DB was quite robust back then !

Recently had a fun time with backing data up. A semi competent computer user asked me to help him get a Linux instead going on one of his PCs. Having set up a previous machine of this with Linux in the past, I said sure, helped him decide what distro he wanted and came over armed with a live USB.

I checked with him that he had everything important backed up, he had. So I plug the USB stick in and let it run the pre configured install while we talk.

I walk him through the steps so he can do it himself next time, including the part where the drive gets repartitioned.

At the end I said we were ready to put his backed up stuff on the machine and asked him for it. D:\backup he says. He'd created a new partition to put his back up on. It was of course nuked during the install.

I've learned to always ask for the backup before even starting the machine.

My approach for this is to tell the aspiring Linux user to buy another disk off Amazon. They're well under $100 and will arrive the next day.

I then carefully disconnect and remove his existing OS disk, replacing it with the newly purchased one, on which I do a clean install of Linux.

We can (or not) copy his files off the old drive, which then remains "on the shelf", in case he has a change of heart and decides Linux is not for him. This also provides him with the comforting knowledge (he can see it right there on the shelf) that his decision to try Linux is completely reversible. Very handy when converting a less-than-knowledgeable friend or family member from Windows to Linux (for reduced incidence of service calls)

I do this for myself as well, when upgrading every few years. Handy to have a complete backup drive, which was perhaps getting long in the tooth, and a fresh new, (often faster and larger) drive for the new install.

Don't fib. That wasn't a SMALL prayer.

It was 1996, one of those "fix windows " programs deleted a file Windows needed to boot the GUI. I went into "DOS MODE" and found a backup. I renamed the *.bak" file to "*.bat" then rebooted. It worked and saved us having to buy a Windows 95 install CD, since the machine didn't have one as it came with Windows 95 pre-installed.

I was 13 at the time and a totally noob with computers, who do you think had run that "fix windows" program in the first place?