The sad tale of the Alpha massacre
- Reference: 1731310211
- News link: https://www.theregister.co.uk/2024/11/11/who_me/
- Source link:
This week's hero is a veteran we'll Regomize as "Gandalf" for such is the grayness of his beard nowadays. Back in less gray years, though, Gandalf worked for a significant – but now defunct – database maker.
Primary development was carried out on SunOS, but the developer also maintained releases for HP-UX, AIX, Tru64, Siemens Nixdorf UNIX, SCO UNIX, Linux and Windows NT among other exotic operating systems. Gandalf was on the porting team, whose mission was to back-port releases and fixes to the relevant platform branch, run the full suite of tests, and prepare software for release.
[1]
Got that so far? Good.
[2]
[3]
To aid in this endeavor, the team had a set of Quality Assurance tools for each specific database version for each platform. To test, they would simply copy the relevant set of QA Tools from a shared directory and place it on the local filesystem. Then they would set the shell variable QATOOLS to point to the location of the QA Tools. For example: set QATOOLS=/opt/qatools/
The /qatools/ directory contained all the necessary binaries, configuration files, log file locations and other necessary information to test that particular product suite: $QATOOLS/bin, $QATOOLS/etc, $QATOOLS/incl, $QATOOLS/var.
[4]
Of course it was important to ensure that they were using the right tools for the right version of the database on the right system, so each time before they began testing the needed to ensure the old version of the toolset had been cleared. This will become important in a minute.
[5]Relocation is a complete success – right up until the last minute
[6]I made this network so resilient nothing could possibly go wro...
[7]Linux admin asked savvy scientist for IT help and the boffin blew it
[8]Compression? What's that? And why is the network congested and the PCs frozen?
One fine day, Gandalf and team were tasked with testing a new release of the Tru64 port on a DEC Alpha installation located remotely (in fact, it was in Menlo Park, California – a possibly unnecessary detail, but Gandalf included it so you may as well know). They logged in as root, and issued the command to clear out the past version of the tool kit:
rm -rf $QATOOLS/bin $QATOOLS/etc $QATOOLS/incl $QATOOLS/var
Shortly thereafter, Gandalf noticed that things stopped working. Most notably, the telnet connection (yes, we are well into the before times here) went dead and could not be revived.
Investigation revealed a genuinely horrific error: before typing in the very powerful command above, they had failed to point the QATOOLS variable at the correct location. Or indeed at any location.
[9]
If you recall anything about Alpha, you'll know what that command did. As Gandalf put it: "The resulting carnage was as swift as the DEC Alpha was powerful." And he recalled one of the graybeards at the time telling him: "That machine was dead before the sweat of your brow hit the keyboard!"
In short, everything was gone.
Thankfully the sysadmins in Menlo Park were able to reconstruct their devastated system and ultimately there were no serious repercussions for Gandalf or his team. Just an important lesson learned – and a heck of a war story gathered for the retelling.
Who Me? needs your war stories! Our mailbag is getting very low and dusty, so if you have a tale of tech support gone awry or lessons learned the hard way, please [10]click here to send an email to Who, Me? so we can possibly share your adventures on some future Monday morn. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/11/04/who_me/
[6] https://www.theregister.com/2024/10/28/who_me/
[7] https://www.theregister.com/2024/10/21/who_me/
[8] https://www.theregister.com/2024/10/14/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: don't try this at home...
Of course dd in the wrong direction is also a rite of passage for every sysadmin
Re: don't try this at home...
Always, always do a "ls -l" before "rm -r" whatever...
How many of us have typed "rm -rf *" without double checking where we are in the filesystem?
For me, fortunately, the system was so slow that it didn't delete much and I could easily restore the root partition from tape.
You're not a real sysadmin if you haven't learnt the hard way to use set -u ...
Real admins don't... they cry silently .
Or worse, trying to get rid of the hidden files by doing "rm -rf .*"
Which matches, of course, all hidden directories.
And the current directory "."
And the parent directory ".."
Fun.
I once wrote so many files that rm -rf> crashed because all it does it chain together rm $FILE commands.
I soon learnt to tell Sun Grid Engine not write logs :)
rsync is pretty good at dealing with that situation. Yes, I speak from experience.
rsync an empty directory into a full one (using --delete, natch) can also be quicker than rm
Almost...I did a chown -R user:user .* but thankfully not sudo so it was unable to mangle other home directories. Lesson learned...
Too much space
> How many of us have typed "rm -rf *"
Not quite. But one time (and one time only) I accidentally inserted a space in the worst possible place when clearing out a directory.
Here's what I meant: rm -rf /tmp
And this is how it turned out rm -rf / tmp and it goes without saying this was all done as root.
Whose bright idea was it to make the space bar so big?
It sounds like Gandalf was an Alpha male
Gandalf the Grey
No, not an Alpha male until after he had turned white......
It's the sting in the tale that did it.
$QATOOLS/bin, $QATOOLS/etc, $QATOOLS/incl, $QATOOLS/var Everything beyond this point was superfluous, because we could all tell exactly what was going to happen.
To be honest, I thought there'd be a space sneaking in between the variable name and the "/bin"...
I tend to put a trailing / on all my directories which can mitigate this sort of thing (unless already in /), thus (in bash) there would be the possibly missing
QATOOLS="/opt/qatools/"
and then
rm -rf ${QATOOLS}bin
Also, it just occurred to me that (again in bash) you could even try to remember to type (or script)
rm -rf ${QATOOLS:-QQQQ}bin
so that the rm shouldn't get anything ever, unless you really like directory names with Q in them:-)
Yup, knew immediately when I read that line what was about to happen...
Always, ALWAYS, wrap up procedural things like this into a utility script, no matter how simple. Never try to wing it each time because you will fuck it up eventually no matter how good you are (or think you are).
When the script messes up due to unforeseen circumstances you can strengthen it ($?{VARNAME} syntax etc.) so the next guy is better protected.
And that is why...
you always do a DIR, ls, SELECT * FROM or whatever before any significant delete command.
Re: And that is why...
I had to delete a directory with a name like &NAME&, no problem use backslashes, but muscle memory took over..
rm -rf /&NAME/&
Luckily I was not root, and the machine was slow, so I could kill it (warning messages gave me a clue). Dev machine, not too bad to get back together. I now use double quotes
Re: And that is why...
Tip: tab completion will make the shell fill in the escaped characters correctly, and works with just a \ to distinguish "hello world" from "hello" by adding a \ after the hello and pressing tab.
It will also flag a lot of other errors by virtue of not working unless it matches an actual file, so is a useful habit to get in to.
Installing programs from C:/
I'm not saying it was good form, but sometimes I was just too lazy to make a /temp directory to unzip a program installer into, so I ran it from C:/
Mind you, I wouldn't let pkunzip fill my root with scores of files; this would just be to uncompress a monolithic executable installer.
Until I met the one which installed it's payload and then cleaned up by deleting everything in the install-from directory. You know. Everything.
Re: Installing programs from C:/
Back in either 1989 or 1990 I was visiting a friend, and he wanted to put a couple of files on a floppy (yes, a 5 1/4", following on from the discussion about what disks could legitimately be called "floppy" from a couple of weeks ago) for me to take home. He popped it in, did a dir a: to see that there was nothing important on there, and then a quick del *.*. As it took a few seconds to complete I could see the colour draining from his cheeks. He was on his dad's PC, and had just deleted all the files in his C:\ directory...
Luckily, his dad was an organized person so the only things in that folder were the things necessary to boot up, all work files were properly tucked away in subdirectories, so the biggest pain was recreating AUTOEXEC.BAT and CONFIG.SYS, including finding the right control characters and codepages to get his printer to print Scandinavian characters again (CP 850, echo Alt-135 (ç) > LPT1: in case any archaeologists trying to do just that find this post.). Luckily this was before either of us had a Soundblaster card (PCSPKR.DRV FTW) so we at least didn't have to go interrupt chasing.
Re: Installing programs from C:/
That is such a common issue that NSIS has an explicit warning to installer developers to absolutely never do that...
And this is why ...
Gandalf was scared of the power of the one ring
"One rm to rule them all"
I'll get me coat. The one with the grey, wide-rimmed hat, please. (not pointy, however, just my regular Tilley)
Re: And this is why ...
Have a merry time!
Re: And this is why ...
I'm just wondering, was this on a Tolkien Ring network?
I think...
if it were on a tru64 system QATOOLS=/usr/opt/qatools as I recall /opt was reserved for third party kernel packages.
Alway a startlingly good idea to script these things... even more startling if you don't as this then young wizard discovered .
Presumably his employer's database product was packaged using the native tools for each supported platform - setld kits in this case. I would have thought packaging the qatools software etc would have made sense as even setld -f -d ... couldn't come close to rm -rf / . I recall initially creating kits was tedious but definitely not rocket surgery.
Oddly enough I seem to recall Informix being one of the applications on the Associated Products CD.
Re: I think...
The trouble with scripting is remembering to verify all the prerequisites at the top of the script.
Far too many people... don't.
Because it's just a script.
The worst
is when you have dozens of identical non-networked machines to upgrade, only boredom sets in and your subconscious takes over. Then you forget that just one of these has the only copy of a unique bit of software... until just after you hit 'return'
don't try this at home...
These days, of course, rm -rf / comes with a "are you really really sure?" notice or even "I can't do that right now, Dave" messages.
However sudo dd -if /dev/urandom -of /dev/sda does the job, some would argue, more efficiently/thoroughly and with no safety net.
I suppose you are less likely to type sudo dd... but now you know, if you didn't before, its going to be hard to resist at least once!
Icon cos I love seeing things burn and it was the season for that just recently!