News: 1731310211

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

The sad tale of the Alpha massacre

(2024/11/11)


who, me? Good morning and welcome, once again, to Who, Me? in which Register readers share tales of tech support moments they might prefer to forget. But forgetting is not a way to learn from mistakes, is it?

This week's hero is a veteran we'll Regomize as "Gandalf" for such is the grayness of his beard nowadays. Back in less gray years, though, Gandalf worked for a significant – but now defunct – database maker.

Primary development was carried out on SunOS, but the developer also maintained releases for HP-UX, AIX, Tru64, Siemens Nixdorf UNIX, SCO UNIX, Linux and Windows NT among other exotic operating systems. Gandalf was on the porting team, whose mission was to back-port releases and fixes to the relevant platform branch, run the full suite of tests, and prepare software for release.

[1]

Got that so far? Good.

[2]

[3]

To aid in this endeavor, the team had a set of Quality Assurance tools for each specific database version for each platform. To test, they would simply copy the relevant set of QA Tools from a shared directory and place it on the local filesystem. Then they would set the shell variable QATOOLS to point to the location of the QA Tools. For example: set QATOOLS=/opt/qatools/

The /qatools/ directory contained all the necessary binaries, configuration files, log file locations and other necessary information to test that particular product suite: $QATOOLS/bin, $QATOOLS/etc, $QATOOLS/incl, $QATOOLS/var.

[4]

Of course it was important to ensure that they were using the right tools for the right version of the database on the right system, so each time before they began testing the needed to ensure the old version of the toolset had been cleared. This will become important in a minute.

[5]Relocation is a complete success – right up until the last minute

[6]I made this network so resilient nothing could possibly go wro...

[7]Linux admin asked savvy scientist for IT help and the boffin blew it

[8]Compression? What's that? And why is the network congested and the PCs frozen?

One fine day, Gandalf and team were tasked with testing a new release of the Tru64 port on a DEC Alpha installation located remotely (in fact, it was in Menlo Park, California – a possibly unnecessary detail, but Gandalf included it so you may as well know). They logged in as root, and issued the command to clear out the past version of the tool kit:

rm -rf $QATOOLS/bin $QATOOLS/etc $QATOOLS/incl $QATOOLS/var

Shortly thereafter, Gandalf noticed that things stopped working. Most notably, the telnet connection (yes, we are well into the before times here) went dead and could not be revived.

Investigation revealed a genuinely horrific error: before typing in the very powerful command above, they had failed to point the QATOOLS variable at the correct location. Or indeed at any location.

[9]

If you recall anything about Alpha, you'll know what that command did. As Gandalf put it: "The resulting carnage was as swift as the DEC Alpha was powerful." And he recalled one of the graybeards at the time telling him: "That machine was dead before the sweat of your brow hit the keyboard!"

In short, everything was gone.

Thankfully the sysadmins in Menlo Park were able to reconstruct their devastated system and ultimately there were no serious repercussions for Gandalf or his team. Just an important lesson learned – and a heck of a war story gathered for the retelling.

Who Me? needs your war stories! Our mailbag is getting very low and dusty, so if you have a tale of tech support gone awry or lessons learned the hard way, please [10]click here to send an email to Who, Me? so we can possibly share your adventures on some future Monday morn. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2024/11/04/who_me/

[6] https://www.theregister.com/2024/10/28/who_me/

[7] https://www.theregister.com/2024/10/21/who_me/

[8] https://www.theregister.com/2024/10/14/who_me/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZzHj14p0bT2mC0zlRIeBuwAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



don't try this at home...

42656e4d203239

These days, of course, rm -rf / comes with a "are you really really sure?" notice or even "I can't do that right now, Dave" messages.

However sudo dd -if /dev/urandom -of /dev/sda does the job, some would argue, more efficiently/thoroughly and with no safety net.

I suppose you are less likely to type sudo dd... but now you know, if you didn't before, its going to be hard to resist at least once!

Icon cos I love seeing things burn and it was the season for that just recently!

Re: don't try this at home...

Anonymous Coward

Of course dd in the wrong direction is also a rite of passage for every sysadmin

Re: don't try this at home...

Tim99

Always, always do a "ls -l" before "rm -r" whatever...

A Non e-mouse

How many of us have typed "rm -rf *" without double checking where we are in the filesystem?

For me, fortunately, the system was so slow that it didn't delete much and I could easily restore the root partition from tape.

Korev

You're not a real sysadmin if you haven't learnt the hard way to use set -u ...

b0llchit

Real admins don't... they cry silently .

Joe W

Or worse, trying to get rid of the hidden files by doing "rm -rf .*"

Which matches, of course, all hidden directories.

And the current directory "."

And the parent directory ".."

Fun.

Korev

I once wrote so many files that rm -rf> crashed because all it does it chain together rm $FILE commands.

I soon learnt to tell Sun Grid Engine not write logs :)

Anonymous Coward

rsync is pretty good at dealing with that situation. Yes, I speak from experience.

rsync an empty directory into a full one (using --delete, natch) can also be quicker than rm

Paul Crawford

Almost...I did a chown -R user:user .* but thankfully not sudo so it was unable to mangle other home directories. Lesson learned...

Too much space

Pete 2

> How many of us have typed "rm -rf *"

Not quite. But one time (and one time only) I accidentally inserted a space in the worst possible place when clearing out a directory.

Here's what I meant: rm -rf /tmp

And this is how it turned out rm -rf / tmp and it goes without saying this was all done as root.

Whose bright idea was it to make the space bar so big?

Korev

It sounds like Gandalf was an Alpha male

Gandalf the Grey

Mast1

No, not an Alpha male until after he had turned white......

It's the sting in the tale that did it.

Vincent Ballard

$QATOOLS/bin, $QATOOLS/etc, $QATOOLS/incl, $QATOOLS/var Everything beyond this point was superfluous, because we could all tell exactly what was going to happen.

Joe W

To be honest, I thought there'd be a space sneaking in between the variable name and the "/bin"...

Paul Kinsler

I tend to put a trailing / on all my directories which can mitigate this sort of thing (unless already in /), thus (in bash) there would be the possibly missing

QATOOLS="/opt/qatools/"

and then

rm -rf ${QATOOLS}bin

Also, it just occurred to me that (again in bash) you could even try to remember to type (or script)

rm -rf ${QATOOLS:-QQQQ}bin

so that the rm shouldn't get anything ever, unless you really like directory names with Q in them:-)

munnoch

Yup, knew immediately when I read that line what was about to happen...

Always, ALWAYS, wrap up procedural things like this into a utility script, no matter how simple. Never try to wing it each time because you will fuck it up eventually no matter how good you are (or think you are).

When the script messes up due to unforeseen circumstances you can strengthen it ($?{VARNAME} syntax etc.) so the next guy is better protected.

And that is why...

GlenP

you always do a DIR, ls, SELECT * FROM or whatever before any significant delete command.

Re: And that is why...

Ken Shabby

I had to delete a directory with a name like &NAME&, no problem use backslashes, but muscle memory took over..

rm -rf /&NAME/&

Luckily I was not root, and the machine was slow, so I could kill it (warning messages gave me a clue). Dev machine, not too bad to get back together. I now use double quotes

Re: And that is why...

Mike007

Tip: tab completion will make the shell fill in the escaped characters correctly, and works with just a \ to distinguish "hello world" from "hello" by adding a \ after the hello and pressing tab.

It will also flag a lot of other errors by virtue of not working unless it matches an actual file, so is a useful habit to get in to.

Installing programs from C:/

AustinTX

I'm not saying it was good form, but sometimes I was just too lazy to make a /temp directory to unzip a program installer into, so I ran it from C:/

Mind you, I wouldn't let pkunzip fill my root with scores of files; this would just be to uncompress a monolithic executable installer.

Until I met the one which installed it's payload and then cleaned up by deleting everything in the install-from directory. You know. Everything.

Re: Installing programs from C:/

Flightmode

Back in either 1989 or 1990 I was visiting a friend, and he wanted to put a couple of files on a floppy (yes, a 5 1/4", following on from the discussion about what disks could legitimately be called "floppy" from a couple of weeks ago) for me to take home. He popped it in, did a dir a: to see that there was nothing important on there, and then a quick del *.*. As it took a few seconds to complete I could see the colour draining from his cheeks. He was on his dad's PC, and had just deleted all the files in his C:\ directory...

Luckily, his dad was an organized person so the only things in that folder were the things necessary to boot up, all work files were properly tucked away in subdirectories, so the biggest pain was recreating AUTOEXEC.BAT and CONFIG.SYS, including finding the right control characters and codepages to get his printer to print Scandinavian characters again (CP 850, echo Alt-135 (ç) > LPT1: in case any archaeologists trying to do just that find this post.). Luckily this was before either of us had a Soundblaster card (PCSPKR.DRV FTW) so we at least didn't have to go interrupt chasing.

Re: Installing programs from C:/

Richard 12

That is such a common issue that NSIS has an explicit warning to installer developers to absolutely never do that...

And this is why ...

Michael H.F. Wilkinson

Gandalf was scared of the power of the one ring

"One rm to rule them all"

I'll get me coat. The one with the grey, wide-rimmed hat, please. (not pointy, however, just my regular Tilley)

Re: And this is why ...

Korev

Have a merry time!

Re: And this is why ...

Korev

I'm just wondering, was this on a Tolkien Ring network?

I think...

Bebu sa Ware

if it were on a tru64 system QATOOLS=/usr/opt/qatools as I recall /opt was reserved for third party kernel packages.

Alway a startlingly good idea to script these things... even more startling if you don't as this then young wizard discovered .

Presumably his employer's database product was packaged using the native tools for each supported platform - setld kits in this case. I would have thought packaging the qatools software etc would have made sense as even setld -f -d ... couldn't come close to rm -rf / . I recall initially creating kits was tedious but definitely not rocket surgery.

Oddly enough I seem to recall Informix being one of the applications on the Associated Products CD.

Re: I think...

Richard 12

The trouble with scripting is remembering to verify all the prerequisites at the top of the script.

Far too many people... don't.

Because it's just a script.

The worst

Will Godfrey

is when you have dozens of identical non-networked machines to upgrade, only boredom sets in and your subconscious takes over. Then you forget that just one of these has the only copy of a unique bit of software... until just after you hit 'return'

Power corrupts. Absolute power is kind of neat.
-- John Lehman, Secretary of the Navy, 1981-1987