News: 1740988935

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Techie pulled an all-nighter that one mistake turned into an all-weekender

(2025/03/03)


Who, Me? Well, would you look at the calendar? It’s Monday already, and by lunchtime any fond memories of the weekend will have been erased by work worries of the sort The Register celebrates each week in “Who, Me?” – the reader-contributed column that tells your stories of making messes and somehow escaping.

This week, meet a reader we’ll Regomize as “Alejandro” who told us he once worked for a major multinational company and in the early 1990s was trained to perform a mainframe migration project. The plan was to bring a payroll app into the late 20th century by moving from a 1970s-vintage IBM box to a new NCR machine that ran Unix.

To make this happen, Alejandro trained for five weeks and made sure he was confident in Unix, as the old IBM box used one of Big Blue’s ancient Oses.

[1]

The project was described as a “remote unboxing” for the NCR machine, an odd term that Alejandro explained by telling us “The system was located about 1,000 miles away, in Colorado, and I had to install and configure all the system and user packages, user home directories, set up system security, initiate databases, inter-system file transfers, and so on.”

[2]

[3]

To do all that, he needed to use telnet to access the NCR Box. Readers of a certain age will recall that telnet was a tool that allowed remote access to virtual terminals.

Alejandro planned to use two virtual terminals for this job, so fired up telnet on a Friday night and got to work.

[4]

“I pulled an all-nighter and by Saturday evening I was done, and it was time to clean up,” he told Who, Me?

[5]Untrained techie botched a big hardware sale by breaking client's ERP

[6]Techie pointed out meetings are pointless, and was punished for it

[7]Does this thing run on a 220 V power supply? Oh. That puff of smoke suggests not

[8]CompSci teacher sets lab task: Accidentally breaking the university

At the top of the cleaning list was removing the packages he’d used during the installation process, which Alejandro knew were in a certain directory. With his newfound Unix skills he typed the command he thought would wipe only the temporary files.

But with almost 24 hours of unbroken work behind him, Alejandro was a long way up the directory tree. His rm rf * command was now busy wiping the entire system he’d just built.

“I was really tired and didn't think twice about it,” he told Who, Me? And then the telnet session he was using went pear-shaped.

“It stopped responding and some strange errors were popping up,” he told Who, Me?

[9]

The obvious thing to do was check the other telnet session, which he soon saw “was busy deleting files.”

“The memory of the command I had typed returned and I felt the ground sinking below me and my chest tightening,” Alejandro wrote.

Thankfully, Alejandro knew the chap who handled backup tapes in the Colorado office. He reinstalled the OS, leaving Alejandro to once again telnet in and install the apps.

“The job was finished before start of day on Monday and my boss did not fire me,” he recalled. “And I was very happy the weekend was over.”

Have you made a mistake that meant you had to re-do a job? If so, [10]click here to send us an email . The “Who, Me?” mailbag is in a sorry state and could really use your story! ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2025/02/24/who_me/

[6] https://www.theregister.com/2025/02/17/who_me/

[7] https://www.theregister.com/2025/02/10/who_me/

[8] https://www.theregister.com/2025/02/03/who_me/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



Best training ever

TWB

I doubt he ever made that mistake again. Good that his boss kept him on.

Re: Best training ever

Evil Auditor

... doubt he ever made that mistake again.

Sure. Except, this commentard sometimes has a flat learning curve...

Ouch!!!

Michael H.F. Wilkinson

Rules learnt the hard way #2: double check, and double check again before using rm -rf *

Rule #1 is of course: Do not act incautiously when confronting little bald wrinkly smiling men!

Doffs hat to the late, great sir Terry Pratchett

Re: Ouch!!!

SVD_NL

Sleep deprivation does funny things to your brain. The other day i had a narrow escape, i ran a command i was going to pipe to a force delete, but because i wasn't 100% familiar with it i wanted to check the output first.

...30 seconds later...

hmm, this is taking longer than expected, what's up?

Turns out i typed /* rather than ./* i guess that's quite a few files to go through...

sad... they fixed that

42656e4d203239

rm -rf * has a safety net around it these days.

For those that care dd is now your friend (for special values of friend)

Re: sad... they fixed that

OhForF'

I don't trust any system to have that safety net in place to safe me. My approach is never to do a "rm " directly but first use "find " and if the result lists those (and only those) files and directories i expect i append "-print0 | xargs -0 rm -f" to the find command.

dd needs triple checking - way to easy to zero out your hard disk instead of the flash drive.

All nighter

elsergiovolador

In the past, to incentivise employee to pull all nighter to get that feature in before the deadline, was to pay them handsomely for it.

Now, to make employee come for all nighter is to pay them not enough for them to afford heating or even rent.

Punctual employee is the one who lives in their car, in company's car park.

Re: All nighter

Phil O'Sophical

Why does anyone think that an all-nighter is somehow a good thing, no matter what the pay is? It's pretty much guaranteed to leave you so tired you make stupid mistakes, just like Alejandro did.

If it's really that urgent (and it almost never is), put a team together and work shifts. Solo heroics are a recipe for disaster, and Alejandro's manager shouldn't have allowed him to do it.

Re: All nighter

elsergiovolador

This happens at companies hiring "rockstar" developers in fast-paced environment to work with dynamic teams.

Re: All nighter

Anonymous Coward

Tell that to any bookkeeper, anywhere.

In addition, cutting staff reduces outgoings for which management gets rewards, whereas consequential failures are always blamed on the overworked staff left after that exercise.

If you want to fix that, find a way to rescind bonus payments when the consequences arrive. I wish you luck..

Re: All nighter

Anonymous Coward

Allnighters mean night pay. It's cheaper to do stuff during office hours if possible.

Keeping a backup...

big_D

I was working on a COBOL based finance system in the late 1980s, running on a VAX. I booked out the code I needed to adjust (it was part of the preparations for the Y2K, turning 2-digit years into 4-digit years). I did some work and then had a brainwave...

So I stopped what I was doing copied the .cob files I was working on to .cobol, then went back to the .cob files and re-wrote the code. 2 days later, I had tested the code and the way I had made the changes was better than what I had originally started, so I deleted the .cob files... NO WAIT! GAH! I deleted the actual changes! I had to copy the .cobol back to .cob and start all over again. At least I knew what I had done the first time around, so it only took a few hours, not 2 days, to write the code again, in fact I came up with some other improvements along the way as well and the code was even more efficient...

- "-r" (Thankfully!)

An_Old_Dog

One fine day my task was to install WordPerfect onto a multi-user Xenix system which had about 30 simultaneous users.

I was introduced to the client org's Big Boss' assistant, Todd, and was left to get on with things. I asked, "Todd, will you please show me your most-recent backup tape?"

Todd (in a paranoid, demanding tone): "Why?! What's wrong?! What happened?!"

Me (in a reassuring tone): "Nothing's wrong. I haven't even started yet. I just want to see the tape and check it out. If I had said, 'Todd, will you please get out your most-recent backup tape?', then you'd know something had gone wrong."

With suspicion and reluctance, Todd went into a little closet, opened up a cardboard box of DC-600A tape cartridges, and handed me one. It looked new and dust-free; the label had been written on only once, with a pencilled notation of yesterday's date. I popped it into the tape drive of the Xenix server, logged in, su 'd to root, rewound the tape, and did a tar -tv /dev/st0 . The tape drive made noises, and after a few monents a list of directories/files flew up the screen. All good. I popped the tape and handed it back to Todd.

The WordPerfect package included a pile of 3.5" diskettes and a beautifully-designed and printed manual. I turned to the installation section and got started.

There was a long list of steps, and part of me wanted to tick them off with a pencil after I completed each one, but I could not bring myself to deface the customer's beautiful manual.

Soldiering on, I did this and that, then (per instructions) removed all the files from a temp directory via rm * , then (per instructions) did a cd .. And for good measure, did -- not in the instructions -- a pwd and an ls .

I was at the filesystem root, and there were no files. Subdirectories, yes, but no files. I yanked my hands away from the keyboard so fast I and my wheelie chair rolled backwards all the way across the room. I listened for the screams of angry users, but heard none. I sat there thinking carefully, and concluded:

(0) I had lost my place in the instructions, and had cd 'd up to '/' from '/tmp' before I done the rm * ;

(1) The files in the filesystem root are needed only for the boot process. Once the OS kernel is in-core (well, 'in-RAM'), those files aren't needed again until the next reboot (I glanced over at the UPS quietly humming on the floor);

(2) I should be able to restore just the files in '/' and everything will be good again; and,

(3) From here on, I will mark up the client's beautiful manuals (in pencil)!

I found the Big Boss' assistant, and asked, "Todd, will you please get out your most-recent backup tape?"

He didn't run for reelection. "Politics brings you into contact with all
the people you'd give anything to avoid," he said. "I'm staying home."
-- Garrison Keillor, "Lake Wobegone Days"