Techie pulled an all-nighter that one mistake turned into an all-weekender
- Reference: 1740988935
- News link: https://www.theregister.co.uk/2025/03/03/who_me/
- Source link:
This week, meet a reader we’ll Regomize as “Alejandro” who told us he once worked for a major multinational company and in the early 1990s was trained to perform a mainframe migration project. The plan was to bring a payroll app into the late 20th century by moving from a 1970s-vintage IBM box to a new NCR machine that ran Unix.
To make this happen, Alejandro trained for five weeks and made sure he was confident in Unix, as the old IBM box used one of Big Blue’s ancient Oses.
[1]
The project was described as a “remote unboxing” for the NCR machine, an odd term that Alejandro explained by telling us “The system was located about 1,000 miles away, in Colorado, and I had to install and configure all the system and user packages, user home directories, set up system security, initiate databases, inter-system file transfers, and so on.”
[2]
[3]
To do all that, he needed to use telnet to access the NCR Box. Readers of a certain age will recall that telnet was a tool that allowed remote access to virtual terminals.
Alejandro planned to use two virtual terminals for this job, so fired up telnet on a Friday night and got to work.
[4]
“I pulled an all-nighter and by Saturday evening I was done, and it was time to clean up,” he told Who, Me?
[5]Untrained techie botched a big hardware sale by breaking client's ERP
[6]Techie pointed out meetings are pointless, and was punished for it
[7]Does this thing run on a 220 V power supply? Oh. That puff of smoke suggests not
[8]CompSci teacher sets lab task: Accidentally breaking the university
At the top of the cleaning list was removing the packages he’d used during the installation process, which Alejandro knew were in a certain directory. With his newfound Unix skills he typed the command he thought would wipe only the temporary files.
But with almost 24 hours of unbroken work behind him, Alejandro was a long way up the directory tree. His rm rf * command was now busy wiping the entire system he’d just built.
“I was really tired and didn't think twice about it,” he told Who, Me? And then the telnet session he was using went pear-shaped.
“It stopped responding and some strange errors were popping up,” he told Who, Me?
[9]
The obvious thing to do was check the other telnet session, which he soon saw “was busy deleting files.”
“The memory of the command I had typed returned and I felt the ground sinking below me and my chest tightening,” Alejandro wrote.
Thankfully, Alejandro knew the chap who handled backup tapes in the Colorado office. He reinstalled the OS, leaving Alejandro to once again telnet in and install the apps.
“The job was finished before start of day on Monday and my boss did not fire me,” he recalled. “And I was very happy the weekend was over.”
Have you made a mistake that meant you had to re-do a job? If so, [10]click here to send us an email . The “Who, Me?” mailbag is in a sorry state and could really use your story! ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2025/02/24/who_me/
[6] https://www.theregister.com/2025/02/17/who_me/
[7] https://www.theregister.com/2025/02/10/who_me/
[8] https://www.theregister.com/2025/02/03/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z8WL1TK4FuHbq-6fef7KfQAAANU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Best training ever
... doubt he ever made that mistake again.
Sure. Except, this commentard sometimes has a flat learning curve...
Ouch!!!
Rules learnt the hard way #2: double check, and double check again before using rm -rf *
Rule #1 is of course: Do not act incautiously when confronting little bald wrinkly smiling men!
Doffs hat to the late, great sir Terry Pratchett
Re: Ouch!!!
Sleep deprivation does funny things to your brain. The other day i had a narrow escape, i ran a command i was going to pipe to a force delete, but because i wasn't 100% familiar with it i wanted to check the output first.
...30 seconds later...
hmm, this is taking longer than expected, what's up?
Turns out i typed /* rather than ./* i guess that's quite a few files to go through...
sad... they fixed that
rm -rf * has a safety net around it these days.
For those that care dd is now your friend (for special values of friend)
Re: sad... they fixed that
I don't trust any system to have that safety net in place to safe me. My approach is never to do a "rm
dd needs triple checking - way to easy to zero out your hard disk instead of the flash drive.
All nighter
In the past, to incentivise employee to pull all nighter to get that feature in before the deadline, was to pay them handsomely for it.
Now, to make employee come for all nighter is to pay them not enough for them to afford heating or even rent.
Punctual employee is the one who lives in their car, in company's car park.
Re: All nighter
Why does anyone think that an all-nighter is somehow a good thing, no matter what the pay is? It's pretty much guaranteed to leave you so tired you make stupid mistakes, just like Alejandro did.
If it's really that urgent (and it almost never is), put a team together and work shifts. Solo heroics are a recipe for disaster, and Alejandro's manager shouldn't have allowed him to do it.
Re: All nighter
This happens at companies hiring "rockstar" developers in fast-paced environment to work with dynamic teams.
Re: All nighter
Tell that to any bookkeeper, anywhere.
In addition, cutting staff reduces outgoings for which management gets rewards, whereas consequential failures are always blamed on the overworked staff left after that exercise.
If you want to fix that, find a way to rescind bonus payments when the consequences arrive. I wish you luck..
Re: All nighter
Allnighters mean night pay. It's cheaper to do stuff during office hours if possible.
Keeping a backup...
I was working on a COBOL based finance system in the late 1980s, running on a VAX. I booked out the code I needed to adjust (it was part of the preparations for the Y2K, turning 2-digit years into 4-digit years). I did some work and then had a brainwave...
So I stopped what I was doing copied the .cob files I was working on to .cobol, then went back to the .cob files and re-wrote the code. 2 days later, I had tested the code and the way I had made the changes was better than what I had originally started, so I deleted the .cob files... NO WAIT! GAH! I deleted the actual changes! I had to copy the .cobol back to .cob and start all over again. At least I knew what I had done the first time around, so it only took a few hours, not 2 days, to write the code again, in fact I came up with some other improvements along the way as well and the code was even more efficient...
- "-r" (Thankfully!)
One fine day my task was to install WordPerfect onto a multi-user Xenix system which had about 30 simultaneous users.
I was introduced to the client org's Big Boss' assistant, Todd, and was left to get on with things. I asked, "Todd, will you please show me your most-recent backup tape?"
Todd (in a paranoid, demanding tone): "Why?! What's wrong?! What happened?!"
Me (in a reassuring tone): "Nothing's wrong. I haven't even started yet. I just want to see the tape and check it out. If I had said, 'Todd, will you please get out your most-recent backup tape?', then you'd know something had gone wrong."
With suspicion and reluctance, Todd went into a little closet, opened up a cardboard box of DC-600A tape cartridges, and handed me one. It looked new and dust-free; the label had been written on only once, with a pencilled notation of yesterday's date. I popped it into the tape drive of the Xenix server, logged in, su 'd to root, rewound the tape, and did a tar -tv /dev/st0 . The tape drive made noises, and after a few monents a list of directories/files flew up the screen. All good. I popped the tape and handed it back to Todd.
The WordPerfect package included a pile of 3.5" diskettes and a beautifully-designed and printed manual. I turned to the installation section and got started.
There was a long list of steps, and part of me wanted to tick them off with a pencil after I completed each one, but I could not bring myself to deface the customer's beautiful manual.
Soldiering on, I did this and that, then (per instructions) removed all the files from a temp directory via rm * , then (per instructions) did a cd .. And for good measure, did -- not in the instructions -- a pwd and an ls .
I was at the filesystem root, and there were no files. Subdirectories, yes, but no files. I yanked my hands away from the keyboard so fast I and my wheelie chair rolled backwards all the way across the room. I listened for the screams of angry users, but heard none. I sat there thinking carefully, and concluded:
(0) I had lost my place in the instructions, and had cd 'd up to '/' from '/tmp' before I done the rm * ;
(1) The files in the filesystem root are needed only for the boot process. Once the OS kernel is in-core (well, 'in-RAM'), those files aren't needed again until the next reboot (I glanced over at the UPS quietly humming on the floor);
(2) I should be able to restore just the files in '/' and everything will be good again; and,
(3) From here on, I will mark up the client's beautiful manuals (in pencil)!
I found the Big Boss' assistant, and asked, "Todd, will you please get out your most-recent backup tape?"
Best training ever
I doubt he ever made that mistake again. Good that his boss kept him on.