Scripted shortcut caused double-click disaster of sysadmin's own making
- Reference: 1696836494
- News link: https://www.theregister.co.uk/2023/10/09/who_me/
- Source link:
This week meet "Ricardo" who spent some time in the 1990s working for a large management consultancy business. One of the big items on Ricardo's tech support to-do list was setting up laptops for deployment to clients. This task was repetitive and could sometimes involve configuring a lot of laptops in a single batch.
Naturally he'd configured CD-ROM containing a standard OS image, so the job basically did itself once he inserted the disc, ran a script, and clicked “OK” on a few prompts. The contents of the target machine’s drive would be overwritten by the contents of the CD within minutes.
[1]
Over time our hero realized that many if not all of those prompts were pro-forma and he was essentially wasting precious seconds clicking "OK" when the answer was never anything else. So he refined the script so that the prompts were suppressed – all of them – and all he had to do was double-click and then get a hot chocolate while waiting for the install to complete.
[2]
[3]
Efficiency, thy name is Ricardo.
One afternoon he was making a tweak or two to his brilliantly affective script on his production machine before burning a new version of the CD-ROM, when he remembered a small change he'd intended to make. Without even thinking, he double-clicked on the script …
[4]
Er, should that have been "right-clicked on the script?"
Yeah, it should have been. But it wasn't.
[5]Lost your luggage? That's nothing – we just lost your whole flight!
[6]No, no, no! Disco joke hit bum note in the rehab center
[7]Chap blew up critical equipment on his first day – but it wasn't his volt
[8]Watt's the worst thing you can do to a datacenter? Failing to RTFM, electrically
Rather than right-clicking on the script and selecting "Edit" to make his small change, Ricardo had instead executed the script. On his production machine. The machine that stored all of his carefully constructed scripting – not to mention absolutely everything else he needed to do his job.
And of course there was no "Are you sure Y/N?" to save Ricardo's skin, was there? With a fraction of a second, the script merrily started eating away the hard drive … and Ricardo's livelihood.
In desperation, our hero even pulled the plug out of the wall to stop the script from completing its mission, but too late. The master boot record was gone, and with it Ricardo's dignity.
[9]
Of course it was just one machine, and not a server or anything – Ricardo's only victim was himself. And of course there were backups for everything but the most recent work, so it wasn't too big a disaster.
But when Ricardo rebuilt the script, he left in at least one prompt to accept before wiping the disk. Lesson learned.
Ever learned an important lesson the very, very, hard way? Tell us about it in [10]an email to Who, Me? and we'll share your learnings with your colleagues. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZSPPRCjhyqvi0O9XWgwi4gAAAI4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZSPPRCjhyqvi0O9XWgwi4gAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZSPPRCjhyqvi0O9XWgwi4gAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZSPPRCjhyqvi0O9XWgwi4gAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/10/02/who_me/
[6] https://www.theregister.com/2023/09/25/who_me/
[7] https://www.theregister.com/2023/09/18/who_me/
[8] https://www.theregister.com/2023/09/11/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZSPPRCjhyqvi0O9XWgwi4gAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Lesson learned
Yeah, he learnt to be more careful whilst doing CD things in the office...
Re: Lesson learned
Wouldn't that be, lesson wiped?
"we'll share your learnings with your colleagues."
"Learnings" - yes from orbit -->
Yes
Learnt that for exactly the same reason.
Is there anyone
With more than a few days' experience in IT that hasn't learned that kind of hard lesson?
My own was back in DOS and Netware days (thankfully also on a workstation and not a server):
Format c:
Are you sure?
Of course I am bloody sure! I know what I'm doing ffs!
Oh hang on...that was C: not D:
Ah crap.
Re: Is there anyone
Memories of a colleague back in the day, when we used 3.5" floppies quite extensively. He was muttering and grumbling and someone asked him what the problem was. He said that all he was trying to do was format a floppy disk so he could re-use it, but his computer had started asking loads of "are you sure?" type questions. Unfortunately he made the final confirmation a split-second before someone asked "are you sure you didn't type 'format c:' ?"
Re: Is there anyone
Just about to post this very thing.
Oh so very cock sure, so very confident, so very fucking naïve!
Re: Is there anyone
I've recounted this before, back in the days when MS-DOS <> IBM Compatible Apricot used A: for the HD then B: for the floppy. Switching between different machines you'd go to format the floppy, type in FORMAT A:, Y to Are you sure, then oh sh*t (or other expletive). This was followed by frantic hitting of Ctrl-C and then reaching for the Norton UNDELETE utility which, fortunately, worked at that stage of the format. You had to know the first characters of the deleted filenames but the systems were a fairly standard setup.
Re: Is there anyone
Forty years ago - so memory is hazy - I was very inexperienced with the Basys newsroom system. I mistook a couple of commands and instead of just restarting the single user I had intended, I reset the entire system: newsroom, gallery, control room... at about five minutes before the broadcast.
Fortunately scripts were always printed, and most of them had been, and also fortunately the system restarted - just - in time. But the screams were loud...
flak jacket--->
Decades ago, in my first job as a software engineer, I was also asked to administer my company's Sun-3 workstations since I was the only person in the company who had any Unix experience. One day I started to get complaints about one of the machines not working properly, and when I investigated I found that /tmp was pretty much full (the SunOS installed assigned /tmp to its own partition, but when it filled up all sorts of applications stopped working). So, metaphorically putting my System Administer hat on, I logged on as "root" and entered the following command "rm -rf / tmp/*".
Note critical space in an unfortunate place. I certainly did and hit Control-C within a couple of seconds, but too late to save the OS. Fortunately, this was early in the working day and I had performed a routine backup the previous night, so I was able to get the installation tapes (yes, *tapes*) out of the cupboard, re-install the OS and re-load the backup. Downtime was only a couple of hours, but tense ones for me.
Lesson learned - always double check the commands you are entering as "root". Also, the next time this happened I used the command "cd /tmp && rm -rf *" - much safer!
cd /tmp && rm -rf *
Any time I have to enter rm -f * (let alone -rf) I do the cd first, then make sure I am where I think I am and that ls returns what I expect, before entering the rm.
I wouldn't have the balls to enter it all in one go as above.
The Sorceror's Apprentice
Reading the article I had an image of a young Mickey Mouse trying to stop enchanted broomsticks from interfering with the laptop.
It is mandatory to have a Oh No Oh Second experience
We all need to have had that sudden panic at least once in our lives, preferably on our own/test systems.
It teaches the fallibility of human condition in the clearest terms.
It is where we get our fear of things we don't fully understand and has us reaching for the manual, making n+1 backups and only when forced do we ignore this sagely wisdom "Draco dormiens nunquam titillandus"
Re: It is mandatory to have a Oh No Oh Second experience
"an Oh No" not "a Oh No"
Re: It is mandatory to have a Oh No Oh Second experience
The Ohnosecond, that mysterious time dilation quantum effect. Wen measured by any measuring device it is only registered as a tiny fraction of a second while the human just realizing the full levity of his fuck-up and the full consequences of the action just performed, experiences it as an eternity. Having all the time he or she needs to contemplate the meaning of life and their place in the cosmos.
"The master boot record was gone, and with it Ricardo's dignity."
Sums it up perfectly. And who amongst us has not had that sinking feeling at one time or another?
Re: "The master boot record was gone, and with it Ricardo's dignity."
Or the syncing feeling when a screwup has been automatically propagated to backups.
brilliantly affective
Nice use of language, made me smile, have one of these.
Are you sure
"Are you sure" wouldn't stop many people from making the same mistake.
Of course I am bloody sure! I just run it you muppet!
*proceeds to delete the "are you sure" prompt*
Confirmation bias
Was with someone in the early days of DOS. She put in the dreaded "del *.*" on her C: drive. But its OK. She realised the error "OH, God, I didn'y mean to do that".
DOS prompts "Are you sure?"
And she replies, whilst talking to me, "Yes I am sure I didn't mean to do that".
Sorry - But I did laugh
Lesson learned
Indeed.
I would even say, lesson engraved.