Funny, that: Handy script for wiping directories is capable of wreaking havoc beyond a miscreant's wildest dreams
- Reference: 1598858105
- News link: https://www.theregister.co.uk/2020/08/31/who_me/
- Source link:
"O" spent the early part of this century in the capable hands of Windows Server 2003, and was always keen to find new ways of keeping his systems ticking over: "Being the only experienced professional running the systems with a few 'independent consultants' who looked down at any really hard server work," he said, modestly, "there was always an area to improve."
The consultants managed only one of the company's five or so sites and while complexity was swerved where possible, they were "very vocal about any issue that made them look good".
The ad-hoc spaghetti of the gang's various solutions had been beaten into submission through "Quality Control" and "Change Management" although disk space woes remained a constant. The issue, recalled O, was "due to a surprisingly high resistance to RAID5 (and the crazy cost of those SCSI disks!)"
Upgrading the primary server (lurking in the same location as the consultants) was to be a simple job. O would fly in, plan, start and finish the task. Everyone was happy.
Well, not quite.
This PDP-11/70 was due to predict an election outcome – but no one could predict it falling over [2]READ MORE
"A new cluster with plenty of disk space seemed to draw out the complaints on disk space across the other regions," O explained. So he put together a script that could read a list of user-approved sacrificial directories when disk space was getting low, and wipe the things.
A cautious fellow, he also had the script check which server it was running on ("to prevent misfires!") and added a "reasonable amount of intelligence and automation".
As with so many things in the IT world, O's handy script sat in the background and was soon forgotten about as new nodes, BDCs and the like were added with the expansion of the company.
It wasn't until a planned weekend outage that the wheels first showed signs of coming off. O noted that some servers were blocking updates due to disk issues.
Not a problem. He remoted into the consoles of the servers, double-checked where he was and ran his handy clean-up script.
"I watched it start," he said, "then went off to do the other patching reboots."
The calls began eight hours later. A newly added BDC in the south of the country had died, showing winnt errors on reboot. Odd, but after fettling a boot disk (and thanks given to how the disk system, NTFS, managed Access Control Lists) things were soon back up and running.
"Initial root cause was bad disk," said O, "because of the lack of investment into RAID5."
Time passed, and even the BDC incident began to fade into memory. And then another call came in, this time from the site with the cluster and that team of consultants.
"A failover had happened, and things looked bad," explained O. "A PDC going down was bad news, but adding to the complexity was the main site and ... the consultants"
They looked at the problem and helpfully determined that, no, this wasn't RAID-related. Instead the passive node's C: drive looked like somebody had run del *.* /s on it.
How on earth could such a destructive command (which would have a crack at deleting all files in the current directory and all subdirectories) possibly have been run?
"Looking back at the 'bulletproof' script in hindsight as I was cleaning my desk," sighed O, "the script was not designed to run where it should not run.
"It was designed to check for the server name and execute based on where it was meant to run."
However, if it didn't find the name, it did not simply exit. Oh no.
"The script added ' del specific drive\path\ ', but when running on a new node for which it hadn't been told about it became only ' del \*.* ..."
As well as the /s parameter, O added /f (to force the deletion of read-only files) and /q (to stop Windows asking if the user was sure about that wildcard). /s , /f and /q – the three options of the apocalypse.
It was, he said ruefully, "the cherry on top".
Ever created what you thought was the neatest utility ever, only to realise that you have unleashed a data-destroying monster? Or conducted an impromptu test of your company's backup and restore strategy? You have? Then an email to [3]Who, Me? will clear your conscience. ®
Get our [4]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://www.theregister.com/2020/08/24/who_me_pdp_11_election/
[3] mailto:whome@theregister.com
[4] https://whitepapers.theregister.com/
Re: ElReg asks ...
Today is Pungenday, the 24th day of Bureaucracy in the YOLD 3186
My contribution ...
Looking after my company's collection of Sun 3 workstations 30 years ago. Someone had been doing some development work that had resulted in /tmp filling up, with all of the assorted problems this gives. So one day when I came in (I was often the first in the office in those days) I logged on to the server using the root account and issued the immortal command rm -rf / tmp .
Note significant space.
It took me about 2 seconds to realise what I had done and hit
'fessed up to my manager and spent the rest of the morning re-installing from scratch. Fortunately I had done a backup just before the screw-up, so I could restore everyone's user directories with no loss.
Re: My contribution ...
Cheer up. I don't know of a single long-term un*x admin who won't (eventually, it might take a couple beers) admit to have done the same thing. Sometimes a couple of times.
Always at the worst possible time, of course.
capable hands of Windows Server 2003
Yeah, I've seen those "capable hands" running at a small software house where I was temping back in 2005. I don't know who set up the server but they had to reboot it at least twice a week because it would just curl up and die for no logical reason. What was worse was that it had done so since it had first been installed. No one ever investigated any further - they'd just reboot it without question.
Near the end of that temping stint (about 5 weeks), they did offer me a full time job there. Having been there long enough to see first hand how the way the place was (mis)managed - and I don't just mean the IT side of it - I politely declined. They went bust a couple of years later. No surprise.
Re: capable hands of Windows Server 2003
Wasn't in the 2000s when it was discovered or at least rumored that Microsoft used Linux for their own servers? I remember that being a thing people said a lot back then but I do not know if it was true.
We've all been there
We had a QA group that did something similar, a long test job that ran for hours, and cleaned up with something like "rm -rf $(LOGDIR}/".
Inevitably some unexpected failure (is there any other sort?) in the test job resulted in this being run with LOGDIR unset. As a user in the same GID group as the staff. On a lab system which had the default automounter config which NFS-mounted all the user home directories, many of which seemed to have 775 permissions...
The sysadmin finally twigged when he realised that the calls from people saying "some of my files have disappeared" were coming in in alphabetical order of username. A hasty shutdown of the home directory NFS system was followed by some forensic network access to find the guilty system.
Fortunately the overnight backups & regular ZFS snapshots meant that the QA team responsible got away with an apology, and buying a few beers.
Are we including sabotage?
Of a completely unfortunate accidental sort of course.
A departing employee completely accidentally left a floppy disk in their laptop, which was set as the boot drive and which by complete accident had its autoexec.bat consisting of just one command that would delete the contents of the C: drive if it was completely accidentally booted from.
The great thing about booting from a floppy is that it is the sort of thing that you can hear and stop with a magic word and a mystical gesture before it does any of that completely accidental damage for which the git knew full well I would be the one blamed.
.
On the confessional front, have I deleted stuff I didn't mean to delete? Yep. On the other hand I still feel the stinging lesson of a disk failure that happened the day before the dvd blanks arrived upon which the much put-off backup of said disk was to be stuck.
Re: Are we including sabotage?
Quote
Yep. On the other hand I still feel the stinging lesson of a disk failure that happened the day before the dvd blanks arrived upon which the much put-off backup of said disk was to be stuck.
OH fudge.. I knew there was something I had to do friday before beer time.....
I'm writing a backup/restore script now, and completely paranoid that the backup won't backup or the restore won't restore or the person using it won't see the error message or someone will try to run it as a cron job with half the environment variables missing or something. There are more ifs with quotes and $? than there are lines that actually do stuff.
Have you considered...
#! /bin/bash -xe
# Prints each line of what will be executed after expansion but before execution
# Exit on any error outside a condition
# -x -e will not work: Everything after the second space is considered a single word
There, but for the grace of God,
go I...
Philosophical questions
If data is deleted, but no one is around to notice, was anything lost?
I've sort of nearly done the same thing once, but only because someone was using a script they shouldn't have nor use.
My contribution...
I was doing a temp stint at a place that shall remain nameless to protect them from the hoardes of angry IT folks that would surely set fire to the HQ if said name were made known. Suffice it to say it was one of those huge juggernaughts that we all love to eviscerate with vitriol at every opportunity.
I was in a server room tending to an old cluster that needed some TLC. Archaic disks that hadn't been backed up in aeons, software so old it had probably been given first drafts on clay tablets by Egyptian clerks wondering how to spell all the buzzword bingo bullshit, managed by monkies in feisty knickers.
I had verified that the server was no longer connected to the internal network by order of the manager I had been assigned to. I made sure that it wasn't running any jobs that hadn't been marked as non critical, temporary, or otherwise able to be sacrificed without need for panick. At which point I start searching the disks for where the largest concentration of files (sizes, numbers, etc) were to be found.
Imagine my surprise when the largest by an exponential margin turned out to be a personal directory full of porn. I dutifully made an offline copy for *cough* Reasons and began backing up the entire system to the specific NAS unit dedicated for just that cluster & purpose.
I get done, detach the NAS, lock it in the drawer the manager indicated, and began cleaning up/out said server for repurposing to other tasks.
I'm about halfway through when some guy I don't know barges into the server room in a wide-eyed frizzy-haired state & heads straight for me like a laser beam.
"What have you done to my server?!" he roars as if he were Zeus & I'm about to get smote with lightning.
I explain what I'd been brought in to do, show him the paperwork from my manager giving me authorization to do it, & explain that I've just given the machine a fresh, legally licensed copy of the OS (complete with drivers) to prep it for reuse elsewhere.
"You can't DO that! That's MY server!" he roars again.
Unimpressed I show him the paperwork that expressly says I most certainly can & have $Manager's orders to do so.
Back & forth, back & forth, him roaring, me not giving a shit. I'd made *damn* sure I was on the machine $Manager specified to prevent me from fucking anything else up, so to have a different manager berating me for doing my job leaves me wanting to smack him upside the head with a NAS.
He shouts he'll talk to $Manager, I nod & say to go ahead, & while he's off to go have some more shouting, I'll finish the job I want to get paid for.
Turned out that Old Yeller had been some up-and-comer nepotistic bugger that had hit his Peter Principle limit. He was used to coming in to work, vanishing into his office, & surfing porn all day. Actual work? Don't make him laugh.
My having taken down his personal porn server was seriously putting a crimp in his pseudo-productivity & he was Having Words with $Manager about why said interference Was Not Allowed.
Except Old Yeller really should have talked to whomever he'd been related to first. Because that person no longer worked at the company, upper management was sick & fekkin' tired of the useless dolt, & this had been their shot across the bow to get him to actually DO something for a change.
$Manager showed their paperwork authorizing said work & sent Old Yeller up the ladder. At some point Old Yeller was told to stuff it.
$Manager thanked me for not giving in to the idiot, for having had the forethought to keep all the paperwork I'd needed to deflect said idiots anger, and for giving him back seriously needed resources.
I was quite pleased with the trip to the pub for lunch & a pint on $Manager.
I was even MORE pleased with the fat brown envelope he offered for his own copy of what I'd found on the machine.
"It's all on that NAS you locked in the cabinet." made him grin like a shark swimming through a cloud of fresh chum.
I was even MORE pleased when I uploaded the entire trove to my various torrent accounts (Demonoid FTW!) & watched my street cred go through the roof.
Ahhhhhh... fun times!
ElReg asks ...
... "Remember when September seemed so far away?"
Yeah, but then I'm old ... It's been September for nearly 10,000 days.
Today is Monday, September the 9862nd 1993.