One person's shortcut was another's long road to panic
- Reference: 1706516954
- News link: https://www.theregister.co.uk/2024/01/29/who_me/
- Source link:
This week meet a reader we'll Regomize as "Bart" who once held the grand title of "Scientist" at a research lab which did a lot of data-crunching for orgs like NASA and ESA.
Such orgs obviously produce a huge amount of data. Bart told Who, Me? that at the time he left there was at least 2.5 petabytes of storage on hand, with that total growing fast.
[1]
The machines that doing all that processing had also grown rapidly and a bit haphazardly as and when need arose over the years, so the system was, let's say … quirky. Sometimes, for no apparent reason, processing jobs fell over. Because they were processing near-real-time data and had contracts to fulfil, priority was placed own getting stuff done fast rather than figuring out why a particular job crashed.
[2]
[3]
Thus, if a particular processing job failed, the system would leave it in a junk directory and start again. Storage capacity was not at a premium – time was.
Of course, even with all that capacity, the useless junk directories full of half-processed data would eventually build up and occupy considerable space. For a while, Bart would just tell people to delete their junk directories – but users were often in no hurry to comply. Doing it manually himself was also not a good use of Bart's time – he had more Scientist-y things to do than clearing out other people's junk.
[4]
So Bart created a shell script that he could run periodically to scan the workspace directory, dig down into any subdirectories, find the junk and clear it out.
How would it find the junk? Well, it was quite simple. You see, once a job was fully processed on the workspace directory it was transferred to one of the storage servers, where it would be collected by whoever owned it. Thus, the only data on the workspace directory should be work in progress. Anything that didn't have a "running" flag on it was obviously one of the failed jobs and could go.
Quite simple. Quite clever. And it was quick – usually a couple of minutes to free up terabytes of space.
[5]
Of course, the trick was only ever to run the script on the workspace server, where all the live work was happening. Who, Me? mentions this for the sake of what we in the business call "dramatic foreshadowing."
[6]Poor communication led to complete lack of communication
[7]WTF? Potty-mouthed intern's obscene error message mostly amused manager
[8]New year, new bug – rivalry between devs led to a deep-code disaster
[9]PLACEHOLDER ONLY Someone please write witty headline here
One day, when the servers were starting to look kind of fullish, Bart ran his script on the workspace server and headed off to grab a coffee and talk to colleagues. Twenty minutes later, when he came back, the script was still running. Concerned, he killed it to find out what was going on.
Reader, you will not be surprised to learn that bad stuff was going on. For some reason, one of Bart's colleagues had left a symlink – kind of like an alias or a shortcut – from a directory on the workspace server to the root directory of one of the storage servers. The script had found the symlink, followed it as if it were a subdirectory, and begun deleting the contents of the storage server.
Because of course nothing on the storage server had a "running" flag. No-one would ever be crazy enough to do live processing on the storage server, would they?
It's important to note that there's no good reason Bart could think of why such a symlink would exist. The whole system was designed to keep live data on the workspaces away from the storage server. The script could have been written to ignore symlinks, but who would even think to do that?
Bart never found out who created the symlink, nor why. What he did find was gigabyte after gigabyte of empty disk space where data ought to be.
Data for which his employer had contracts .
Thankfully it didn't take long to work out that much of what had been deleted was superseded data, and the rest could be reasonably quickly reconstructed by re-processing the raw data. Nonetheless, Bart had some serious egg on his face.
If you've ever found yourself wearing a facial frittata after deleting the wrong data, look on the sunny side: you can share it with other readers via [10]an email to Who, Me? and we'll make you (anonymously) famous. Go on, don't keep all your eggs in one basket – we could use some fresh tales to tell. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZbeFXmW47fMNOW@9pnTWhAAAAAk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbeFXmW47fMNOW@9pnTWhAAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbeFXmW47fMNOW@9pnTWhAAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbeFXmW47fMNOW@9pnTWhAAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbeFXmW47fMNOW@9pnTWhAAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/01/22/who_me/
[7] https://www.theregister.com/2024/01/15/who_me/
[8] https://www.theregister.com/2024/01/08/who_me/
[9] https://www.theregister.com/2023/12/18/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Oops!
I would like to say that an IT professional would use the tools that already exist for tasks like this, like `find`, which by default does not follow symlinks. :)
Re: Oops!
In all fairness, Bart was employed as a scientist not a sysadmin.
Ignoring *all* symlinks might have had side-effects, there could have been legitimate symlinks inside the data directories.
To be robust, you'd have to check all symlinks and ignore only those that go outside a specific level. And repeat the check if it's a symlink to a symlink...
Re: Oops!
This problem was encountered pretty much as soon as symbolic links were introduced in 4.2BSD. Each utility that traversed the filesystem (du, find, etc) had to have a flag added to indicate whether symbolic links should be followed. I remember a version of SunOS in the mid-1980s whose cron job to remove old files in /tmp followed symbolic links, with predictable results.
So, a symlink set off a whole chain of events...
Until the chain broke?
You can make similar cockups with the /MIR function of Robocopy . Its reluctant to follow Microsofts version of symlinks though.
Lucky escape there for Bart , although hopefully the 2.5 petabytes of storage was backed up.
Genuinely curious...
...why this colleague made a symlink to the root folder of a different server in the junk folder.
It seems very unlikely this was an accident, the only reason i can think of was that he wanted to quickly move folders from the junk folder to the storage server (perhaps incentivised by Bart's agressive purging strategy).
Also, IMO the best strategy here would've probably been running the script as a user or service with access limited to the junk folder. This limits any and all damage to the junk folder, no matter how stupid other people are!
Ouch!
I do remember one script I wrote to back up stuff getting into an infinite loop because someone had made a symlink loop in their directory structure. This resulted in loads of extra copies on the backup drive before I could stop it. Changed the script to ignore symlinks. Fairly harmless, but annoying as I had to clean up the back-up manually
Re: Ouch!
Ah, joy of joys, the self taught rsync users and their (i.e. my!) self written backup scripts...
"teehing troubles"... right?
Meh. Long time ago. And storage _was_ a premium, my time wasn't. I need a drink, I think. Too early, though, and have some things that are not compatible with day time drinking. Not like that time when we had some time to kill after a conference, and we went on a tapas tour in the late morning (until the evening, when we had to head to the airport...) in Honululu, eating small dishes and drinking Mai Tais...
As a old fart, I always expect someone else's stupidity.
Also as an old fart, I expect myself to make stupid mistakes, too (I'm only human!), and program accordingly.
"Only two things are infinite, the universe and human stupidity, and I'm not sure about the former." —Albert Einstein (supposedly)
"Apart from hydrogen, the most common thing in the universe is stupidity." —Harlan Ellison
"There is more stupidity than hydrogen in the universe, and it has a longer shelf life." —Frank Zappa
Hey
At least it's better than the complete scuttling of a roughly $300m mission because someone forgot to convert between Imperial and Metric units.
So much free space
Did this one on a Unisys A-Seried painframe. As far as I can remember there were precisely zero ways to prevent other users from accessing each others' files. There was a directory with subdirectories per user where everybody could store their personal scripts, text files, and other artifacts. One day I wanted to clean up my personal directory but I specified the top-level directory instead, and there went everybody's data. It went pretty quickly because there wasn't much in those directories – the week before somebody else had accidentally done exactly the same thing.
Check before Deleting
As a newly-employed graduate I spent time in various design offices 'doing the rounds' as part of the ongoing training. These offices were highly technical and manned (including ladies) by experienced, skilled engineers who had been brought up using the slide-rule for numerical calculations. A very new, very expensive electronic desk-top calculator was brought in for those calculations that didn't need to go to the 'Computer Department' for longer, iterative processes. Although programmable, it had limited memory for recording the programme and it was quite common to write and run the calculation each time. The answer was shown on an LCD display. This desktop device was shared amongst an office of eight and usage was governed by gentlemanly enquiry.
It was after lunch, a warm afternoon with the sun streaming in through the windows and several of the engineers were in deep, deep thought. Very deep. Eyes closed as they theorised or even dreamed of new mechanisms, slack jaws on chests in relaxation. Tony had even let his pipe go out. Rather than disturb these thinkers I used the calculator without checking...... When Tony's thoughts returned to earth he was most upset that the calculation he had been working on before lunch had been wiped out. He never forgave me for 'not checking before using'.
It wasn't long before personal hand-held calculators became ubiquitous so the lose-programme problem didn't persist. Much later, air-conditioning reduced/stopped the afternoon 'thought reorganising'.
Oops!
I would like to say that IT professional would have built a symlink-check into their code to ensure that it did not scan outside of the "junk" folder. Sadly, in my experience, that is too often not the case ...