What do you do when all your source walks out the door?
- Reference: 1649662208
- News link: https://www.theregister.co.uk/2022/04/11/who_me/
- Source link:
Today's story, from a reader Regomised as "Al", concerns his time at a company in the 1980s. The company was working on a project to replace thousands of ageing "dumb" terminals with PCs. "The Great PC Invasion and Distributed Computing Revolution were under way," Al observed.
"The company had hired a collection of experienced PC and minicomputer programmers who were led by a management team of Mainframe Gods (as they viewed themselves)."
[2]
We know just the type.
[3]
[4]
"As a bunch of hotshot PC and UN*X types," he went on, "we demanded a version control system and a tool for backing up the source tree. In their wisdom, the Mainframe Gods chose not to invest in spurious tech like backups and version control, therefore each programmer had a personal responsibility to back up their source code."
It went about as well as you might imagine. Some staff followed the process for a bit, but after a while nobody bothered. Nobody, that is, except for the person who did the builds. "Dave" (for that was not his name) had all the current production code on his PC. Everything. In one place.
[5]
It was fine at first. Dave worked hard and also wrote a lot of code. Al couldn't tell how good it was; the words "Code Review" were alien to the company. But the builds happened and the terminal emulation software was delivered. Everyone was happy. Even though Dave had the only copy of the "official" source code.
Al described Dave as "a big guy." He was softly spoken and tended to (mostly) keep his opinions to himself. "He had his eccentricities," remembered Al, "such as a fondness for rifles that he kept in the trunk of his car."
Of course, the inevitable happened. After what Al delicately described as a series of "issues," Dave quit or was asked to leave the company (both mysteriously happened at the same time).
[6]
However, rather than march Dave directly out of the building, the geniuses in management gave him the rest of the day to finish up his work.
"During the afternoon of that day, Dave's manager looked out the window to see Dave loading boxes and boxes of floppy disks into his car," said Al.
A curious thing to do on one's last day. Perhaps Dave was just doing a final clear-out of office stationery? Or perhaps...
[7]If you fire someone, don't let them hang around a month to finish code
[8]Debugging source is even harder when you can't stop laughing at it
[9]The IBM System/360 Model 40 told you to WHAT now?
[10]Prototype app outperforms and outlasts outsourced production version
The manager scurried over to Dave's PC and found it coming to the end of a FORMAT operation. The hard disk containing the only complete copy of the source had been wiped. The floppies Dave was loading up into his car, nestled among the rifles, were the only backups in existence.
"To his credit, Dave's manager talked him off the cliff and got the backups returned to the office," Al told us. Dave was then politely escorted to a coffee shop to complete his last day while a panicked staffer managed to restore the backups to the now blank PC and the project could continue.
"The project did eventually succeed, but we did have further harrowing moments with more conventional causes," said Al. "We did establish regular backups and duplication of key source code."
"The moral of the story: back up your data. Trust but verify."
We like to think Dave had no malicious intent and was simply tidying up after himself. Surely not a final act of vengeance? And considering what else was in the trunk, it could have gone very, very differently.
Ever been tempted to dash out a quick FORMAT C: or a sudo shred just for giggles? Confess all with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/Who,%20Me?/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YlP8VskEZ6bE3eRW0VRiKwAAAUY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YlP8VskEZ6bE3eRW0VRiKwAAAUY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YlP8VskEZ6bE3eRW0VRiKwAAAUY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YlP8VskEZ6bE3eRW0VRiKwAAAUY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YlP8VskEZ6bE3eRW0VRiKwAAAUY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/04/04/who_me/
[8] https://www.theregister.com/2022/03/28/who_me/
[9] https://www.theregister.com/2022/03/21/who_me/
[10] https://www.theregister.com/2022/03/14/who_me/
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
I have tried the 'sudo rm -rf /' on an old machine that was due to be reinstalled just to see how it would go.
The answer was very far indeed! Eventually the GUI display lost all text when the fonts were deleted, and when the disk LED finally went off I rebooted in to a live CD to view the results - pretty much everything had gone bar a few in-use directories like /proc and similar. rm had even rm'd itself!
accidentally
I wanted to remove hidden files on a linux machine (you all know where this is heading, right?). Unfortunately some idiot (yeah, me) had done some work as root (yeah...), and some of the hidden files were actually owned by root. So some knucklehead (yeah, me again) did some "hard thinking" (not really) and regexp'ing, and the hidden files, beginning with a "." and then letters following, or maybe even numbers (not that said idiot cared), would surely be matched by ".*". Unfortunately the command rm -rf .* also matched .. and everything under that one as well.
Even spinning rust is really quick when it comes to just unlinking the references to the inodes to clear up space.
I did not lose too much stuff, but I learned a valuable lesson that day.
Even doing that as non-root pretty much screws the system up to the point a rebuild/restore is necessary.
Had a user do that. Twice. With a crappy script (his script took the directory as a parameter and he forgot to supply one, so the directory ended up being "/" where he then tried to run an rm -rf).
When we recovered his machine the first time, he went to show me what he had typed and promptly hit the return key and ran it again!
I was NOT amused.
I've posted this before, but I think it fits here.
Just over a billion years ago ...
... as the Internet measures time (call it roughly 1984), I received a brand new Sun 2/160. It was a dual pedestal beast, with all of 8 Megs of RAM and a pair of 380 Meg CDC SMD drives. Roughly 65 grand worth.
I decanted it from the boxes-on-pallets, plugged all the cables in, and fired the thing up. Into a beautiful new GUI on the Sony Trinitron monitor, just as advertised. Logged in as root, on purpose as there were no other accounts as yet(!!), using the default password(!!!!) ... and poked around. All was well, near as I could tell.
The plan was to repartition the disks to better suit our needs and then reinstall the OS. So I made absolutely certain I had the correct tapes, and did the one thing I had never done as a sysadmin ... closed the GUI, and from the # prompt ran rm -rf / intentionally. I was curious to see how long it would take to lose it's tiny little mind. It trundled away to itself for a few minutes, but seemingly was still working fine, enough of vmunix and the shell were in RAM and the swap partition to keep doing simple stuff. I was quite surprised, but that wasn't really what I was there for ...
So I shut her down, went and got a cuppa coffee, reached for the first tape and went to fire up the machine ... only to discover it didn't ship with a tape drive, despite one being listed on the packing list. It had a lovely bezel that LOOKED like it might be a tape drive, but the space behind it was empty. Oops. So there I was, 8AM and stuck with 65K worth of dead Sun hardware that I was supposed to demo for the Brass at 4PM.
Fortunately the 1980s Sun had Clues about customer service. One call, and their field service rep had the SCSI tape drive, the requisite cables, and a couple of VMEbus cards, (E)EPROMS and spare OS tapes "just in case" on my desk in under forty minutes. She even hung out and made certain that the system worked properly after we took it apart to install the bits that needed installing, and then partitioned it and re-installed the OS.
I made the 4 o'clock deadline ... and bought the Rep the first of many well deserved dinners.
Silly Con Valley was a very small place back then ... Sometimes I miss it.
Yes and no.
My old boss was a dick. The kind of person who would sit next to you and send you an email rather than turn his head to have a conversation. The kind of boss who would somewhat loudly criticize you and never apologize when proven wrong (like the time he had a go at me for requesting an order for WORM media because we were running low - because he wrote a bit of code that told him we weren't and didn't need any. Until I looked over his shoulder and pointed out an error in his code).
The week before I left I had a genuine failure on my workstation and had to rebuild it. I just didn't copy back my documentation. I got a panicked call a couple of weeks later because he realized he didn't have any docs at all.
So here's the 'no' bit. I made sure the guy I worked with who was left behind had all my docs. I'd spent time going through some of the stuff with him. I even took calls from him if he got stuck (he was a friend of mine, and it made him look good to be able to swoop in and save the day).
Formatting pc's in the days of DOS and Windows 3.1
Went to a consumer electronics store with a mate who was looking to buy or build a decent PC in the mid 90's to be used for accounting and salaires on his father's farm. Sales guy was very pushy trying to upsell the most expensive machine he had in the store, pushing some big name box with an Intel MMX over the faster but way cheaper Cyrix, because it had a gaming graphics card and refusing to answer questions about upgradability - like getting a cyrix machine and adding RAM and an extra drive would be cheaper than the top of the line at the time gaming rig (that was nothing special other than the high price), and ignoring our question that storage space was more important than cpu and graphics power but the sales guy obviously knew more about IT than 2 young geeks who talked Intel technical manuals over breakfast at high school. After his agressive speech and ignoring our actual questions for like 15 minutes and finally worming out of his grasp and deciding to roll our box, we wandered around the computer section, logging out of windows on all 6 machines on display and quietly running a format c:
Petty revenge from a bunch of 16 year olds. We were young and dumb at the time. (Now I'm just old and dumb :) )
Re: Formatting pc's in the days of DOS and Windows 3.1
Another favouite of mine in the world of completely unprotected display models of 90's PC shops...
Open autoexec.bat
:lol
echo I am a fish
goto lol
Then reboot and go home
Kids will be kids
Educational Only
We were all getting made redundant and as our notice period got shorter, our workload lessened. As I am someone who can't sit around and do nothing in work (despite the circumstances!), I got hold of one of many spare laptops and over a couple of weeks tried to work out the deadliest and quietest way to trash a Windows 98 install using just basic OS commands. If I remember correctly, I ran a silently-run batch script on start-up which deleted particular directories in a specific order and the end result was that after about five minutes of silent deletes, the filesystem had so many things missing that it just stopped working.
As I said, it was purely for educational reasons and to while away my time - I had declared all my documentation "up-to-date and accurate" (ahem) the previous week - as even back then trashing systems on departure was a perilous thing to do.
Never get the chance to do it again
At the end of the life of the Met. Office's IBM Power 7 775 supercomputers, which I had supported the entire time they were there, the Met Office data security team asked me to carry out the complete logical destruction of the 4 systems that made up the environment (I had the appropriate security clearances and technical knowledge and their trust as I had been there for the entire life of the systems. This was so they did not have to monitor the process too closely, beyond checking the various processes would definitely clear any restricted data, and a quick interview when it was completed to sign off that the checklist had been followed).
Through a combination of primary disk initialization to destroy the RAID arrays, through scrub operations on some of the particularly sensitive disks, and format operations on the disks of the Hardware Management consoles, over the course of the first three days of the process, I systematically destroyed two supercomputers that were still in the top 200 of the top 500 list, and two smaller systems that were for test and collaboration purposes.
It made me very sad that the systems I had invested nearly 5 years of my working life to keep running had to be destroyed, and I had mixed feelings about the being person doing the logical work (and some of the physical work, as I was helping the IBM decom. team who had flown in from the US as I was 'escorting' them to make sure they did not touch the other systems). The entire decommissioning and removal process stared on the Monday morning and the systems were on the trucks removing them, and the floor tiles replaced by Thursday evening. The only thing that was left was the water cooling infrastructure and floor bracing, which the Met Office were removing themselves.
Sad that that gig had to end, but very happy I had the chance to be involved at the centre of the project. It was a chance very few people get to enjoy.
Re: Never get the chance to do it again
For the curious, is it possible to say vaguely what could be on a met office supercomputer that would be quite so sensitive? The only thing I can think is they're really protective of their models (for commercial reasons?).
I've worked at several places where the source code was really the engineer who wrote it, and despite having the code left behind, the intellectual process that created it was gone and the projects withered on the vine without it. Too many managers only consider their "digital assets" to be of value.
Just the once.
When we drew up the terms of a startup, I included language that indicated that I would be sole owner of all the software that I generated for the new company should the company go under within three years. As is typical for most small startups, we went under before two years was up. Bad management. The Bank shut us down and took possession of all equipment (complete with Santa Clara County Sheriffs ensuring we didn't walk out with the kit).
I managed to start the SCC disk formatting and my computers wiped of all traces an hour before they got there ... There were no backups, because expensive. (Have I mentioned bad management?) However, I was quite secure in the knowledge that I had already transferred the entire code-base to my coloed server under Bryant Street in Palo Alto. The new-fangled Internet was handy for that kind of thing.
It took 'em a week to realize that the code no longer existed, and that without it all the hardware was so much scrap. As the only IT guy in the former company, they figured it must be all my fault, and the Bank sent me a very threatening letter. Which I immediately passed to my lawyer (brother). Who sent them a copy of the initial terms. I never heard from them again ... but I did find some of our kit at Weirdstuff Warehouse, selling at scrap value. I ignored it. The whole project had left a bad taste in my mouth.
I still have the source, but it's useless now. We were trying to fill a niche that was filled by a couple of multi-nationals a couple of years later. It's a shame, really, we didn't even have patents on our tech, because spendy. Perhaps I should mention bad management?
A company I worked with (hence AC) had a software engineer who died unexpectedly on holiday, she must have been only around 50 and no signs of ill health so a real shock to all of us.
However, said company decided to wipe and re-purpose her PC shortly after. About 6 months later I got a request if I had any information on some software, eh?! Turns out she had made changes and built/deployed the new version that fixed some things but the changes were only checked in to the local disk, not the central source management system.
Had they only put in a new HDD to re-use that PC!
Backup AND Restore
Of coure, having a backup process in place is only half the story.
I once worked in a government scientific organisation that used networked Windows PCs and had a process that backed up certain parts of the shared drives. One directory of the shared drives contained a large number of small files of geophysical data totalling about 600 MB which was input data for a tsunami simulation system. According to the powers that be, and the file system viewer, these files were backed up. But I had my suspicions, so I moved one of those files to multiple other places and then requested that the file be restored from backup because "I had accidentally deleted it". I was never told the details, but I was told that the file could not be restored. As my contract was ending, I just alerted my boss to the problem before departing.
Moral: backups are pointless unless you test the restore process.
"Ever been tempted to dash out a quick FORMAT C: or a sudo shred just for giggles? "
Actually done it, no. Tempted to do it, several times a day.