Whatever you do, don't show initiative if you value your job
- Reference: 1655109010
- News link: https://www.theregister.co.uk/2022/06/13/who_me/
- Source link:
Our story comes from "Harry" (not his name) who was working for a major medical products company back in the days when Windows XP was the hot new thing and software was, frankly, a bit simpler.
We've also turned the Regomizer on a new hire assigned to Harry. We'll call him "James" since we imagine he'd rather his true identity never be made public considering what happened...
[1]
However, back to Harry for now. Harry was a line engineer, responsible for ensuring "the enormous series of complex machines spat out the product in just the right way to avoid any defective products getting out of the factory."
[2]
[3]
This was important, not just because of the whole medical product thing, but because certain countries operated a three-strikes-and-you're-out rule for products with issues turning up in the market. So those machines had to be in tip-top shape, along with the software that ran them.
Harry was given a new hire, James, to train up on the equipment. "New hires would usually be fresh engineering graduates," he said, "and things went as they usually went, having a test line where mistakes could be made, and, importantly, used a simplified software stack, running on Windows XP (of course)."
[4]
When training was complete, James could be let loose on the real thing. He was expected to know the lines inside and out but, crucially, not the software stack. That was very much a black-box affair, updated manually when a CD-ROM arrived with updates.
The time came to do the updates, and James was instructed to run them while the machines on the lines were being cleaned and recalibrated.
James was the only one on the night shift and, looking at the update and thinking he knew a bit about PCs, decided to use his initiative. He ran the updates on all the lines. He then decided to see if he could make the PCs a bit more efficient. The disks could always benefit from more headroom, so he ran a utility to look for duplicate files.
[5]Brute force and whiskey: The solution to all life's problems
[6]Keeping your head as an entire database goes pear-shaped
[7]Beware the fury of a database developer torn from tables and SQL
[8]September 16, 1992, was not a good day to be overly enthusiastic about your job
Oh my, there were lots. James had full system access and so after each update removed the duplicate files. Much better.
"The lines were fired back up over the following days," remembered Harry, "getting ready to get back to full productivity. But they didn't."
[9]
Instead each line popped up a console window on their displays, all with the same message: FILE NOT FOUND
The message repeated over and over again, and the factory was eerily silent.
"Turns out," said Harry, "that the software demanded a precise duplicate of a large database in order to perform its normal duties, but James had destroyed all of them."
A fresh install had to be mailed out in order to get the lines restarted. Harry was hauled before the bosses but successfully protested his ignorance by virtue of simply not knowing how one would go about doing such a thing.
James was not so lucky, and found himself marched out of the door once his helpfulness was discovered.
"Access to control systems was thereafter heavily limited," said Harry, "and updates had to be pushed by a senior engineer only, on pain of death (well, firing anyway)."
So was James truly the guilty party? Or should Harry have pointed at the mysterious boxes driving the line and said "Here be Dragons" to ward off his protégé? Have you ever taken the initiative, only for it to turn into a catastrophe? Confess all with an email to [10]Who, Me? ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YqcKyaY50qf0uYZbvJBHzwAAAEI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YqcKyaY50qf0uYZbvJBHzwAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YqcKyaY50qf0uYZbvJBHzwAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YqcKyaY50qf0uYZbvJBHzwAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2022/06/06/who_me/
[6] https://www.theregister.com/2022/05/30/who_me/
[7] https://www.theregister.com/2022/05/23/who_me/
[8] https://www.theregister.com/2022/05/16/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YqcKyaY50qf0uYZbvJBHzwAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: "So was James truly the guilty party?"
Ehm, and the company with none or very bad change-management procedures for critical production systems. The company and its management is clearly at fault here.
Re: "So was James truly the guilty party?"
Sounds like there's plenty of fault to hand around here. This wouldn't have occurred if any of the following had happened:
- James had not had his brilliant idea / decided to check with someone else before doing it on a critical system
- Harry had realised that letting a newbie loose on a critical system was a disaster in the making
- The company had a decent change-management / permissions system on their critical machines
All the holes lined up on the swiss cheese model here.
Not sure if this was a fire-worthy event though. Yes James made a very stupid mistake, but he made it with good intentions and the experience he got from it was hard-bought. If he was in any way competent, he is unlikely to ever make a similar mistake for the rest of his career. The next guy they get on board to replace him might. If this was the biggest in a line of similar mistakes for James though, yes he should be fired. And an even heavier look should be taken at whoever let him near those machines.
Where were the procedures ?
I agree that James launched himself into his own troubles, but where were the documented procedures on how to do what ?
Of course, this was back in the day and this is exactly the kind of thing that made people understand how important written procedures are.
Re: Where were the procedures ?
In that sort of environment it would be surprising if there weren't documented procedures. They're not much use if they're ignored and it sounds as if James was just the type to do that. I suppose he never found himself anywhere near the emergency stop button.
jumping out of plane with no parachute
"Oh my, there were lots. James had full system access and so after each update removed the duplicate files. Much better."
Ouch, mate, there's a *heavy* assumption behind this happening with no catastrophic impact !!!
Like jumping out of the plane with no parachute, hoping you'll eventually find one holding up, here in the air, while descending !
How could he be this careless ?
Re: jumping out of plane with no parachute
"How could he be this careless ?"
Human nature. Are you new to the working life?
Backup and restore capability for mission critical files?
Another case of "we've not heard of it".
If this really was such a "major" manufacturer, then the senior staff should have been the ones for the chop for skimping on the importants. Alternatively they could have adopted the true engineering approach of using the incident as a learning experience and putting procedures and preventions in place to stop it from happening again. If anything, they should have been grateful to the fresh faced youngster for helping to reveal and eliminate a major risk to their business.
Re: Backup and restore capability for mission critical files?
Backups? Those were what James deleted.
This was a newly installed version - installed by James - and, reading between the lines, it appears that the install created backups - AKA duplicate files - and at startup checked tor their existence. It seems that the system was designed to do exactly what you suggest and did it so as to be idiot proof until nature produced a bigger idiot.
Well meant vs. well done
Taking the initiative to do something helpful without knowing all details doesn't often resultin a job well done.
This "who, me" reminds me of the scientists in university that helped adminstrating the departments machines by cleaning up the clutter of files with zero length in /dev ..
Result was half a day of work for the admins to restore the OS (Apollo Domain OS if memory serves) from tape.
"So was James truly the guilty party?"
It seems that James was cocky and arrogant enough to wreck the system because 'he new best', so Yes, he was guilty and being shown the door wasn't unjust.
However, the idiot who left him on his own overnight with administrative access to critical systems, when he was still wet behind the ears, should bear more guilt. If that was Harry, then he should have been shuffled off too. Assuming he was expendable, of course.