Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working?
- Reference: 1667804234
- News link: https://www.theregister.co.uk/2022/11/07/who_me/
- Source link:
This week's hero, whom we shall Regomize as "Sedgwick," worked as a systems engineer in the R&D department of a major pharmaceutical company. The company was at the time in a fevered race (possibly literally) against a huge competitor, searching for a breakthrough drug to treat a particular ailment.
Got that? Systems essential, downtime bad.
[1]
The company had just upgraded its "minicomputer-based system that did simultaneous data capture and reduction for many channels of chemical analysis instrument," and Sedgwick was tasked with training the users in its operation.
[2]
[3]
The customer had already been using a turnkey version of the system. The upgrade allowed far finer control of and access to the underlying system. The multi-user system had an account for each user, and each account was assigned a numeric privilege level. Each command had at least one required privilege level, and sometimes two. For example, to terminate one of your own programs required a medium privilege level, but to terminate any program in the system required the highest level. As Sedgwick puts it: "With so much power at his fingertips, clearly the system manager needed to be trained, as otherwise havoc might ensue."
Indeed it might, Sedg. Indeed it might.
[4]The boss worked in a fishbowl, so office tricks were a treat
[5]Data loss prevention emergency tactic: keep your finger on the power button for the foreseeable future
[6]Loathsome eighties ladder-climber levelled by a custom DOS prompt
[7]PC component scavenging queue jumper pulled into line with a screensaver
Once it was all up and running – on live data, we should add – the system manager asked for a demonstration of the privilege system.
So, naturally, Sedgwick created an account for himself with medium-level access that ought not to have been able to terminate many programs on the system. That's only sensible.
[8]
And for the demonstration he also created a dummy high-level process that his demo account would be unable to terminate, but of course would have no effect if the demonstration went wrong, yes?
Sadly, no.
Sedgwick issued the command to kill the main data capture coordinating process – on live data – secure in the knowledge that the system would never allow such a thing.
[9]
Unbeknownst to poor, poor Sedgwick, the latest revision of the system had introduced a bug – soon patched – in the command privilege tables.
At this point, we shall assume readers are familiar with the unit of time known as the "ohnosecond", which defines the elapsed time between making an error and realizing you've made an error.
After not much more an a single ohnosecond, the first irate chemists were banging on the windows wondering where their work had gone.
Sure, it was hardly Sedgwick's fault there was a bug in the system. But his confidence that there was no such bug was utterly misplaced. Always assume there's a bug.
In the end, both the company Sedgwick worked for and its competitor were beaten to the breakthrough by a Swedish concern, and ended up merging. Who knows? If not for one fateful killed process …
Have you ever changed the course of history with the press of a button? Prevented (or hastened) a major advance through accidental heroism? Tell us all about in in [10]an email to Who? Me and we may share your tale. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y2jlS1uaF0vYqLO7PoNvwQAAAQY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2jlS1uaF0vYqLO7PoNvwQAAAQY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y2jlS1uaF0vYqLO7PoNvwQAAAQY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2022/10/31/who_me/
[5] https://www.theregister.com/2022/10/24/saved_by_the_analog/
[6] https://www.theregister.com/2022/10/17/who_me/
[7] https://www.theregister.com/2022/09/26/who_me/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2jlS1uaF0vYqLO7PoNvwQAAAQY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y2jlS1uaF0vYqLO7PoNvwQAAAQY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Father Dougal
Accidents are one thing, but there are some people who do this sort of thing deliberately, as told [1]in this tale of our own Father Dougal
[1] https://forums.theregister.com/forum/all/2017/11/24/on-call/#c_3356383
Ah yes, Big Red Buttons...
A gift that never seizes giving.
icon should have been red, but the red ones try to stop one
Don't knock the big red button..
In the Uni computer lab I used to work in, there was a big red button on the wall that killed the power to the student computers. I *think* it was there from when the room hosted several large, and potentially dangerous machines, it's just when the electricians installed the new electrical mains, they ran them through the same BRB.
Now, when closing the lab, I always gave warnings half an hour before closing, then ten minutes, 5 and finally 2, so I'd argue they did get fair warning.
95% of students would leave after the first or second warning. TBH, by the time you got to the third, you were down to the hardcore students who wouldn't leave no matter how many warnings you gave.
After a few weeks, I changed the final warning to "If you don't pack up and leave within 2 minutes, I'll go to the big red button, hit it, and you'll lose any work that isn't saved". Weirdly, no one ever tested me on that..
Warning: Language nazi at work!
It never ceases to amaze me when people use a phonetically similiar but incorrect word in a sentence.
--------> Why does he look so much like Mark Labbett?
Re: Warning: Language nazi at work!
Thanks for clarifying the meaning...I just skipped over it, a little confused. Well it is Monday morning.
Re: Warning: Language nazi at work!
See also https://forums.theregister.com/forum/all/2022/11/07/who_me/#c_4562836
Working as it's supposed to
> the latest revision of the system had introduced a bug
After all, gotta keep the support staff in jobs.
This security software has great quality..
I remember working at a customer site, and by law they had to have some security software installed. The manufacturer was really pleased to quote their defect rate ... single figure defects per year, installed on thousands of systems.
During my visit an audit was taking place. I heard the conversation went a bit like this.
Auditor:Do you have this security software installed
Customer: Yes.
Auditor: Is it enabled
Customer:Not as such
Auditor:Please may I see it working, and that it stops someone without authorisation from doing something.
Customer: Not during production hours
They turned on the security system on a test system, and the performance dropped significantly, there were abends and other errors, it did not scale to hundreds of users, it just about did what it said on the box (after a lot of complex setup).
A short while later the Government updated the law to say the software had to be installed, enabled, and working.
The result was so many defects and problems were found the manufacturer had to withdraw it as it needed a redesign.
Re: This security software has great quality..
Aha, the unbreakable truth: apparently-bug-free software just isn't being used.
Bug reports are a Good Thing for software that people find useful: it's a sign that people are using it (and quite probably for things you didn't envisage)!
Nothing so drastic, but I did have to travel to a customer site to set up a room for on-site training. It was a last minute request (only about a day's notice). I was informed that everything was ready. It had to be done on a Friday, and this was on the Isle-Of-Wight. The company shut up shop early in the afternoon.
So I get there and start my work - only to find the requirements weren't met. They were missing software. Queue people rushing around trying to
a) Blame someone else
b) Find someone who could authorize the software purchase so late in their working "day"
Luckily I am covered (I simply produced the statement from them that all was ready and the requirements were met), and they managed to find someone willing to cough up for the software. Install done just in time, and I got a trip down memory lane (had to ride on the chain ferry - haven't done that in decades).
I realise this isn't always feasible, but this is why I normally advise people, if they are doing development work on a system, to have a test install of the system running somewhere, probably with slightly out of date data.
A system I'm working on at the moment runs on two VMs. One for the production system, and one for the test system. The test system uses a copy of the database from the main system that is about 24 hours behind. We do all our development, testing and demonstrations on the test system, safe in the knowledge that if the company hosting the systems is doing their job (and generally, they do) properly, our data is safe.
That said, another system I'm working on, we didn't pay for a test/development server, so I've learned to be *very* careful on it..
"ohnosecond"
Reminds me of the flight instructor who bet his students that it is not possible to retract the landing gear while on the ground. One Saab 340 was totalled in the process.
Oh please - send the link to the You Tube video of this one!
A near miss
Some timeago I worked for a software development company that had a product to archive data from relational databases. It worked by running a search for candicate records to be archived and then found all the releated data. So, you could archive customers who had not ordered anything for five years, and it would also find all the invoices, payments, shipping, correspondence that related to the customer(s).
As the tech person I always insisted that the client test their scripts on a test sustem before running it on production.
One client proudly proclaimed that they didn't need a test system. Nor a development system. They just ran production. So I asked how they did development of the application and specifically new database tables (etc). The answer was "Oh, Malcolm here just gets on the main console and keys the changes directly in to the database. On production. And if it goes wrong, we are front page news next day. Ha!" We were out of there so fast as I explained to the samesman that no amount of profit was worth the risk that if anything did go wrong we were bound to be the fall guys.
Oh, yes. The system? HMRC's tax computers.
Apologies for the lack of detail, I wasn't present in the room but did see the effects. One Thursday, there was a lot of grumbling at the end of the day when an unfortunate IT engineer took down the main data centre by accidentally hitting the big red button, causing people to lose unsaved work. The following Monday morning, the same thing happened again, with similar grumbles.
I later learned that the second incident happened *because* the engineer in question had fitted the big red button with a cover. They were proudly showing it off to their boss by sharply hitting said cover, which hadn't been properly fitted and gave way just enough to trigger the button again.