When we asked how you crashed the system we wanted an explanation not a demonstration
- Reference: 1671433206
- News link: https://www.theregister.co.uk/2022/12/19/who_me/
- Source link:
This week meet someone we'll Regomize as "Hal", who recounted for us a tale from the epoch of the mainframe, when glass cathedrals full of sacred boxes stood protected and air-conditioned. In this faraway time, none but the appropriately credentialed could pass through the sliding doors and gaze upon the IBM-emblazoned bezels of these mighty boxes.
Well, the appropriately credentialed and their interns. More on that in a sec.
[1]
One night, Hal tells us, a disaster happened. The mainframes had, all at once, shut down. The terminals to which they spoke had shut down. The peripherals and various doo-dads with which they doo-did had all shut down. The entire business, effectively, had ground to a halt.
[2]
[3]
Support crews were awoken and called to the site to reboot the system and diagnose the fault. Senior technical staff were alerted that a meeting would be held at 07:00 AM to figure out what had gone so very very wrong.
At that morning meeting, the support crew determined that the system had not crashed at all. In fact the machines had not encountered any fault and had behaved exactly as they were designed to in the event of an emergency shutdown.
[4]
A what's that did you say? Emergency shutdown? What emergency shutdown?
It transpired that an intern, who should never have been alone in the glass cathedral in the first place, had spotted an "ambiguous message" on one of the consoles and panicked. Fearing an emergency, he fled the room and slapped the big red button on his way out.
[5]Server installer fails to spot STOP button – because he wasn't an archaeologist
[6]Programming error created billion-dollar mistake that made the coder ... a hero?
[7]How not to test a new system: push a button and wait to see what happens
[8]Job 1: Get the boss on the network. Job 2: Figure out why Job 1 broke the network for everyone else
This was, of course, not the protocol. Clearly, something had gone wrong in the training process to put an intern in such a position that they hit the big red button instead of grabbing a senior member of staff to explain the "ambiguous message" on the terminal, right?
So the intern was in big trouble, as was whoever was supposed to be in the room with said intern. But senior people needed to know exactly how this happened so it could't happen again.
They asked the intern to walk them through what had happened, what had he been doing, and all of the steps he had taken.
[9]
Which he did – in great and demonstrative detail for every step.
Even the one when he slapped the big red button.
Again.
As the mainframes once again shut down, taking with them all the work the support crew had done through the night getting everything up and running again, the supervising tech "lost all sense of decency" as he lifted the intern bodily and placed him in a nearby garbage receptacle.
Needless to say this incident became the abrupt end to an internship, though thankfully he suffered no serious injuries. Indeed he was taken on as a programmer – possibly, Hal suspects, to head off any legal action stemming from the whole tossed-in-the-trash thing.
Have you ever gone a bit too far in investigating a tech fault? Fixed something only to break something else? Don't keep these stories to yourself – the green thing to do is to recycle them with [10]an email to Who, Me? We promise not to throw them in the trash.
®
Editor's Note: Who, Me? will resume on January 9.
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y6BET86WvxYdpUzeFX9TCgAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y6BET86WvxYdpUzeFX9TCgAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y6BET86WvxYdpUzeFX9TCgAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y6BET86WvxYdpUzeFX9TCgAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2022/12/12/who_me/
[6] https://www.theregister.com/2022/12/05/who_me/
[7] https://www.theregister.com/2022/11/28/who_me/
[8] https://www.theregister.com/2022/11/21/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y6BET86WvxYdpUzeFX9TCgAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: ... half a brain
I once had to throw myself across the room to physically restrain a colleague who was about to put the third (and last) copy of a Novell server backup tape into the drive that had just chewed up the previous two. Brain-fade can, and does, happen…
Timing
And I wonder why the management had to organise a meeting at 7am, presumably while the system was still being sorted and before any real analysis could have been carried out. Not too early for a witch-hunt to be held, obviously.
Re: Timing
As an ex support engineer and tech support manager I would have booked a 7 am meeting, that's the earliest I can expect some of the guys to get in in the morning and have a handover from the poor sods who have been up all night. In reality this means they could probably get to go home at about 9 am. I've done thew whole 'work the problem until it's resolved' thing too many times in thee past, my longest stint was from 8 am Friday to 2 PM Sunday afternoon I don't put my teams through that its much better to cycle people through and make sure they are rested and can actually think.
I did drive home on the sunday afternoon as we only had one car and the weekend marked the end of a long week with a few rest days to follow but it was not a safe drive.
Not Only But Also
Deploying a new PC (That's covered in other oft repeated tales).......
Background Guy One: Ohh just got a email from Sanjay (For that was his name), telling me where he is now working...... I don't know why he bothers as invariably hes got the sack 1 week after sending it.
Background Guy Two: Great [/sarcasm]
Background Guy Three: Who what eh? [/Confused]
Background Guy One: OK we had a chemical spill incident, down to him, all those that got splashed had to have cold water showers outside & sent home in paper overalls, our clothing destroyed.
Background Guy Two: I remember all to well [/anger]
Background Guy One: So we had the Elfin Safety autopsy & Sanjay was asked to show us what he did leading up to the incident.....
Background Guy Two: I remember that all to well as well [/real_anger]
Background Guy One: So Sanjay went through the steps right to the point where he was supposed to carefully lower a weight into the chemical vat...... & for the second time he just heavily dropped it in, splashing us & this time the entire Elfin Safety team who were dutifully watching up close & personal, after a repeat of the cold showers & paperwork....he was unsurprisingly sacked!
Icon - WTF's just splashed into me eye!
Re: Not Only But Also
Story is good, but the +1 is for the creative icon usage!
A simple solution
> But senior people needed to know exactly how this happened so it could't happen again.
Just disconnect the big red button.
Or if that is too easy, rig it so that it has to be pressed twice. And that after the first time, flashes up a message saying Please do not press this button again
Re: A simple solution
Our ask little Molly for a solution!
Re: A simple solution
This only works with BBBs - big black buttons labelled in black with a black background.
Re: A simple solution
Hoblack Desiato would approve.
HAL!
That's a computer's name... or Malcolm's father!
Re: HAL!
I'm sorry Dave, I did it again.
Whoops - lets do it again and watch this time.
Someone told me about an incident where their father worked. It was metal press factory, where they had big sheets of steel and made them into boxes. The machines had safety equipment, but one person managed to get the end of his middle finger sliced off! Lots of blood etc, and the very tip of a finger.
There was a H&S review, and people came to find out why the safe guards didn't work. They asked him to go through the same process as he did before. There were no problems - until they said they were going when he cut the end of the middle finger on the other hand!
Apparently the injured person had very narrow fingers which went through the mesh of the protective cage. With the audience watching he was extra careful; as they were going he went back to normal operational procedures.
The mesh was changed to have even smaller holes.
Re: Whoops - lets do it again and watch this time.
And this is where Standards come from. The blood of the unlucky.
Which is why abolishing them is only ever proposed by MPs like the Minister for Workhouses, Rees-Mogg.
Mr intern, do exactly what you did before.
And then what precisely did you do after you saw that message,………
Unanswered question
All I can think of is: What was that "ambiguous message" ?
You cannot blame anybody for pushing the fine red knob when it reads "Disk on fire, reading flames." .
Well, can you? Can you?
Re: Unanswered question
https://en.wikipedia.org/wiki/Lp0_on_fire
Re: Unanswered question
Server reply: HTTP/1.1 418 I'm a teapot
> This week meet someone we'll Regomize as "Hal"
...aaaaand just like that, my minds eye conjured up the image of Hal from "Malcolm in the Middle", running away in panic from an enraged Craig Feldspar, after breaking his home entertainment system.
Fully reproducible event: that's the first step on solving the issue, isn't it?
You'd really want to treat this one as an off-by-one error. It is self correcting after that one error.
Reminds me of an operator running some trainees through some basic commands on the console of our Sperry/Unisys mainframe. But of course not actually hitting Enter for those commands which could cause an issue. Finally he got round to showing them the command to reboot by keying in $! (Dollar-Bang), forgetting that this was the one command that did *not* need you to hit Enter for it to be actioned! :o
I was, for a moment, offended that they gave the intern a job programming as a potential sorry for being (rightly) put in the bin after being a dope twice .
But then I remembered that's what you tend to do with people who aren't up to the task. You give them a promotion.
I like the idea of a user bin. We could even call it /usr/bin
Let me show you how it should be done......
We were replacing the three generating sets used under emergency to charge batteries at a power station. Each diesel-alternator set was about 1MW. I asked why the old sets were being replaced, because after all, they hadn't done much work over the years and the technology had not advanced significantly.
One of the operators took me into the 'old' equipment room where the three engines were open but with tarpaulin covers over the crankcase. Each engine had a broken crank...... The first broke due to an auto-synchronisation fault, allegedly. The second broke during a demonstration of manual-synchronisation with the grid. The third broke when the expert took over to show 'how it should be done'.
The replacement sets were a good order for us so ----> all round.
... half a brain
seriously not required?
I could totally see me doing the same procedure, even if the powers that be frown upon it nowadays. Spoilsports.
I once had a sandal chucked at me (deserved it, every bit). I learned from the best, so to say. Said guy would have flying-tackled me, should I had tried to push the BRB...