Your trainee just took down our business and has no idea how or why
- Reference: 1713511812
- News link: https://www.theregister.co.uk/2024/04/19/on_call/
- Source link:
This week, meet a Reg -admirer we'll anonymize as "Leanne" who once worked as a senior architect in a mainframe environment – a role that saw her keep an eye on several of internal chats in which colleagues would share news of glitches and oddities she would be asked to check out.
One day a senior database administrator pinged one of those chats with the unwelcome observation that they were pretty sure some files had gone missing. Could Leanne check the mainframe?
[1]
Leanne's console told her all was well with the big box. But multiple threads soon reported problems. Severe problems.
[2]
[3]
"People couldn't log in, the business was down, SEV-1 tickets were stacking up. Alarm bells were ringing," Leanne told On Call.
But according to the screen she saw, the mainframe was alive and well.
[4]
A little digging into system logs revealed the mainframe was upset because requests to the mighty machine's direct-access storage device (DASD) were failing.
Leanne got together with colleagues and her employer's outsourced maintenance provider to investigate the situation.
The problem was obvious: the DASD had been deliberately taken offline. The mainframe was therefore unable to access its only source of storage.
[5]
Who had done this deed?
[6]Support contract required techie to lounge around in a $5,000/night hotel room
[7]Techie saved the day and was then criticized for the fix
[8]Do not touch that computer. Not even while wearing gloves. It is a biohazard
[9]What's brown and sticky and broke this PC?
A trainee from the mainframe vendor admitted they had flipped the switch.
"I put in the wrong command, and then the system wouldn't let me put in the right one," was the rookie's excuse.
Leanne assessed the situation and decided the trainee "had no idea what she was doing and did not know what she had done."
What she had done was take a big chunk of a Fortune 500 company offline. On Monday morning, just as the week was getting busy.
Happily, Leanne's position meant she had privileges that allowed her to reconnect the DASD.
Which she did ASAP, averting disaster.
"The vendor was read the riot act for putting an unsupervised trainee on the console, and the trainee got to listen to her management get in trouble with the customer," Leanne told On Call, rating the incident as "Fun, fun all around."
And maybe, also, one of history's great debrief meetings and/or lessons in giving a vendor a kicking for doing exactly what they're paid not to do.
Has an ignorant kid broken your boxes? Or has an outsourced supplier brought you undone? On Call wants your story so we can share it here on a future Friday, and the good news is sending it to us is as simple as [10]clicking here to send us an email . ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZiJAxiDzx9FxbJkKV2AB5gAAAFM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZiJAxiDzx9FxbJkKV2AB5gAAAFM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZiJAxiDzx9FxbJkKV2AB5gAAAFM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZiJAxiDzx9FxbJkKV2AB5gAAAFM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZiJAxiDzx9FxbJkKV2AB5gAAAFM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/04/12/on_call/
[7] https://www.theregister.com/2024/04/05/on_call/
[8] https://www.theregister.com/2024/03/29/on_call/
[9] https://www.theregister.com/2024/03/22/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Whoopsie!
We looked after a small operations servers (machines on customers site, in broom cupboard, connected via leased line) and there were a stack up updates to install which required multiple reboots (manual and automatic) so it was decided that the updates would be applied over a bank holiday weekend when servers pretending to be yo-yo's wouldn't impact staff.
The first couple went fine, connected in using VNC applied the OS updates (auto reboot) and then applied software updates (manual reboot).
3rd machine went fine until the manual reboot, "Start", "Shutdown" "Enter"..... Shit, that said shut down not restart!!!!
And that is how I ended up having to drive 200 miles to press a power button
Re: Whoopsie!
> And that is how I ended up having to drive 200 miles to press a power button
...and in a perfect world, a followup discussion about the upgrade led the clients to use servers with a baseboard management controller, like HP's iLO or Dell's DRAC; so (almost) everything can be done remotely.
For an extra fee to the server vendor, of course.
Re: Whoopsie!
Agreed, this is definitely the primary fault of the vendor. You could argue that the rookie/trainee should have asked for help instead of pressing the big red button, but the point is if you give a trainee the power/ability to do this then don't provide sufficient support for them, you're asking for disaster to occur sooner or later.
Re: Whoopsie!
And yet at some point the trainee has to fly solo. Nobody said it was easy.
Re: Whoopsie!
" Fortunately, it failed over to the other node "
Which means... nope, totally not a brain fart. You were conducting a resilience test to ensure that the system is working as it is supposed to.
Re: Whoopsie!
You'd think the vendor would have had the forethought to tell the trainee not to do something unless they were certain they knew what they were doing, but more importantly, to check with the client first to see if it was ok to actually do anything.
You'd think so, but I have experienced (and witnessed) that particular form of genius management whose idea of training is to yell at the new kid to get it sorted out and to not embarrass anyone by asking "stupid" questions. Which tends to result in chaos, predictably. Well, predictable to anyone but that particular type of manager. The article kinda reads like that, and it might explain the outcome, too...
Leanne's console told her all was well with the big box
If the only storage for the mainframe was offline yet "Leanne's console told her all was well with the big box" then it tells you there was something not right in how she was monitoring things.
Re: Leanne's console told her all was well with the big box
Unless Leanne's employer was the Post Office.
I hope the kid wasn't sacked.
We've seen enough of On Call to realise no one has protection from the hurt ego of a manager.
I hope they weren't sacked, but I would imagine an embarrassed manager would have gleefully signed the kid's P45.
so the kid had enough privileges to take the storage offline but not enough to bring 'em back online - that in itsself is somewhat dumb to my way of thinking
but no getting away from the fact the kid should never have been let loose unsupervised...
The "privilege" for taking something offline is rather easily obtained. You can also do it with a screwdriver. However, bringing it back online... that is a different story.
> You can also do it with a screwdriver
You need to mention that to El Reg readers?
(Cough) grandmothers, sucking eggs (cough)
A couple of decades ago, we were showing the new DBA around the datacentre and he logged onto the production server console - he hadn't got his laptop yet - and checking out his permissions, etc. He went to logout, but instead of clicking on "logout" the idiot moved the cursor to "shutdown" and we weren't fast enough to stop him. Luckily it was a cluster and failover worked so users had a temporary blip. He never really lived that down...
A few years later, at a different vendor, a hardware engineer was walking towards the DC door carrying his toolbox and various spare parts and reached out to press the door unlock, but for some reason known only to themselves, an idiot had put the emergency power down near to the door - and the engineer pressed that by mistake. Cue suddenly silent DC and sounds of panic as the helpdesk was swamped with calls from 5-6 councils and a few businesses...
Also fire alarms
I've not seen that with the emergency power, but I've seen loads of fire call buttons next to light switches near doors.
Reach round frame to turn lights on... oops...
Not so bad if they have Molly Guards.
"silent DC and sounds of panic as the helpdesk was swamped with calls"
Providing you can power up quickly enough the trick is to tell them not to answer the phones for a few minutes then tell them to check again, it's all runng here so it must have been a comms failure at their end. Providing the customers don't get together to compare notes.
Whoopsie!
You'd think the vendor would have had the forethought to tell the trainee not to do something unless they were certain they knew what they were doing, but more importantly, to check with the client first to see if it was ok to actually do anything.
Personally, I once had a brain fart and shutdown a node of an Exchange 2003 cluster. Fortunately, it failed over to the other node as it should have done and I had to connect to the ILO and power the server back on.