Junior techie had leverage, but didn’t appreciate the gravity of the situation
- Reference: 1705653125
- News link: https://www.theregister.co.uk/2024/01/19/on_call/
- Source link:
This week, meet a reader we’ll call “Robert” who told of his time working for a local government organization in the UK.
Robert gave one of his junior minions the job of installing a 6kVA uninterruptible power supply (UPS) in one of the server racks he serviced.
[1]
On Call just searched for UPSes of that capacity. We found 4U machines that weigh in at 65kg – 7.47 adult badgers according to The Register ’s famous [2]standards converter .
[3]
[4]
“The minion duly wandered off along the service corridor, and vanished into the depths of the server room,” Robert told On Call. And a couple of hours later the minion returned with news he’d done the job, other than hooking it up.
Robert decided he and a colleague who served as senior networks engineer should check the work had been done correctly.
[5]
It had not – the 65kg UPS had been installed in the topmost space in the middle rack, definitely not a good spot to put something that heavy.
“Said minion was called back, grilled over the choice of installation space, and told to put the UPS in the lowest free space in rack number one of seven,” Robert wrote.
The appropriate hoist was duly retrieved so the minion could do the job again and Robert again retired to await its completion.
[6]
That revery was rudely interrupted after around 20 minutes by a thundering crash.
[7]While we fire the boss, can you lock him out of the network?
[8]Tech support done bad sure makes it hard to do tech support good
[9]People power made payroll support in putrid places prodigiously perilous
[10]CEO arranged his own cybersecurity, with predictable results
The next sound Robert heard was shouts of complaint about the network being down.
"The minion had decided to pull the UPS out on rails at the top of the cabinet before getting the server crane into position, resulting in a huge imbalance," Robert revealed.
As the giant battery reached tipping point, it pulled seven server racks to an angle that meant many of the fibres and cables connected to kit they carried snapped.
“Suffice to say, a bollocking of biblical proportions was levelled at the minion, and his server room access was removed,” Robert wrote. Security was summoned and the minion made a walk of shame out of the office, as he’d been suspended from work pending an enquiry.
The fix took three weeks, and the minion was reduced in rank until such time he could be trusted again.
And Robert? He knows it was a mistake to let the minion do the job unattended.
Has trusting a job to a junior turned to disaster? If so, [11]click here to send On Call and email and we may feature your story on a future Friday.
Keep the yarns coming, folks. On Call always needs more candidates – especially as we close in on our 500th edition! ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZapWXsPRXf4mTLB9h6sHDwAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://www.theregister.com/Design/page/reg-standards-converter.html
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZapWXsPRXf4mTLB9h6sHDwAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZapWXsPRXf4mTLB9h6sHDwAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZapWXsPRXf4mTLB9h6sHDwAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZapWXsPRXf4mTLB9h6sHDwAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/01/12/on_call/
[8] https://www.theregister.com/2024/01/05/on_call/
[9] https://www.theregister.com/2024/01/01/on_call/
[10] https://www.theregister.com/2023/12/29/on_call/
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
Re: Curious 6000kva?
Yes, you would need to take roof off and crane it in! 6000kva is roughly 6 MW capacity.
Re: Curious 6000kva?
It probably has a lid on top to pur in heavy water, and a dial to retract the carbon rods...
Can I go to the pub now?
(drat, they don't open that early, this side of the channel the "pub lunch" is not a thing...)
Re: Curious 6000kva?
And a nice blue glow around it when suspended in water, I assume
Re: Curious 6000kva?
Swimming-pool reactors and Cherenkov radiation. Man, I love those colors!
Re: Curious 6000kva?
Here in Spain there aren't many pubs, but there are plenty of bars and cognac for breakfast is not unheard-of in the heading-for-retirement generation and above.
Re: Curious 6000kva?
6kVA or 6000VA would make sense, 6000kVA would need multiple cabinets on its own.
Personally I think the wrong person was suspended - the junior should never have been tasked with the installation on their own.
I agree completely. This is the fault of management. An experienced person should have been present to ensure that the junior understood the requirements and pitfalls of the task.
It would seem logical to install a heavy equipment as low as possible - I know I certainly wouldn't have thought of getting it to the top of a rack, even when I was only 20. But the human mind is a curious thing, and who can say what criteria the junior had in mind for the task ? Maybe he was thinking that it would be more practical to plug something into the top rack, instead of crouching down to the bottom rack. It's an argument that is not without merit, but you have to discount the weight. Experience tells you to never discount the weight.
Not to mention that a 65kg 6kVA UPS must have a certain cost associated to it. An experienced engineer should definitely have been overseeing the operation.
Logically the electrons benefit from gravitational help to keep the rack running smoothly.
But won't it be harder to push them uphill when recharging?
Overhead transmission from the grid sees to that!
It would seem logical to install a heavy equipment as low as possible
It would, but when the then-new City hospital was being built in Belfast many moons ago the architect put the X-ray unit on the top floor of the tower, seemingly unaware of the weight of lead screening. That was a wee bit harder to fix.
Too heavy
Southampton University had a related problem in the 60s when the MechEng tower block had to be balanced by a suitable distribution of equipment.
Human Nature
It's perhaps a bit too easy to flog Robert, too hard, for his part in this. Smart people frequently have a hard time imagining how non-thinking less-intelligent people can be ("Ain't i' fookin' obvious t'at ya don' put t'e heavy t'ing in t'e top 'o t'e rack 'cause t'at makes it EXTRA -tippy?!!").
That said, Robert's job was supervisory, and he should at least have evaluated the minion's previous experience and typical thought processes (or lack thereof) before turning him loose. Perhaps Robert did previously evaluate the minion, with RESULTS==OK, but the minion had two brain-farts that fateful day.
It would seem logical to install a heavy equipment as low as possible
Indeed, but the minion was "told to put the UPS in the lowest free space in rack number one of seven". They may have taken "lowest" as meaning lowest number with slot numbering going top-to-bottom rather than lowest meaning physically lowest to the ground.
I know from experience of dealing with people who are 'on the spectrum' that they can tend to interpret instructions in a way which is technically correct but not what others would intuit.
More than enough blame to go around
> He knows it was a mistake to let the minion do the job unattended.
"Robert" was lucky that his role as supervisor remained intact. Places I have worked with *big* datacentres do not allow people to work alone.
Re: More than enough blame to go around
Ah yes absolutely true, but you forgot "working for a local government organization in the UK." The usual modus operandi here would be to quickly look around for someone to pin the blame on, and ideally the most junior person they can find.
On the plus side, it was a good test of their disaster recovery procedures! I wonder if the UPS survived to be used again?
Re: More than enough blame to go around
> working for a local government organization in the UK
You would hope the union would have something to say
Just one question - how did the minion install said UPS the first time round without tipping things over?
Friday, almost time for a walkabouts and a beer at the local. ---->
Obviously, to crane the thing up, he had to have it loaded correctly on the ground. That worked.
He just didn't load it correctly to get it back down.
Agree that the wrong person was suspended. Purely from a manual handling/H&S perspective that surely should have been a two person job.
Pun-ishment?
Report to the torturer for some time on the rack...
Our training programme is designed to stretch our employees.
Curious 6000kva?
"installing a 6000kva uninterruptible power supply (UPS) in one of the server racks he serviced."
Wondering whether that is 6kVA rather than 6000kVA?
A serious ask of 4U box weighing 65kg I would have thought?