Boss broke servers with a careless bit of keyboarding, leaving techies to sort it out late on a Sunday
- Reference: 1669361529
- News link: https://www.theregister.co.uk/2022/11/25/on_call/
- Source link:
This week, meet "Dennis", who in the early 2000s worked for a managed service provider that was in the midst of a datacenter migration.
"Our racks were in a literal cupboard and a newer room was being built next door," Dennis explained.
[1]
While the build progressed, Dennis and his colleagues started moving odd bits of equipment around and tidying things up.
[2]
[3]
"All was going well," Dennis reminisced.
Until it wasn't.
[4]
"On Sunday evening we noticed that things were running slow but didn't think much of it – we just assumed an automated process was taking longer than usual and things would calm down."
Things did not calm down. Instead, they heated up to the point at which everything crashed.
"Late that night everything died. Our firewalls, servers, the VSA platform we used to manage clients' kit, and an on-prem email server."
[5]IT manager's 'think outside the box' edict was, for once, not (only) a revolting cliché
[6]Go ahead, be rude. You don't know it now, but it will cost you $350,000
[7]Hot, sweaty builders hosed a server – literally – leaving support with an all-night RAID repair job
[8]No, I will not pay the bill. Why? Because we pay you to fix things, not break them
Dennis's colleagues dragged themselves awake and into the datacenter, where they found the air conditioning had shut down and server fans were roaring.
The culprit was a keyboard.
[9]
A keyboard?
"The boss had been in earlier that day to do more work on the rack, and had left a keyboard on the floor standing on end, leaning against the fans," Dennis recounted. Combined with pressure from a door, the keyboard had enough force to do some damage.
Dennis found some upside in the meltdown.
"We intended to replace a pair of servers with newer models when the new server room was ready," he explained. Thankfully those servers were already in the building. Not so thankfully, the keyboard incident meant Dennis got to spend Sunday night and most of Monday rebuilding everything.
Dennis also got to respond to some customer complaints about the outage with an explanation that started "The boss came in on Sunday and left a keyboard lying around …"
Has seemingly innocuous equipment caused you grief? If so, [10]click here to send On-Call an email and we may thank you by using your story in a future column. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y4CgSiJoWJTyytjKXAVZbQAAAAI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y4CgSiJoWJTyytjKXAVZbQAAAAI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y4CgSiJoWJTyytjKXAVZbQAAAAI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y4CgSiJoWJTyytjKXAVZbQAAAAI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2022/11/18/on_call/
[6] https://www.theregister.com/2022/11/11/on_call/
[7] https://www.theregister.com/2022/11/04/on_call/
[8] https://www.theregister.com/2022/10/28/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y4CgSiJoWJTyytjKXAVZbQAAAAI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: "an on-prem email server"
It is always on-prem somewhere, isn't it? Yeah, drives me crazy as well. OK, moderately vexed, I guess...
Bosses shouldn't touch stuff...
Having grown from the PFY to the BOFH, to the manager, I'm keenly aware that I can still help out a qualified engineer with crazy ideas or helping to carry a switch that is too heavy to carry alone, but unfortunately I also know not to rummage around anything all by myself, because I've grown into the single most scary thing to any engineer:
I have enough knowledge to do vast damage, but not enough knowledge any more to recover from the damage I caused myself...
Re: Bosses shouldn't touch stuff...
"unfortunately"? I'd call that "fortunately" and probably prefix with "very".
I'm in a similar position myself, having taught a number of PFYs everything they know. However, I was always careful not to teach them everything I know.
Re: Bosses shouldn't touch stuff...
I have enough knowledge to do vast damage, but not enough knowledge any more to recover from the damage I caused myself...
Something that I live by, as it's a good few years since I've been been properly on the tools
There are a few systems here which I sometimes have to refer to colleagues to get me information about - how many records are in such-and-such a table, what's the disk utilisation, etc. All stuff that readily accessible through a management console, but I explicitly said that I didn't want to even have a login for that.
For my colleagues the nuisance factor of me bugging them with the occasional question is far more preferable to me trying to answer the question for myself and accidentally doing some damage that causes them even more work.
Re: Bosses shouldn't touch stuff...
but not enough knowledge any more to recover from the damage
Important to keep in mind at work, but I can't be the only person who got started in IT by breaking things, and then having to learn how to fix them (Quick! Before mum and dad get home!).
Re: Bosses shouldn't touch stuff...
Agreed. I once wiped my Dad's AUTOEXEC.BAT and CONFIG.SYS accidentally by trying to install a game that we'd picked up from Finmere market that day.
I was 9.
He was not happy.
Cold and hot
Yeah, the server room is a strange place... cold in some places, like on the floor against the opposite wall the airco is mounted on. We used to put our lunch boxes there. One day a colleague brought in some goodies for a birthday party. They asked if they could put the bag in the server room. At this point in history, my office was adjacent to the server room, and had a door right from my office to the server room, which, back in the day was unlocked during the day. My colleague popped into the server room and deposited his bag of party snacks and was on his way. Later, I had to swap the backup tapes, and when I entered the server room, there was this odd smell of melted plastic. The shopping bag my colleague put in the server room, wasn't on floor where we all put our lunches, but behind a rack, so not to be seen by hungry eyes... It hadn't started burning or anything, but was a good approximation of shrink wrap on one side of the bag.
Needless to say, my boss at the time ruled out anyone other than server admins storing their lunch in the server room. I'm sure I should have had a good lecture from that, but I think if our colleague had asked my boss instead of me, my boss would have done the same thing, say yeah, and let our colleague proceed...
Re: Cold and hot
> anyone other than server admins storing their lunch in the server room
I love that the solution to this wasn't "food should not be in the server room"
Belt up
I heard about someone who wanted to move some disks - about 6 inches - while they were running; this was to make some space to get some more kit in.
They carefully put their backs against the disks, bent their legs (health and safety was important) and pushed. That worked fine until someone stood up. He caught his belt in the Emergency Power Off button - a "Pull" button, (to prevent accidents with people pushing it) and suddenly it went quiet.
This caused an immediate outage, and about half a day to recover. You cannot just power it on again.
Fortunately the guy's boss was one of those pushing the disks, and he got into trouble,
"an on-prem email server"
Or "email server" as we used to call them.