Nobody does DR tests to survive lightning striking twice
- Reference: 1688715011
- News link: https://www.theregister.co.uk/2023/07/07/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Karl" who once worked in the services limb of an enormous and venerable three-letter corporation's global services operation.
"We supported the mainframes and datacenter operations for a large international hotel and resort conglomerate" Karl told On Call.
Nothing quite enhances the senses like a dark, quiet and smoky datacenter
Karl and his colleagues were required to clock 45 hours a week on their timesheets, but he rated the gig "fairly cushy" and mostly involving "server crashes and failed disks."
But the night a thunderstorm rolled past the hotel conglomerate's main datacenter was not cushy.
[1]
"Phones and pagers started going off around the world," Karl recalled.
[2]
[3]
Upon arrival at said facility, Karl found a tree split in half near its exit. That was a graphic clue about what had happened: lightning had struck, backup power had failed, and acrid smoke had everyone spooked. The main casualty was a power conditioner near the exit door, and its demise caused a power surge that "jumped every safeguard, and took out nearly every Ethernet port in the building too."
Under emergency lighting – which survived the lightning strike – Karl observed that "Nothing quite enhances the senses like a dark, quiet and smoky datacenter."
[4]
In that spooky atmosphere, Karl and his colleagues "spent 12 hours restoring service, swapping parts from spares, and parts from the local depots, alternating shifts to keep people fresh." Happily, the mainframes came up, and seemed no worse for wear. After a long slog – performed under the pressure of the hotel chain's desire for mission critical apps to resume operations – the end of the job was within sight.
[5]Quirky QWERTY killed a password in Paris
[6]False negative stretched routine software installation into four days of frustration
[7]That old box of tech junk you should probably throw out saves a warehouse
[8]Cunningly camouflaged cable routed around WAN-sized hole in project budget
We opened this On Call with the adage that lightning never strikes twice.
As he prepared to finish the restore job, Karl learned that's not true.
"A second lightning strike sent a second power surge coursing through the building, throwing all power out the window, and we literally just sighed, and flipped our playbooks back to page one, assessing the situation from scratch."
"The same fire crew showed up, the same techs were called, the same parts from stores were pulled out, the same careful fingers-crossed power cycle of the mainframes occurred and amazingly, they all came up without a problem."
[9]
After more than 24 hours of disaster recovery, Karl could pack up and go home.
"The hotel chain lost a little revenue, and the big global services firm got a fat check," he recalled.
But Karl and his pals got nothing.
"Not a pat on the back, a pint at the pub, or any overtime pay."
Which was why, not long after this incident, sysadmins sued for not paying overtime.
"They lost, we won. But didn't really win either," he observed, before adding: "The moral of the story is 'Nothing stops lightning, and no one wins but the lawyers'."
Have you ever worked on a job that turned into double trouble? Or not been properly paid for your troubles? [10]Click here to send On Call an email detailing your experiences and we may give it a run here on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZKfiSBJCarbLiPg-uku5XQAAAgE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZKfiSBJCarbLiPg-uku5XQAAAgE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZKfiSBJCarbLiPg-uku5XQAAAgE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZKfiSBJCarbLiPg-uku5XQAAAgE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/06/30/on_call/
[6] https://www.theregister.com/2023/06/16/on_call/
[7] https://www.theregister.com/2023/05/26/on_call/
[8] https://www.theregister.com/2023/06/02/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZKfiSBJCarbLiPg-uku5XQAAAgE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: No Good Deed Goes Unpunished
I did the same thing and after a while I pointed out that because of the amount of TOIL I was getting (same as you, time and a half, double at weekends), it'd be worth my while doing some small bits and pieces on the side for the clients. My boss pointed out that I can't do that because there is a clause in my contract prohibiting me from performing any work for clients on the side...so I read the clause and the clause in the contract read "...within the scope of works as set out in the client contract...". Since the sales team and legal team were very keen to stay ahead of any edge case complaints, they made the scope of work for each client extremely specific and classified anything outside of the extremely specific scope as "best endeavors", as in, they will assist with it, but it isn't deemed as "in scope" therefore if anything goes wrong, no claim can be made against the company regarding poor service, negligence etc...but here is the nice bit, all the really valuable work (server side admin, hardware upgrades, Linux stuff etc etc...basically anything that wasn't desktop support or really easy), was outside of the scope...I pointed this out to my boss..."yeah, but the scope of work in the customer contracts is really specific, there's a ton of work I can take on during TOIL and I can make a killing without having to take money out of the business"...well he went white as a fucking sheet, the lawyers stepped in and insisted on giving me a 30% payrise and they cancelled my acces to TOIL in favour of paying me 2.5x for each overtime hour I did...and yeah, you bet your ass I took the piss...we had a three shift pattern, 7am til 2pm, 9am to 5pm, and 11am to 7pm...you can bet your ass I never took the 9 to 5 slot...I took the 7 til 2 slot and booked 3 hours of overtime a day, 5 days a week for years.
Re: No Good Deed Goes Unpunished
We had a weird problem with some kit on another continent and we had to send a development engineer out to fix it instead of one of the normal support team. We estimated a 3 day trip and he was gone over 2 weeks and left a very happy customer. When he put in his expenses his engineering manager wouldn't authorise a lot of the expenses cos they were outside policy. E.g. he'd only packed enough clothes for a couple of days so he'd had his clothes laundered a few time by the hotel and bought some other odds and sods he hadn't packed. The customer PM went ballistic and read the riot act to the engineering manager, who signed off the exes. The engineering boss, in a fit of contrition, sent some flowers to the engineer's wife to apologise for him being away so long. She sent them back to him with something pretty unpleasant written across the card.
I guess computers have moved on and many now have a dedicated lightning port...
Not for much longer, now the EU has poked its fingers in…
It's all Thunderbolt™ now...
Naah.. Universal Surge Bus 3
This is getting a bit too Bohemian Rhapsody for me.
Cue Galileo and fandangos...
In the future all lightning must be cloud based.
At least you fixed the problem.
I was once at the TalkTalk colo datacentre in Milton Keynes (don't know if its still there) during a downpour, decommissioning a load of customer kit to relocate it to another DC...the roof caved in and there was water gushing into the datacentre everywhere...thankfully though, not on my racks, which were almost dead centre...I was there all day working on my own kit and I observed precisely zero people coming in to sort out the racks that got drenched...I had a wander over to some of the, by this time, dead racks...and I noted at least a few million quids worth of kit that was now soaked and ruined...most of which were storage arrays...so quite a few people, somewhere were having a really shitty day.
I really hated that datacentre...pretty much any datacentre in MK to be honest...it's far enough away from London to be relatively cheap, but also just far enough away for the trip to have a long boring journey...I've also never been in a nice datacentre in Milton Keynes...they're all miserable...several of them have pool tables but they all have at least 1 ball missing, cues with no tips...broken coffee machines...backed up bogs...a miserable twat on the front desk...walking around them is creepy, like walking around a derelict wreck from the 90s.
Re: At least you fixed the problem.
To be honest that description of the DC extends to most of Milton Keynes in general
Re: At least you fixed the problem.
"To be honest that description of the DC extends to most of Milton Keynes in general"
Thanks, just what I needed to be reminded of while packing my bags to go and visit the family in MK!
Though Gallery MK hosts some good exhibitions and the National Museum of Computing offers many moments of nostalgia to older Commentards.
And this looks potentially interesting: https://miltonkeynesmuseum.org.uk/collections/view-our-collections/#Telephone
A good weekend to all Commentards -->
Re: At least you fixed the problem.
I'd concur on both TNMOC and MK Museum. Last time I visited the latter it was with someone who'd actually operated the manual exchanges displayed during/shortly after WW2.
Re: At least you fixed the problem.
There's always Bletchley Park too...
Re: At least you fixed the problem.
I was once at the TalkTalk colo datacentre in Milton Keynes (don't know if its still there)
TalkTalk and Milton Keynes are still with us, sad to say...
II've also never been in a nice datacentre in Milton Keynes.
Datacentre and Milton Keynes should set off very loud alarms warning you that there isn't going to be any niceness.
Donkeys years sinceupon, when 8 inch disks were still a thing, a work chum was despatched from our Luton office to take a raft of bits to a water treatment works in Diss, Norfolk, which had been on the fizzy end of a lightning strike.
I got involved when he rang in to say that he'd found the problem - an 8 inch floppy unit had been fried and he'd replaced it. Everything was then fine he said, until the second lightning strike fritzed the replacement.
Guess who got stuck with a panic run to Diss and back.
I think we did something about power arrangements but I've been asleep a few times since then and I'm buggered if I can remember what.
Sadly my old chum is no longer with us but I do remember him in the occasional lightning storms we get here.
No Good Deed Goes Unpunished
Not paying for the overtime? Wow, that is really shitty.
In a previous role I provided out-of-hours and on-call support and the official company policy was to book extra work as overtime in the first instance, or take time off in lieu (TOIL). After booking my first batch of out-of-hours overtime my manager said there wasn't a budget for this so I'd have to take it as TOIL. I checked for the company's policy document on this to see how much TOIL to take - the equivalent of the hours worked as overtime, or the equivalent of the hours worked at the overtime rate, which was at time and a half. There was nothing in the document to specify what TOIL to take, so I booked it as equivalent to the overtime rate.
Naturally my manager queried this, so I explained that if the company were unwilling to pay me, say, four and a half hours pay for three hours overtime at time and a half, then they would need to give me four and a half hours of time off in lieu. I referred him to the policy document and said it wasn't detailed in there so I'd interpreted it at face value, but that I was more than happy to go and have a chat with my union representative to get their take on it. My manager said it was fine and that's how I booked my TOIL for over five years.