Oh! A surprise tour of the data centre! You shouldn't have. No, you really shouldn't have
- Reference: 1630654691
- News link: https://www.theregister.co.uk/2021/09/03/on_call/
- Source link:
Today's story comes from "Guy" (not his name) who was enjoying his second weekend of being on call.
He'd been told that Saturday was the only day when the phone might ring, and so had planned a relaxed Sunday in central London with a new significant other. Not too far from the office should the unexpected happen.
[2]
And happen it did. Just as he disembarked the train, ready to impress his date with the sights and sounds of the big city, his phone went off. It was the on-call DBA and one of the most important database servers was very, very unhappy. Rather than its usual eight CPUs, it was struggling along on only two. Could he check it out, please?
[3]
[4]
The office was located in London's Docklands, so Guy's date was treated to an impromptu tour of the Docklands Light Railway and a demonstration of Guy's skills of persuasion in gaining access to the server room.
"Opening the door to the machine room I was hit by a wall of hot air," Guy told us. "The thermometer just inside the door was reading 40°C, it should have been at 5°C."
[5]
The aircon had failed, although strangely none of the supposed hourly checks by building security had noted it. Almost as if they opted for a peruse of the Sunday papers rather than doing their rounds. Surely not.
Guy reported the failure and proceeded to have a look around the stifling room. Almost every server was showing the yellow service light of distress but amazingly none seemed to have actually died.
There was, however, that problem reported by the DBA. So Guy dug a little deeper.
[6]
"The majority of the machines used were Sun E4500's," he told us, "and unusually E4500's vented cooling air from left to right, rather than front to back."
Of course, the database server was located at the right-hand of a rack. Guy couldn't get too close – the air coming off the server was hot . A thermometer registered it as 70°C.
"What had happened was the DB server was one of three E4500's all in a line. The left-hand server was the dev DB server and the hot air it was sending out was being sucked into the UAT DB right next to it (literally a 2" gap). It further heated the air before it was sucked into the Prod DB server, again only inches away."
[7]Fix five days of server failure with this one weird trick
[8]So the data centre's 'getting a little hot' – at 57°C, that's quite the understatement
[9]See that last line in the access list? Yeah, that means you don't have an access list
[10]Breaking Bad or just a bad breakpoint? That feeling when your predecessor is BASIC
Guy got the go-ahead to turn off the poor thing in order to allow it to cool down. He also shifted it to the shelf above (sadly his new girlfriend was not present to witness him hefting the weighty chassis of one of Sun's finest above his head) while he waited for the aircon engineers to deal with the issue.
With his plans for a lovely day in London now in tatters – it would take hours for the server room to cool down – Guy pondered what to do. The answer was obvious – rope his date into helping him identify all the unhappy servers so he could diagnose their issues.
The old romantic.
As it transpired, only a few power supplies and disks had been properly fried, and those were just on some of the dev servers and so could wait until Monday for repair.
As it turns out, that surprise date in the data centre was a smart move as Guy and his then-girlfriend are now just a few years shy of their 20th wedding anniversary. "She doesn't ask much about my work, though," he added wryly.
And the security guards? Their fate is shrouded in the mists of time, although Guy did note that their discussion with the building manager was "somewhat interesting."
One might almost say "heated."
We've all had a candlelit moment spoiled by the trilling of the On Call phone. Then again, the same persistent ping can also rescue one from the most awkward of rendezvous. Tell us about the time the pager went off at an inopportune time with an email to [11]On Call . ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YTHyYJ4u-tpNtzOyZnfnUAAAAAg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YTHyYJ4u-tpNtzOyZnfnUAAAAAg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YTHyYJ4u-tpNtzOyZnfnUAAAAAg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YTHyYJ4u-tpNtzOyZnfnUAAAAAg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YTHyYJ4u-tpNtzOyZnfnUAAAAAg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2021/08/27/on_call/
[8] https://www.theregister.com/2021/08/20/on_call/
[9] https://www.theregister.com/2021/08/13/on_call/
[10] https://www.theregister.com/2021/08/06/on_call/
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
Once I was on holiday in the USA when I got an SMS from a client asking me why I wasn't answering my mail. I had to phone at 6AM (California time) to explain that I was out of the country.
The interesting thing (from my point of view) was that my associate at the time was also in contact with the client, and he apparently did nothing to handle the issue before they texted me.
He is no longer my associate.
Cliche
I once got called in to see why one of services is down. They had a brand new server delivered a week before and it was plugged in on a table in developers' room. They were desperate for any extra bit of processing power, so they installed everything before racking it in the server room and they planned to move it on a "quiet" day (hope you catch the irony). Anyways. Friday night, got a text that service is down, go check. Indeed there was no ping, so I went to see what's up.
Yep, cleaner has disconnected the network cable.
Re: Cliche
I think it's some sort of special training cleaners get - be sure to unplug everything before you start cleaning.
Re: Cliche
Builders as well. Another time, we were having additional AC put into our server room, as it was being expanded to accommodate a couple of new mini computers...
The builder just strolled up, unplugged one of the microVaxes and plugged in his drill!
The "real" VAXes were fine, they were hard wired and had a huge, physical switch to power them off. But the microVAXes were using standard 13amp sockets.
Re: Cliche
That is how I learnt to save my work regularly...
I was writing documentation on a Mac Plus and it was getting late (18:30). I had everything finished and was putting in the index flags into the document - every word on every page that should appear in the index had to be individually flagged! (I would have thought, creating an Index list would have been more sensible, but, no, instead of typing out a thousand words to index, I had to go through 400 pages of documentation and flag each occurrence of each word individually!
I had been struggling through and saving the document would take a couple of minutes, so I had not saved for a while, to decrease the wait for beer-o-clock...
Then the cleaner bustled in and phup! My screen went blank!! She'd just yanked out the cable for the Mac, so she could plug in the vacuum cleaner! GAAAAAHHH!
I was once contacted to assist in the remote diagnosis of a server that refused to boot. It took a few of hours to get everything straightened out. Mu's company is based in the United Kingdom, but I was standing outside the Olympic Stadium in Montreal (Canada) at the time, which the wife and two progeny had just entered to take a look around.
were you working with the user UCAP at the time? He had a similar issue!
kayaks2fish apparently has a tendency to take other people's posts and recopy them almost verbatim.
I really don't understand that line of thinking.
Echo echo ...
kayaks2fish echoed UCAP on the [1]Perserverence story earlier today too.
[1] https://forums.theregister.com/forum/all/2021/09/03/perserverance_rover_succesful_rock_drilling/
It's like Groundhog Day
It's like Groundhog Day all over again
Once had to sit in the car outside a distillery in Scotland while on holiday explaining to a senior software engineer for a German provider of Mainframe Laser Printers that yes, JES2 got restarted when you IPL the mainframe.
This was in the 90’s and I had the misfortune to be in the only place there was a Vodafone signal in miles – ‘not’ helped that the phone was in it’s car cradle so using the cars antenna to pull in the weak signal.
I did catch up with the family before they went on the tour, after taking the phone off the cradle and switching it off.
I got chewed out on a Friday morning once for not answering the on-call phone the previous evening.
It turned out that the pub I'd gone to do the quiz at was in a dead-spot for phone signal on the network that the on-call phone was using, so it never rang. I filed that information away for further use in case I didn't want to take a call again in future...
One Christmas feast with the family and that poor IT sod got a phone call from a user who noticed a CryptoLocker taking over. Only months later, the IT was fully functional again. And with such incidents, I'm rather glad not to be on the operation side anymore.
I was called out on Boxing Day 2015 after the floodwaters in Leeds entered the datacentre for one of the telephony providers for the customer (ironically the government department that deals with flooding and other environment issues). Cue 8 hours on conference calls agreeing that we couldn't do anything until the telephony was sorted... But it was at double-time and it got me out of the sales :)
Gotcha
I was called in to look at why the warehouse was not running and all of the warehouse computers were showing a critical system error dialogue.
Traced it to the existence of a critical error file in a directory, deleted the file the warehouse came back online.
{takes hero stance}
I had to do it because the usual guy was away on holiday.
Looking further into the issue I found a little program running on a timer that created said file at randomised times good few days between creations.
Turns out you get paid extra for being on call and being called and mucho gravitas by getting the warehouse back up and running, so El Sid had written a batch program that sat there and created the “incident”.
Now that’s mortgage driven development.
year 2000
Year 2000 support. I wasn't called out, but I had to go to the US on Jan 1st - to provide support when the US banks opened.
Jan 2nd I had an email from my wife. Something went "ping" at midnight - what is is?
Jan 3rd another email from my unhappy wife. WHAT IS GOING PING - as it wakes her up. She was tired from doing support.
Jan 4th email from my very unhappy wife. She had taken contents of my bedside table and spread them on the bed and floor and sat up till the witching hour.
We had all been given pagers for the year 2K support - just in case. I had thrown it in the drawer besides my bed and forgotten about it.
At midnight all the pagers were sent a message like "the lottery numbers" etc.
She took the batteries out - and it took a long time for her to forget.
Response time
A conversation I earwigged, not being involved personally. But due to the nature of the issue I could not help but overhear.,
The on-call support person on a particular friday night was someone who "knew their rights" and worked to the letter of their job description.
At the time THE PHONE rang, they had just sat down to a romantic dinner at a local establishment. This was a time when any sort of mobile phone was quite a novelty, before the era of smartphones and laptops. It was not recorded whether that "romance" involved anyone else, but I digress.
It turned out that the support person knew that the conditions of being on call required them to respond within 45 minutes. So 40 minutes after the phone rang (and apparently rang several more times) they called to find out what was interrupting their carefully planned, to coincide with being on call, night out.
The details of the emergency were not part of the post-mortem, but downtime of the main server was said to be in the order of £5-figures per hour. What was apparent was the difference in interpretation of what "respond to a call" meant.
It was explained to the support person that responding meant presenting themselves in body and mind at the premises to perform whatever was necessary to resolve the problem. This explanation was made loudly and in words that left no room for misunderstanding. And that when the on call phone rang, they were pretty damn well expected to answer it instantly.
Production issues of this nature were quite rare, so the depth of the support person's understanding was never tested.
I did get a call once to help remotely diagnose why a server was refusing to boot. Took a couple of hours to sort out. Mu company is in the UK, but at that time I was standing outside the Olympic Stadium at Montreal (Canada) which the wife & 2 progs had just gone into to have a look around.
Never did see the inside of the stadium