So the data centre's 'getting a little hot' – at 57°C, that's quite the understatement
- Reference: 1629444965
- News link: https://www.theregister.co.uk/2021/08/20/on_call/
- Source link:
Our story takes us back to the halcyon days of 2014 and a medical facility where our hero, Regomised as "Chris", worked.
As has been the trend over recent years, the on-premises data centre for which Chris was responsible was not large ("a couple of rows of servers," he recalled) but still needed cooling and environmental management.
[2]
On the night in question, Chris (being on call) was notified that there was a problem with the air conditioning. A co-worker, who lived closer than Chris, has already been in and said "things were getting a little hot."
[3]
[4]
An understatement, as it turned out.
Chris instructed his colleague to shut down whatever he could and embarked on the journey to the site.
[5]
Awaiting him was what remained of his colleague: a sweat monster. Things were indeed "getting a little hot."
"After a few well-placed words from myself, we started to evaluate it even closer."
The AC wasn't just broken, it was actually blowing hot air into the room, heating things up even further. "The room was close to 135° Fahrenheit."
[6]
That's over 57° Celsius. Not ideal. Forget clean shutdowns; the servers needed to be turned off now .
The engineering team was consulted on exactly how bad hitting the Big Red Button would be, although in the end the new power distribution system (installed recently) was used "to bring servers down quicker but not cleaner."
How did Chris know the exact temperature reached in the room? "Half the sprinkler heads popped," he told us.
Which raises the question of why Chris and his co-worker were not treated to a soaking by a suppression system convinced things were on fire. "The engineering people had done a test at one time and forgot to close a valve," he told us.
[7]See that last line in the access list? Yeah, that means you don't have an access list
[8]Breaking Bad or just a bad breakpoint? That feeling when your predecessor is BASIC
[9]Malware and Trojans, but there's only one horse the boss man wants to hear about
[10]Exsparko-destructus! What happens when wand waving meets extremely poor wiring
"So instead of dousing servers and storage with water, it all went out an exit pipe."
"I was very glad they had not made that work right," he confessed.
"Sadly, though, they did tell me later on they would fix it so it would work later.
It took a 23-hour stint on site to bring everything back online.
"Hopefully," said Chris, "they never 'fixed' the sprinkler after I left."
We can but hope that whatever the eventual repair was, the engineers took the opportunity to move on from just water in the DC.
Ever had your bacon saved from a frying pan encounter thanks to negligence of others? Share your story with an email to [11]On Call . ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YR99YZCeJW0mXzoPBVHyzAAAAEY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YR99YZCeJW0mXzoPBVHyzAAAAEY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YR99YZCeJW0mXzoPBVHyzAAAAEY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YR99YZCeJW0mXzoPBVHyzAAAAEY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YR99YZCeJW0mXzoPBVHyzAAAAEY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2021/08/13/on_call/
[8] https://www.theregister.com/2021/08/06/on_call/
[9] https://www.theregister.com/2021/07/30/on_call/
[10] https://www.theregister.com/2021/07/23/on_call/
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
That reminds me...
of something very similar that happened when I was a server engineer. There was an urgent call from the DC that the AC had fallen over (turns out the seawater feed pipe that drew water in from the nearby harbour had been blocked) and things were getting rather toasty. Our solution was a lot easier (and didn't require turning anything off): we opened the windows once Security had been notified that the alarm was about to go off. Only downside was that they faced a street so we had to post people by each open window to ensure nobody tried to get in.
Temps came down rather quickly and an interim solution (big fan, open internal doors) was put in place until the pipe was unclogged.
Have posted this story before, but always worth repeating :)
Re: That reminds me...
"so we had to post people by each open window to ensure nobody tried to get in."
Also, to ensure nobody tried to get out...
Re: That reminds me...
We had the same idea when there was an AC failure. Unfortunately, the systems were graded "UK Restricted", so security weren't that happy, even though this was on a campus site.
They allowed us to man the open windows and doors for just long enough for us to do as clean a shutdown as we could, and then we had to close everything up again and do an asset check to make absolutely certain everything was still present in the server room.
Be careful with your sprinklers
Setting the scene: Three standby generators in individual cells; Set one complete, set two semi-complete, set three in assembly with covers off and engine open.
The customer wanted to check their sprinkler system before accepting the first plant into service (even thought the station need two to be operational). So the machinery in the 'completed' cell 1, engine, alternator, etc. was wrapped in polythene and tightly protected for the test. Naturally, the customer wanted to witness the event and they donned their deluge-protective clothing: think Hazmat suit although Captain Birdseye might be more appropriate. All other personnel had to clear the building.
We guessed something was amiss when my shoe floated out of cell 3. In their co-ordination meetings between the various contracts, they had omitted to label the cell-machinery in a uniform manner. Sprinkler cell 1 was not Engine cell 1........
As the dogged, suited Captain, still completely dry, stepped from cell 1 and into cell 3 he kindly offered his handkerchief to help dry-off a 1MW alternator spouting pretty waterfalls. Early pub lunch for us.
I once had to do something similar in a Skoda...
Nose to tail traffic because there's nothing like having a busy A road go from dual carriageway to single and back again... And the engine overheating light comes on. Slow enough not to cool the engine, fast enough that I couldn't stop the engine.
Cue turning the heating on full blow - high heat. Did the trick in saving the engine but cooked the occupants somewhat.
Re: I once had to do something similar in a Skoda...
In a similar situation with my 1st car, I had the windows & sunroof open (top spec when new - £100 to me :-))
Re: I once had to do something similar in a Skoda...
I've had to do the same - unfortunately in a tropical climate, which didn't improve the passenger experience.
Re: I once had to do something similar in a Skoda...
I once had a similar experience in sunny New Mexico in the late 80s or early 90s - a somewhat older vehicle was standing by the side of the highway showing clear signs of overheating. We stopped and offered assistance and showed them the trick of running the heater to help cool the engine.
They drove off happily, and we followed for a bit to make sure all was well. Sure enough, after a little while, they pulled over again. Turned out it was too hot in the car with the heater running (it was nearly 40 degrees outside), so they'd switched on the A/C, and a late 70s/early 80s A/C compressor represents a significant engine load...
Advised them to roll down the windows instead, which got them to the next exit.
Re: I once had to do something similar in a Skoda...
Nothing ICT related, but that car story reminds me of the time, many years ago, my (now ex-) wife and I were taking a caravan from near Bristol to her parents who had moved into a former croft near Inverness. It was an old twin axle 16 beast and my 1200cc Mk1 Cortina would have stood no chance even shifting it. We borrowed my brother-in-law’s “tweaked” Triumph 2000 (triple carbed, high lift cams, etc). It managed to tow fine (even in overdrive on the motorway).
All well until we turned onto the final bit of the journey, off the main roads. We came to a steep hill (1:4, I found out later, when I got the map from the missus). All still OK until we came up behind a coach. The car’s temperature gauge started to creep up, something my brother-in-law had mentioned as a warning. Nothing for it but pull out - straight section of road, nothing coming - and floor it in bottom gear. The look on the coach driver’s face as we passed was a picture…
I got to respect that car!
Re: I once had to do something similar in a Skoda...
Going back about 50 years that wasn't all that unusual. A relative had to drive everywhere for months with the heater on full blast to prevent the engine overheating. Very unpleasant on a hot day on the M1 with all the windows wide open.
I can't remember now (if I even knew) whether he ever fixed it or scrapped the car.
I can finally tell this one!
Back in the 90's the company I worked at upgraded the mainframes from 2 ICL ME29's to a nice shiny ICL Series 39 Lv40 with the corresponding disk cabinet and tape drives for backup. The problem was that the replacement took longer than expected so the air-con in the room was struggling to cope.
Initially this wasn't a problem, the 2 ME29's would run the production work while the S39 was being commissioned, things got a little warm but the S39 was shut down at the end of the day. However once production jobs started running on the new box we couldn't shut it down as easily. As everyone except project managers knows, projects always have snags, in this case it was that some of the batch jobs needed more work before they could be moved over which meant that for a few weeks we would be running both MEs & the S39 from 8am to 11pm.
In the first day it got warm, then hot, then hotter! Then the over-temp alarm in the S39 started going off, a quick check of the manual showed that the alarm would be followed by a shutdown, this would be sub-optimal so a call to the friendly ICL field engineer followed. He told us that there was a reset button located just under the top vents in the box & that pressing it would silence the alarm for 30 mins and cancel the shutdown.
For the next week there was a step-ladder beside the S39, after about midday whoever was on shift would leave the computer room because it was too hot to work in for more than a few mins at a time. When the temperature alarm in the S39 went off, someone would dash in, go up the step-ladder, reach in and hit the alarm reset. This would then repeat every 30 mins throughout the day until all the batch jobs had finished and everything could be shut down. Remote operator consoles were used to monitor things like tape requests etc & these would be done as fast as possible to avoid the heat in the room.
After a week the management had a big chiller unit installed. Of course, after the old Me29s were decommissioned the room was usually too cold to work in because the new chiller was not linked to the air-con and constantly blasted chilled air into the room.
'Dave's syndrome. Poor bastard.'