News: 1667547128

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Hot, sweaty builders hosed a server – literally – leaving support with an all-night RAID repair job

(2022/11/04)


On Call As two precious days of free time loom, The Register offers a two-minute taste of the energy-draining drama that makes weekends so important in another instalment of On-Call, our weekly reader-contributed tale of the messes IT people are asked to address.

This week, meet "Oswald" who sent a story of pulling an all-nighter to restore an email server in around 2003.

Oswald had not planned to work all night. Indeed, the office in which he toiled was no place for it – part of the building was being reconfigured to increase the size of the server room.

[1]

"The project involved building the new server room first, moving the kit in, then demolishing the old one so the main building work could begin," Oswald explained to On-Call.

[2]

[3]

While all the necessary wiring was arranged, the main aircon was out of action. A temporary unit was placed in the new server room and a substantial hole was cut in the wall so its hot air could be vented to the rest of the construction site.

This arrangement worked, and the email server Oswald tended kept humming. Until it didn't.

[4]

"I'd built the email server myself and it had always been reliable, so I went down to the server room to have a look, avoiding all the rubble and dust as I walked through the building site," Oswald explained.

He then pieced together the problem.

"It was a hot summer, and the builders had understandably started to tire of having hot air con exhaust blown over them as they worked. So they figured a solution: add more ducting – around ten meters worth – and redirect the hose out of a window."

[5]Switch survives three hours of beer spray, fails after twelve

[6]No, I will not pay the bill. Why? Because we pay you to fix things, not break them

[7]To make this computer work, users had to press a button. Why didn't it work? Guess

[8]No, working in IT does not mean you can fix anything with a soldering iron

That sounded like a splendid plan – who wants to encounter hot, sweaty, builders?

For Oswald, that terrifying prospect may actually have been preferable. As it turned out, that ducting was the reason his email server died.

[9]

"The portable air con didn't have the power to push its exhaust down that length of corrugated tubing," he explained. So lethargic hot air met construction dust and the server's air filters clogged, things heated up, and its disks degraded until email crawled to a halt.

"I proceeded to spend nearly 24 hours (including taking the server home in a taxi that evening, and back into work the next morning) coaxing the bits off the disks onto new ones." That enormous effort was needed because, while Oswald's employer had backups, the incident would have cost about 12 hours of emails.

"The company would rather be without email for a day than lose data," Oswald explained, so "In the end I got every bit off the old RAID disks and reinstalled the server early the following morning with no data loss."

The word "hero" gets used too much these days. But what the hey, The Register might just use it now.

Have builders ever botched your job? Have attempts to extend infrastructure come up short? Let us know by [10]clicking here to send a mail to On-Call and we'll try to feature your tale here on a future Friday. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y2Twy80IAyEyYxhyQiYzuAAAAFU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2Twy80IAyEyYxhyQiYzuAAAAFU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y2Twy80IAyEyYxhyQiYzuAAAAFU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y2Twy80IAyEyYxhyQiYzuAAAAFU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2016/04/29/an_ethernet_switch_under_a_beer_tap_what_could_go_wrong/

[6] https://www.theregister.com/2022/10/28/on_call/

[7] https://www.theregister.com/2022/10/21/on_call/

[8] https://www.theregister.com/2022/10/07/on_call/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y2Twy80IAyEyYxhyQiYzuAAAAFU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/



Botched Aircon

GlenP

The setup I inherited in my current role was that the server cupboard (it was too small to be called a room) had two heat pumps, not even proper aircon. At the time we were located in a small factory unit with office on the front so rather than do a proper job the "exterior" units for the heat pumps were installed inside the factory, just under the metal roof, above a hot plastics extruder. Of course that meant that just at the time they were most needed they couldn't cope. I did have to make an emergency trip to the garden centre for a large plant trough to catch the drips from the units!

Eventually they failed completely and were replaced with a proper aircon unit that was somewhat overspecced for the job.

Re: Botched Aircon

Kevin Johnston

With something like a cooling unit there is a very large zone before it becomes over-specced although I understand what you mean. The biggest risk with having spare capacity is that someone has a bright idea on how to make use of that excess and in no time at all you need to uprate everything.

Re: Botched Aircon

Anonymous Coward

One of my clients has absolutely insane aircon in the comms room. You need a jacket in the summer in there. Absolutely bonkers.

Re: Botched Aircon

GlenP

Being positively ancient I'd consider that the norm. 16C was the maximum back when I started so keeping a sweater or jacket handy in summer was standard practice.

Re: Botched Aircon

MiguelC

I once did a stint at an insurance company during a hot western European summer. At the time, wearing business suit was mandatory, even for IT. So I wore the jacket on my arm, got into my "office" (the server room), put my jacket on, buttoned it, and toiled away. And the end of the day I would take my jacket off and go home... Fortunately it was just for a pair of weeks, but it felt really awkward.

Re: Botched Aircon

PM from Hell

When ICL release their S39 range of mainframes the temperature window for the CPU's was very narrow, we often had to install an underfloor air dam to reduce the airflow through the CPU cabinet as they were getting too chilled and pulling errors. They had been designed for the expected DC temperature range but in a big DC the sir through the flow would often start off very cold at one end in order to keep the other end cool enough

Re: Botched Aircon

WonkoTheSane

TWO heat pumps? You were lucky!

When our new server cupboard was built as part of a factory extension, either the builder or the architect had assumed that a bathroom vent fan would be adequate cooling!

State of the server racks if I hadn't complained up the manglement chain --->

Re: Botched Aircon

Anonymous Coward

Having struggled with lack of cooling in a DC hosting 7 mainframe systems it finally came time in the organisations capital programme to replace it. Having been 'done over' at my budget expense by the property department I was determined it was not happening this time and insisted on seeing the details of the systems they had specced up.

Needless to say they wanted to replace the 3 Huge Airedale Aircon systems with 2 much lower powered system's.

The old systems had been designed to provide a 2+1 operation with us cycling through the 3 aircon systems with only 2 operating at any one time but we had increased the number of mainframe systems, hugely increased disk storage and started migrating to UNIX / WIndows systems all 3 aircon units were often in operation to coll the workload.

If our optimistic project plan for the mainframe replacements came to fruition the load would decrease by 20-30% before the aircon was replaced, if we had a delay then the total cooling available would have been insufficient for normal operation, let alone a summers day. I stood my ground and ended up with a 2+1 set up again with an increase in cooling but a reduction in power consumption. As it turned a couple of the old mainframe CPU's were replaced early and thew huge decrease in cooling requirement meant that we could theoretically cool the whole DC with one of the new units having 2 on standby, but that didn't give the optimum airflow for the far end of the DC. The space released by the replacement of the mainframe CPU's and a couple of aisles of DASD by and EMC array was soon taken up by racks of Windows and Unix Servers. I suspect with the increase in power and density of intel based server even the monster cooling Units I installed would now be struggling again

Re: Botched Aircon

KarMann

…two heat pumps, not even proper aircon. I'm sorry, what exactly do you think aircon is?

2 days of free time?

Anonymous Coward

What year is it?

Re: 2 days of free time?

UCAP

What century was it?

Re: 2 days of free time?

TeeCee

You work for Tw@ter and ICMFP!

Oliver Mayes

We had a new data center constructed a few years ago, as it neared completion some of our engineers went out for a final inspection before we began the process of moving all our on-site servers into it. They found it a little warm inside, eventually discovered that the builders had somehow connected the shiny new aircon system up backwards. It was cooling the outside air and dumping the exhaust heat into the building. Great for the planet I'm sure but not a happy place for 30 racks of servers to live.

Builders

Pete B

When we had some building work done that required moving the door to the server room our builders very carefully wrapped all of the racks in sheets of polythene "to keep any dust out" - they didn't seem to grasp the idea of ventilation.

Ah builders

TonyJ

Did a migration about 15 years ago for a re-insurance company. We moved their servers into cages in a pair of co-lo data centres.

At this point in time though, thanks to BT there was no connectivity to the backup site.

One day I was sat in the office in a meeting when the calls went out that "everything had stopped".

A quick check to see and a walk to the data centre (it was only 400 metres down the road) and when I went up to the cage it was immediately apparent what was wrong.

With no permission / notification of works, the owners of the data centre were installing a new cable tray above our cage. The builders, being careful, had draped blankets over the entire row of racks. Of course there was no airflow and eventually due to heat they shut themselves down.

Strong words were had.

Even earlier than that - c1997 - and another server room move for a large shipping company in Southampton. We got word from the builders that the new server room was ready. I was but a junior engineer so above my pay grade, the decision was made to ship the racks.

Only the room wasn't quite finished - there was no suspended floor. The posts for it were there but no posts. The builders had apparently taken a gamble they'd arrive before the kit and lost. And no one on our side of the project had taken time to check what they were saying was factual. Oops. We drove down to be there at 6.30am only to have to turn around and come home!

Back Pressure

Anonymous Coward

Most people underestimate the pressure drop through a pipe, even engineers. Many years ago, I was working for a company that made kit to separate separate liquid droplets from a fast flowing gas. Putting it at its most basic, it worked by temporarily slowing the flow (by introducing a much larger cross-section) and allowing the liquid droplets to coalesce and fall out to a drain. The clean(er) gas was then vented to a safe area - through what everyone assumed was a large pipe. When I ran some check calculations (taking into account the bends needed to get around existing plant) the residual back pressure was enough to bring the whole system down and divert flammable gas to a very unsafe area - where a large “bang” would be very likely. I only signed it off after the vent line was made considerably larger.

Movers

SealTeam6

We once bought several pre-built racks of servers.

They were shipped to us by a moving company which claimed to be 'expert' computer movers.

When the racks arrived we discovered that they were all damaged very badly. The movers had not tied the (wheeled) racks in place on the van so the racks had been flung around inside and crashed into each other like bumper cars!

New Build DC's

PM from Hell

I've been involved in several new built DC's large and small

My three lessons are:

1/ lift every floor tile and check the whole roof void. In one case the builders had swept all the concrete dust, fag packets and other refuse into the center of the floor before fitting the tiles, I've also found lengths of unused cable, packaging and insulation trimmings in roof voids

2/ Sink test the aircon before installing any kit, bring in heaters and stress the aircon, I've had under powered aircon units installed in error, a blocked exit vent (it was hidden in a decorative clock tower and the builders sealed it to keep pigeons out)

3/ Confirm all breaks in the walls ceiling and floors of the 'cube' are properly sealed with the correct fire resistant /expanding sealants' all easy things to fix when the DC is empty and absolutely painful one fully populated.

My computer, my documents, my briefcase, my ASS!

-- Ben Cook