If you have a fan, and want this company to stay in business, bring it to IT now
- Reference: 1676622490
- News link: https://www.theregister.co.uk/2023/02/17/on_call/
- Source link:
This week, meet "Trent" who told us about his time as head of network operations for a payment processing company in Dublin.
Trent's tale took place a day he described to On-Call as "one of those rare occasions in Dublin when the sun shone brightly and the thermometer climbed well north of 20 degrees."
[1]
Which naturally meant everyone in the office was complaining about the heat.
[2]
[3]
Except Trent, who luxuriated in the air-conditioned comfort of the Network Operations Centre (NOC).
Said NOC wasn't just nicely air-conditioned – it was optimally air conditioned. Because on the very day on which this story happened, an air conditioning engineer had come out to talk about selling extra kit to cover more of the building.
[4]
As a demonstration of goodwill he topped up the existing aircon refrigerant tanks.
The small equipment room Trent tended, which he described as "small and crammed full of racks of file servers, payment processing servers, network switches and an ancient Netrix switch running the point of sale" was therefore filled with lovely frigid air as the giant aircon inside went about its work.
Until it didn't.
[5]
"A little after lunchtime … the usual quiet calm in the NOC was suddenly replaced by the strident wail of alarms," Trent recalled.
A glance at the NOC's screens revealed everything was down, so Trent dashed to the equipment room, opened the door and was hit by a hot wave of air and "the unmistakeable stink of refrigerant from the air conditioning unit."
"It appeared that the air conditioning engineer had not closed one of the valves fully when topping up the refrigerant and it had all leaked out of the system."
[6]What's up with IT, Doc? Rabbit hole reveals cause of outage
[7]No, you cannot safely run a network operations center from a corridor
[8]User was told three times 'Do Not Reboot This PC' – then unplugged it anyway
[9]This can’t be a real bomb threat: You've called a modem, not a phone
Trent's first call was to the aircon engineer, suggesting a rapid U-turn and faster journey back to fix the mess he left behind.
Which left Trent with the job of keeping a badly overheating datacenter up and running – an essential chore given that payment companies are all about 24x7 resilience.
"While we waited for the engineer's return, my colleague confiscated every fan we could find in the building and used them to blow air over the racks of overheated equipment," Trent told On-Call.
Which made Trent extremely unpopular, given the weather on that fine Dublin day.
Ten or fifteen minutes later, the fans had cooled the kit down enough for much of it to work.
The aircon engineer showed up about an hour later and normal service was restored.
"Afterwards, the bosses heaped praise on us for our quick thinking with the fans and getting the network back up and working so quickly. But everybody else was less than impressed by our actions and people bitched about it for a considerable time afterwards," Trent told On-Call.
Have you had to co-opt supplies from around the office to keep IT alive? [10]Click here to send On-Call an email that tells your tale and you might find it on this page on some future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y@9eVMZpcPKClMGHKgNjFgAAAMk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y@9eVMZpcPKClMGHKgNjFgAAAMk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y@9eVMZpcPKClMGHKgNjFgAAAMk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y@9eVMZpcPKClMGHKgNjFgAAAMk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y@9eVMZpcPKClMGHKgNjFgAAAMk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/02/10/on_call/
[7] https://www.theregister.com/2023/02/03/on_call/
[8] https://www.theregister.com/2023/01/27/on_call/
[9] https://www.theregister.com/2023/01/13/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Refrigerant or refrigerating fluid?
Could be a slow leak at a service valve. These Schrader valves are similar to the valves on your car or bike tires.
There are typically two valves ... one on the hot high pressure side, and one on the cold low pressure side. If the itty-bitty o-ring that seals one or the other somehow gets damaged during service, it can appear to the tech that all is operating just fine ... but after the system has been running for a while, cycling on and off and generally vibrating, a valve can fail. An old AC tech once told me to always have the Schrader valves replaced when the system is serviced. Cheap insurance.
In my wife's lab, where they do climate modelling 24/7 on their on-premises servers, the air con died during a scorching summer period some years ago, quickly raising room temperature to high 50's. Their immediate solution was similar, although not that effective, as the outside temperature was over 40ºC, but the air con on the adjacent room was turned to its lowest possible temperature, doors wide open and some fans strategically placed to increase airflow between rooms. Instead of having one room at hellish temperature, they got two at purgatory levels, but it held their servers until maintenance came to fix it.
air CON
That reminds me why I hate offices.
The air con. Always something wrong with it.
In some offices, you can feel that you are going to suffocate soon or die of overheating.
And the lack of oxygen - there was always more people than air con specification permitted.
By 2pm, everyone was pretty much in a coma on autopilot just to make it until 5pm.
When you go out for lunch, you can see people leaving the building gasping for air.
Complain to the building manager? Ha. "Our air conditioning is carefully maintained and we can assure you it works to the spec. We constantly monitor CO2 levels and ensure they are the most optimal."
And when you negotiate dropping the temp by 1C, there is always going to be someone moaning it's too cold and they are freezing and they don't understand the concept of wearing a jumper.
Re: air CON
You have airconditioning?
And you are still complaining? About 1C?
Bah.. the snowflakes today
Re: air CON
Pretty much everywhere I've ever worked there has been a long standing feud between two halves of the population that I can't mention without being accused of being sexist ... One side always says it's too hot, the other side always says it's too cold. Facilities says "set it all to 72F, that's what the HVAC is optimized for" ... and so we listen to pretty much everyone bitch about the temperature.
Until one place I worked at upgraded the AC, and all the controls that go along with it (had to do with a couple of new clean rooms). Naturally, the folks installing all the new gear left all the old thermostats in place. They were no longer connected, so why worry about them. A friend of mine noticed one of the secretaries would inevitably turn one of these controls up, and then keep an eye on it from her desk. Within an hour, one of the engineers would stroll by & turn it down again. Then she'd turn it back up, and so on ... This dance went on all day.
So we hatched up a Plan ... with the Boss's permission, we installed unconnected thermostats quite near both the secretary and the engineer ... and removed the one they were "fighting" over. Now both could happily set "their" temperature to whatever they wanted. It worked. Both were happy, and both commented how comfy the office was with the "new, improved" controls. People in their circle of friends made similar comments. The complaining about the temperature stopped, virtually overnight.
That would have been the end of it, except ever since then I've installed faux thermostats for 'special" people. It has never failed to shut them up about the office temperature. However, be warned ... that type can always find something else to bitch about. Don't say I didn't warn you.
Re: air CON
However, be warned ... that type can always find something else to bitch about
That's what lift shafts, stairwells, and open windows are for...
Re: air CON
When it starts in a lift, you have to firmly say "NOPE.".
Usually the person shuts up.
If not, then you can go for the triple NOPE.
If that doesn't help, just call security.
Re: air CON
That's what lift shafts, stairwells, and open windows are for...
Simon? Is that you?
Re: air CON
We have something similar here. Rented building with general open plan layout, where the air-con initially worked fine.
Then various managers decided they were so important that they needed their own little offices, and so at their insistence various stud-wall partitions got put up to make them their precious boxes to sit in. And of course it totally screwed up the airflow in the building, bits were now hot or cold, especially in aforesaid little boxes as of course they had airflow in from the ceiling but no outflow or specific control.
Guess who now routinely complains that things are too hot or too cold...?
Re: air CON
What a great example of people imagining ludicrous nonsense. The oxygen content would have to drop so far, and be replaced by so much CO2, that people would get CO2 poisoning well before the shortage of oxygen became apparent. You're talking complete and utter rubbish.
But everybody else was less than impressed by our actions and people bitched about it for a considerable time afterwards," Trent told On-Call.
So they weren't his fans then...
Boo! Hiss!
Have a beer :-)
Whoops goes the cooling
We were a development lab. We had some building work next to our machine room, and a digger dug through the main pipe from the air conditioning units to our building. It took a week to fix (getting the replacement pipe was the problem)
So no air con - and all but critical systems were shutdown. Also our building had no windows (why do you need them when you have aircon) so the offices soon got very warm with all the warm bodies.
This was before the days of being able to work from home. Some managers were enterprising. Some held meetings in the nice outdoors, under the trees. Other managers said "please work from home - and review the documentation". So people took the paper manuals and reviewed the stuff relevant to their area. Lots of stuff got pruned, and there was a lot of new content.
So overall it was not too bad an impact ( besides the sweaty bodies)
Cooling
We had some variable-speed drives housed in a switchroom which had air conditioning to keep the inverters cool. The inverter rating was about 500 kW so there was a few kW of heat to dissipate. Well the AC was good at cooling the filtered air extracted from outside......but...... The problem was that the site was a sewage works and a bit pongy...... The characteristic 'bad-egg gas', hydrogen sulphide, is very corrosive to copper. These inverters and cabinets had been carefully constructed with uncoated, high-copper-content wire and strips of copper to keep local losses to a minimum but inadvertently creating a huge surface area. Which promptly corroded..... and the corrosion product is conductive.... and flaky..... Inside, the cabinets had a thick layer of black copper sulphide (other spellings are available). All aided by polluted, external air being forced through the cabinets albeit cooled. After a couple of years, gosh it was a mess and the consequential short-circuits were spectacular.
Tinning the conductors could only be a partial cure (tin corrodes as well) so a new closed-circuit cooling system had to be installed with bigger coolers and standby in case of failure. We were fortunate that we had specified the requirements for the cooling air...... But when it was warm/hot weather this was the best building to be in as it was odour-free and blissfully cool.......>
Re: Cooling
That's why we don't allow untreated outside air and inside air to mix in situations like that.
Properly engineered modern air handler systems are truly marvelous to behold.
Re: Cooling
I'd question the wisdom of air cooling motor/servo drivers of that power level with AC air to begin with. Very inefficient. An air-water cooling system with a dedicated water cooling loop would make more sense. Keep the pongy gasses outside and efficiently cool your kilowatts with water (or other liquid coolant). More compact and probably more efficient than an air cooling system.
"people bitched about it for a considerable time afterwards"
Of course. What Trent should have done was shut everything down so as to not inconvenience the desk drones. Customer Satisfaction ? That only counts if I can get a yearly bonus, or better yet, a raise out of it.
There is no I in Team, right ? Well, there obviously was no team there either, so I it was.
Once upon a time, long ago..
.. in Cambridge, when the Internet was still a new idea, someone had not yet come up with the idea of a URL and days were spent using telnet, gopher and other malarky ..
.. we had the exact same problem. At an ISP. Which was addressed with grabbing fans from wherever they were found. Which was also because of a dead airco. Which some of us knew on account of being in the hottest corner of the building. Which we kinda caused ourselves because most of us ran two systems, one Linux (Slackware, just to date this scene) and one with some Windows variant (I think it was 95 because it still needed an IP stack added to work properly and Doom was played the network was always tested with IPX traffic.
I think they were 80486 or something - you still had to set drive geometry in the BIOS if you wanted something to work.
Anyway, I think the ops team managed to keep things from melting out of the racks but I also know it was close..
Re: (Slackware, just to date this scene)
/checks computer os
What, today? :-)
One hot summer our office air con broke down. Quite a few people plus a couple of server rooms meant it wasn't the coolest at the best of times. We hired a few temporary free standing coolers to make the place a bit more workable. I was talking to someone next to one of them when it decided to spring a leak. When a cloud of white gas starts coming towards you, you move. A couple of minutes later it tripped the fire alarm and the place was evacuated.
Freezing in a heat wave
Worked in a company in Perth, Western Australia where there was a fire exit door next to the air conditioning thermostat. Every lunchtime, people would use the fire exit as a shortcut to the car park. Every time someone opened the door, the thermostat would get a blast of hot air. As a result, the air conditioning was working flat out after lunch. It might have been 42C outside but inside we were being blasted by 6C icy winds from the air con!
Refrigerant or refrigerating fluid?
It sounds like fluid and I can imagine that sitting idle behind an open fill valve until the circulating pump kicks in. Refrigerant would, I expect, vent immediately unless there's some strange check valve and the compressor operating boots the pressure over a threshold.