Police ignored the laws of datacenter climate control
- Reference: 1696575728
- News link: https://www.theregister.co.uk/2023/10/06/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Sylvester" who, in the early 2000s, found himself working as civilian IT support for a local constabulary that operated across in several locations – each of which had its own mini- datacenter housing a file server and some other kit.
One of those remote rigs struck trouble and Sylvester was summoned to sort it.
[1]
"This particular machine (a typical small rack mount offering from Dell per the era) was running hot and would reboot spontaneously every hour or so," Sylvester recalled. His initial investigations yielded the info that it was located at a site primarily concerned with roads policing and which included a fully stocked vehicle maintenance bay that put commercial garages to shame.
[2]
[3]
It may also have made cleaners cry, given the detritus produced by vehicle maintenance.
When Sylvester arrived, he was shown through the vehicle repair bays into an adjacent "drying room."
[4]
"Imagine those TV drama scenes in which two dozen police batter down some villain's door down in the pouring rain," he suggested, next asking Register readers to imagine where all that wet gear is placed afterwards.
In a drying room, is the answer.
Sylvester learned that the drying room was also where this outpost of law enforcement decided to install a half-height rack in which resided the servers he'd been asked to fix.
[5]
"Yes, the servers were in a cold damp room whose only entrance was through the vehicle maintenance bay," Sylvester told On Call.
Sylvester suspected that the combination of damp and dirt may have done for the Dell.
His deductive instincts were spot on. When Sylvester opened the rack and looked at the fans, he found "a solid, unrecognizable black lump."
[6]Beta driver turned heads in the hospital
[7]How is this problem mine, techie asked, while cleaning underground computer
[8]Techie labelled 'disgusting filth merchant' by disgusting hypocrite
[9]I'll see your data loss and raise you a security policy violation
"There wasn't a gap in it anywhere for anything even approaching airflow and even if there had been, the fans were never going to move again."
Sylvester informed his colleagues there was no chance the server would be available until new fans were installed, which happened the next day.
"Credit where due, the server fired up again with no issues once the new block was installed the next day," Sylvester signed off.
What's the least appropriate location you've seen hardware located? [10]Click here to send On Call an email and we may share it here on a future Friday. Keep 'em coming, dear readers – the mailbag has lots of nice little stories in it, but we could do with some meatier fare. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZR-axLsWhAocv@fWJFJmNwAAAAo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZR-axLsWhAocv@fWJFJmNwAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZR-axLsWhAocv@fWJFJmNwAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZR-axLsWhAocv@fWJFJmNwAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZR-axLsWhAocv@fWJFJmNwAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/09/29/on_call/
[7] https://www.theregister.com/2023/09/22/on_call/
[8] https://www.theregister.com/2023/09/15/on_call/
[9] https://www.theregister.com/2023/09/01/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Fun with magnets.
Except monochrome CRTs don't have a shadow mask or aperture grille because they only have one electron gun, so there's no need for a degaussing coil.
Plus degaussing a CRT won't fix burn in, so either it was a colour terminal or someone was telling fairy stories.
Re: Fun with magnets.
This one did. I know because I used it. It used to make a loud cracking sound when you degaussed. And the button had degauss written next to it. It was old in 1989 when I first used it.
The magnetic fields in that room twice made my bank cards unreadable.
It sounds like he wasn't the biggest fan of the server's location...
The flow had dried up.
Well someone had to be drafted in
Boom
At least it did not blow up. Or in, or out, or any other direction.
Re: Boom
Well, positioned correctly it wouldn't matter.
Many years ago, a certain air force put one of its air base operations management computers in a portakbin within a hardened air shelter, locked the door and left it to run. That was obviously fairly bomb-proof, from the outside or within. This sort of HAS was a very well sealed against NBC warfare, but because HAS are normally rather cold, nobody gave too much thought to the matter. After abour ten days of continuous running this computer stopped and a technician was sent to reboot the machione or whatever was required. When the doors were opened it became clear that the machine had go do hot that the whole thing had overheated and now featured a lot of melted and resolidified plastic and solder. Nobody was happy about that, but as always the taxpayer picked up the bill.
Convenience
To be fair, if you've got a room right next to the vehicle bay that comes with a heating fan included, why wouldn't you use it as a drying room?
One of my favorites.
Punchdown blocks mounted on the wall of a janitor's closet.
Directly over a mop bucket full of ammonia water.
I am pitching to replacing a switch that's in a swimming pool pump room - hot, humid, full of corrosive chemicals, 24/7, and slowing rusting away.
But I once found - after a network outage - a hub underneath a listed-building's parquet flooring. Literally just sitting under floorboards.
One office didn't come back up and we couldn't work out why because the cable to it was absolutely fine and had been working minutes earlier. Only after much investigation did we find a hidden 10Mbps powered hub was being used to "boost" the length of the cable run, and that hub was literally underneath a floor you couldn't rip up, which nobody had touched in decades, and was plugged into a hidden electrical socket inside the 3-inch gap between the actual floor and the parquet floor. Not documented. Nobody knew it was there. Not visible. Even the electrician had to come out because the plug socket was unknown to them and the building was so old it was a potential hazard (it still had round-pin plugs, toggle switches and those twisted-fabric electrical cables in places).
Found it by sheer chance by tugging on random nearby cables and then hearing something move. Even the LEDs on it weren't bright enough to see any more, probably decades caked in dust. From a networking viewpoint, it was invisible - while it was powered the cable just worked end to end. But if you tried to tone it or test it or trace it with the power off, it just stopped under the floor somewhere.
Now ask me why I've had a top of the line TDR in my toolkit since time immemorial.
Yes, they were very expensive in the early days, but worth every penny. Recommended.
"twisted-fabric electrical cables"
You had fabric? Luxury! We moved into a house where some of the cables were lead-sheathed with paper insulation in the last stages of rotting away.
Long time back now but we moved into an old house which still had flash-cap wiring. For those that have never heard of it a carpenter/joiner(chippie) would run 1"x2" timbers with two channels cut into the length. Sparky would then come along and lay in two bare wires and the chippie would come back put a thin cap over it to protect the residents.
Apparently that was determined to not be a safe method to deploy mains round a house
"I am pitching to replacing a switch that's in a swimming pool pump room - hot, humid, full of corrosive chemicals, 24/7, and slowing rusting away."
I had a car wash as a client with several security cameras inside, all IP68 protected with a long, non-detachable coax+power cable - sold for underwater use in aquariums or pools. All auto-focus (and reasonably good analog image) so they were not meant to be opened, and I didn't.
They all stopped working before two years in the hot and humid (~100%) conditions, probably due to the chemicals involved. Since the cable and camera were single unit, replacing them required recabling in the filthies and slimiest possible cable ducts you can only imagine - in humid conditions and on top of A ladders over a super slippery floor.
Also, seeing really close the filthy and oily rotating brushes that clean cars - I have ever since only opted for touchless wash programs - though I prefer hand wash services.
On the positive side, the cameras produced loads of laugh-out-loud footage of people not heeding the instruction sign posts before entering and thus losing their antennas or rear wings, or not letting the conveyor belt move the car and instead driving and bumbing through like an idiot they were, opening of doors and windows during wash cycles and so on. Very few came back to collect the rear wings for some reason. Some tried to have the car wash owner pay for them with little success.
A field engineer once told me he got sent to a server at a school. As this was a school, the staff didn't want the pupils playing with the server so they built a little box around it. The engineer opened the box and found that the plastic case for the server had heated enough to soften and had sagged onto the motherboard.
Personnally, I'm suspicious of this: Surely if the box had gotten warm enough to soften the plastic, wouldn't it have gotten hot enough to fry the chips?
You'd have thought so, but it's surprising how well many ICs will operate well above their rated temps - perhaps unsurprising given the often very high process temps that silicon dies are processed at, or indeed the temps they'll be exposed to by soldering. Doesn't mean the silicon will always function as intendedoutside it's rating, but it's easily possible for an enclosed processor to push the temps up until something melts - see my little tale above about the computer in the hardened air shelter.
"10Mbps powered hub" ...
Luxury!
We installed "modern" 10base2 with stretches of 10base5 to get extra length ... which obviously had to be installed two hours before we got up and before we licked the road clean with out tongues ...
We even ditched the Apple2 server and its RS232 "network" ... eventually.
Thinnet.
Did the job for small offices (and more than a few large ones) for about two decades.
Was really cheap once NE2000 clones started showing up, and thus became nearly ubiquitous. A couple of techs could crank out a couple hundred seat network in a long weekend, given enough coffee. I must have run several hundred miles of the stuff.
I don't miss it a bit ... but I still run it at home when it makes sense. Why not, I still have all the tools and several 2,500 foot rolls of the stuff. And I maintain a couple of legacy machine shops that use it, along with a few SCADA odds & ends.
When you hear of some of these installations you wonder about the installers. Did they not flag it up as an unsuitable installation? Maybe they'd strayed into the wrong genre - cops and robbers when they should have been in cowboys and indians.
"Did they not flag it up as an unsuitable installation?"
Probably, yes.
But the clueless Boss said "do it anyway, it'll be fine and I've got a quota to fill".
Or perhaps the owner of the shop in question insisted, and was willing to sign a "get out of jail free" card for the installer.
Etc.
Fun with magnets.
A company I worked at had an old terminal (green text type) connected to a digitising table. The terminal needed degaussing quite often (with a button on the side) to clear the text which was semi persistent as did the digitising table with a large magnetic sweeper.
Guess where they stored the back-up tapes from the MicroVAX that ran the office network?