'The computer was sitting in a puddle of mud, with water up to the motherboard'
- Reference: 1703144590
- News link: https://www.theregister.co.uk/2023/12/21/on_call/
- Source link:
A few weeks back On Call featured [1]servers stuck down a mineshaft and we asked readers to detail the dirtiest places they've been asked to work.
With the imminent Christmas break slowing the pace and quality of news, we therefore present a special "On Call: Dirt File" to share the product of our request for filthy stories.
[2]
Let's start with a reader we'll Regomize as "Bevan" who told us of the time he was asked to fix a server for a bank that was old enough to occupy an historic building.
[3]
[4]
He was shown to the server room where he found "a packed dirt floor and a server sitting in a puddle of mud." It got worse: water had risen to the point at which it was lapping on the motherboard.
Bevan managed to avoid being electrocuted, figured out that guttering had overrun and caused the mess, and that the server had survived – but a monitor, keyboard, and networking kit had not.
[5]
Next, meet "Emannuel" who once had a client that operated both a dye works and a horse racing stable.
The two were oddly complementary because the dye works was a hazardous environment in which powered vehicles weren't welcome because they created a risk of fire. When the adjacent horses slowed down, they were sent to the dye works to haul loads.
Which was clever. But also foully pungent, thanks to the horses’ waste production, and the dye itself!
[6]
"I couldn't wait to fix the kit and get out of there," Emmanuel told On Call.
[7]You don't get what you don't pay for, but nobody is paid enough to be abused
[8]Bank's datacenter died after travelling back in time to 1970
[9]Bank boss hated IT, loved the beach, was clueless about ports and politeness
[10]User read the manual, followed instructions, still couldn't make 'Excel' work
Boiler room battle
Now let's meet "Vince" who once served as assistant IT manager for a company that built steam generators for oil extraction.
He described them to On Call as "basically giant boilers."
Vince tended workstations scattered around the plant, several in areas dedicated to welding, sandblasting, and painting.
After one of the machines started sparking, he inspected them all.
"The first was coated with metal particles on the motherboard and in the power supply," Vince recalled. "Others were caked with sand and other stuff."
Plenty were filled with "caked with hardened mud" and one even housed a wasp's nest.
"I chipped out the mud and mess, replaced a couple of power supplies and they were still running two years later when I left," he told On Call.
Finally, meet "Henry" who once worked with a customer that had a datacenter located in the basement of a parking garage.
"To get to the entrance you had to walk through a HazMat area complete with people working in HazMat suits with respirators and everything," he told On Cal.
And that wasn't the worst part.
"The garage was old, and the concrete ceiling started leaking all the water and salt dripping off snow covered cars onto the computer racks."
The datacenter was eventually moved, but when Henry was on the job the solution was to place drain pans atop the racks, to pipe the salty slurry into floor drains.
"That was the all-time low for datacenters I've visited," Henry recalled.
We have another couple of dirt files to share, but could use more! Dish your dirt by [11]clicking here to send On Call an email and we may be able to squeeze in another seasonal special or two in addition to On Call's usual Friday frolic that will appear as usual across the holidays. ®
Get our [12]Tech Resources
[1] https://www.theregister.com/2023/09/22/on_call/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZYQa3H@9QQDde10zCjyuxAAAAFA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZYQa3H@9QQDde10zCjyuxAAAAFA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZYQa3H@9QQDde10zCjyuxAAAAFA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZYQa3H@9QQDde10zCjyuxAAAAFA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZYQa3H@9QQDde10zCjyuxAAAAFA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/12/15/on_call/
[8] https://www.theregister.com/2023/12/08/on_call/
[9] https://www.theregister.com/2023/12/01/on_call/
[10] https://www.theregister.com/2023/11/24/on_call/
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
I once got sent to a print works in *redacted* because the elderly Macintosh Quadra 800 that was running their printer had died. This was back in about 1998, and even back then the Quadra was state of the ark. Don’t imagine a laser printer either - it was running a printing press for the local newspaper (amongst other things), just sitting there, in a cabinet, next to the press, hooked up to a network and receiving the papers for printing from the newer, fancier, Macs in the nice clean offices next door.
I don’t know what the presses for big newspapers like The Guardian or Financial Times are like, but this place was only superficially clean. It looked okay, but behind every panel, under every item, paper dust was thick like snow. I’m amazed the place hadn’t burned down (or blown up). I had a sneaky feeling, even before opening it up, that I knew why that old Quadra had packed its bags and quit.
…and yes, sure enough, when I pulled it from its dusty cocoon and opened the panels I discovered it was quite literally packed with dust. The fans had pulled the dust in, the charge on the components had made sure it stayed. And one day, it got so packed that the fans gave up, and even the minimal cooling that they provided to the thickly blanketed components was taken away… and the computer shut down. And refused to start up again.
Amazingly, all it took to repair the computer was a new power supply and a vacuuming out of the computer. And, in fact, the old PSU worked fine once I got it back to the Apple Centre where I worked, pulled it apart, cleaned it, and reassembled it.
As far as I know, that printer is still there. I wonder if they’ve started cleaning the place?
On a different note, I studied engineering at Uni back in the late eighties/early nineties and we did our CAD work in nice clean offices on Sun workstations. When it came to the CAM part, the designs were downloaded into filthy old Apple IIs which were hooked up to the machines responsible for making our creations. Lathes and so forth. Whilst they had plenty of cooling, there was also oil and general factory grime around - and those old Apples bore the brunt. And to the best of my knowledge, none of them faltered (although I have my suspicions that they keyboards probably didn’t work, then again though, they didn’t have to)
CNC machines
Worst ones I've come across were PCs on a factory floor near dozens of CNC machines. These were leased Dell Optiplex machines, and after 3 years of sitting there in that environment, they were completely coated - outside and in with a thick layer of oil. Every single surface and component had a sticky, slimy coating to it. Before boxing them up for return to the lease company, we used to wrap them in a couple of bin-liners, just to protect our hands and clothes.
The thing is, we had a full list of all the various issues with a return PC that could incur a penalty charge from the lease company. You know, the usual deep scratches, broken casing, faulty/missing components and all that. Yet there was nothing in the list that covered dirt or oil. Hence the lease company used to receive PCs that were bordering on being a biohazard, yet we never received any form of penalty payments for sending back the machines in that state...
Sewerage leak
I was involved in a world wide games event, and so there were lot of temporary facilities used.
One was under a main arena. Because of the thousands of extra people who attended,the toilet facilities could not cope,and there was a sewerage leak into the area where the server's were stored.
The rate of ingress was slow - they reckoned that they had a week before it got into the servers - fortunately the games only lasted another 2 days.
The fall back plan was a taller table and electricity cables tied to the ceiling.
Re: Sewerage leak
I used to look after the small Epson PX-4 laptops that were use to collect the UK Pop Charts (while it was still independent and not run by the industry, for the industry).
I had a laptop exchanged (we used to send a replacement and the courier would replace the new laptop with the broken and then return the broken one to us.)
I once received a laptop back in a sealed bag, that had evidence of a brown leakage, but generally looked pretty clean. On investigation, this was because the shop had diligently cleaned the outside, including keyboard, but it still did not work. The shop had been flooded with raw sewerage, but they were trying to avoid a bill!
Dye Houses
The smell of a dye house is unique. I have been in many over the years in the UK and US and they all had the same aroma.
One particular place in the US was dyeing sheets for major hotel chains. The whole place had a 1 inch layer of cotton fibres covering everything. In harder to reach places it was even thicker. It looked very pretty - like a thick layer of snow, but did nothing for the ventilation of the panels that housed the electronics.
Re: Dye Houses
The "prettiest" one I came across was the small machine room for a semi-rural museum. It was just a small room that had a tiny window to outside that was kept open a crack all year round.
One day I walked in to find a thick layer of colourful dead butterflies covering every inch of every surface. There must have been thousands of the poor things.
They had similar overnight infestations of ladybirds too, over the years. I'm not sure what it was about the room that attracted them.
Re: Dye Houses
The warmth? I accidentally filled my parents' bathroom with ladybirds a couple of years ago when visiting them at Christmas. I opened the window slightly before showering, to avoid steaming the room up, unaware that there were dozens of ladybirds enjoying the warmth and possibly humidity escaping through the cracks in the frame.
I once went to decommission a site that was used to train drug detection dogs.
I took all our gear back to our office so they can be wiped clean and one of them, a network switch, was making a strange noise. I popped the screws and lifted the lid and found every space packed in dog hair. The noise was coming from the fan struggling to breathe!
PLC's by there very definition end up often in filthy areas, and I've seen a few in coal mines, chemical works etc which were caked in various solid products
The worst was Tate and Lyle in Silvertown, London. Its was where the raw sugar was offloaded to be processed. we had a couple of NT servers, we had to maintain and the entire site was caked in a sort of mucky brown layer of sugar, so much that your fillings would start to spontaneously explode after about 30 minutes
Horrible place, I felt really sorry for anyone who had to work there, but it probably put you off sugar for life
Drains
Yes, I've worked in a couple of places with water collection pans over equipment funnelling water to a drain.
One was due to water leakage through decrepit concrete.
The other was in a new building, but the (heavily lagged) liquid Nitrogen pipes to environmental ovens caused so much condensation as to require the water collection above each oven and associated control and test rigs.