Seeing as this story originated in (South) Africa, surely, SURELY, El Reg wouldn't have chosen the name of the main person as "Clarence" because of some hidden Daktari reference?

You wouldn't, would you?!

The worst I've seen involved a builder drilling into concrete in the server room - I don't know why - and the dust causing every hard drive in there to fail, some immediately, others over weeks. Insurance wouldn't cover it either.

I had an issue with mainframe disk failures, my maintainer was saying it was environmental, I was adamant that nothing had changed but decided to get the floor and ceiling voids cleaned to stop the argument

The cleaners found out that somebody had accidentally drilled a hole in the DC floor from below, and never told us, there was still the remains of the pile of concrete dust that my aircon system had been circulating for several months. It took a we had a couple of final drive failures before returning to a normal level of failures.

At a company I used to work at many years ago now, a fire alarm test at a datacenter in a capital city somewhere in continental Europe went horribly wrong, accidentally triggering the release of whatever (presumably outdated) extinguishing agent they were using. All the equipment in the room was covered in a microscopically fine coating of dust. While some initial disaster recovery was done immediately (after scheduling the most emergency power outage I've ever witnessed!) they basically had to send every single device from that room off to professional cleaning services to have them vacuumed inside and out. Took them the better part of a year, running with half capacity, and it was really not that big of a room.

There was a fire where I used to work and as the building was well-designed the flames only affected one room and pretty much burnt itself out before the fire brigade. The smoke; however, got to other rooms in the building and the computers inhaled whatever nastiness was floating around. For some reason they decided to have the laptops cleaned instead of just replacing them. Pretty much all of the affected computers' discs died a month or so later.

Usually not a good combo! Gotta love the architects that scribble IT in as an after thought. At one place, the data cabinet was in the basement directly under the drainage system for the roof of the building, which was flaaaat! Bring on a once in 50 years rainstorm! Yay! Someone call the insurance company!! I couldn't wipe the "I told you so" look from my face for weeks...

Raincoat just because!

There are worse things to flood a basement (and server cabinets) with than rainwater...

At a company where I worked, they had a Tandem non-stop computer in the basement. Due to a leakage, the system was knee-deep in water before someone noticed. Tandem computers advised us, that this was not within specs and did not fall under the non-stop guarantee. However, the system did not stop. Afterwards, some components had to be replaced, but even with the flood, up-time was 1000%.

Tandem - running 240 hours per day.

Random thought: marketing departments could start claiming 1000‰ uptime to catch the eye?

I commissioned a new Mainframe for a customer in a beautiful new DC in a basement. they'd spent about £1.5 million on the DC and equipment, there was a strange looking door in one corner which I had to open, sure enough there was a 5 inch cast iron sewage pipe routed through the DC. Apparently it would have cost £50K to reroute the drains and would have meant that a pipe would have been attached the outside of the wall and this wasn't allowed. I did warn them that they would be up to their chins in sh*t one day

Yep - been there!

During a building move we discovered that the proposed server room location (formerly the gents toilets) had the main potable water tank located right above it, and that they were proposing to remove most of the wall supporting one side of said tank! The tank was, reluctantly, decommissioned and replaced elsewhere but what they didn't tell us about was the open rainwater gutter that also ran above the false ceiling. Fortunately it overflowed during office hours, we noticed and were able to mitigate the problem (plastic tarp and buckets) before the AS/400 got a good soaking.

I looked after the support of on place, some decades ago.

The IT manager was only the IT manager because no one else wanted to be. She was lovely but relatively clueless. However she was also very willing to learn and was a genuine pleasure to work with.

She knew enough to do basic checks (are the coax cables connected and properly terminated, for example) so generally when she called it wasn't something silly.

This one morning I took the call. Nothing was connecting to the server. Of course she'd done the usual checks so it was a case of "can you tell me what the server is saying on the screen?"

At this point it's worth noting that the server lived in the basement so it was a "yep, let me go check and I'll call you back."

Five minutes later the call came in - don't worry, the basement is flooded...

The UK Academic network, JANET's POP in the east of England was in a room in Cambridge beneath a roof garden which had the inevitable consequences... The way around this was to build an internal shed in the machine room! Sadly the photos seem to have vanished from the Internet :-(

I vaguely remember seeing those pictures once, but I couldn't work out if it was a myth and I imagined it.

Thanks for letting me know I'm not going completely insane!

I'm sure the first story was a sub plot in Lethal Weapon 2?

I had more the image of the bathroom scene near the start of the first Paddington movie.

> The story has at least one happy ending: Clarence was paid for the job even though his work washed away.

... and hopefully got another commission to reinstall the replacement system when the museum was reinstated (if it was rebuilt?). Maybe, this time with flood alarms added.

In the dim & distant, I I worked for a mid sized finance house, where all the important stuff ran on IBM Kit. As part of a major system upgrade, the company acquired a swanky new tape robot backup system.

This consisted of an eight sided cylinder, about 10 foot tall lined on the inside with hundreds of tape racks and a forgotten number of drive units. Mounted in the middle of the cylinder was a robotic arm that moved the tapes around, with impressive speed & dexterity. I was part of the WinTel team and had nothing to do with the mainframe stuff, ever. But management decreed that everyone who ever needed to set foot in the computer room HAD to attend mandatory safety training on the new tape library. And very entertaining it was too.

The silo included a half height access door, which could be used to enter the library for maintenance. We were all told about the procedures required to access the insides of the beast, which included removing your tie (finance house), in case it got snagged on anything. But the really interesting topic explained what to do in the theoretically impossible situation of the library waking up, while someone was inside the machine. This basically amounted to standing upright and keeping still, overriding the natural instinct to duck out the way as the arm started moving. The reason, we were told, is that the arm did a slow speed, 360 degree rotation, in both directions, while changing heights, checking for obstructions. If it touched anything during these safety sweeps, power would immediately be cut. The description of the potential consequences of avoiding the arm during its obstruction check have stayed with me for over 25 years - "no one wants the job of cleaning out operator purée. Oh and you'll invalidate your warranty too.."

I can't help feeling that installing a bright red "Stop!!" button inside the cavity would have been more human-compatible.

Ah, is the cat capabale of reading ? (and is at least dichromatic)

"Stand upright and keep still. On no account should you dodge the robot arms, or they will dice you into slurry." :-/

I was shown the massive tape silo library in a bank once and was told a similar thing - stay still and let the robot arm find you, otherwise you'd be hit by it when it got up to its 30mph speed. Also, if you closed the door to the silo without checking first if someone was inside, it was a sackable offence.

When I was in university way back in the 80's, the CompSci department and the associated computer/server room was relocated from the edge of campus to two swanky, recently refurbished floors of the university's main building. This building was (apparently) the largest completely brick-built building in Europe, and included several large shafts that ran from the ground to the top floor that was originally used for heating (that heating system was long gone by my time). The CompSci department had decided to install a relatively new technology called "Ethernet" throughout the university's main building, and they used one of the air shafts as a conduit for the main backbone cable. Not being complete idiots, they used a cable that ran through an armoured steel shell just in case of unforeseen circumstances.

Meanwhile the next couple of floors above the new CompSci department had started to be refurbished (this was an on-going task at the time that lasted several years). The builders had discovered the air shafts and had decided to use them as an easy way to get the demolition rubble down from where they were working to ground floor, at which point they would cart it into the skips. Unfortunately everyone quickly discovered that the armoured Ethernet cable, while cable of surviving anything an drunk undergraduate (or postgraduate for that matter) could swing at it, was not cable of handling 50 tons of building rubble dropped from a great height.

I'll leave it to yo to image the conversation between the head of CompSci, the Vice Chancellor and the building contractor management.

Aston?

Some interesting rooms down in the basement levels of that building, and almost everything surrounding the main hall - especially the voids above it.

I once had a whiteboard on which I wrote "Do Not Clean"

One day the cleaner cleaned the board.

I went to our ever-useful HR droid who later informed me that the cleaner had cleaned the board "because it was dirty".

“Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.” - T Pratchett (Thief of Time)

Do not do X

Signs are so frustrating if they don't tell you why. Some people just can't leave it and need to find out why.

So many disasters could have been avoided if there was a little brief description of why you shouldn't do X.

Curiosity killed Steve.