News: 1670833810

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Server installer fails to spot STOP button – because he wasn't an archaeologist

(2022/12/12)


who, me? Ah, dear reader, once again it is time to greet the day with a tale from Who, Me? – the Reg's welcome to the working week in which readers spill the beans on tech stuff-ups that may or may not have been their fault.

This week, meet a reader we'll Regomize as "Larry" who was commissioned to install a room full of servers at a high-security facility.

Little did Larry know, however, that he really needed skills as an archaeologist to do this job right, because the facility he confronted was built in the 1960s.

[1]

Larry invited his fellow readers to visualize its original décor which boasted glorious wood panelling, magnificent orange and brown carpet tiles, and lots of mid-century industrial tat.

[2]

[3]

Of course the computers of the era were giant machines festooned with blinkenlights galore. And, as was typical for the time, a big red STOP button was present to bring it all grinding to a halt in case of emergency.

Over time, of course, fashions and tastes changed. The tat became tatty. The brown and orange carpet tiles were torn up and replaced with something else that was itself torn up and replaced.

[4]

And the wood panelling got painted.

Badly.

Larry suspects that whichever contractor landed the gig of painting the room had a childhood grudge against the owner of the building. As he put it: "All the sockets, trunking, fire alarms, safety notices and emergency stop buttons were entombed under a carelessly daubed layer of emulsion."

[5]

That's right: even the emergency stop buttons. The result, Larry tells us, was "a comical magnolia plaster cast of a wall."

[6]Programming error created billion-dollar mistake that made the coder ... a hero?

[7]How not to test a new system: push a button and wait to see what happens

[8]Job 1: Get the boss on the network. Job 2: Figure out why Job 1 broke the network for everyone else

[9]Just follow the instructions … no wait, not that instruction to lock everyone out of everything

And that is what Larry found when, some time after the redecoration job, he was called in to fit the room with the latest clutch of computers selected to call its four walls home.

First he had to install new cabling, which involved climbing a ladder to get to the trunking at ceiling level. That completed, he climbed down the ladder, folded it, and leaned it against the wall.

That action generated two sounds.

First came a click.

Next came the unmistakable sigh of many fans decelerating.

It turns out that, concealed amongst the layers of crud coating the computer-room wall, Larry's ladder had found that old stop button … which still worked.

And not just worked, but controlled a circuit that had stuff connected to it. Important stuff.

As Larry was not an archaeologist, nor expected to posses such skills, he was not blamed for the unscheduled power-down.

The situation was resolved with a magnificent kludge: the layers of mess that hid the button were obviously too hard to clear, so a decision was taken to restore it to a bright red hue so it would at least be easy to spot.

Ever pushed a button you oughtn't have pushed? Forgot to push one you ought? Don't worry, we're all friends here – tell us about it in [10]an email to Who, Me? and we'll share the embarrassment together.

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y5cJ1KM6pUGVM3L4FQVB3wAAAFY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y5cJ1KM6pUGVM3L4FQVB3wAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y5cJ1KM6pUGVM3L4FQVB3wAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y5cJ1KM6pUGVM3L4FQVB3wAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y5cJ1KM6pUGVM3L4FQVB3wAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2022/12/05/who_me/

[7] https://www.theregister.com/2022/11/28/who_me/

[8] https://www.theregister.com/2022/11/21/who_me/

[9] https://www.theregister.com/2022/11/14/who_me/

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



Oglethorpe

At university, we were able to do an unrelated STEM module in the first year, leading to quite a few issues in the lab. The most entertaining afternoon came from a microbiology practical where someone had somehow never used a Bunsen before. This led to the gas being ignited at the outlet.

When they tried to turn the gas off for the lab, they discovered the fresh new fitted benches had been installed over the shutoff valve for the room (the outburst from the crusty technician was awe inspiring). This led to several frantic minutes of locating a screwdriver and dismantling the bench before the ceiling went from medium to well done.

slimshady76

Gotta admit your punchline was delicious.

Have one on me! --->

new fitted benches had been installed over the shutoff valve

Anonymous Coward

A while back when we got a new combi installed, I was chatting to one of the installers. He said that when he was refurbing his kitchen (which was in a rear extension), he took up the flooring, only to find an unexpected manhole cover over an access shaft - with a ladder, no less - leading down some considerable distance into what presumably was some of Thames Water's finest infrastructure. I think he said he covered it up again, with the new floor, and just hopes it is never needed as an access point.

Little Mouse

We had an unprotected Big Red Stop Button located right next to the light switch just inside the machine room door. It was second nature to just walk in and hit the lights without even looking.

Amazingly, in the many years I was there, no-one ever tripped the whole room.

Christoph

Ours was on the wall outside in the general office so you could hit it after escaping a halon dump. Shortly afterwards it had a box put over it with a flap opening.

"restore it to a bright red hue"

Pascal Monett

No, the layers are not too hard to clear. That is a lame excuse. Three minutes with a sledgehammer and I guarantee those layers will be gone, along with that part of the wall.

Then you can redo the wall, the connections and the red button, and you can even put a nice clear plexiglass guard to keep said button from getting inadvertantly pressed again.

Because just painting the layers in red in one spot is not going to solve the problem.

Re: "restore it to a bright red hue"

b0llchit

It is not about solving the problem...

It is about the appearance that the problem is solved.

Re: "restore it to a bright red hue"

John Riddoch

Ah, of course - "high-security facility" and the 1960s vintage screams something related to government/defence, so we're talking civil service in all likelihood.

Re: "restore it to a bright red hue"

Phil O'Sophical

I suppose we should be grateful that it wasn't the "launch the nukes" button...

Re: "restore it to a bright red hue"

b0llchit

It still might... Have you looked for buttons behind the other panels?

Re: "restore it to a bright red hue"

KarMann

Or it was the launch button, but the silo doors were similarly painted over, so the missiles didn't launch, we never heard about it, and we're still here to laugh about it.

Re: "restore it to a bright red hue"

Anonymous Coward

Maybe it's a listed building and they were matching the style of the existing decor?

Renovations

chivo243

Gotta love the slop dash that some renovations bring... A room is supposed to look 'better' after the renovations. I once lived in an apartment in Chicago, there were so many layers of "emulsion" on the walls that the electrical outlets looked like Pink Floyd's pig pushing through the wall!

Re: Renovations

KarMann

And at that point, you have to start wondering, how far down are the lead-paint layers in that emulsion?

Anonymous Coward

During the monotonous hours on the night shift, I can clearly remember some awesome butt-clenching moments when objects thrown across the machine room hit the wall perilously close to the infamous red button.

I know of at least one occasion when the button was hit (apparently an awesome but unexpected shot) that left me recovering systems when I rolled up to start my shift.

Paint all over everything, including power sockets and emergency buttons?

jake

Sorry, never happened.

Insurance says no.

Re: Paint all over everything, including power sockets and emergency buttons?

My-Handle

As someone else postulated up-thread, this may have been a government facility. Most of these in the UK are self-insured, meaning that if anything goes wrong at a government site the government pays for it.

Of course, this is conjecture. As is the assumption that the decorator at the time cared about insurance at all.

I would postulate that the decorator in question could well have been some manager's son's new painting business. In this case, any shoddy work would have been swept under the rug.

Re: Paint all over everything, including power sockets and emergency buttons?

jake

At one SillyConValley startup, I volunteered to replace a bunch of power sockets after an insurance company denied coverage due to inappropriate paint. Took an afternoon, and the guy was nice enough to re-inspect it before going home, so we were allowed to occupy the premises on schedule the next day.

At another company, I was in the glass room when an insurance guy found a drip of paint on the BRB and pulled coverage immediately. Heads rolled over that one.

BRBs

Roger Kynaston

I may have told this before - apologies to fellow veteran commentards if I have.

Some contractors had been brought in to do some work on the computer room. Their work involved some heavy hammering work. Some of this was quite close to the BRB and duly triggered it off. Cue the familiar sphincter tightening and loosening followed by lots of frenetic running around and shouting till everything had come back to a state resembling normal.

As a side observation, I do wonder if kit in those older days was more tolerant of such unexpected outages compared to now. Where I work now has all its hardware in a dedicated data centre with resilient this that and everything. Would the modern rackmount stuff come back with as little complaint as was experienced in those (not really) halcyon days?

Sequin

We had a room housing various systems which used to get mysteriously powered off at random times. It turned out that the switch on the ank of sockets was directly in line with the handle on the inside of the door and if you opened it while carying stuff, either by backing against t, or pushing it with your foot the handle hit the switch with just enough force t break the contanct without the switch latching over fully. A strategically positioned door stop eventually fixed the problem.

KarMann

We have a similar, if much less troubling, switch in our kitchen. Open the refrigerator door, let it swing gently against the wall, and a couple of seconds later, the exhaust fan spins up. The slight pause before it's really making noise greatly enhances the disconnect of cause & effect, and the moment of 'what is that noise now?'

I'm a GENIUS! I want to dispute sentence structure with SUSAN SONTAG!!