News: 1669964409

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Two signs in the comms cabinet said 'Do not unplug'. Guess what happened

(2022/12/02)


On Call Welcome once more to On-Call, The Register 's weekly reader-contributed column that tells tales of IT pros being asked to fix things that should never have broken.

This week, meet "Bob" who shared a story from his time as manager of first line IT support for a large retail chain.

Said retailer installed a rack full of kit in each store. Bob proudly described to The Register that those racks were properly resilient: a pair of firewalls and two sets of switches connected to primary and secondary circuits.

[1]

The system was, if not bulletproof, then certainly designed to keep going under most circumstances.

[2]

[3]

Of course, it didn't.

"One day, we got a call from the managed network provider," Bob told On-Call.

[4]

"Every piece of kit in one of our biggest stores had gone down at the same time."

Bob called this a "holy sh*t moment".

Staff from the store called to say everything was broken. Bob chatted to the manager and told him to inspect the comms cabinets.

[5]

"There was a few minutes of silence once he got there," Bob recalled, followed by the exclamation: "Why the f**k has someone unplugged a comms cabinet to charge an iPad?"

About a minute later, the manager reported that all was well.

"The issue should now be sorted, and I will take steps to make sure it doesn't happen again," the manager said.

Bob's remote monitoring tools confirmed that all was indeed well.

[6]Boss broke servers with a careless bit of keyboarding, leaving techies to sort it out late on a Sunday

[7]IT manager's 'think outside the box' edict was, for once, not (only) a revolting cliché

[8]Go ahead, be rude. You don't know it now, but it will cost you $350,000

[9]No, I will not pay the bill. Why? Because we pay you to fix things, not break them

At this point, On-Call must confess to having left out an element of the resilience regime of which Bob was so proud: the plug connecting the comms cabinets to the electricity supply were labelled "Do not unplug". And the socket from which the cabinet drew power also had a label that read "Do not remove plug".

"The incident really highlights that all the (physical) signs in the world sometimes can't stop human stupidity," Bob wrote in his missive to On-Call.

Bob never found out if the iPad owner was fired, and still can't quite believe they turned off a network to charge a device.

What instructions have your users ignored? What damage did they do as a result? Share your story by [10]clicking here to send On-Call an email and we may tell your tale here on a future Friday. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y4na1OMsP90J@qHz4xMOiwAAAJI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y4na1OMsP90J@qHz4xMOiwAAAJI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y4na1OMsP90J@qHz4xMOiwAAAJI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y4na1OMsP90J@qHz4xMOiwAAAJI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y4na1OMsP90J@qHz4xMOiwAAAJI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2022/11/25/on_call/

[7] https://www.theregister.com/2022/11/18/on_call/

[8] https://www.theregister.com/2022/11/11/on_call/

[9] https://www.theregister.com/2022/10/28/on_call/

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/



Not only two signs...

chivo243

But in two languages, and the plugs still magically managed to free themselves...

Re: Not only two signs...

simonlb

When I was doing my apprenticeship many years ago I saw an electrical switch on a wall with a load of gaffer tape over the switch (which was turned on) and a label taped to the wall next to it which read, "This switch must be left switched on permanently all the time."

Pascal Monett

If it's a switch, I'm sure an electrician could just connect the wires behind and leave the switch as bait for the clueless.

Anonymous Coward

Get one with a metal surface and wire it to hot mains. Leave the crispy bodies behind as a warning to the next person considering touching it anyway.

You do this, of course, for educational reasons (and to weed out the dumb ones), because just wiring something in as a spur with a fuse as interrupter is better but at best half as entertaining.

jake

Because "This switch must be left switched on permanently all the time." doesn't apply to ihe cognizant engineer doing routine or emergency maintenance. He sometimes needs the on/off switch, or even to unplug the kit physically, for example if the original needs replacing NOW! .

Re: Not only two signs...

Hubert Cumberdale

[1]This sort of thing would probably be a better idea.

[1] https://www.securitysafetyproducts.co.uk/security/protective-covers-cages/clearstop-antitamper-locking-electrical-double-plug-socket-cover-dementia-care.php

Re: Not only two signs...

Trygve Henriksen

It's a CLEAR cover. That won't stop anyone stupid enough to mess with the cables at all.

If the socket can't be hidden under a raised floor, use a 60309-something instead.

https://en.wikipedia.org/wiki/IEC_60309

Then use distribution rails in the rack with IEC 60320 C13 sockets. Most Rack-mount kit can work with a standard C13 - to - C14 cable.

Well, besides Cisco...

Re: Not only two signs...

Anonymous Custard

Oddly enough, the copious addition of duct/gaffer tape to such a situation was my first thought as well.

And failing that, attach it copiously to the iPad (ideally sticking it somewhere unreachable), and if possible also to the idiot owner.

It really is the universal fix-it for anything...

Re: Not only two signs...

Dave K

And this is why comms cabinets should (wherever possible) be located in a secure, locked room to which only limited authorised personnel have the key.

Where that isn't possible, (ie, small comms cabinets), the power socket should be located as high up as possible to discourage anyone from reaching for it to plug something else in.

Re: Not only two signs...

TchmilFan

This is fine until “management” decides they need some extra storage space and you come back after a holiday only to find the combi lock’s been removed and combustible non-IT stuff piled on the shelves.

I was commenting on twitter the other day…

Gordon 10

The thread poster was detailing all the ways Twitter could go wrong with half of the staff sacked.

I pointed out the only one they missed was unplugging the one plug that MUST NEVER BE UNPLUGGED.

This actually happened to me in the early days of my career working for a specialist transaction processors. They used clusters of zOS mainframe to serve the transactions with the amount m of <3s latency world wide before ubiquitous fibre was a thing. This set up optimised speed over stability as the main thing under their control was processing time not network latency.

In those days unplugging the master terminal would bring the whole edifice crashing down…. Guess where the cleaner plugged in her hoover for the monthly ops room clean….

Re: I was commenting on twitter the other day…

chivo243

monthly "oops" room clean….

Re: I was commenting on twitter the other day…

Anonymous Coward

Back when I was doing customer support for ICL we had a huge number of calls of small mainframes getting disc controller errors recovered read and write fails to discs in regular patterns across many machines.

The Small Series 39 machines were low level cabinets (god know why) and being small mainframes had 2 channels to each controller and a manual switch on the cabinet bevel to allow operators to swap the interface if it was throwing erros and didn't route switch automatically (a fantastically rare event). The switches were beautifully designed and had little stalks poking out of the bevel of the cabinet, an enthusiastic cleaner polishing the cabinet could change controller routes several times in a few seconds causing the errors and the hardware itself to route switch again when the switches stopped moving.

The fix for persistent offenders was a perspex cover for the bevel which required either removal or the use of a bic biro to move the switch

When I left ICL and managed a tech support team one of my first purchases was a hoover for he ops team, it was stupidly expensive as it was supposed to produce no static and hoovering the ops room became late shift responsibility once a week. the improvement in reliability of a number of comms routers and small servers improved miraculously.

Red Dwarf did it.

Dave 126

An episode from Season IV begins with most of the crew on the bridge of the eponymous space ship during a red alert, frantically battling with their consoles. Suddenly all their computers go blank. Terror. Then the Cat strolls in using a hair dryer that he's just plugged in in place of all the battlestation consoles.

Re: Red Dwarf did it.

b0llchit

Priorities, sir, priorities.

(miauw)

Physical Methods Trump Signs in Any Language

Richard Jones 1

For critical, 'Do not unplug' equipment, we resorted to using specialised plugs and sockets, that were hard to unplug and into whose sockets, only the specialised plugs could be inserted. That defeated the cleaners and sundry hardware users. It did not stop one determined PC user who inserted his floppy, (with a month's work on it), into an off limits machine that ran processes 24 hours a day. It used the floppy drive as a sneaker-net output device. He tried to run what he wanted, the PC took over, trashed his floppy and overwrote it with the machine's process data.; lesson learned, there was no way back.

Re: Physical Methods Trump Signs in Any Language

Sceptic Tank

That sounds like 1x poorly designed software system. (a) It relies on a diskette accidentally being present in the drive to write ouput. (b) It just trashes whatever it finds in the drive without warning?

Re: Physical Methods Trump Signs in Any Language

lglethal

I dont know. Part b) seems like a definite BOFH action. Assuming the machine is well signposted as being completely off limits. Then trashing anything that gets inserted into it, seems more like the poor machine defending itself from an unauthrosied penetration... ;)

Re: Physical Methods Trump Signs in Any Language

LosD

#metoo

Re: Physical Methods Trump Signs in Any Language

Gerhard den Hollander

plugs that only a specialised plug could fit into still get unplugged to see if it will take a charger.

I even had a cable that was not only marked ``do not unplug'' but even duct-taped to the scoket being removed just because someone needed a wall socket, and this one was nearest

Re: Physical Methods Trump Signs in Any Language

brotherelf

I wonder if there's a market for plugs and wire ends that look dodgy. Like, "I can see bare copper, I better not touch this, it might electrocute me" dodgy.

Re: Physical Methods Trump Signs in Any Language

Anonymous Coward

The soon appearing stack of steaming corpses will show you that "I better should not touch it" is not really a thought that occurs in the vacuum of the heads of a disappointing amount of people.

Worse, if it's fake it will only encourage them to touch the next one that isn't, so it's really better for humanity to use the Real Thing and just keep some tools handy for occasionally scraping the carbonised remains away.

Nah, I'm a practical man. Why?

:)

Re: Physical Methods Trump Signs in Any Language

UCAP

"I better should not touch it" is not really a thought that occurs in the vacuum of the heads of a disappointing amount of people

The most common thought is "what happens if I press this", often followed shortly after by "oops".

Re: Physical Methods Trump Signs in Any Language

Norman Nescio

It's not only idiots who do apparently stupid things: [1]xkcd: The DIfference

[1] https://xkcd.com/242/

Re: Physical Methods Trump Signs in Any Language

Anonymous Coward

The problem with 'specialised plugs and sockets' is that unless they are really different then the power-hungry idiots are still liable to unplug them just to see if their plug fits

I remember a company that ordered a lot of UK 13A-type plugs with odd pins (rectangular pins were rotated 90 degrees) but abandoned that idea because the idiots still saw them as standard plugs until it was too late

UPS?

GlenP

One wonders why there wasn't a UPS installed in the cabinet with an alert configured, given the steps they'd gone to over redundancy.

Not that a UPS is a cure-all, especially when someone (not me) manages to plug *both* power leads for the Exchange server into the same distribution panel from the single UPS and it's the plug on that panel that fails! I was leaving anyway so wasn't bothered but I did then have to explain the basics to the new staff.

Re: UPS?

Anonymous Coward

Or why a "rack full" was powered off a single standard mains plug?

For this sort of thing, either a commando socket or a fused spur would be far more appropriate.

Re: UPS?

Anonymous Coward

When I took over running a small office network (around 1999, just one large Compaq tower acting as mail and file server - only a dozen users), the only unplanned shutdown was when the UPS failed! Cue a new server with twin power supplies and two UPS boxes.

Don't forget mischief

Roger Greenwood

Yes, yes idiots are everywhere, but so are naughty folks. I have an e-stop outside a lab to kill the power, for emergency use only. Labelled of course, but even so it kept getting pressed with no one admitting to it. It now has a very sturdy steel cover over it, painted red, screwed to the wall so hard to accidentally hit. Still gets pressed.

Re: Don't forget mischief

Little Mouse

Some people are simply compelled to press the button or touch the wet paint. They literally can't stop themselves.

I live with one such individual. It certainly makes life interesting.

Re: Don't forget mischief

jmch

The way that human brains are wired, there is focus on a subject, and negation is not instinctively processed. A sign saying "Do not switch off" or "Do not unplug" simply draws the brain to the possibility that there is something that could be switched off / unplugged. In the same way, if you tell someone "don't think of a pink hippopotamus", they will immediately think of a pink hippopotamus (even if it's just to be able to comprehend what it is that that they should not be thinking about).

Re: Don't forget mischief

tatatata

I once saw a fence around a building site. The fence had a red button with a sign "Do NOT press this button". Around the corner was the display of the counter how many times the button was pressed. It was over 10 000 when I walked by.

Re: Don't forget mischief

lglethal

Just for Info, you can buy such buttons that come with a die pack installed, that stains the hands and clothes of whoever presses the button.

My college at Uni (Halls of residence if you want), had a problem with drunken people hitting the fire alarm button, forcing a late night evacuation, the arrival of the fire brigade, and a massive bill (~$5000) for the college each time (apparently you dont get billed if it's an actual fire, which did happen once!).

They installed the die pack alarm buttons, and funnily enough the late night fire alarm presses stopped. People being aware that they will be identified, and face a $5000 bill, somehow seems to remove the "fun" factor of waking everyone up with a fire alarm.

So maybe such a device might help in your case. So long as you also advertise that the device carries the dye and they will be held responsible for unnecessary presses...

Re: Don't forget mischief

LosD

If you're willing to suffer one more press, it might be a good idea to wait with announcing it until after the idiot is identified and fired/billed/mocked.

Re: Don't forget mischief

jake

Some establishments put a gel (called a tamper dye) on the top of the fire pulls. Activating the pull gets the dye on the fingers. Attempting to wash it off spreads the dye and turns it a rather fetching shade of blue ... the myth almost always says "purple spray", but I've only seen topical blue tamper dye. There is no squirter or "dye pack" built into any fire pull that I am aware of, and I've spent a good deal of my life mucking about in the wiring of schools, hospitals and other such institurions, both new and old. I've been asked to apply the dye to new kit on several occasions, but I've always declined, citing allergies[0].

https://www.american-time.com/product/syringe-tamper-dye-for-fire-alarms/

[0] 'tis true, kinda ... I'm allergic to setting booby traps. Totally uncivilized, IMO.

Re: Don't forget mischief

lglethal

The ones we had were not a fire pull, but a "break glass and push button" type. I assume the push button also acted as a spray release like a spray can, but I never tested it, nor saw the after effect. As I said, once they were installed, and everyone knew about them, no one was quite stupid enough to be willing to risk a $5000 bill just to see how effective the dye was...

Re: Don't forget mischief

jake

Perhaps such a thing exists. Or perhaps just telling people it has been installed is enough.

I went looking for pull stations with a dye pack the first time I had a client request them. I could not find such a beast. So I called the local Fire Department, to ask them if they knew of any buildings that had them so I could get a manufacturer and part number. They told me that such a device would be illegal in that jurisdiction because knowing that you would get coated in dye would be enough to stop most people from pulling the alarm in a real emergency. It would also be a hazard in the mandatory yearly testing (it's an insurance thing). That was around 40 years ago. Since then, I've probably had over a dozen clients ask me to install them, I always say "Sure! Get me a manufacturer name and part number and I'll spec 'em for the build." They have never got back to me, and we always put in standard alarm pulls (and the topical dye in some cases).

However, I'm always willing to learn something new. Got a manufacturer and part number?

It's Friday ... this round's on me.

Re: Don't forget mischief

lglethal

Sorry Jake, it was more than 20 years ago and I was merely a student and not involved in the installation in any way shape or form.

They definitely had people come in and replace all of the old button presses with new ones, but maybe they just spread the rumour that they had a dye spray in them and that was enough to stop people "testing" them.

A quick google hasnt found anything, but it seems to default to American style handles, and maybe we had something different in Australia back then. Sorry.

¯\_ (ツ)_/¯

Re: Don't forget mischief

Anonymous Coward

A CCTV camera might have been more effective than that steel cover.

Re: Don't forget mischief

jake

These days that is indeed the preferred method.

You're worried about turning off the server...

Neil Barnes

Don't go to hospital: https://www.theguardian.com/world/2022/dec/01/germany-woman-held-on-suspicion-of-turning-off-hospital-roommates-ventilator

A plug as a single point of failure

Emir Al Weeq

With the redundancy used, I'm surprised they didn't put a bit of thought into the power distribution and connect each side to the mains with a separate plug. I appreciate that the setup may not deserve independently resilient redundant feeds but, with the design used, the fuse in the plug was a SPoF, sign or no sign.

A helpful hint to the readership at large ...

jake

For future reference, they make locking outlet covers that fit over an inserted plug or plugs, preventing the removal of same. The locks are trash, easier to pick than a file cabinet, but they work for this kind of thing. Under twenty bucks, and usually in stock at your favorite purveyor of sparky stuff. They make more expensive and harder to defeat versions, too. Recommended.

ElRegioLPL

I like that this story is from at least the last 12 years and not any time after 1930

My-Handle

We had a similar issue at an ex-employer's. A network relay lived in one corner of the boiler room and connected to the next building over. Signs aplenty on the plug and socket it was plugged into. Until the spanner-monkey that came in to service the boiler decided that the sign didn't mean him and he unplugged it to plug in his drill. Next building over lost all connection to several site servers and the world at large. The IT director (whose office was in the next building over) was in there like a shot and tore the guy a new one, and he wasn't typically a shouty man :)

"can't quite believe they turned off a network to charge a device"

Mishak

So strong is the draw of social media...

Re: "can't quite believe they turned off a network to charge a device"

jake

And they claim the shit's not addictive. Hah!

Re: "can't quite believe they turned off a network to charge a device"

lglethal

It's important to be up with current affairs, at least for those not-so bright sparks who use Social Media. If they dont keep up to date, they wont know when to do a volt e face...

Tubz

I blame the installer, should be hard wired to the mains and the big red on/off switch in the locked cabinet behind a plastic shield to protect from accidents.

Kevin Johnston

This was exactly my thinking when I read the story, why does all this critical equipment use standard wall mounted mains sockets which are just begging for some numpty to mess about with? Surely you have two mains socket strips inside each locked cabinet to distribute the primary and secondary mains power to the units and these strips are fed from a fused system which is also behind lock and key (or maybe a numberpad lock)

Never tell people how to do things. Tell them WHAT to do and they will
surprise you with their ingenuity.
-- Gen. George S. Patton, Jr.