Exsparko-destructus! What happens when wand waving meets extremely poor wiring
- Reference: 1627025413
- News link: https://www.theregister.co.uk/2021/07/23/on_call/
- Source link:
Our story, from a reader Regomised as "Andrew", takes us back to his time working for a certain telecommunications company, "still well known in the ISP/Telco hardware world," he said," but not as big as they used to be."
Andrew's workplace enjoyed an on-site data centre with all the bells and whistles one could hope for. HVAC, UPSes – it was not short of the initialisms and acronyms so beloved by the IT world. It also featured the usual Emergency Power Off (EPO) buttons and actual humans to prowl the corridors at night to check all was secure.
[2]
"To keep everyone honest," explained Andrew, "the guards would carry a wand that would read RFID tags at various locations to record the time, date and position." Thus management would know that the guards were doing their rounds.
[3]
[4]
The data centre contained all manner of exotic gear, including a Sun Enterprise 10000 (or E10K) setup, which carried a price tag usually ending in lots and lots of numbers. As such, the guard had to pay a visit to the data centre floor to verify everything looked ok.
To ensure the guard actually went into the data centre, the RFID tag was on a panel away from the door. This panel also contained HVAC controls and alarm relay hardware. Oh, and also an EPO button, which was sensibly covered and sealed.
[5]
You can probably guess where this is going.
Going forward to the night in question…
Doubtless making swooshing sounds, the guard wielded his RFID wand like a lightsaber, bringing it down on the tag with an almighty thump. The contents of the panel rattled disapprovingly and the RFID tag was registered. The guard continued on his rounds, oblivious to the mayhem his exuberance had unleashed.
[6]Try placing a pot plant directly above your CRT monitor – it really ties the desk together
[7]Where's the boss? Ah right, thorough deep-dive audit. On the boardroom table. Gotcha
[8]Go to L: A man of the cloth faces keyboard conundrum
[9]Ouch! When the IT equipment is sound, but the setup is hole-y inappropriate
Since nothing had apparently happened, nothing was reported. And nothing was reported in the logbook either. However, something had happened, and every pager for everyone on-call (including our hero) went off. Sun's hardware had been abruptly shut down. Attention was demanded and, bleary eyed, the SysOps and Networking teams rushed to the aid of the suddenly stricken system.
"By the time I got there the lights were back on and every server was sitting waiting for FSCK."
[10]
But what had happened? Once things were back online the guard was questioned and swore that he'd only administered a sharp rap on the RFID tag. Logs were pulled and pointed to the EPO being pushed "but the covers around each EPO button were still in place and the seals were still intact."
"How could this be?"
It transpired that the RFID tag had been stuck to the panel housing the EPO relay. "With great caution the cover was removed to find... the sketchiest relay install known to man."
The relay itself was suspended in mid-air, only supported by wires from the EPO and PDU transfer units. A contractor had installed it, thought "job's a good'un," closed up the panel and left it, dangling, waiting for a guard to pretend he was Luke Skywalker or Inigo Montoya and thump the tag on the beige case, thus shorting out the wires.
The fate of the contractor is lost to the mists of time (we suspect that with such competence and attention to detail, he must be a political advisor by now.) Andrew wasn't sure what became of the RFID panel that triggered an On Call pagerdemic either.
However, in terms of data centre disasters, he noted: "Not all buttons are red but beige boring panels can easily do the job."
We've all had to deal with the results of somebody else's poor handiwork, but have you been bitten on the behind by the leavings of a shonkmeister? Or were you the leaver of the shonk? Let us know with an email to [11]On Call . ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YPqTPPr5uDxoUdGZJPYKEgAAAE4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YPqTPPr5uDxoUdGZJPYKEgAAAE4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YPqTPPr5uDxoUdGZJPYKEgAAAE4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YPqTPPr5uDxoUdGZJPYKEgAAAE4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2021/07/16/on_call/
[7] https://www.theregister.com/2021/07/09/on_call/
[8] https://www.theregister.com/2021/07/02/on_call/
[9] https://www.theregister.com/2021/06/25/on_call/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YPqTPPr5uDxoUdGZJPYKEgAAAE4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
Re: Hands up
I remember the cabling risers I dealt with from that era. 19 year old me would've shimmied down the riser with some cutters; they were certainly big enough. They were a little intimidating as they ran the full height of the building and I was based on the 8th floor.
Re: Hands up
You are John McClane and I claim my £5
Re: Hands up
Wow, they were big cutters, 8 floors long : how do you get them to open in a riser ?
Less poor wiring than poor building
Working on a studio many years ago in the UN building in New York... while climbing through the ceiling voids looking for cable runs I discovered that the 'secure' wall between the studio complex and the presumably highly secure bank branch who had the next suite, er, stopped at the ceiling tiles.
They were not amused when I told them...
Re: Less poor wiring than poor building
Hmm. You would really have though that was something the bank's security team should have checked well before moving in!
Re: Less poor wiring than poor building
Not an issue in modern housing due to the way it's thrown up, but in older terraces It's not unknown to have dividing walls between premises not go into the roof space (and a fire risk as the dividing wall is the main firebreak). Offices are a different matter as they're rarely occupied in the way originally planned.
Re: Less poor wiring than poor building
Cheaper version of this. My office backs onto a lecture classroom. The wall is plasterboard & studs, but with offset studs for better sound isolation (decouples the walls). I complain that I can hear the bass of every lecture (despite loudspeaker not being mounted on that wall) , so during re-vamp of the classroom they put in a third, offset stud wall.
I can still hear the lecture, so do a bit of prodding myself.
Yup: stud walls go up to beyond the (early 2000) ceiling tiles. But only as far as the 1950s ceiling studwork for attaching an earlier set of now-removed acoustic tiles. Sealing that gap sorted the problem. (too thin for students to crawl through to escape the lecture)
Re: Less poor wiring than poor building
Similar here.
My old office suite (I was in one of three) was created by dividing a meeting room which is used for, er, meetings along with student seminars etc.
The sound echoes through the void above the stud partition. Warned them at the time but what would I know. I does work both ways though as they discovered when I was on the phone to a car insurance company that had incurred my displeasure.
The centre manager and I also tried to get some services (network and power) installed at the same time but were overruled by the Facilities Dir. Now I've moved office and they want to rearrange and put two extra desks along that wall but guess what's missing?
Not quite the same but in a previous role we had a Notes server supplied and managed by a sister company's IT people who were very precious about it.
Following an office move, when I was just seeing out my redundancy notice and handing over, the server power failed, cue annoyed calls from the Dutch, what's wrong, why didn't the UPS kick in, etc. It turned out that when things had been recabled in the new office (not by me) they'd plugged both PSUs on the server into a single C14 socket strip connected to the UPS, it was the socket strip wiring that had failed. Why it was cabled that way I don't know, given that I'd deliberately installed two such strips to allow some degree of redundancy as budgets didn't run to a second UPS*.
*Ideally we'd have had one PSU to the UPS and one direct to the mains but that wasn't considered acceptable
I submitted a "Who Me?" to El Reg a couple of years back where I experienced a similar issue. It's always great when someone installs a server and doesn't realise that the redundancy offered by two PSUs is limited somewhat when you plug both of them into the same power distribution unit.
I recall having to explain to a senior manager why we needed a second UPS when he'd signed for a replacement server with two PSUs (for a small office, ~20 years ago). I had to point out that one reason we needed a new server was because the other one (basically a Compaq tower PC with a bigger HDD than normal) had lost power when the UPS failed!
(I hasten to add I had only taken on the IT role part-time and temporarily when the IT manager, who had previously worked for a multinational, and probably had had limited hands-on with actual kit, decided to retire.)
Bodge it and quickly
Facilities had a moan once because the crew I’d organised to run Ethernet through a campus building had pinched / snagged a power cable in a metal surface trunking. I asked why there was metal surface trunking in the first place. Weeks of blaming with them insisting they wanted to run the Ethernet with me countering, it was agreed the electric wiring wasn’t safe and the metal trunking would be replaced with new electrical wiring and Ethernet re termination.
Everyone was just lucky the metal trunking was earthed.
On inspection there had been evidence of arching in lots of places and after plastic was installed the stories came out how that building was plagued by electric circuit trips for years previously.
Not so much poor wiring, just a sparky drilling a hole through a wall into the comms room and managing to get the drill bit smack in between 2 cables, stripping the insulation and shorting the wires as he went. One loud bang later (not the Paris kind) and we had a room full of dead comms kit...
And a charred, smoking sparky? Or was he somehow spared voluntary electrocution?
Tagline
Shouldn't the tagline be "Hello. My name isn't Andrew. You killed my data centre. Prepare to die" ?
And was the guard left-handed?
Re: Tagline
Inconceivable!
Re: Tagline
You use that word a lot. I do not think it means what you think it means...
Er ...
Not quite the same, but I got called out to a site where their in-house "electrical genius" had installed a number of "Emergency Stop" buttons demanded by Health & Safety.
Shunning the usual methods, he had wired cheap push button switches across the mains at strategic points. Pressing the switch shorted the supply, tripped the circuit breaker and cut off the power.
Eventually one test welded shut the contacts of the 32V rated switch involved, making it impossible to reset the breaker.
Re: Er ...
sounds like a datacenter in center of Oz crapital with some, er, response sensitive departmental servers in it and an ISP ITIRC. Plumber tightening a cooling pipe dropped a spanner which with the unerring accuracy such situations demand, across the terminals of 415 V going into the center. Cue loud bang, some of steel wrench becoming gaseous and rapid silence on servers as UPS due for maintenance failed with enthusiasm, or so the story goes. Not there, heard it second hand
Re: Er ...
also reminds one of the simplest project a merkin electronic magazine published in 1970s, but to a higher standard. Metal box with one single pole switch, rated 110 V 10 Amp. Plug it in flip switch, blow fuses. Why ? GOK
RFID wand
They should have known giving a terminally bored security guard something that could be used as a sword or light saber would eventually bite them in the ass somehow!
If it works, it isn't stupid!
Contractor was engaged to install a bit of networking kit (Centillion 100 for those old enough). When it came time to go Gig-E, it all had to go. We found a C100 cargo strapped underneath an AC duct in the ceiling of a room. It was the only place where all of the Ethernet cables would meet.
Nothing New
If I had a pound for every bodged bit of wiring done by 'professionals' that I've had to put right over the years I'd be a very rich man!
{we won't talk about the bits I did myself in the early years}
Cage nuts and cable monkeys...
We've outsourced our WAN, and this works... mostly...
(after we got rid of the Frame Delay setup that was delivered at first)
And it's the company we outsourced it to that sends out techies to replace routers and stuff when there's an upgrade.
One of these 'techies' placed the cage nuts on the front of the rails, mounted the router onto them, and placed the modem and PSU on top of the router.
My guess
Either a Monday morning job, too bleary eyed to know or care, or Friday afternoon at Pub 0'clock!
Hands up
My first job in IT as a 19 year old learning the ropes.
Helped the IT manager with some cabling twinax cabling work for AS/400 that ran down 4 floors in a conduit shaft, problem was, we didn't have access to 2 of the floors as they were owned by other companies, somewhere they were cable tied very firmly so couldn't move them,.
had the bright idea of cutting the cables at both ends to which we had no access so they could be moved.
Then, by stripping back the plastic, twisted the cables, soldered them together and covered them in insulating tape and left it at that.
Wasn't any complaints on the Monday morning from the AS/400 people so job well done and dodgy cable soldering hidden.
I was the 19 year bright spark who said to cut the cables and resolder, just fortunate there was enough slack at either end to do it.