A time when cabling was not so much 'structured' than 'survival of the fittest'
- Reference: 1641207851
- News link: https://www.theregister.co.uk/2022/01/03/who_me/
- Source link:
Today's story, from "Keith", takes us back to the heady days of the 1990s and the expensive funnelling of market data to customers keen to maintain their edge.
The markets were about to close on the day in question when all the alarms in the world seemed to go off at once on Keith's console. Important customers had suddenly lost connectivity and that vital data had stopped flowing. He glanced at the time – maintenance windows never opened earlier than half an hour after the markets had closed – so something bad must have happened.
[2]
He sprinted to the communications room, flung open the door ... and found the head of comms standing sheepishly beside a rack of suddenly dead modems and multiplexers and a screeching UPS. A screwdriver was in his hand. Oh dear.
[3]
[4]
The targeted job? Changing a modem. Simple enough, but thanks to the evolutionary manner in which the cabling had been built up over the years, not as straightforward as one might have hoped. All the power leads had been cable-tied into a bundle.
No matter, the comms boss had decided to get ahead of the game and remove the cable tie. Sure, the markets were still open, but it wasn't as if he was going to unplug anything, right? Just a bit of prep work.
[5]
Of course, the cutters were on the other side of the room. But he had a screwdriver, so to save time he had opted to break the tie by just inserting the screwdriver, twisting, and ... oops.
"The tip of the driver had slipped," said Keith, "and cut through a random power cord and sent a spike through that whole rack, and the UPS was complaining about an over/under voltage spike, shrieking to wake the dead."
[6]Ooh, an update. Let's install it. What could possibly go wro-
[7]How to destroy expensive test kit: What does that button do?
[8]When civilisation ends, a Xenix box will be running a long-forgotten job somewhere
[9]A tiny typo in an automated email to thousands of customers turns out to be a big problem for legal
Usually, the circuit breaker in the power strip would have prevented too much damage, but so severe was the shock that even that had been broken. Keith's team broke into two groups – one to find a spare strip and reconnect it, the other to recover the hardware.
"Many of them were single power," he said ruefully. "Not only that, but almost every device had a blown fuse in it."
The newer devices had their own circuit breakers, but much of the kit was of the "replacement deferred due to budget" variety; technically still within their lifespan, but very long in the tooth.
[10]
"Not a problem, right?" thought Keith. "We had a box full of fuses lying around somewhere?"
No, they did not. What they did find – after scouring the stores, closets, and basements – were fuses of entirely the wrong type and rating. In the end, decommissioned devices were torn open and their fuses scavenged.
"I think we finally got that rack fully back up at about 8pm, some four hours after the market close," recalled Keith, "and that had included upgrading two devices from new install hardware that had been slated for other use.
"Several devices required a reconfiguration by hand since of course we had no backups of the configs, except a few random printouts."
The following morning a meeting was convened that featured the very biggest of cheeses, including the CEO and president. As head of operations, Keith was present. As was Sir Screwdriver himself, the head of comms.
Unsurprisingly, the action items came thick and fast. "Comms and ops had to check each other's wiring for compliance to the standards," remembered Keith, "which I got the job of writing down from the word-of-mouth lore that we had been using."
The work included checking out the devices and power supplies. Not a problem for the 100 or so servers in the ops room and their tidy cabling. It was an altogether different challenge for Keith in the comms room and its less-than-structured approach to cable management, however. He turned up a dozen miswired devices and racks running from two power strips on the same UPS – all thankfully fixed and documented before Sir Screwdriver could return to his time-saving ways.
Some things really weren't better in the old days.
Have you ever stuck a screwdriver where you shouldn't? Or tried to recover a configuration located only in the memory of the longest serving member of staff? Tell us your story with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YdMruA53b-hbIq0BQXeSEAAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdMruA53b-hbIq0BQXeSEAAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdMruA53b-hbIq0BQXeSEAAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdMruA53b-hbIq0BQXeSEAAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2021/12/13/who_me/
[7] https://www.theregister.com/2021/12/06/who_me/
[8] https://www.theregister.com/2021/11/29/who_me/
[9] https://www.theregister.com/2021/11/22/who_me/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdMruA53b-hbIq0BQXeSEAAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Don't mix power tools and alcohol
My Dad managed to do that once. He was drunk, and my Mum was upset about a mirror not having been put up. She really should have saved that outburst till later. He decided this was the perfect time to embark on such a task, and proceeded to drill into the wall, hitting those two wires with astonishing precision. The drill was plugged into the very socket those wires were powering, so there was a loud bang, followed by purple smoke. Mercifully he wasn't hurt, and thankfully this sort of thing was not a common occurrence in my house.
Re: Don't mix power tools and alcohol
Given the way electric drills always seem able to find hidden infrastructure with pin point accuracy, I'm surprised the military haven't funded research into the phenomenon. It's certainly better than any targetting technology the military actually uses for weapons.
Re: Don't mix power tools and alcohol
The military should have done some serious testing on my dad before he died. That man could locate every wire in the place with nanometer precision, but only after he'd grabbed the wrong power tool for the job. Need a hammer? He'd grab a power auger & !ZAP! blow all the breakers in the state. It got so bad that mom flat out told him "If you trip the breaker you'd better start running before I can target your ass with a nail gun."
Or maybe the military could examine her instead, she could target hitting me with a thrown shoe for misbehaving from across the property & thwap me no matter how hard I tried to duck & dodge. =-Jp
Re: Don't mix power tools and alcohol
Yes, when I was watching something recently about some dowsing rod nonsense with my SO, and it was saying that they have a device that supposedly locates underground pipes and wires, I naturally had to say 'I know! A backhoe!'
Then I had to explain/show what a backhoe is, because either they're called something else here in the UK, or she just didn't happen to be familiar with that particular wire-targeting device.
ETA: Now that I think about it, I believe it was something on a recent ElectroBOOM, which makes it even more relevant to this discussion. He certainly shares that knack with the head of comms & backhoes.
Re: Don't mix power tools and alcohol
Backhoes are known in the UK as JCBs, after the name of the best known manufacturer - Joseph Cyril Bamford.
Re: Don't mix power tools and alcohol
Ah, I know of JCBs, I've driven past at least one of their plants a few times, but didn't realise the name might be specifically applied to backhoes and their ilk, rather than just any of that yellow construction equipment. Thanks!
Re: Don't mix power tools and alcohol
If you wanted a term that was not brand-specific, you could say 'digger' or 'excavator', as many equipment hire companies in the UK do.
Re: Don't mix power tools and alcohol
That specific style is also often known simply as a "digger". ie it has the big wide bucket on the front and the smaller articulated bucket on the back. I'm not sure why it's called a backhoe in the US though as the bucket digger on the back is nothing like any [1]hoe I've ever seen :-)
[1] https://www.google.co.uk/search?q=hoe&source=lnms&tbm=isch&sa=X&ved=2ahUKEwi48Y2F95X1AhXvQkEAHRg8BikQ_AUoAXoECAIQAw&biw=1366&bih=564&dpr=1
Re: Don't mix power tools and alcohol
Nothing like any ho' I've ever seen either.
Did that in our new house. I was putting in skirting boards and drilled a hole in the wall in the bedroom, only to short the wires coming up from the ground floor into the first floor, everything went dark!
Somehow, I managed to just brush the wires or the spinning drill caused enough interference to break the circuit. The drill was still fine and after resetting the breaker, I could carry on, putting the drill back in the hole was fine, but spinning it up caused it to trip again - so not a normal power surge.
Luckily an electrician friend of the family was there putting in some new plugs and light switches downstairs and he could quickly repair the cables in-situ, without having to break open the whole wall...
"The drill was still fine and after resetting the breaker, I could carry on, putting the drill back in the hole was fine, but spinning it up caused it to trip again - so not a normal power surge."
WTF? See icon ----------------->
For full clarification, you drill a hole, cause a power outage, turn the power back on and do it all over again in same hole? Really? Maybe you ought to have asked your electrician friend to look into it BEFORE you put the drill back into the same hole.
Logic dictates ...
... that people don't try it again. Bit I've seen loads of occasions when someone, for eg, touches something live, shrieks 'ow!' and then does it again to see if it really was live.
I've done it myself when brain was offline. Recently I released a gate which acted like a massive spring, bloodied my nose and launched me backwards into a pile of shit ... which I should've done from the /other/ side so it couldn't get me.
You live and learn, as long as you continue to live.
Re: Logic dictates ...
"You live and learn, as long as you continue to live."
...and the rest get [1]Darwin Awards :-)
[1] https://darwinawards.com/
"BEFORE you put the drill back into the same hole."
Check there isn't an old bloke with a beard and Victorian clothes hanging around.
He mat just be waiting to give you an award, posthumously!
Given all the other work going on, the first time, I though we had just overloaded the circuit. We'd had about half a dozen outages over the previous few days, due to too many drills, jack hammers etc. being used at the same time.
Be kind. He's probably in the IT biz. Let's turn it off and back on. And see if it does it again.
Guilty as charged
Decades ago, I was asked to swap a cable round on the back of a server.
Said server had a LED on the back, which would light if you pressed the matching button on the front, to help identify it when you were round the back.
I was told to press said button, which I did.... which resulted in a catastrophic power-off. Which idiot designs two buttons next to each other?
Re: Guilty as charged
Which idiot designs two buttons next to each other?
Potentially there's about 7.7 billion of them.
Sounds like us (sic)
A good story El Reg - I've seen this happen too. Messing around with AC power is something that people do not worry about in the USA, 120V is a little itch, not a shock so this story sounds like it might have happened in the US, not the UK. I see people do live wiring all the time over here, putting wires into terminals and tightening screws with a little spark most of the time.
I remember my first 240V shock in the UK, I was a kid and wet myself when I picked up the box I was working on and my finger went over the back of the power switch. Ouch!!! OK, I need to change my trousers!
Re: Sounds like us (sic)
When my father was posted from University College, London, to University of Louisville in 1963, he caused an uproar when he checked to see if a socket was live by touching one terminal with his finger. All of the medical staff had fits, "You mustn't do that, it'll kill you". "Nah, it's only 110V, it just tickles" was his reply.
Personally, I used to work for a large electrical manufacturer, we produced equipment for 1100V, 3.6kV, and 7.2Kv, and I was authorised to work on live equipments at all of those voltages. I wasn't authorised for 11kV because that kit was produced at another site, and we didn't have any test sets for that voltage anyway. I still work on live equipment to this day, but the only voltages I touch are 415V or less, but I still take all the precautions I had drummed into me back at GEC.
Re: Sounds like us (sic)
Yeah, 240v can be unpleasant. I once got a nasty belt in my parent's old house. We had a hanging ceiling lamp downstairs in the hallway with 3 bulbs in it and one of the bulbs had blown. I changed out the bulb and switched it on. All looked well! I noticed the lamp fitting was swaying back and forth due to my recent work on it, and reached up to steady it. Cue a nasty zap from the fitting which felt like someone had yanked my arm off. Thankfully, no worse damage to me than that.
Later that evening, we removed the fitting from the ceiling and found that the genius that had fitted it originally had just twisted the bare wires together from the fitting and the house wiring (no connector block, not even some insulating tape around them), then screwed the (metal) fitting of the lamp over the top. Unsurprisingly, one of the cables had come to rest against the light fitting. Result? Every time the light was on, the whole fitting became live.
Re: Sounds like us (sic)
I remember my first 240V shock in the UK
I got my first experience with 380V when I moved to Belgium and had to disconnect an electric cooker - the plug came apart in my hand and I accidentally brushed over a live contact. I found 240V bad enough, but I was very glad I did this one handed because that didn't half pack a punch, enough to encorage me to not ever doing that again without fully cutting power first..
Screwdrivers in wrong places.
Not IT related but I've got a couple here:
1) Back when I was a teenager my second job was an apprentice TV engineer. Poking around in the high voltage areas of a TV with a screwdriver and not a lot of teenage wisdom would occasionally result in accidentally earthing part of said screwdriver's shaft to the chassis while the tip was in contact with something very much NOT at earth potential. The resulting spark would often temporarily spot weld the screwdriver to the chassis! Good job this was back in the valve days as those things were far more resilient to weird voltages being thrown around than the transistorised replacements that had started coming along around the same time.
2) A few years ago I had a guy out to do the regular yearly service on my gas combi boiler. He'd cleaned everything up and partly re-assembled it and was testing how well the water was heating up before putting the main case back on. At the same time he was looking around for a screwdriver that he'd misplaced. Then he noticed a "slight" irregularity through the small window that gave a view into the burner - he'd found his screwdriver! A hasty disassembly and the screwdriver, whose plastic handle was by now slightly melted, was recovered! The following year he was still proudly using the same screwdriver...
Re: Screwdrivers in wrong places.
Ah yes, the old days when, after your first shock, you always remembered to short out the CRT tube before pulling it out to replace it after the CRT heater died. Sure, it was kilo volts stored in the CRT but only a few micro amps.
Re: Screwdrivers in wrong places.
When I was just a teenage nerd, I acquired an old all vacuum tube Dumont oscilloscope along with complete service documentation. One Saturday morning I decided to give it a full service/alignment. Not being totally stupid, I knew I needed to discharge the CRT. However, I basically jumpered a large insulated handle screw driver to the chassis without any kind of limiting resistor to allow the charge to drain off in a controlled manner. As I moved the tip of the screw driver close to the terminal, an enormous spark jumped the gap and made a CRACK sound loud enough to wake the dog upstairs and set it to barking.
Evolutionary manner
One place I worked at in the early nineties decided to finally embrace the notiion of a PC on every desk.
It was a small 4 story building from the thirties and still had the original wiring overlaid with five decades of upgrades and telephony so the general mangler decided that ripping all of the old redundant cables out for a fresh start with the modern age.
Six weeks later, and a couple of skips the old stuff had been removed the floors and walls reinstated and a start made on the new network, the second day one of the installers discovered an old mains incomer in three phase, with his drill, that tripped the local sub-station due to the bit being welded between two phases.
The incomer didn't go anywhere, it was just taped up and left in the wall under the stairs.
They had it coming.
> The work included checking out the devices and power supplies. Not a problem
> for the 100 or so servers in the ops room and their tidy cabling. It was an
> altogether different challenge for Keith in the comms room and its less-than-structured
> approach to cable management, however.
Unfortunetaly, this is exactly what I have come to expect from comms teams and their racks. Sure, there are offenders in both camps. Tidiest are the mainframe guys. Comms and telephone tend to be on the other end of the spectrum, with the excuse usually being "Oh, we don't have time for that now."
Re: They had it coming.
To be fair, if I hear the stories of the contractors we occasionally get in that could very well be accurate. They're not given much time for a job.
Re: They had it coming.
Problem is, if you don't have time for it now, you'll have even less time for it later on, when Murphy strikes. Do it right the first time hardly ever comes to bite you. It is just a bit sad for the work invested when plans change and you have to undo it prematurely. It is rather easy to live with that.
Re: They had it coming.
"They're not given much time for a job"
There's a whole world of wisdom tied to that. Whenever I find a cock-up in some corporate process, from cleaning to completely rebuilding a school there will always be a point where some fuck witted bean counter has said one of "We need to get that done in n number of days", fewer than the experts have assessed it, or reduced the budget to less than it's possible to complete the work with, or said "We'll find money for that bit later". ( They never fucking well do, though.).
So you end up with inadequate storage space/cupboard doors that don't quite fit/ the promised and essential electronic locks that weren't installed/network sockets with no computers ( or vice versa), but staff who need to use them and so on and so on.
I managed to to this with a panel pin
Putting up a picture hook, tap, tap, tap...Hmmm, why have all the lights gone out?
It had gone between earth/neutral so it was the main RCD that cut out. The embarrassing thing is it was a cable I'd put in the wall myself some 20 years previously.
Re: I managed to to this with a panel pin
Don't feel bad. I cannot remember things I did 20 months ago - sometimes 20 weeks ago ;-) It's a good sign that you had better things to think about than holding onto where you buried some cable for 20 years!!
Stuck a finger...
I stuck a finger where it shouldn't matter.
German (and most European) sockets have an exposed Earth connector, so that plugs are earthed as they are inserted. Sort of the opposite of the UK system of the Earth prong being longer and pushing a blanking plate out of the way for the life and neutral.
This exposed earth connector is also very handy for earthing yourself when working on delicate electronics. Just connect your earthing strap to the socket. Job's a goodun!
Anyway, years of no problems grabbing earth connectors. I was standing in my office concentrating on my whiteboard on the other side and I lost myself in thought and felt myself falling forward. I reached behind me and managed to get my thumb on the window ledge and 2 fingers in a power socket, where they touched the earth connector. BANG!
I got a jolt up my arm, my shoulder was killing me and I was a little shaky. I went to reception to report the incident and called up the technicians. They laughed and said I was imagining things. We then went to my office and they tested the row of sockets. Earth dead, earth dead, earth dead, "see, you imagining things", earth LIVE, "oh f*** what the hell?"
It also hadn't tripped the breaker, for some reason! They turned off the power to my office, then dismantled the socket and found the electrician (some 10 years earlier) had somehow managed to cross-wire one socket and, in those 10 years, somehow nobody had ever used the socket to plug anything in. Unfortunately, they wouldn't let me plug in my rickety of PC, before they re-wired it.
I had a lucky escape that day.
Re: Stuck a finger...
Back when I was an apprentice TV engineer whenever we had a lot of work on we'd call in a retired engineer to help out. The guy was in his late 60s and, having worked in the trade for donkeys years, he seemed to be completely immune to electric shocks. I once saw him check to see if a ceiling light bulb socket was live by deliberately sticking his fingers in it. "Yep," he said, a few seconds later - so we replaced the bulb.
Re: Stuck a finger...
Back when a student earning a few pounds in a TV shop doing deliveries and repairs, we regularly worked on TVs when live. A screwdriver held about 1/2” from the cap of the line output valve would draw an arc - if it didn’t you knew the fault was in that section (usually the valve or the LOPT). Was seconded to a workshop in a neighbouring town and tried the same trick on a TV with a suspect line output fault. Unfortunately they ran their workshop off an isolating transformer and ELCB - killed all power on the premises. Easily sorted…
On another occasion (when still a student) I was sent to an office to check why a microfiche reader wasn’t working. First check is for mains power so eased the plug partially out of its socket and put a “mains test” screwdriver on the live pin. A loud bang and whole ring main on that floor went down. I hadn’t noticed, in the dark corner, that the socket had one of those fancy brushed steel covers!
A third tale, from when I was a toddler. My gran’s house had pre-13A wiring and one of the sockets on the wall beside the tea-table was a co-axial mains affair (never seen one anywhere else and nobody knew what it was for). I poked my finger in (as one does as a toddler): luckily, my finger just shorted live and neutral so the current had a short path across the tip of my finger and no current went through me. A jolt and slight burn.
Re: Stuck a finger...
We had this in a building in Malta - at odd moments, the RCD would trip, kicking the office into backup generator power, with the equivalent of an HGV engine rumbling two floors up supplying us with enough juice for systems and airco until we flipped it to live again, only for this to repeat itself hours later.
After a few weeks of this, electricians began to check every newly installed socket and the very last socket they tested was wired incorrectly (typical). As it was only feeding something small, it apparenly hovered just around the current required to trip the RCD.
That really was a swine to find.
Nothing like a disaster...
to get your off your ass and after the disaster... As I took on more responsibility, I found lots of things that had been bodged just good enough, and it was as far as installations had gone. I found keeping things neat, and allowing for maintenance was a good balance!
Re: Nothing like a disaster...
@chivo243
It's a new year so I have upvoted you even though you have spelt "arse" wrong...
Of course a screwdriver is not enough, you need duck tape too. And a knife, and the whole toolbox and of course reading the darn manual.
Who the hell reads the manual?
Anyways... handy tip #1 for a long and happy life
Never trust a power circuit after telling the apprentice which cable to pull out of the socket....
oops bit a mistyping there.... long and happy life for your apprentice
Its not the voltage...
Once dropped a wrench (spanner in the uk) while working above a 48V battery bank, used as primary power for the local city telco. Did it bounce and hit the floor, oh no, did it take out the power, or trip a breaker, oh no. As is 'normal' in these cases it dropped perfectly on to the 2 1/4"x4" copper rails running the length of the building. taking quite a notch out of said rails, and turning the wrench in to a puddle of molten globules flying through the air. The batteries and associated equipment didn't even notice.
Also, back in the day, was disposing of a bunch of retired CRT's, procedure was take to dumpster, and break the nub off the back or the neck to safely pressurize them. Unfortunately I discovered that a CRT can hold quite a charge for a very long time, as I picked one up, and happened to grab on to where the cap would be fitted. Not wanting to drop a very large glass object in the shop, I just stood there as it drained through me. Only a second, kilo-volts at micro-amps, but it was an interesting experience.
Oh
I remember using a screwdriver to lever EEPROM chips out of their sockets when developing software for SBCs in the early 1980s. Never had a problem until the "pre-production, limited issue" DEC board which I discovered, slightly late, had a resistor or similar located under the EEPROM.
Have you ever stuck a screwdriver where you shouldn't?
Yes, but I cannot elaborate ^^
Shocks
A little off topic but still remember an amusing incident regarding a cement factory. Scenario; the kiln is down but due to be re-lit later that night. The daywork Electrical crew were tasked with doing routine maintenance on the starter gear to a fully isolated 3kV induction fan, located several hundred feet away from the kiln floor at the back of the kiln. Cue keen young shift manager (yours truly) who did not want a repeat of the last attempted light-up of this particular kiln where the aforementioned fan motor arced over owing to excessive dust in the motor. Said shift manager dispatches shift electrician (normally lazy bugger) to blow the motor out and check all was well with motor - we all wanted a quiet night. In a completely out of character move, the shift electrician goes over and beyond required duty and decides to megger out the cables as well - the screams of terror from the sparks working on the starter were really quite impressive!
Should be: Whew It Wasn't Me
Based on the story, this installment should be titled "Whew! It wasn't me!" When you investigate an issue and realize that it's someone else who cocked everything up. Even better when that someone else is your boss.
Something similar!
Someone putting up a bracket in the machine room for something - drilled through the drywall perfectly between 2 wires, shorted them out and the whole comms room went dark (which happened to be the room on the other side of the wall).
Fun times!