To err is human. To really screw things up requires a wayward screwdriver
- Reference: 1641803350
- News link: https://www.theregister.co.uk/2022/01/10/who_me/
- Source link:
The latest tale takes us back to the 1970s, "when computers really were computers and a megabyte was a LOT of data," recalled today's contributor, Regomised as "Mark."
Our hero was a newly minted site engineer, working shifts at a data centre housing multiple mainframes running 24 hours a day, seven days a week.
[2]
The beasts in question were of the ICL 2900 variety and worth millions (even in those days). Now long obsolete, some units still endure in museums – [3]The National Museum of Computing, for example, has a 2966 replete with three original ICL terminals. The Central Processing Units occupy five large cabinets alone, and a dedicated mains power supply is required to keep the behemoth supplied with juice.
[4]
[5]
Which is where Mark came in, some 40 or so years previously when the ICL 2900 series was at the height of its power.
"One Sunday morning," he told us, "towards the end of a night shift, the lead engineer said we would carry out preventative maintenance on one of mainframes.
[6]
"Even now the word 'preventative' starts a nervous tic..."
It was a simple task. Work through the 30 or so cabinets and clean the fan filters and check the PSUs. Plus any other bits of maintenance that needed doing.
As he was making his way through the boxes, Mark noted that an indicator light on one panel was dead. "No problem!" he thought and, with the enthusiasm of the newly minted, decided: "I can change this without a power down."
[7]A time when cabling was not so much 'structured' than 'survival of the fittest'
[8]It's the day before the grand opening but we need a firmware update. It'll be fine
[9]Wi-Fi not working? It's time to consult the lovely people on those fine Linux forums
[10]How to destroy expensive test kit: What does that button do?
The indicator was powered from a terminal block, along with a great many other connections. As one would expect, Mark took great care when removing the old neon. The same care was taken with screwing in the new one but... it turned out it wasn't enough.
The screwdriver slipped...
[11]
"There was a flash brighter than the sun, and the noise of a large mainframe juddering to a sudden halt," he said.
"I was out of the computer room doors and walking back in with a puzzled expression on my face by the time the operators realised the thing had crashed."
All in all, the mainframe was down for 24 hours and displayed a voracious appetite for spare parts before finally coming back to life.
And Mark, his misdeed noted, had to face the music.
"Amazingly," he said, "I was only given a reprimand rather than ending my IT career... the site manager saying that my intentions were good, and I would have learned from my mistake."
Ever dropped a screwdriver where you shouldn't? And was your boss as understanding as Mark's? Let us know with an email to [12]Who, Me? ®
Get our [13]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YdwR3APgYQSk@P4COMC7XQAAAJA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.tnmoc.org/icl-2966
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdwR3APgYQSk@P4COMC7XQAAAJA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdwR3APgYQSk@P4COMC7XQAAAJA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdwR3APgYQSk@P4COMC7XQAAAJA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/01/03/who_me/
[8] https://www.theregister.com/2021/12/27/who_me/
[9] https://www.theregister.com/2021/12/20/who_me/
[10] https://www.theregister.com/2021/12/06/who_me/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdwR3APgYQSk@P4COMC7XQAAAJA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] mailto:whome@theregister.com
[13] https://whitepapers.theregister.com/
Re: Harrisberger's Fourth Law
It is well known that killing equipment is rewarded with experience points.
Re: Harrisberger's Fourth Law
> Experience is directly proportional to the amount of equipment ruined.
Yes, you always need to keep your skills current
I would like to say No.
Who would make such a stupid mistake, and do so more than once in their career - Yes me!
Sometimes the hardware harms you...
I had a friend whose screwdriver "slipped" while he was repairing a PC and managed to hit him in the eye. That was the account his employer got, at any rate. The facts of the matter may have been that he was practicing the essential tech skill of spinning his screwdriver in the air and catching it like some kind of crosshead throwing knife.
Re: Sometimes the hardware harms you...
My favourite is when working in underfloor ducts. In order to prevent too much knee an back creaking a pile of recently used tools collects within arms reach. Frequently near a desk in such a way as anyone approaching cannot see them, but fortunately in a place where someone could not step on them. Without approaching you first and then backing away from you so they stand on the one tool that rests on another and all the others rest on the other end of it and so are launched in an arc of steel that would make a good Roman helmet topper. And strangely they all come down (or go up) point first except for the network crimping took which never comes down anywhere!
mm yeah
Reminds me of a conversation I once had with a visiting Injection moulding contractor (we were wrestling with some complicated relay logic ladder)
Al do you know anything about relay ladders? he pauses to reflect for a moment... 'well I know how to get my welded on screwdriver off the cabinet'.
Wise words indeed.
Working as a field engineer in the City of London back in the late 80s, one of our customers was Barings Bank (remember them before Nick Leeson?). I was on their dealing room floor replacing a faulty fusing assembly on an HP LaserJet II. The fusing assembly was a mains powered Halogen bulb which heated a metal roller that fused the toner powder onto the paper. Step one of any repair was to make sure the power lead was out of any equipment before working on anything - especially the fusing assembly of a LaserJet.
I was stood with my back to the office, screwdriver in hand, deep in the printer chassis removing the fusing assembly screws when there was a bright flash, a loud pop and from behind me the sound of a couple of dozen PCs spinning down and the crackle of freshly turned off CRT monitors. Yep - the printer was on, I'd shorted the halogen bulb to ground and popped the earth leakage breaker on the circuit. It took down half the dealing floor and I learned my lesson. I still have the screwdriver which bears a blackened pit on the driver shaft and blobs of molten tempered steel running down it.
I experienced a brief and luckily harmless fireworks display the other day while changing the alternator on my car without disconnecting the battery.
Not quite the same stakes as the machine in the story.
Accidentally leaving a socket wrench on one of the driveshaft bolts is a great way to bend chrome-vanadium steel bars.
Accidentally leaving the socket on the wheel nut is a great way to produce a rattle from within the hubcap and a swift return of the car to the offending mechanic.
Two events spring to mind - one computing related, one not.
The first was where a Newbury Labs terminal engineer used snipe-nosed pliers to slide sleeving over a mains transformer terminal - with the PSU connected to the mains and switched on. He attempted to claim for replacing his glasses which had a sputtered metallic coating... This was declined, but he was let off the cost of replacing his now snub-nosed pliers.
The second was an Irishman who dropped his metal mallet into the uncovered, ceramic fusebox in the roadway in Covent Garden. Each 'bang' relaunched the mallet into the air again, accompanied with what sounded like 'Feck'. There were many iterations, and people scattering in all directions.
Overtime opportunities
As a very junior Operator working shifts on an ICL 2980 at Bath University, I heard a very persistent rumour of an ICL Engineer caught "earning" overtime by judicious use of a 9v battery dragged across the backplane of said mainframe.
Apparently, this used to happen close to the end of his shift, because, of course, once he started the "repair", he had to stay and finish the job. And it happened too often to be just bad luck.
It seems he got caught when, after suspicions were aroused, his manager hid in an empty cabinet and caught him red-handed.
As I said, just a rumour, perhaps an urban myth, I have no confirmed facts, but it was a very popular tale at the time.
is there a reason
....they don't shroud the shafts of screwdrivers with plastic or some form of insulating paint?
Re: is there a reason
I've got a screwdriver with a notched end where it caused a bit of a short circuit, it now also has heat shrink covering the shaft and is used for anything that might be live.
I think this is also the screwdriver found under the raised floor of a computer room.
Re: is there a reason
Yes. They can charge more for the VDE version.
Re: is there a reason
No-one wants the lawsuit coming from its failure.
I once went to change a broken light fitting in the toilet at home. I told everybody what I was going to do and switched off the mains at the fuse box. Switching on another light switch confirmed the power was off.
I went to undo one of the screws and there was an almighty flash, with soot marks across the ceiling, a slightly shortened screwdriver and a tingling in the arm!
I went to double check the fuse box... it was still off! I then noticed another rusty box in the gloom about 12ft off the ground... yes, it's sole purpose was to serve this one light fitting. (I think it may have dated back to when the outside privy had been converted to an inside toilet by the simple application of a sledgehammer)
This is why you carry and use one of those glowing screwdrivers.
Same here. Went to change the mains socket nearest my dining room table. Had the plugs all turned off. Some lazy arse had taken a spur off the kitchen.... A nasty nip was had.. Use my multimeter all the time now
I had something similar in my aunt's house. Went to install a ceiling fan, lights on then flipped the MCBs in the basement one by one until the light went out and my assistant yelled down to me.
Flipped all but the last one back on, confident I had found the circuit (they were all unlabelled of course), went to change the fitting and as I was working the hairs on the back of my hand stood up. Checked the circuit... still live. Some utter muppet had taken the ring out of one MCB and brought it back into a DIFFERENT MCB. It needed BOTH off to kill the lighting ring. I figured that the lights still being on in another part of the kitchen was down to the kitchen having two rings or something weird like that (it was a HUGE house, built to house a 70's pop sensation, so it had had a recording studio in it at one point in time - one reason my aunt bought the house was that it had a soundproofed bunker room and she has a fair number of special needs children so it made an ideal playroom for the noisy ones / isolation room for the autistic ones - the distribution board was about 2 metres long and had three separate metered supplies, presumably for business and tax reasons).
I had to tell my aunt to have the place checked and rewired professionally - that cost a good few grand even back in the day.
I moved into a new (to me) house and the breaker box was labeled in Egyptian hieroglyphics so I figured I'd update it by turning on all the lights, turning off all the breakers, and flipping them one by one, then going into the house and seeing what lit up.
There was a lot of "HUH, that's INTERESTING" said that day, especially seeing when one room lit up, as well as the closet in an unconnected room. And when one breaker turned on the light and one outlet in the "family room" but all the other outlets were on another breaker.
Granted, this is a house old enough for a NEMA 10-30 for the dryer, which has a neutral but not a ground (which is why it's no longer code)
A TV related bang!
Another tale from my apprentice TV engineering days.
I was sent out with an engineer called Mack to help collect a TV that was no longer required by the family renting it. The TV was on one side of the fireplace, the double mains socket on the other. At some point in the past the mains lead had been extended (using a now-illegal taped up connection) by someone in the family. The extension part of the lead was pinned to the skirting board by cable clips before it ran underneath the carpet and around the fireplace to disappear behind a cabinet before reaching the wall socket. Mack asked if they wanted to keep their extension wire - they did - so he asked them to unplug the other end, which they did. He got out his side cutters and chopped through the wire next to where the taped up joint was. There was a loud bang, Mack was suddenly wearing a very surpised expression and the side cutters, whose handles were thankfully insulated, now had a hole where their blades had previously met.
"Oh," said the member of the family who had done the unplugging, "I think it might have been the other plug."
Ah, yes! Warning lights!
A long time ago, in a Big Broadcasting Company...
It was nearing the end of my shift as a videotape engineer, my jobs were all completed and I was awaiting the nod to depart from the shift supervisor. Suddenly, he bursts into the control room, with a 90 minute, 2" tape in his hands and a wild look in his eyes. Naturally, his eyes landed on me!
"Presentation forgot to book a machine for the trailers! They are due on the air in 5 minutes! Get this tape on VT 13, and ready for transmission as fast as you can!", thrusting the tape into my hands.
Now back in those days, all the videotape machines were 2" quadruplex behemoths that normally required around 30 minutes to prepare for transmission! No time for proper checks or alignment, just throw the tape on, put everything in "auto" and put your faith in the system!
Now VT 13 was unique in the department in that, at the time, it was the sole machine made by RCA. All the rest were Ampex machines. The RCA had a number of quirks, one of which was quite weak spooling motors. It took quite a bit longer to spool through the huge 2" tapes than the equivalent Ampex machine, and naturally the wanted trailer was at the tail end of the tape!
Another quirk was that the transport controls for the machine were at the end of this giant monstrosity, conveniently out of reach of the internal 'phone! Because of the panic, the control room was still plugging up all the cables necessary to connect VT13 to the presentation studio, so there I was trying to ring the studio on the phone, whilst trying to find the right spot on the tape, and not having arms long enough to do both simultaneously!
I managed to get the machine on cue with seconds to spare, and hit the "play" button on time. Fortunately - or maybe not - the colour monitor in the booth had been left looking at the "off-air" signal rather than the output from the machine, and it was soon obvious that something was severely wrong! All the skin tones were bright green, instead of the usual fleshy pink! That was the point at which I realised that the machine was on the wrong pulse train (used to synchronise the machines to the various studios). After some terse exchanges with the studio, I decided the only option was to change pulses live, on-air!
I hit the pulse selector to the correct one for the studio. The VT gave an almighty hiccup and lurch before stabilising again. Now all the skin tones were cyan instead of pink...!
There was nothing else to do but to let it run out the rest of the trailer - mercifully only a minute or so - and let the presentation guys apologise to viewers afterwards.
Naturally, there was an inquest immediately afterwards, and I was asked why I hadn't noticed the machine was on the wrong pulse train. Well, the indicator bulb had blown on the pulse selector, so there was nothing to draw my attention to it. Whether I would have noticed in the panic, I don't know, but at least I was not held responsible for the fiasco. However, it was deemed that this fault had to be rectified NOW, before it could catch anyone else out!
Cue the entry of a very disgruntled maintenance engineer, who, like me, had been about to depart for home. Changing the bulb required a major dismantling of a very fiddly row of selector buttons, which he achieved in record time!
Having replaced the bulb, he started punching the buttons in sequence to make sure all the others worked. At this precise moment, the doors to both transmission suites (network 1 and network 2) flew open and the senior engineers ran out shouting "Who is f*!&ing about with the pulses?!?"!
Yes, both networks were crashing about and reframing on air!
It turned out that this pulse selector in VT 13 had yet another issue, in that when switching pulse trains, it momentarily shorted two pulse trains together! Not enough to cause any electrical damage, but certainly enough to seriously disturb the transmission machines!
The tale of woe in the fault log the next day made for very interesting reading....!
--
Pete
Rather boring in contrast, but not so long ago...
.... I fatally short-circuited the Dell T20 mainboard in my home/home office server while trying to reseat a card that I unjustifiably assumed was causing problems. Since then I do remember well what I knew in theory but had always ignored—that a "poweroff" command doesn't really turn anything off, without me actually switching off the PSU...
Of course that happened on Friday afternoon.
I was lucky enough, though, to find someone selling an identical box within a two-hour drive on that same weekend, so the downtime remained bearable. And a few weeks later I even found a ridiculously cheap replacement mainboard to revive the old box, which then became my new desktop PC.
While I (so far) never repeated that specific mistake, every time I get off cheaply I wonder whether that might be detrimental to the learning effect.
Ah good old ICL
While doing my engineering degree we tried to learn Fortran IV on the uni ICL 1902 (IIRC). I lived 2 miles from it and in trying to do my thesis which involved programming I'd sometime walk there and back 4 times in a day, arriving to be greeted by a large sign on the wall lit up with 'HARDWARE FAULT'. After a term of this I was allowed into a small office where there was a PDP11 and Fortran and rather than writing a bit and then coming back later to fix the errors I could do it within a minute. No physical explosions but my brain exploded after that!
Harrisberger's Fourth Law
Experience is directly proportional to the amount of equipment ruined.