If we plug this in without telling anyone, nobody will know we caused the outage
- Reference: 1708928949
- News link: https://www.theregister.co.uk/2024/02/26/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Padraic" who spent some time in the 1990s working in a manufacturing facility on the west coast of Ireland. The firm, which is now defunct (no fault of Padraic's we're sure) made various items of telecommunications equipment, some of it custom PCBs destined for call centers.
One fine day, Padraic and a colleague were required to install a PBX in the server room, to test new software. The server room wasn't really built for such things – the PBX was a chunky 32kg box, and the room was tightly packed with five or six racks of Sun SPARC Enterprise servers. As Padraic puts it, "there was barely enough room to swing a Celtic kitten."
[1]
Nonetheless, they managed to drill holes for rack mounts, heave the PBX into place and install some PCBs – all before lunch.
[2]
[3]
At which point they couldn't help but notice some frantic activity from sysops who seemed to be running about frantically.
The servers were down! The production line had stopped! Hundreds of thousands of punts (that's olde-timey currency for you youngsters who don't remember life before the Euro) were evaporating by the minute!
[4]
Well, thought Padraic and his colleague, we'd best be out of the way while you sort that out.
[5]Self-taught-techie slept on the datacenter floor, survived communism, ended a marriage
[6]'Crash test dummy' smashed VIP demo by offering a helping hand
[7]Developer's default setting created turbulence in the flight simulator
[8]One person's shortcut was another's long road to panic
Returning from lunch, the place was still just as chaotic. Management was involved now, looking furiously at the sysops who were unable to restore normality.
Padraic and pal resumed installing PCBs in the new PBX and, as they did so, noticed a big fat SCSI cable on the back of one of the machines that was just ever so slightly off-kilter and therefore not making a clean connection. Clearly, one or the other of them had nudged it out of the way at some point during their morning's labors.
Padraic looked at his colleague, who looked back at him. They exchanged a quiet understanding between them, and pushed the cable the few millimeters required to get it back in place. Not a word was spoken – especially not to any of the frantic sysops.
When they finished installing the PBX, Padraic did venture to ask someone about the server situation. All fine! Everything just came back on line! No-one even knows why! A complete mystery – one of those things that's probably best left unsolved.
[9]
Now, you may be thinking to yourself "Didn't SCSI cables usually have clips on them to stop exactly this sort of thing from happening?" And you'd be right. But as Padraic explained: "In this multi-million punt, state-of-the-art installation, nobody had bothered to fit the little 10p clips on the SCSI sockets which are designed to hold the cable connectors in place."
So it wasn't entirely his fault, to be sure.
Ah the 10p saving that ends up costing thousands – classic. If you've ever found yourself in a mess that could have been prevented by some spare change, let us know about it by [10]clicking here to send an email to Who, Me? and we'll use your story to soften some future Monday morn. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZdxvW4wHBaL4a122C7PycwAAANE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZdxvW4wHBaL4a122C7PycwAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZdxvW4wHBaL4a122C7PycwAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZdxvW4wHBaL4a122C7PycwAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/02/19/who_me/
[6] https://www.theregister.com/2024/02/12/who_me/
[7] https://www.theregister.com/2024/02/05/who_me/
[8] https://www.theregister.com/2024/01/29/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZdxvW4wHBaL4a122C7PycwAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Let's Check the Server Room Access Log
They didn't bother to clip the SCSI connector in, and couldn't detect that a drive/device was disconnected during who knows how long a period of frantic troubleshooting, so I'd only give it 50/50 odds at best that there was a server room access log.
Re: Let's Check the Server Room Access Log
Or the access logs were themselves saved on the downed storage device ;)
This story reminds me of the old saying "SCSI isn't magic. There are fundamental technical reasons why it's necessary to sacrifice a young goat to your array every new moon"
Re: Let's Check the Server Room Access Log
My recommendations of a ritual sacrifice have never been approved.
On projects which become 'awkward' I undertake a procedure where mantras are invoked to aid the excision of demons. Those nearby think I am just muttering..... I leave a very small text file to maintain protection and also make an appropriate libation ---->
Re: Let's Check the Server Room Access Log
Access logs in the 90s?
You mean the mechanical code lock on the door....if it was even that secure?
Re: Let's Check the Server Room Access Log
System stopped while they were there but continued when they weren't & started when they were there. Problem doesn't correlate well with their presence. Couldn't be them.
The sure fire 10p investment...
Ah the 10p saving that ends up costing thousands – classic. If you've ever found yourself in a mess that could have been prevented by some spare change, let us know about it
Buts lets exclude 10p invested in a firearm round. We all have stories that could have been best avoided by that simple investment.
Re: The sure fire 10p investment...
let me guess... you're from the western side of the Atlantic Pond.....
Re: The sure fire 10p investment...
TBF there are a few lads on the Emerald Isle who favour a calibre based coercion approach
Ugh I hated SCSI cables
The OG SCSI-1 Centronics style connectors with the clips were fairly secure (if they were properly clipped) but they went away when SCSI went wide. So by the 90s those had fallen by the wayside for wide/ultrawide SCSI using the high density connector which was a nightmare!
I can't count the number of times that intermittent problems could be traced to a dodgy 68 pin UWSCSI cable that had one pin that was slightly bent or had been pushed in a bit, so that that it didn't make solid contact. It could be properly connected and have the screw things on the sides actually screwed in (which were another problem altogether...) but still be in a situation where wiggling the connector or pushing it hard up or down or in could make an intermittent problem go away. But heat cycles or the slightest bump of the cable would undo that magic and it would again become intermittent. The only real fix was to trash the cable in favor of a new one, and hope that once it is plugged in it is never moved - because repeated plug/remove cycles were what created the bent/pushed in pins in the first place!!
I'm glad I got into storage consulting when fibre channel was going mainstream, so I only had one contract where I ever had to deal with servers connected to a Symmetrix with SCSI! Unfortunately JBODSs were still attached with SCSI so you still had to deal with those terribly designed 68 pin HD connectors for server boot drives/scratch storage.
Re: Ugh I hated SCSI cables
Ahhh the bent pin nightmare. I had a custom long nose plier that I'd ground the end into a flat nose, specifically so that I could get in and straighten them back up.
Re: Ugh I hated SCSI cables
I kept a wirewrap tool for exactly that purpose. It was unusual to find a plug where I could not gently slide the tool between the surrounding pins and straighten any bent ones.
Re: Ugh I hated SCSI cables
I'll see your 68 pin SCSI and raise you a fully populated 120 pin EDAC - complete with a lovely tangle of A/V cabling.
At my last job, we had to be very, very careful when working on one of our racks, because one of the fibres didn't quite fit soundly in the back of one of the servers.
We all knew this, so we never had a problem.
Well, we didn't until the aircon engineer visited.
The aircon engineer who dropped his screwdriver.
The screwdriver that just happened to graze that exact fibre as it fell to earth.
Cue lots of frantic teachers calling because they'd lost access to all the vital powerpoints held on that server...
Philosophical question:
If the fasteners would have avoided the outage, is it really Padraic's fault?
... or is it the fault of the people who failed to install the fasteners, albeit with a really long lead time?
Re: Philosophical question:
THAT! Would be an ecumenical matter.
Let's Check the Server Room Access Log
Someone is going to figure out that the problem started while these guys were in the server room and then later stopped when they went back!