Fix five days of server failure with this one weird trick
- Reference: 1630050426
- News link: https://www.theregister.co.uk/2021/08/27/on_call/
- Source link:
Our story, from a reader Regomised as "Will", takes us back a quarter of century to the back office of a well-known UK bank. You know the one – it had a strapline that Will noted was "an absolute nightmare of a slogan if we ever refused anyone a loan."
It had a substantial amount of Unisys B-Series gear, in this case one server per 20 connected clients. "We had nine of these server/client configurations," Will recalled. And he was responsible for supporting them.
[2]
The servers were of the B39 variety. Veritable balls of fire, with a mighty 20MB of RAM serving the demanding 80386-based processor. It had a floppy drive and an exposed SCSI bus for attaching peripherals. As was the custom, the grey, plastic box housing processor and RAM was called the "CPU" and Will's usual configuration added one or two hard disks, a tape drive, and some communications modules hanging off the SCSI bus.
[3]
[4]
All was good until a fateful Monday, when the server crashed overnight with a SCSI tape drive error. Not unheard of, and after a reboot all seemed OK.
Tuesday, the same thing happened again. This time Will escalated the problem. A call was made to Unisys, his word was accepted ("I had a bit of a reputation at the time," he modestly told us) and a new drive shipped. Will fitted it, tested it, and declared the job good.
[5]
Wednesday rolled around and once again the server tripped up with a tape drive error. Faulty replacement parts are not unheard of, but Unisys opted to play it safe and sent out an engineer to do the switcheroo. All went well until the engineer was packing up and the new tape drive errored. Again.
"Long story short," said Will, "the rest of Wednesday was spent trying to get this tape drive to work, and it didn't."
The Unisys engineer arranged for a complete, known working server to be shipped to the office.
[6]
On Thursday the server arrived. The hard disk from the suspect box was swapped.
The new server crashed with the same error.
Will had a bright idea – there was a decommissioned server still running in the office. How about using that?
Nope – still crashed.
Will and the Unisys engineer then tried building a single server from bits of other known working computers. The only constant was that hard drive.
And still it crashed.
Will's tale predates [7]Craig David's "7 Days" , but the experience certainly sounds like one to which the singer could relate.
[8]So the data centre's 'getting a little hot' – at 57°C, that's quite the understatement
[9]See that last line in the access list? Yeah, that means you don't have an access list
[10]Breaking Bad or just a bad breakpoint? That feeling when your predecessor is BASIC
[11]Malware and Trojans, but there's only one horse the boss man wants to hear about
The week was coming to an end at the bank, and still the server was fighting Will and the called-out Unisys engineer. The hard drive couldn't be changed (for obvious reasons), but everything else had.
Except one thing. The box received power from a brick on the floor. "This is going to sound weird," Will hesitantly said to the Unisys engineer. "I know the power supply shouldn't have anything to do with it because everything else is working, but we've changed everything else, so…"
"Yes, you're right," replied the engineer, "it is weird."
"But why not?"
As Will bent to inspect the black power brick, the system responded by immediately crashing.
"It turned out that the power adapter was faulty and sending bad voltage to the server and then to all the peripherals via the SCSI bus."
It transpired that the CPU and hard drives could handle it, but the tape drive was a bit fussier and responded in time-honoured fashion: generating an unhelpful error and crashing the server.
"The power brick and cable were taken from our spare," said Will, "and it worked faultlessly thereafter."
Sadly, the line "and then I changed the power supply" does not feature in the climax of Mr David's opus. It might have improved it immeasurably, perhaps creating an anthem for all those cursed to be called out to solve more mysterious IT problems. Share your experience with an email to [12]On Call . ®
Get our [13]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YSi3354u-tpNtzOyZndhAwAAAAQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YSi3354u-tpNtzOyZndhAwAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YSi3354u-tpNtzOyZndhAwAAAAQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YSi3354u-tpNtzOyZndhAwAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YSi3354u-tpNtzOyZndhAwAAAAQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.youtube.com/watch?v=P5lmb5-tnM0
[8] https://www.theregister.com/2021/08/20/on_call/
[9] https://www.theregister.com/2021/08/13/on_call/
[10] https://www.theregister.com/2021/08/06/on_call/
[11] https://www.theregister.com/2021/07/30/on_call/
[12] mailto:oncall@theregister.com
[13] https://whitepapers.theregister.com/
Re: The "inspector"
Upvoted for being a toddler and realising the fan should be turning... though I'm now guessing that that knowledge was acquired from a previous incident involving fingers and pain...
Re: The "inspector"
Fingers and pain is why I still remember that capacitors retain charge even after the device they are in has been unplugged.
That and picking myself up from the other side of my bedroom.
Re: The "inspector"
Had something similar when I was younger. I would have been around 14 years old and during one school holiday, my dad took me into work with him for the day. Most the kit there was SGI Indigo2 workstations (lovely machines back in the mid 90s), but they also had a Win95 PC in the corner of the room for running some applications not available under IRIX.
The Win95 PC (a Pentium 200 if I recall) was universally hated amongst the staff as it was very unreliable compared to the SGI workstations. I used it a bit and agreed. In fact, it was unreliable to the point that something had to be wrong with it. On a whim I popped off the case and pointed out the dead CPU fan that was causing the thing to repeatedly overheat and crash.
We shut it down and popped off the old heatsink and you could actually see the CPU's model number on the scorched heat-sink it had been getting that hot. With a replacement cooler fitted, it ran fine after that. I was actually quite amazed that the CPU hadn't been damaged by the prolonged heat if I'm honest.
Re: The "inspector"
I once had a client's PC showing typical overheating behaviour (working fine first thing, but crashed around 10am)
Took the cover off (honestly, isn't that what you do with ANY problem?) and could see that the fan had seized, but then the heatsink had got so hot that it had melted the plastic clips that held that fan in place. The fan and plastic frame were hanging off the heatsink to the point that even if the fan had spun it wouldn't have cooled much.
I didn't have appropriate spares with me so I replaced the fan and cable-tied it in place as a 'temporary' fix. That lasted about another 3-4 years until the PC was replaced.
Power supply on the floor?
Never seen a server with a separate "power brick", but I'm surprised they didn't think to swap it out sooner when they had swapped out everything else. Old school SCSI gear was surprisingly finicky about power.
Though I have to admit I was expecting the tale to end with everything having been replaced except the SCSI cable, and one pin was going to be ever so slightly shorter than the rest. Having a pin that gets "pushed in" just a bit, rendering a cable as working but not 100% of the time, was something I ran into more than once back in the day. Though IIRC with Ultra Wide SCSI cables that used more / smaller pins not the original brand of SCSI with the stout pins where it would be easier to notice when something like this occurred.
Re: Power supply on the floor?
SCSI cables always seemed to be as pernickety as a Siamese cat. Unless they were seated Just So they would sit there sulking until they were.
Centronics cables, on the other hand were made of sterner stuff. I say someone perform the impossible task of putting one in upside down - it required considerable force and a complete lack of mechanical sympathy, but they managed it. After realising their error they pulled it out, fitted it in the correct orientation and carried on printing.
Re: Power supply on the floor?
I received an Adaptec SCSI controller and 3 Quantum Fireball SCSI drives (20MB, 20MB and 40MB, ISTR). A friend had had them and had tried "everything" and it was "just broken" and he was throwing them out. I grabbed them and plugged them into my PC and had a little look.
He had missed the terminator on the Adaptec controller for the second SCSI bus. I plugged in a spare terminator I found at work and it worked fine. A real stonker of a set-up!
Bent pins
I won't be the first person to have performed the miracle of 'restoring a monitor to full colour' by bending straight the connecting pin.....
I'm guessing TSB
TSB were fond of reminding us in their adverts they were "The bank that likes to say 'Yes'."
I had to scurry off to Wikipedia to remind myself of when Lloyds and TSB merged. 1999 to save you a trip and 2013 when the two demerged back into their separate entities.
Re: I'm guessing TSB
And look at the fallout that caused :)
Re: I'm guessing TSB
Warning: NSFW.
It made me think of [1]this :-D
[1] https://www.flickr.com/photos/stillunusual/14284327157
Re: I'm guessing TSB
I went to look at the link and the cookie policy was so complex I gave up (see the gdpr thread today)
What could go wrong
And look at the fallout that caused.
Saw some first hand.
"an absolute nightmare of a slogan if we ever refused anyone a loan."
Arghh!!!! You've brought back horrible memories of doing a skit of their advert in Drama class at school, which went down like a wet pudding. Crickets, tumbleweed, bemused looks.....
In a former life I received a call from a shop floor: funky harddrive. It made some spinning and scratching noise but otherwise didn't work. I changed it for a brand new one. And a few days later the same symptom appeared again. I changed it again and gave them stern warning not to physically abuse computers. Given this was a shop floor and knowing suspecting that they'd rather take a hammer to solve problems, was not a totally unreasonable assumption. Anyway, the harddrive crashed again shortly after. I admit, I did not believe their affirmation that they didn't hit or even touch the machine but I agreed to take it to the lab to have a closer look. Turns out the hdd controller developed a habit of crashing drives. And I not to jump to conclusions. Nah, just kidding - I still do.
I used to use that B2x B3x kit when I worked for Unisys.
It was actually badge engineered Convergent Technology kit sold by Burroughs/Unisys and a few other big names IIRC.
It had an unusual design in that all the peripherals came in uniform grey box design about the size of a thick book; you stacked them beside each other then operated a plastic lever that cranked them together and connected the buses. It took a few seconds to do; it was really easy.
The thing is, it wasn't really something that needed to be easy; saving a few minutes of hardware install time with that mechanism wasn't really cost effective. What it did do, however, was encourage hardware sharing by penny pinching managers. Things like tape drives would wander round the office daily. What ought to have been a one time operation on initial install ended up happening daily.
Aside from that it was good, if expensive, kit. And the OS was way ahead of its time. But when the PC came along, it couldn't complete.
Now that bus clip together design already wasn't quite robust enough; it would flex along with whatever it was sitting on. Repeated operation of the connection mechanism swapping kit around just made things worse. The things ended up becoming really sensitive to the evenness of the surface and how much it flexed. It got to the point where someone sitting on the edge of your desk could cause errors.
I was sure that it would turn out to be that damn bus connector.
Dodgy power supply, Gigabyte?
The "inspector"
This reminds me of a time in my early days tinkering with electronics and pcs, i was literally a little more than a toddler, and my dad had a small pc selling and repairing business.
One customer brought his pc in for support, because it was "randomly freezing and had to be powered off". It was a big full tower with 5 5inch bays and 2 or 3 floppy bays in the front. My dad was trying to investigate a software issue, after all these were the days of the freshly shipped Windows 95, but it seemed nothing was out of place, all the drivers were looking fine and a fresh re-install of windows didn't solve the issue.
Meanwhile, being a curious creature, i crawled under the desk to look inside that chungus while the case was open (no shiny glass side panels in those, the two sides and the top came out as one piece)..... and EUREKA, the little cpu fan (think Pentium I 133MHz) was not spinning.
A quick replacement, et voilà, the machine got back to full performance.
I even had to make the call to the customer telling what had been wrong with his pc.
I miss those days, now i'm a sysadmin in a medium-to-large company and i have responsibilities... oh well, TGIF, cheers!