News: 1726212369

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Muppet broke the datacenter every day, in its own weighty way

(2024/09/13)


On Call By Friday the weight of the world presses down upon even the most enthusiastic IT pro, which is why The Register uses the last day of the working week to lighten the load with a new instalment of On Call – the reader-contributed column in which we tell your tales of struggling out from under tech support burdens.

This week, meet a reader who asked to be Regomized as "Harp" and in the 1980s worked for an insurance company that ran an exotic database coded in Fortran.

"The system had a ton of cables, data points and the like running under a raised floor in a sealed, heavily air-conditioned room, with a lead back to the mainframe," Harp explained.

[1]

So tangled and elongated was that stuff under the raised floor that the system was known as "Snuffleupagus" – after the Sesame Street character that has fur, a trunk of sorts, no ears, and maybe-kinda looks like a woolly mammoth.

[2]

[3]

Like his Sesame Street namesake, Snuffleupagus was not very reliable. Most evenings, the system would mysteriously power down.

"That led to complete shutdowns and restarts taking upwards to an hour to recover, reboot and reset," Harp recalled.

[4]

Investigations ensued, electricians came and went, and no root cause was found.

[5]To patch this server, we need to get someone drunk

[6]A nice cup of tea rewired the datacenter and got things working again

[7]Tech support chap solved knotty disk failure problem by staring at the floor

[8]Client tells techie: You're not leaving the country until this printer is working

Eventually it became apparent that the problem only happened when a chap named Bob was on duty with four other workers.

Harp checked the logs to see if Bob or anyone else was doing something that crashed the rig – but never found anything amiss.

"I was sitting in a chair trying to figure out what was wrong with Snuffleupagus when Bob got up from his desk, walked across the raised platform, and – Thhzzzp! – the system went down."

Bob, as described by Harp, was "One of those 'larger than life' characters with a low center of gravity, ginger beard, and spectacles."

[9]

He was also hefty. So hefty that when he walked across the raised floor – which turned out to have had some panels replaced with extra-thick tiles – his weight meant the floor sagged just enough to press the tops of some metal cable connectors.

"Someone had run phone lines (long unused but still active) in the junction tunnel and over time, they had been smashed so that their wires were exposed to the couplings, creating a short."

Harp lifted the panels, saw the issue, wrapped the wires with electrical tape, and tucked them safely away from the floor tiles. By doing so, he stopped Snuffleupagus from crashing.

"I ran into Bob several years later," Harp told On Call. "He owned his own line of health clubs, had dropped nearly 70kg, and shaved the beard."

If size has mattered enough to create a technical problem you had to fix, [10]click here to send On Call an email and we may tell your tale here on a future Friday. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZuQNRWw1q7ksbMC_IZ_01AAAAcU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZuQNRWw1q7ksbMC_IZ_01AAAAcU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZuQNRWw1q7ksbMC_IZ_01AAAAcU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZuQNRWw1q7ksbMC_IZ_01AAAAcU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2024/09/06/on_call/

[6] https://www.theregister.com/2024/08/30/on_call/

[7] https://www.theregister.com/2024/08/23/on_call/

[8] https://www.theregister.com/2024/08/16/on_call/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZuQNRWw1q7ksbMC_IZ_01AAAAcU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/



This kind of semi-random, intermittent error is such a pain to diagnose

Michael H.F. Wilkinson

A friend of mine once worked on a 68020-based computer (quite a beast in its day, I think it had a whole 16 MB of RAM), which would be on the fritz from time to time. It turned out, there was a tiny break in one of the connections on the motherboard, which sort of closed when the board was slightly bent one way, but opened if the board was seated slightly differently. Some hawk-eyed technician found it, put a little solder on the break, and the board worked flawlessly.

Re: This kind of semi-random, intermittent error is such a pain to diagnose

Pascal Monett

Now that is eagle-eyed diagnostics !

Re: This kind of semi-random, intermittent error is such a pain to diagnose

IanRS

I once had a graphics card with a similar problem. Occasionally the whole system would crash, and I noticed while poking (literally) around while the system was running that pushing it one way would almost guarantee a crash. I put a small length of wood between it and another PCI, or maybe ISA back then, card to keep it slightly flexed the other way. Improved reliability and ran for many years that way.

Korev

Bob, as described by Harp, was "One of those 'larger than life' characters with a low center of gravity, ginger beard, and spectacles."

So the moral of the story is to not let bespectacled gingers into the data centre?

Anonymous Coward

That's me out of a job then.

Sorted

Julian Poyntz

Gone bald, keep clean shaven - though there is now a decent quantity of grey in the stubble these days

Muppets

Korev

So was the current stopped by a circuit Beaker?

Re: Muppets

Flightmode

It was clearly bork bork borken.

Re: Muppets

UCAP

Sounds like he was a bit of an Animal.

'larger than life' characters with a low center of gravity, ginger beard, and spectacles.

Anonymous Coward

This makes me wonder whether the ginger beard and spectacles are mentioned because their extra weight tipped the balance. This also raises some interesting questions.

Could this have been fixed by shaving that character?

Would switching the spectacles to contact lenses help?

Do ginger beards weigh more than other colours?

Actually, I'm genuinely intrigued by that last one now.

Muppet... broke... heavy way.

Bebu

And here I was uncharitably thinking Miss Piggy. :)

Prst. V.Jeltz

Thats quite the diagnostic feat , even though he just got lucky and saw Bob walk over the floor , its still impressive to join those two dots.

You consider a lot of factors when diagnosing , but when you have to start thinking external stuff like about who's on duty , or what day it is , or what the weathers like AND its intermittent ... then its gonna be an uphill struggle

.

(its not always the cleaner unplugging something!)

I'm meeeeeelting!

KittenHuffer

I once had a pair of photophobic mice!

They had worked fine for about 3 months (in the Widdle of Minter) but once sunny afternoons started happening they would suddenly go on strike. They were opti-mechanical mice (ball turning roller with windmill/LED/sensor) that the manufacturer had housed in a wafer thin mechanical mouse shell. The shell was not thick enough to stop bright sunlight from swamping the sensors.

'Fortunately' we had a set of mole-rats who worked all day in an orifice with no external windows (so not fortunate for the mole-rats). A quick swap of mice with them cured the problem.

In truth, I think they were just mice that had turned to the dark side!

Lazlo Woodbine

One place I worked, we had a couple of Compaq servers that would intermittently reboot.

The reboots were completely random, and server logs gave no clues whatsoever.

We had support guys in almost every week running all kinds of tests.

One day, the servers both rebooted while the tech guys were in the server room (room is rather grand, basically a large cupboard, and by in, I mean the were standing there with the door open looking at the servers). This meant they spotted exactly what happened as the servers rebooted.

The manager had walked past and answered her cordless phone, this was an old analogue cordless, the EMP as she answered the phone must have been enough to trip the servers.

The solution, we got her one of the new DECT cordless phones and never had a random reboor problem again...

Hidden intermittent faults

Inventor of the Marmite Laser

I've posted this before, so I'll apologise to those who've already seen it but it's pertinent to the tale of obscure intermittent faults.

Had a site visit donkeys years since upon - pre PC days, in fact. The client, a research bit of a major oil company, had an ISC3651 colour desktop computer. 8080 based, with a staggering 16k of plug in ROM for the operating system on the A3 sized mainboard and a 50 way parallel expansion port on the back, which we'd plugged into our interface.

This is a similar beast: https://vintagecomputer.ca/intecolor-3600-series-computer/

The client had been complaining of random crashing and muggins was sent up to have a firkle.

Disconnected the expansion cable and extracted the mainboard. Had a close look and no obvious issues. Maybe it was one of the myriad connectors. Reassembled the machine, tested and crash. Rinse and repeat several times, increasing the scope of connector unplugging/reseating.

Crash, crash, crash.

After the umpteenth time I had just reassembled and was powering up when I realised I'd forgot the expansion cable. I reached across the top of the machine to plug it in (ok to do live). As I leant on the top of the case - crash. Lift off hand - back to life. Repeat with the same outcome.

Interesting.

Pulled mainboard again and got a magnifier. Started scrutinizing the board with a really intense scrute.

Found one data pin in one of the plugin ROM holders had never ever been soldered. The pin was just pressed against the side of the plated thru hole

30 seconds with a soldering iron and it was fixed.

And that thing had been running an engine test bed for 2 years like that.

Holmes deduces

Sam not the Viking

Not quite in the same vein, but we had a job overseas where there was clearly a temperature-issue. The site equipment had air-conditioning but overnight the temperature would rise causing a trip at anti-social hours. It always reset and then operated normally. Sometimes we would stay on site to try and catch the event but it never happened when we were there.

The site was protected 24 hours a day by a security company and they reported no unusual events. We asked these security people to make a log of a few temperatures; indoor air, outdoor air, just so we could try to get a handle on what was happening. Every hour, as they did their rounds they would record the thermometer readings.

These logs showed remarkable consistency in the values. After the initial set of figures which we recorded before leaving site, the temperatures would remain constant all night. Correctly for the indoor temperature but, my dear Watson, surely the outdoors should have a bit of variation?

An unannounced visit in the small hours revealed the problem: The security people were putting up covers over the air vents in the equipment room. You will already have guessed that this made their 'office' cooler and they could have a nap (for 8 hours!) in a beautifully cool environment. The temperature logs were just copies of the first set of figures written up before going to the land of nod.

Sizing

Anonymous Coward

Never actually had weight cause a problem, but in my younger days I did carry around a fair bit of excess. There have been a few times recently where I've needed to gain access to enclosed spaces and the thought has gone through my head that in the past I wouldn't have been physically able to fit.

Indeed in one instance the client had an electrician who also needed to access said enclosed space... and couldn't. Knowing that I'd done it already, he asked me to sub for him.

Be cheerful while you are alive.
-- Phathotep, 24th Century B.C.