A nice cup of tea rewired the datacenter and got things working again
- Reference: 1725002770
- News link: https://www.theregister.co.uk/2024/08/30/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Kevin" who told us of his time working as a hardware engineer for Norwegian minicomputer vendor Norsk Data. His employer had acquired a word processing outfit called Wordplex (and, according to [1]contemporary accounts , ran it into the ground and milked its customers – just like tech acquisitions work to this day).
Kevin was asked to fix a Wordplex installation that he described as "a cluster system with a host based on eight-bit Z80 technologies, 10 to 20MB Winchester HDD, 5¼-inch floppy, green screen, and two slaves attached over RS-232 with no internal storage."
[2]
No, it probably couldn't run Crysis. It's a wonder it could run anything.
[3]
[4]
And sometimes it didn't.
Kevin was summoned when the Wordplex suffered intermittent and inexplicable crashes that he had been unable to address on two previous visits. But he suspected not all was well with the electricity supply, so on his third visit brought along an electrician.
[5]Tech support chap solved knotty disk failure problem by staring at the floor
[6]Client tells techie: You're not leaving the country until this printer is working
[7]Techie told 'Bill Gates' Excel is rubbish – and the Microsoft boss had it fixed in 48 hours
[8]Customer bricked a phone – and threatened to brick techie's face with it
Upon arrival, the sparky opened his bag of tricks and prepared to work. The client kindly offered a cup of tea, which Kevin gratefully accepted.
He then heard two things: the kettle click into action, and the word processing machine crashing.
[9]
The electrician investigated, and soon reported that the client had recently upgraded its electrics.
Badly.
"The upgrade had introduced a fault that meant a heavy load – such as all the salesforce coming in for a meeting while the aircon ran and all lifts were in use – stressed the wiring," Kevin told On Call.
[10]
When the kettle went on, it was the final straw – at least for the circuits that supplied juice to the Wordplex.
Kevin and Norsk Data were off the hook. Better still, the kettle still worked so Kevin got his tea!
"It was one of the most satisfying cuppas a customer ever made me," he told On Call.
Has tea saved your tech? Or cursed it? Share your stories of the world's second best hot non-alcoholic beverage – we're not saying what comes first but feel free to debate it – by [11]clicking here to send On Call an email so we can feature your story on a future Friday. Or if you don't have a story, feel free to brew me a mug of strong Earl Grey with half a teaspoon of sugar and full cream milk. Again, feel free to debate that. ®
Get our [12]Tech Resources
[1] https://www.techmonitor.ai/technology/norsk_data_accompanies_new_nord_mini_unix_line_with_wordplex_on_ats_88000_commitment
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZtGYRHiiTKXjGL8ymUs8QQAAAEE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZtGYRHiiTKXjGL8ymUs8QQAAAEE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZtGYRHiiTKXjGL8ymUs8QQAAAEE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/08/23/on_call/
[6] https://www.theregister.com/2024/08/16/on_call/
[7] https://www.theregister.com/2024/08/09/on_call/
[8] https://www.theregister.com/2024/07/26/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZtGYRHiiTKXjGL8ymUs8QQAAAEE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZtGYRHiiTKXjGL8ymUs8QQAAAEE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
Re: There is a proper way to do most things
As another commentard said on here many moons ago, "Brew what thou wilt shall be the whole of the law".
Re: There is a proper way to do most things
A mug is fine - provided its a porcelain mug, of course, and the contents are lapsang souchong, no milk, no sugar.
Though this is also accepable --->
Re: There is a proper way to do most things
You drink beer out of a porcellain mug?! (sorry...)
I drink Darjeeling FTGFOP[1] (far too good for ordinary people :p ), but 1. quite strong 2. in large quantities and 3. out of a mug. No sugar no milk, no lemon.
Life is too short to drink bad tea.
[1] ok, or autumnal - which has a more aromatic taste. Or nice gunpowder tea, or - what my tea shop recommended - Java (which does not become bitter in a thermos mug, which I sometimes use). Earl Grey is an abomination unto Nuggan and shall be shunned.
Re: There is a proper way to do most things
+1 for the PTerry reference!
Re: There is a proper way to do most things
Darjeeling is my preferred tea in the afternoon, but at work I stick to Keemun black tea. No matter how strong you make it, it never turns bitter
Re: There is a proper way to do most things
Thompson's Punjana, which can handily also be used to treat fencing. I've tried Barry's (the southern equivalent, no, not England), and will concede it also does the job. Fortnum and Mason (yes, England) do an Irish Breakfast tea and while they seem to have got the general idea they haven't quite got the spirit of it.
(Okay, will admit to drinking Thompson's signature or Irish breakfast in preference, as more than one cup of punjana has a bizarre drying effect, I swear it's got stronger.)
Re: There is a proper way to do most things
There is only one way .... and that is the [1]standard way!
---------> Just nipping out for a hot one! What do you mean, I'm [2]not allowed ?!?
[1] https://en.wikipedia.org/wiki/ISO_3103
[2] https://www.bbc.co.uk/news/articles/cdey4zzl529o
Re: There is a proper way to do most things
> "feel free to brew me a mug of strong Earl Grey with half a teaspoon of sugar and full cream milk"
It's just trolling. Ignore it
Great story and it also shows why all modern servers should be running with UPS. If that had been case here, the log would have shown that UPS had to fix the power every now and then which is much more easy to diagnose than random crash out of blue.
And it might be onpopular opinion but I actually prefer my Earl Grey tea with nonfat milk. I think using milk with fat hides some of the flavor of the tea. And I use at least one full spoon of sugar, too.
A cup of tea didn't literally save the day, but it did put everyone in the right frame of mind after a major unexpected power outage.
We had time to plan the various steps of the power up - Lots of steps, lots of dependencies. What needed to be checked before we moved onto the next step, etc. All scribbled onto a whiteboard ready for when the power came back. But then we had to wait for an unknown number of hours, knowing that management would be on our backs to get everything back on as soon as possible the moment the lights came back on.
The one key risk that we oldies wanted to avoid was flipping from half-asleep boredom to unruly excitement in a heartbeat when that happened - Too much potential for excitable engineers to power up the wrong thing at the wrong time and set us all back.
So step one on the chart was - "Put the kettle on and have a brew.", and we took that 10 minutes to calmly go over The Plan one more time. One of the best spent 10 minutes of my working life.
Yep, rush at your peril...
Every test plan bar one that I wrote for customer IBM HA/CMP clusters included a "get well away from the keyboard" (go get tea/coffee, walk around the building, whatever) step between test phases. Because without that there was a tendency to try and get through the tests as quickly as possible, which didn't always give the cluster time to settle properly (background processes completing etc.).
The 'bar one' that I didn't write this in for was the one where the customer's team demonstrated what happened if you didn't wait long enough.
I hope he didn't bring his tea into the datacentre!
Also is el Reg doing a site traffic census today. Mentioning tea, coffee, beer or bacon sandwiches is a sure-fire way to bloat the comments!
Re: I hope he didn't bring his tea into the datacentre!
Dip someone mention beer and bacon sandwich’s?
Re: I hope he didn't bring his tea into the datacentre!
I've found you can only get so much beer into the sandwich before structural integrity is compromised.
back in the early days of the UK's national lottery i worked for a company in bracknell that was involved with the networking for this new lottery; their datacenter was a glorified portacabin on stilts in the warehouse
Everything worked well but i remember shortly before i left that we needed to add another power circuit for a digital alpha machine.
I was down there with the sparky when he opened the large electrical distribution box at the back of the cabin; sitting in the bottom was a mug of very old, mostly dried out coffee
"i always wondered where that mug went" was all he said, and then simply carried on adding the new wiring.
needless to say i checked that distro box before it was sealed back up...
Tea in the machine room
Yonks ago I worked on an IBM mainframe installation in Kuwait. At that time (early 80s) the economy of Kuwait was founded, not on oil as many mistakenly believed, but on car crashes, photography and sticky green tea. Whenever we had a visit from an IBM hardware engineer the staff would assiduously bring him his sticky green tea in a small glass which he slammed down on top of whatever bit of kit he was working on. Having come from a very controlled data centre in west London I found this shocking and scary but I have to admit that to my knowledge it never caused a problem. At least he went out of the machine room to smoke.
I'm alright, Jack.
I've mentioned this before, but when our industrial estate suffered a general power-cut, the only things that kept operating from the UPS were our book-keeper's laptop, phone-charger, radio, desk lamp and foot-warmer. She alone was bathed in light, comforted by Radio 2...... Not for long, the UPS was over-loaded.
Her 'office' was in the same room as the server and she had 'modified' the electrical connections to suit her personal arrangements.
Ye olde country house
Many years back I was working in a lab contained in an outbuilding of an old country house, as seemed to be the fashion among Brit tech companies back then. On arrival I was shown the 3COM file server (I said it was a while back) with attached UPS. The team had discovered that putting the kettle on over-stressed the feed to the outbuilding. Likewise hoovers etc.
Of course tea makes things work better
Especially when used as a strong Brownian-motion generator with an atomic vector plotter suspended in it. Fire up the old Bambleweeny 57 Submeson brain!
The one with the HHGTTG radio play cassette tapes in the pocket please.
Re: Of course tea makes things work better
Ah, so that is where my tapes went
:)
Recently we suffered a power failure to one of our sites.
Turned out the installed battery and inverter was powered by a simple two-pin plug, and not a proper high-amperage plug.
Of course it melted and burnt a wee bit :)
I miss PhotonicInduction, he should come back to YT and do some proper entertainment.
When everything has gone to the hot place in a handcart...
A deep breath, the least repellent cup, mug, vase left unwashed in the tea room sink with hot black coffee (the buggers never replace the milk, and the 43 rat dropping brand really needs milk) and popping outside for a cancer stick with the ersatz coffee cooling gave one the chance to carefully consider the various options available.
I wonder whether over time the frequency of serious IT cock-ups is inversely related to the incidence of lung cancer?
After DNS it is almost always the power supply if you include wiring, leads, plugs, socket, demented UPSs and the very odd* gormless sparky. ;)
* as in rare. Lacking gorm in a sparky is longevity limiting.
There is a proper way to do most things
"feel free to brew me a mug of strong Earl Grey with half a teaspoon of sugar and full cream milk"
What the Hell!!!!!
Even using a mug is just plain wrong. However, if you've found something that works for you, then so be it.