Server broke because it was invisibly designed to break
- Reference: 1671175754
- News link: https://www.theregister.co.uk/2022/12/16/on_call/
- Source link:
This week, meet a chap we'll Regomize as "Kris" who arrived at his desk one Monday morning to find the phone ringing off the hook, a full voicemail inbox, and his pager pinging with urgent messages from his manager.
The cause of all that urgency was a dead server that powered an important application.
[1]
Kris quickly inspected the machine, which was connected to power and had working UPSes. No tell-tale smell suggested something inside had expired. Turning everything and anything off and on again had no effect.
[2]
[3]
“The unit was dead as a doornail, as they say,” Kris wrote.
The only thing to do was therefore to call the company's service provider for help.
[4]
"This was one of those situations where I really didn't want to deal with them because in most instances by the time they sent over a tech, the problem was already resolved either by the user or by myself,” Kris told us. But at this stage a second opinion was the only option.
"About five hours later the tech shows up smelling as if he lived in his car and slept in his ashtray," Kris told to On-Call. Within five minutes he'd diagnosed the server as broken and new power supplies as the fix.
That hardware arrived two days later and – after Kris and the techie grunted and hefted the server to get it installed – failed to fix the dead box.
[5]Boss installed software from behind the Iron Curtain, techies ended up Putin things back together
[6]Two signs in the comms cabinet said 'Do not unplug'. Guess what happened
[7]Go ahead, be rude. You don't know it now, but it will cost you $350,000
[8]No, I will not pay the bill. Why? Because we pay you to fix things, not break them
Kris confessed to The Register that part of him didn't mind this at all, because seeing the smug techie brought down a peg was quite fun.
However by this time – three days into an incident that deprived the company of an important application – Kris was under more than a little pressure.
[9]
So he concurred with the visiting tech's blankly amused observation that the server remained broken and that the motherboard must be the real problem.
Three days later a new motherboard arrived and – after more lifting and sweat – did nothing to restore the server to working order.
But all that work did set Kris on the way to finding a fix, because handling the case set in train another thought: were the interlock switches working?
Interlock switches, for the uninitiated, are safety mechanisms that stop current flowing when cases are open. Which is a fine idea because nobody should be electrocuted while working on a server.
It transpired that one of the switches had broken on this server, but the fault was invisible and undetectable.
"I'd never opened up the server in my time at this job," Kris said. "I made a quick trip over to R&D and one of the engineers pulled a similar switch out of their parts and gave one to me. The tech wired in a new switch and we were good to go."
This incident is not a typical On-Call triumph.
That came – or rather, didn't come – in the following weeks and months, when Kris's inbox never contained an invoice for the attempted fix by his external service provider. So while the incident was unpleasant, at least it didn't cost Kris's employer a cent!
On-Call will run in its usual Friday timeslot on the 23rd, then offer some seasonal specials. So if you have any stories like the one from Kris, or tales of holiday time tech support [10]click here to send On-Call an email . ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y5xPz1AYnV7Ie4i7EHl@CgAAAME&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y5xPz1AYnV7Ie4i7EHl@CgAAAME&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y5xPz1AYnV7Ie4i7EHl@CgAAAME&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y5xPz1AYnV7Ie4i7EHl@CgAAAME&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2022/12/09/on_call/
[6] https://www.theregister.com/2022/12/02/on_call/
[7] https://www.theregister.com/2022/11/11/on_call/
[8] https://www.theregister.com/2022/10/28/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y5xPz1AYnV7Ie4i7EHl@CgAAAME&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
A service provider that doesn't bill because their attempted fixes failed?
Surely this is a service provider which no longer exists, because the ones that last are the ones that nickel and dime you for everything. Even when they are wasting your time with failed fixes, they are billing you $300/hr for that service if it isn't covered under the normal contract/warranty.
At best if a replacement part doesn't fix the issue they won't charge you for the part, though I've seen them insist on putting the original part back to avoid such a charge!
I recall at one consulting gig the company I was working for had a few obsolete Clariion arrays that were under T&M support. I'm not sure of the details, but apparently IT typically did internal billing at a set price per gigabyte to the various departments, but these arrays were legacy arrays that had come when they acquired another company and the manager of that unit didn't have to pay the internal billing so long as they kept using the arrays they "owned".
So something went wrong and the IT support people called EMC, who sent out someone who said he hadn't been trained on hardware that old so he said they'd send someone else. Then he came back the next day because they didn't have anyone in the local field office trained on such outdated hardware! So he muddled along, running up huge charges, and finally and belated fixed the issue. IT of course made the "owner" of that array pay that hefty bill, and it wasn't long before those legacy arrays were gone from the floor! I imagine one big bill can wipe out months or even years of "savings" on that per gigabyte charge.
If EMC didn't charge for all the failed attempts at fixing it, I'm sure that bill would have been a lot smaller. But T&M is T&M, and EMC is legendary for making you pay every penny of that charge when you are not on a support contract!
Re: A service provider that doesn't bill because their attempted fixes failed?
Not hardware but...
When I took over my current role one of the things I inherited was a non-working, but critical, VAT report which was preventing sign-off of a major project.
The consultant asked if they could bring a trainee with them, which was fine by us, we're always happy to support youngsters and their development. After a day on site the report still didn't work correctly and was still preventing sign-off, yet they billed us for 2 people's time for a full day! I passed it back to the finance people who'd arranged all of this (before I started) stating, in polite language, where I felt the provider could stick their invoice but I suspect it was just paid.
In the mean time I'd fixed the report anyway - seeing *0.15 in a VAT report was a bit of a giveaway, especially as the rate had changed recently. Why the **** of a consultant had hard-coded the calculation when the database had a TAX field with the correct value is a mystery. It was never going to work anyway even if the rate had been correct as some items, e.g. fuels, carried a different rate.
Re: A service provider that doesn't bill because their attempted fixes failed?
”Surely this is a service provider which no longer exists”
My thought were that this story is improbable, if a service provide doesn’t even try to bill. Maybe there were some pre-emptive yelling by the company’s directors?
The article says: ”at least it didn't cost Kris's employer a cent!” but the server run a piece of important software. The incident probably cost the company quite a lot of money.
Re: A service provider that doesn't bill because their attempted fixes failed?
Depends on the size of the client. Some service providers won't bill if they know they've effed up and they know the client knows that. They'd rather take that small hit than risk losing someone who will definitely pay them more in the future,.
It then occured to me ....
.... that this was an open and shut case!
-------> Never had an open and shut case!
Re: It then occured to me ....
You don't usually get quantum superposition on an object this large. I'm sure there should be a Nobel in that.
> Which is a fine idea because nobody should be electrocuted while working on a server.
'Elf and safety gone mad
“nobody should be electrocuted while working on a server”
Where has all the fun in the world gone?
Where has all the fun in the world gone?
It got electrocuted?
It has been sucked out of the world by a bunch of vampires that were rejected by HMRC for being 'too nasty and petty to be tax employees'!
-------> The AVERAGE H&S employee!
It died in a freak HVAC accident.
Surely the ones working on the server should be left alone, and the electricity supplied to all the shoulder-surfers and manglement interrupting to ask why it's not fixed yet?
It is a fine idea
Having walked in on someone doing the 50 Hz boogie, fortunately with no long term impact on the 220 V AC disco dancer, it's not a fun experience for anyone involved.
He (and I) were fortunate that there was a wooden-handled broom near to hand to allow me to nudge him free.
Re: It is a fine idea
I still have the scar on my thumb that resulted from an inadvertent connection to the national grid. I'm just glad my Mum hadn't spotted me on the way into the house and demanded that I remove my wellies! If I hadn't been so well insulated then my feet (along with the rest of me) would probably be six feet under at the moment!
Re: It is a fine idea
At the age of 9 I was flung across a room when my Dad asked me "just hold these multimeter probes in this socket while I figure out which breaker I need to switch off".
Some years later, while changing a light fitting with the power still on, I connected live to earth via my thumb. No shocks, but the smell of cooking meat. I had a small area of my thumb that was completely dead for several years.
I learned. Eventually....
had a printer with the same fault
I had a old dot matrix printer that wouldn't print. The interlock switch (just a leaf spring pair of contacts) thought the lid was always open. Fix was to bend the contacts a little.
Re: Fix was to bend the contacts a little.
I've got a dehumidifier whose "full bucket" sensor is like that. Every now and again I have to bend the contact strip out a bit, and the fault is resolved. Eventually it'll just go "snap", and that'll be it, I suppose, but it's done quite well...
Safety interlocks and fault protection
I had many call-outs to an FM transmitter that was extremely reliable apart from safety interlocks and the airflow detector for the cooling fan, one of which had a tendency to switch the whole thing off at a very unsociable hour. So now I tend to think of interlocks first in any totally-off situation.
The Interlock and Health and Safety
We supplied some equipment to a well-known process contractor in the food industry and their procedure was that all MCC's had to be commissioned by their own staff 'for safety reasons'. No problem, I had a trainee with me and it would be good to see someone else doing the job "with the highest regard for safety of personnel and equipment". We had to undergo a two-hour H&S 'induction' to cover the general, local and specific hazards, which included not carrying two cups of coffee and never carry a tray on stairs etc. as well as the usual hazards of electricity and the like. We were examined on the knowledge so transferred.
Eventually, suited, booted, gloved and glasses, we meet the 'Appointed Person' who would be doing the hands-on pre-checks etc. before starting the plant and after further words about safety he proceeds to open all the panel doors, then switch on the isolating circuit breakers using a spanner....... My trainee and I, stood well back...... Of course, nothing worked and after a frustrating half-hour or so where the expert would take no advice, I tentatively suggested that the door-switches might be causing, correctly, an interrupt in the circuit? He proceeded to by-pass the switch whereupon the breakers slammed in and the interior became live. It was mind-bogglingly dangerous. We couldn't work or even observe under these conditions so we went back to the project manager who sighed and said they had 'a bit of a problem' with this guy.
We returned the next week to repeat the exercise, albeit with a reduced H&S Induction, this time successfully, with a different 'expert'. I still have the first guy's name in case our paths ever cross again.
guesswork
the server remained broken and that the motherboard must be the real problem.
So just take another guess then? and wait three days for parts to find out? (leaving customer without their "critical application" )
This is the difference between "Part Fitter" and "Engineer" , whatever type of hardware is involved.
Car dealerships often do this and charge you truly eyewatering prices for each wrong guess - despite being in a far better position than anyone else to do the job properly due to having all the knowledge and proprietary diagnostic equipment on hand.
Shoulda got a ....
[1]Magic switch then!
------->abra.cud@bra
[1] http://www.catb.org/jargon/html/magic-story.html