News: 1641544505

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Nothing's working, and I've checked everything, so it must be YOUR fault

(2022/01/07)


On Call The customer is always right. Except when they're not. Here we have a story from the [1]On Call archives concerning connectors, telephones, and a user blessed with a little too much confidence.

Today's tale takes us back to the 1990s and comes from a former employee of a now-defunct telecom equipment vendor. We'll call him Felix, for that is not his name.

"I got to do a lot of on-site troubleshooting for the specific subsystem of the telecom kit that I had expertise in," Felix explained. One of the UK's telephone operators of the era was going through a period of modernisation and replacing decade-old kit with something decidedly more modern.

[2]

The modern devices were considerably faster and less power hungry than the elderly hardware. However, the rollout also necessitated some serious reconfiguration of the older gear. "A particular blade in the old kit," Felix told us, "could be made to operate in either an active or passive state depending on the orientation of a plug-in connector block.

[3]

[4]

"To change the state all you had to do was pull the blade, change the connector block orientation and plug the blade back in."

The hardware was pretty fundamental when it came to keeping the UK PSTN network going and a failure could mean that some end users might find themselves with dead landline phones. This being the era before the dominance of the mobile phone, "not having landline service was a VeryBadThing™," said Felix.

[5]

The call came in at 4:30pm on Friday, because they always do. It came directly from the tech lead of the operator's modernisation programme. Felix recalled the very much one-way conversation.

"It's all gone TITSUP!" Felix recalled him screaming down the phone, "Location X is failing to sync with the rest of the network, I've checked all the hardware and the config and it should work. But it's not!"

All we know about Location X was that it was a very, very important customer and it being down was costing the telephone operator an unthinkable sum per hour in terms of lost revenue as well as reputation.

[6]

Felix adopted the diligent approach familiar to all those who take such calls: "Has anyone changed anything? Are you absolutely sure the config is correct?"

"Of course it is!" bellowed the caller, "and we need someone down here to fix it ASAP!"

"Of course they did," sighed Felix.

[7]You've stolen the antiglare shield on that monitor you've fixed – they say the screen is completely unreadable now

[8]Who you gonna call? Premium numbers, but a not-so-premium service

[9]The Filth Filter is part of the chipset, honest. Goes between the TPM and SEP. No, really

[10]What came first? The chicken, the egg, or the bodge to make everything work?

Being possessed of the shortest of straws, it fell to Felix to source a company car and all the equipment required to diagnose what had befallen the gear before undertaking the 250-mile (402km) trip to site.

"By the time I was ready to start work it must've been close to midnight," he said.

But at least he was there.

He checked the clock signals coming out of the kit and confirmed that, yes, it was definitely not synching with the rest of the network. Next, he looked at the hardware. Out came the blade and…

"Lo and behold the connector block was in the 'passive' orientation."

"If ever I've seen the face of someone who wanted the ground to swallow him up," said Felix, "Mr Tech Lead was him."

Wordlessly, Felix moved the connector block to the correct orientation and replaced the blade. Everything sprang into life. There had been no need for all the expensive test equipment nor, to be honest, his presence. He hadn't even had time to book a hotel in the panic to get to site.

"So," he surmised, "basically I travelled 250 miles on a Friday evening, with a car full of tens of thousands of pounds worth of test gear, under threat of 'the world is ending' only to unplug a plug-in connector, turn it through 180 degrees and plug it back in again...

"Back then I could claim call-out and overtime, though, so at least it didn't seem quite so bad."

As for who had done the deed, Felix reckoned it was more likely a local tech being helpful. It wasn't until the upgrade programme had got under way and other blades had been changed out that the problem reared its head.

Still, we've all had that call from the customer who was convinced everything was correct until a quick eyeball confirmed it most definitely wasn't. Did your connector cockup involve a blue flash or a red face? Tell all with an email to [11]On Call . ®

Get our [12]Tech Resources



[1] https://www.theregister.com/Tag/on-call

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YdgdXQV9yL6ScFhZQL33FAAAAAk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdgdXQV9yL6ScFhZQL33FAAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdgdXQV9yL6ScFhZQL33FAAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YdgdXQV9yL6ScFhZQL33FAAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YdgdXQV9yL6ScFhZQL33FAAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[7] https://www.theregister.com/2021/12/31/on_call/

[8] https://www.theregister.com/2021/12/24/on_call/

[9] https://www.theregister.com/2021/12/17/on_call/

[10] https://www.theregister.com/2021/12/10/on_call/

[11] mailto:oncall@theregister.com

[12] https://whitepapers.theregister.com/



Bit of a twist to the ending

phy445

See title

Re: Bit of a twist to the ending

Korev

They should have thought about Occam's razor whilst looking at these blades...

Blue flash

Mishak

That would be the time an engineer moved a 1000A, 80V PSU (about 1m x 1m x 2.5m) without unplugging it, which ripped the three-phase socket off the wall, causing the phases to short together. Luckily, the protection for the circuit kept the bang reasonably local.

Another time, the same guy was having trouble getting some thermocouples to stay in place on a power board (open) he was working on (it was connected to one of above PSUs, but it was only set to 48v). He decided that a large lump of solid brass would do the job, so put it on top. The PCB was a charred, smoking mess by the time the block had stopped jumping in the air and throwing molten metal in all directions...

Re: Blue flash

My-Handle

Sounds like one of "those guys". The kind whom disaster stalks daily.

I once knew a farm-hand / grounds-keeper at a large stables who, among other things, managed to tip a tractor twice within as many months. No-one else had managed it in the memory of those currently working there, and he did it two completely different ways. He was a nice guy, eager and hard-working... but his decision-making left a little to be desired.

Re: The kind whom disaster stalks daily

Pascal Monett

Seems like disaster hardly needed to stalk him, more like just look around and, "ah, there he is".

Moving an industrial electrical equipment without shutting it down and unplugging it first ? Where did he learn the job, at McDonalds ?

Printer Cartridge

Wally Dug

Not quite the same level of importance obviously, but my father-in-law swore blind several times that he had removed the orange sticky label thing from the head of the new Canon printer ink cartridge, yet it was still not printing.

My round trip to remove said orange sticky label thing was only about 50 miles and it was during the day, but at least he had the good grace to be completely and utterly embarrassed. And, yes, we still bring it up every now and again!

Re: Printer Cartridge

James Wilson

My father asked for help as the stupid new cartridge he'd put in his printer wouldn't come out at all. I eventually managed to extract the black cartridge from the colour slot. In his defence the cartridges on that printer are the same size, though slightly more damning is that he had needed to force it past the plastic part that is there to stop you putting one in the wrong slot.

Humble

Archivist

Over the years I've usually been the one doing the fixing, but on those occasions I've asked for assistance, I've learned that it pays to be humble and admit up front that it might be something I've done wrong. That way I feel less of an idiot!

Re: Humble

Sam not the Viking

And if you can ease the embarrassment of a colleague, boss or customer, you'll never be forgotten.

Several times...

DailyLlama

The worst was a 400 mile round trip from Uxbridge to Newton Abbot to restart a server after someone swore they'd done it, only to find out they'd turned the monitor off and on again.

Re: Several times...

Tim 11

reminds me of a story from a friend of mine. His dad had some problem with the computer that couldn't be resolved over the phone so friend asked the dad to bring the computer over next time he visited.

Dad turned up carrying just the monitor which he assumed, since that was by far the biggest component, that was where all the gubbins was.

Re: Several times...

Prst. V.Jeltz

You can tell when a user is doing that if they're doing it while you're on the phone by the speed.

If they are just claiming they did it earlier , and it not pinging etc ... decision time

Re: Several times...

Sandtitz

About 15 years ago I was on-call and client calls from a middle-of-nowhere "highway oasis": one restaurant computer isn't connecting. Client declines to troubleshoot beyond rebooting and asks for onsite service (per agreement).

After driving the company van for 6 hours and 500 km I arrive only to immediately discover that someone had unplugged one end of a patch cable in the comms room cabinet. Re-connected, checked the operation and explained the problem to the manager. 15 minutes and off I go for another 6 hour trip back to HQ.

A tiring day, but at least writing the bill gave some satisfaction.

---

On another occasion I was to install the computer equipment at a restaurant even further away. Waking up before 5:00 for early morning flight, and arriving five hours later I can see (and I'm told) that the builders are about a month behind schedule with missing walls/electricity/network/everything. Can't do anything so back to the hotel I had booked, spend the day and take the first available flight back.

Apparently the construction company had communicated the change of schedule several months earlier and the news had reached my inept manager who kept it to himself and tried to blame me afterwards. Feckin eejit!

Too Many...

GlenP

I've probably recounted these before but they're from back when I worked for a company doing support for the consultants at a major insurance company.

There was the user who swore blind she'd checked the power lead, including unplugging it and reconnecting it so we called an engineer, her boss swore when he received the call-out invoice for "plugging in power cable!"

Then there was the one that insisted the power to her computer hadn't gone off (we always knew when it had, due to the corruption of the database). An hour later she phoned back and apologised, her consultant boss had unplugged the computer for some reason then plugged it back in, restarted it and not told her.

After nearly 37 years in the industry I've learnt things rarely go wrong for no reason, it's nearly always something that's been changed, and I'll confess on some occasions it's been me that's made the change.

I think that step is optional

ColinPa

I was on a help desk working with a customer who was having problems getting some software configured. The problem had been going on for about a week. In the end I went back to the beginning and the conversation went like

Me: Do step 1 and send me the output

Him: Here you are

Me: Do step 2 and send me the output

Him: Here you are

Me: Do step 3 and send me the output

Him:I don't think we need to do that, as it has no effect.

Me: Please do step 3 and send me the output

Him:Ok then but you are wasting your time, just wait and see

...

Me: Now start it

Him:It works!

A month later I met the guys boss at a conference and we had a conversation. The manager said the guy was often like that - not following instructions, and then it not working, and blaming it on the products. I wrote an email for the manager describing the situation - and an estimate of the costs of my time etc. I believe the managed had a career changing conversation with the employee, because we never heard from the employee again.

Prst. V.Jeltz

To be fair it sounds like an easy , bloody obvious thing ,but is also something you'd never guess over the phone because its ... too obvious / simple / easy

Maybe if its a "user" you'd check there wasnt a powercut in progress , but this was some kind of "tech lead ".

Maybe I'll get replies along the lines of "never underestimate idiocy " etc

But you simply cannot treat all comers like morons all the time .

There is a science to asking idiot proof questions without embarrassing the customer / higher up

Sometimes you just have to "assume" some things as a given

This will doubtless lead to a very specific reply:

“assumption is the mother of all fuckups"

well let me say in advance "fuck off 50/50 hindsight wielding bastards"

Phone problems

Giles C

User stating in a hotel in the USA in the early 2000s (we were uk) rings up adamant that the modem in the laptop isn’t working.

Talk him through software checks (convinced he is right) but no dial tone.

After a few minutes someone asks how is he speaking to us - at which point he says on the hotel room phone line - then the penny drops as he realises there is only 1 phone in the room………

The louder they yell.....

Coastal cutie

I've found that the likelihood of the problem being with something the user has done/not done/not checked even though they say they have is directly proportional to the volume with which they are yelling at you that "OF COURSE I'VE DONE THAT/NOT DONE THAT/CHECKED THAT, I'M NOT SOME SORT OF IDIOT" (excuse the shouting).

What garlic is to food, insanity is to art.