On Call 2024 has commenced, but in today's edition of On Call – The Register ’s reader-contributed tales of tech support strife – a reader we'll Regomize as "Stuart" shared a tale caused by a temporal anomaly.

"About two years back I was working for an online trading company as IT support manager and was on call for the week," Stuart told On Call.

At 8:00PM one evening, the dreaded support line rang with news that a price information feed was down. That sort of outage meant traders couldn't make money, so Stuart rushed to his laptop and tested the connection to the price feed provider.

[1]

It was down, and no amount of troubleshooting could reveal the cause.

[2]

[3]

Stuart described the source of the price feed as "a really big name in the financial trading world" – the sort of outfit that prides itself on resilience and therefore operated a support service of its own that was also on call at all hours.

So Stuart called, and quickly learned that his employer had once subscribed to a private feed – but a couple of years previously had asked for it to be terminated. It had been left in place for reasons nobody could recall.

[4]

Which was no use to Stuart, who rightly demanded a fix – fast.

Thankfully there was good news: a public feed of the same information was available, and all Stuart needed to do was point his server at the IP address listed in a manual.

Which didn't work.

[5]

At this point, the feed provider escalated to its second-level support team which – after failing to diagnose the issue, referring to the manual and doing all the stuff Stuart had already tried and shown did not work – escalated to third-level support.

At this point, the clock had ticked well past midnight and nobody was their best self, nor enjoying things one jot.

[6]People power made payroll support in putrid places prodigiously perilous

[7]Superuser mostly helped IT, until a BSOD saw him invent a farcical fix

[8]'The computer was sitting in a puddle of mud, with water up to the motherboard'

[9]You don't get what you don't pay for, but nobody is paid enough to be abused

Stuart was particularly unhappy as at this point traders had been hobbled for several hours, and he was beginning to imagine how he would explain the outage to bosses the next day.

Third-level support then asked a crucial question: Which version of the manual was he looking at?

Suffice to say it was an old one that contained an incorrect IP address. And the storied org that provided the price feed had been shown to be shabby at documenting its own affairs.

Stuart eventually learned the cause of the incident: the public price feed was just a little cheaper than a private feed, so his employer shifted and saved a few bucks. But nobody had bothered to finish the job. For years.

So when the crunch came, only L3 support was able to do the job.

On Call is pretty sure this is not best practice IT management!

Stuart escaped the incident without censure.

"We got half a day off with no questions asked," he told On Call.

Has tech support made it hard for you to deliver tech support? If so, [10]click here to send On Call an email and we'll try to tell your tale in a future column. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZZfhVs0SVtuT7XcQwnVCtAAAAQk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZZfhVs0SVtuT7XcQwnVCtAAAAQk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZZfhVs0SVtuT7XcQwnVCtAAAAQk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZZfhVs0SVtuT7XcQwnVCtAAAAQk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZZfhVs0SVtuT7XcQwnVCtAAAAQk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2024/01/01/on_call/

[7] https://www.theregister.com/2023/12/22/on_call/

[8] https://www.theregister.com/2023/12/21/on_call/

[9] https://www.theregister.com/2023/12/15/on_call/

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/