News: 1769758215

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

In-house techies fixed faults before outsourced help even noticed they'd happened

(2026/01/30)


On Call Welcome to another instalment of On Call, The Register 's weekly reader-contributed column that shares your stories of weird and wonderful tech support jobs.

This week, meet a reader we'll Regomize as "Callum" who sent us a story from his time working for a company in the north of England that ran an 8-CPU Sun server which used pairs of processors stored on removable cards.

"It was running the Oracle database that underpinned their sales platform, and was considered sufficiently critical that there was a five-figure leased-line connection to the support vendor who 'constantly monitored' the server for issues and was paid to send an engineer within an hour," Callum explained. A nearby cache of spare parts meant that replacement hardware would usually arrive before the support tech!

[1]

This arrangement was sensible because the server was old and occasionally unreliable.

[2]

[3]

"We would have issues from time to time, generally CPU cards having tantrums," Callum wrote. "But no problem, that's what the support contract was for!"

The arrangement also had two flaws.

[4]

One was that the designated contracted on call support lived sufficiently far away that the slightest bit of bad weather – and there's plenty of that in the north – meant road conditions became so bad he could not safely arrive within an hour as required.

The second was that the monitoring system wasn't very good at noticing when the servers went down but was excellent at detecting startups.

[5]Engineer used welding shop air hose to 'clean' PCs – hilarity did not ensue

[6]Help desk read irrelevant script, so techies found and fixed their own problem

[7]User found two reasons – both of them wrong – to dispute tech support's diagnosis

[8]User insisted their screen was blank, until admitting it wasn't

Callum told us those quirks meant incidents usually unfolded as follows:

A CPU card would experience a fault;

The server's OS responded to losing a quarter of its CPUs by rebooting;

The server would not reboot, because one of its CPU cards was broken;

The contact center would complain to IT;

Callum, or whichever other IT worker was on call, would drive in to remove the faulty card and reboot the server;

The server would resume operations;

The support contractor would call to report a server outage and promise to send someone within an hour.

After this sequence played out two or three times, Callum said they told the contracted IT support guy not to risk rushing in to fix the server.

"Most of the time we could get the box up and running ourselves, and in any case the world wasn't going to end if we couldn't sell stuff for a couple of hours," Callum wrote. "So we told him not to risk killing himself and/or others by attempting to get to us in an hour."

The support vendor came to the party, too, by waiving the cost of the leased line for two years in recognition that it wasn't doing its job!

[9]

Have you outpaced outsourced help? If so, [10]hurry up and click here to send The Register an email so we can become your third-party narrator in a future edition of On Call! ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aXyPVaCBdMEen3oeUohl4AAAARM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aXyPVaCBdMEen3oeUohl4AAAARM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aXyPVaCBdMEen3oeUohl4AAAARM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aXyPVaCBdMEen3oeUohl4AAAARM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2026/01/16/on_call/

[6] https://www.theregister.com/2026/01/09/on_call/

[7] https://www.theregister.com/2025/12/19/on_call/

[8] https://www.theregister.com/2025/12/12/on_call/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aXyPVaCBdMEen3oeUohl4AAAARM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/



Korev

One was that the designated contracted on call support lived sufficiently far away that the slightest bit of bad weather – and there's plenty of that in the north – meant road conditions became so bad he could not safely arrive within an hour as required.

What happened if said engineer was at another customer's site?

Analog mobile phones

Caver_Dave

A long time ago I lived in an area of no analog mobile phone coverage (and still no digital, although I can physically see a mast without using binoculars), but for one week in the month I had to have the out-of-hours support phone and be on-call for the organisation that did much of its warehousing and distribution work over-night.

I informed the person who was imposing this rule that there was no mobile signal and that I would never receive the call, and they just shrugged. So, I accepted the phone (no extra payment unless called into the office) and never had a call in the 2.5 years before I left.

Re: Analog mobile phones

Admiral Grace Hopper

On call with a pager in the 80s, one of the most important documents held by the team was the list of nightclubs with no reception for the pager.

Re: Analog mobile phones

Jou (Mxyzptlk)

> and never had a call in the 2.5 years before I left.

You'd need more upvotes.

Question: Was it ever noticed that you never had a call, for whatever reason?

I.e.: Nobody called you vs. nobody reached you vs. nobody reached you and nobody noticed vs. nobody reached you and nobody noticed 'case nobody gave a sh?

Almost believable

matjaggard

I was with you right up the the contractor waiving a fee - took the story from "incredibly likely" to "apocryphal".

GlenP

What happened if said engineer was at another customer's site?

The perennial problem with such contracts, especially away from major urban areas.

Not mine but one from the days when my brother was a manager at an IT Support Provider who had little presence in the north of England and none in Scotland:

Sales - We've just won a major support contract!

Engineering - Oh yes, where is it?

Sales - Aberdeen!

Engineering - What's the response time?

Sales - A guaranteed two hours!

Engineering - You do realise our nearest support engineer is based about 4 hours away, and the next one nearer 6? We could barely promise next working day.

Too often engineering would get ignored and told to just get on with it but I believe in this case the contract was refused.

Michael H.F. Wilkinson

This reminds me of the notion of an "appeasement engineer" (someone with an engineering diploma so fresh the ink is only just dry), in what I believe is [1]the very first BOFH episode (certainly the first I read).

[1] https://bofh.bjash.com/bofh/genesis1.html

"Well, as you're on site...."

Sam not the Viking

The first company I worked for made a variety of technical products, with increasingly complex control systems as the world progressed towards electronics. After completing training, I was sent to sites to get new machinery/controls up and running and handed over to the customer/end user. More than occasionally, I was called to 'have a look at' some other piece of equipment which was 'giving a bit of trouble' which was not in our supply and sometimes had been installed a long time ago. Often the problems were simple and my (excellent) training covered the general principles behind the processes involved. Being helpful was good customer-relations and considering that these customers were responsible for future orders we assisted where reasonable.

I drew the line at one site where the contractor asked me to set up a large number of automatic actuators; I showed him how it was done and left him to it..... whereupon he called in the manufacturer, at some cost.

Normal behaviour

Denarius

In-house usually know about issues and often have fixes close to hand. Its the Outsaucery that has its own "process" that buggers things up

Re: Normal behaviour

BartyFartsLast

As I always point out, you get the level of support you pay for.

It is on us to get the contracts right and hold their feet to the fire if they don't live up to them..

Flightmode

At a previous job, we had a series of routers - Cisco GSR 12012 - break down on us in pretty rapid succession. It was deemed this was due to a poorly designed backplane that would bend out of shape due to the warm air circulating around in the metal box, which would lead to poor or intermittent connectivity to the inserted line cards. Cisco accepted that the fault was in their construction, and when we asked to have the troublesome boxes replaced on our 4 hour Advance Replacement contract, they said that "Sorry, we only offer 7 business days replacement on the chassis". Which was of course met with severe skepticism from those that negotiated our support contracts and knew exactly how much we were paying per annum for 4-hour replacement of the chassis.

As a compromise (since they didn't have any of the big metal boxes in our nearby depots) they eventually renegotiated our support contract, waived a bunch of fees and send us a stack of backplanes to replace the faulty ones. A colleague of mine replaced one and came back exasperated to the office - "You know how many screws there are in one of those?! I counted 136 before I gave up!" - so Cisco eventually agreed to send an engineer to do the remaining replacements for us, on their dime.

Christoph

Way back in the 60s the keyholder for a particular device lived well out of town and had no car and no telephone.

The device was the siren to give four minutes warning of a nuclear attack.

Prst. V.Jeltz

"five-figure leased-line connection to the support vendor"

so minimum £10,000 ( pa presumably)

Thats amazing , times have changed eh ?

and thats just for the support vendor to ping it every 5 mins? not the main architecture ?

It doesent say when this is , but "sun server" gives an indication , thats a probably a years salary

"Few sins are less forgivable in polite society than offering poor
people products they actively seek."
-- Katherine Mangu-Ward, "Education for Profit", Reason Magazine, July 2008