News: 1642149005

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Software guy smashes through the Somebody Else's Problem field to save the day

(2022/01/14)


On Call A warning from the past in today's [1]On Call . Helpfulness is not always rewarded with a pat on the back and a slap-up meal on expenses. Sometimes the Somebody Else's Problem field* is best left alone.

Our tale comes from a reader Regomised as Derek and concerns his time working for a multinational with plants at multiple locations in the UK. He was part of the application support team and toiled under a 24/7 call-out arrangement. A cry for help had to be answered within the hour.

One of the plants would start production at 0400 on Monday mornings, but started work four hours earlier to make sure things were up to speed. It had two main buildings. One was an office unit, housing the comms and server rooms. The other had an equipment room, with switches and patch panels as well as an operations room with client PCs and expensively large monitors.

[2]

The two buildings were connected by a fibre-optic cable that ran in a culvert beneath the yard, through which large trucks (Derek told us they could weigh as much as 38 tonnes) would rumble.

[3]

[4]

"The plant required the applications to be running," Derek told us. "No manual overrides so no system = no production.

You can probably see where this is going.

[5]

Sure enough, at 0130 one Monday morning the call came in. Nothing was working.

"I reluctantly dragged myself from my warm bed to drive the 10 miles to the plant," recalled Derek, "to be greeted by the sombre production team."

It wasn't looking good. All the screens were showing "connection error" messages and a simple ping test showed that the network between the two buildings was down. Derek's heart sank as he pondered the cause. The culvert running under the yard – the one over which those heavy trucks passed over and over again – had probably been stoved in, severing the connection.

[6]

Worse, for reasons known only to management, the infrastructure team responsible didn't provide out-of-hours support. Sure – the applications team was always on the end of the phone. The rest? Only 8 to 5.

What to do? At this point Derek could have played the "Not my job" card and left production hanging. Instead, he decided to see if there was anything he could do to help.

He went to the equipment room where, sure enough, the fibre connection light on the switch was dark. "So I followed the cable from the rack," he said, "and found it went to a wall-mounted box where it converted from the patch cable to the main multicore cable that ran under the yard..."

Hang on. One of the connectors looked like it was sitting a bit lower than its partner. Derek gave it a push and twist to seat it properly and hey presto! The plant burst into life.

[7]Nothing's working, and I've checked everything, so it must be YOUR fault

[8]You've stolen the antiglare shield on that monitor you've fixed – they say the screen is completely unreadable now

[9]Who you gonna call? Premium numbers, but a not-so-premium service

[10]The Filth Filter is part of the chipset, honest. Goes between the TPM and SEP. No, really

"I suspect the connector had never been fully home since being installed about five years previously," he told us, "it had been hanging, but had finally dropped to the point where the connection broke."

Derek was the hero of the hour, and the recipient of many beer tokens from a grateful production team.

His boss, however, was less than impressed. Not with the slipshod connection, but with Derek's helpful assistance. Instead he found himself receiving a dressing down: "That's not your job!" snapped the boss.

"Such is life," said a now-retired Derek.

Derek's story reminded your correspondent of a fibre connection in a previous life where contractors left a mere 2cm of cable outside of the shielding. 2cm that proved all too attractive for sharp rodent teeth. Sadly, fiddling with the connectors didn't resolve the rat-induced hardware trauma.

But that's another story.

Have you ever stepped out of your comfort zone to fix somebody else's problem? Were you garlanded with praise or berated for your efforts? Let us know, with an email to [11]On Call . ®

*Somebody Else's Problem (SEP) field: documented in Douglas Adams' Life, The Universe and Everything and concerns something being invisible because it is, er, somebody's else's problem. Not to be confused with the Steve Jobs Reality Distortion Field.

Get our [12]Tech Resources



[1] https://www.theregister.com/Tag/on-call

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YeFX3NT2xzNAYNL-P2dGzgAAAIQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YeFX3NT2xzNAYNL-P2dGzgAAAIQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YeFX3NT2xzNAYNL-P2dGzgAAAIQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YeFX3NT2xzNAYNL-P2dGzgAAAIQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YeFX3NT2xzNAYNL-P2dGzgAAAIQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[7] https://www.theregister.com/2022/01/07/on_call/

[8] https://www.theregister.com/2021/12/31/on_call/

[9] https://www.theregister.com/2021/12/24/on_call/

[10] https://www.theregister.com/2021/12/17/on_call/

[11] mailto:oncall@theregister.com

[12] https://whitepapers.theregister.com/



El blissett

Boss is right - by showing there is a business case for 24/7 infrastructure call out support, Derek has destroyed the factory's shaky bottom line and maybe the whole company.... or even worse, denied the Boss's mates in infrastructure a nice call-out fee in a few days' time!

About nice call-out fees

MiguelC

I've told this before, did not get a stern talking but also could not correct the problem.

During the Y2K project I was in, my team was correcting an application consisting of almost a 100 programs chained together (just as COBOL likes) and we were made aware that that particular batch chain would break every single night, meaning an hefty on-call bill. And is had been going like that for years.

So I analysed the failure point and it was seemingly simple to correct - IIRC there were numeric conversions that would fail under certain conditions, conditions that would almost certainly be met once or twice per run.

That correction was deemed out-of-scope by our client rep, who, curiously, was also that application's IT boss.

I recognise the story

Caver_Dave

I was possibly standing outside when the story teller was getting the dressing down (it certainly sounds very similar to the story I could hear through the door!)

I was awaiting my turn for fixing the "application error" by putting paper in the printer - again not our job! I was told the unions would be up in arms, but I heard nothing more.

Re: I recognise the story

Roger Greenwood

I had a bollocking once for changing a 13A plug:- "you're not an electrician, the unions will not be happy if they find out". Ah the 1970s....

Re: I recognise the story

PM from Hell

I was tech support manager for a County Council. When the legislation on working on electrical equipment changed I had to organise a course on how to fit a plug on a mains lead for my comms team. The same team who literally built the x.25 switches for each site at the time from the chassis up :(.

Our site electrician was actually embarrassed to have to do this for us but we needed the sign off of the H&SE ream were going to ban us from working on any electrical equipment. The Desktop team didn't have to have the training as all their mains leads came with molded plugs and were never re-wired.

By the nature of the role members of both teams would regularly be working on hot devices with the case removed when performing testing.

What Derek needed to do...

lglethal

... was point out to his Boss, that he could perform an internal charge at Callout rates (plus plus plus...) to the Infrastructure Team for fixing their Failure. If the Infrastructure Team objected, pointing out to the CEO that had the failure not been corrected when it was, the loss of production would have cost $X (almost certainly multiple factors of 10 higher than the Charge on the Infrastructure Team). Of course the Boss could also then request additional resources for his department as reward for keeping the wheels spinning when other teams where letting the company down...

Solidarity in management only extends to the point where one's own department could get a leg up...

Re: What Derek needed to do...

Fred Daggy

Some days you're the pigeon, some days the statue.

Re: What Derek needed to do...

Anonymous Custard

And good deeds never go unpunished...

Re: What Derek needed to do...

Anonymous Coward

I usually use "some days you're the dog, other days you're the fire hydrant", but I like yours too.

SEP squared

b0llchit

So Derek has managed to shift the SEP field from "cabling is SEP" to "the boss is SEP" .

That is a very commendable way of dealing with a boss.

Dodgy connection

Mark #255

Obviously not in the same league, but I once spent a couple of hours trying to get my laptop plugged into the big meeting room screen. But my HDMI cable seemed dodgy (which was odd, because I'd used it elsewhere without issue). Tried another, and another (by this point I'm "borrowing" them from vacant screens in the office outside). Some just don't work, others drop the connection every few seconds.

Eventually, I realised that the HDMI cable between my laptop and the wall socket was not the only one, and checked behind the screen to find one of the plugs of the other cable wasn't fully seated.

Then I just had to return all the borrowed cables...

Ultimate Jobsworth

Ikoth

Around the turn of the millennium I did a contract stint for a local authority on a project to hook up all their schools to the internet. The job was split into phases – the comms crew would hit site to install the CPE, a week or so later the cabling guys would arrive to CAT5 the building and then my team would turn up to install the (singular) Cisco switch and get some lights blinking. Another crew would follow in our wake to install the (singular) school server and back office PCs.

On one occasion me and my (staff) partner arrived at one site to find that the cabling crew hadn’t plugged network cables into the patch panel in the comms rack, where we were installing the switch. Instead, they’d just left a box of unopened cables in the corner of the room. My sidekick immediately pulled out his phone to talk to his manager about “the problem”. I literally could not believe what I was hearing. I pointed to the box of cables and mimed opening a zip-lock bag, uncoiling the cable and plugging it into the patch panel and switch. Still in deep conversation with the boss, he ignored me. So I went over to the box, picked out a cable and pulled the bag open. At this point he bellowed STOP!!!! at the top of his lungs, muttered something into his phone and thrust it into may hand. At which point I got told in no uncertain terms that I was not, under any circumstances, to plug in the patch cable, as it “was not our job” and that the cabling crew would be scheduled to go back and plug the missing cords into the patch panel.

We then had to explain the problem to the school secretary and apologise that it would be another 4 – 6 weeks before their school would be connected.

I vowed to never take another public sector contract again.

Andy The Hat

Strangely, I've never been able to identify the boundaries of the SEP field and obviously I'd never knowingly circumvent the system ...

I always exist in the IBFI (Its Broken Fix It) aether ... "You're a computer tech - fix my computer chair ...", "You're the technician, fix my glasses", or even "You're the technician you can fix my intimate hair trimmer"

GlenP

Working for smallish companies over the years I've found that. If it has a switch it's probably IT's problem!

As I'm also the person who tends to have hammers, screwdrivers, etc. so minor repairs also fall to IT.

Anonymous Coward

Unfortunately for me, my missus is one of those people who bug me with:

1) You're an engineer. the Internet isn't working. Why can't you fix it?

2) The smart TV isn't working. You're an engineer, why can't you fix it?

3) The washing machine isn't working. You're an engineer, why can't you fix it?

etc etc etc.

The Angel of Competence

dak

https://dilbert.com/search_results?terms=angel%20of%20competence

Pretty much the same

AdamT

Yup. A company I worked for had developed a digital printing system for the foil that goes on the back of tablet/pill blister packs. It had stopped working after the pharmaceutical company it had been supplied to had moved the packaging machine it was attached to and they needed to get it working again. All the people who had worked at our end had moved departments so I - the relatively new guy - got sent to the US with a laptop, the source code and the old compiler that was needed.

Took me a while to hunt it down and, in the end, I had to resort to tracing every single wire from every single sensor and, yes, someone unknown had removed a panel which had a ribbon cable attached. Obviously the ribbon cable was too short, they'd half yanked it out of the IDC clip and then quietly put the panel back. So I carefully removed it, quietly took it to a vice, gave it a nice squeeze and then put it back. Everything now worked.

Not sure what the bill was (consulting rates, flights, hotels, car hire, etc.) but that was probably pretty expensive for just using a vice.

Or to be more accurate "for knowing which bit to squeeze in the vice"...

It's a sad day for this IT rag...

Kane

...when the SEP field requires a footnote to explain it's origins.

Re: It's a sad day for this IT rag...

JeffB

I tried following Hitchhiker's, both the BBC2 series and the film, but found them to be great cures for insomnia, so it has never received more than 10 minutes of my attention.

Re: It's a sad day for this IT rag...

b0llchit

So... your SEP field has been very strong. It works!

Re: It's a sad day for this IT rag...

42656e4d203239

Try the radio transmissions... they were the best. After that the A/V franchise went into a bit of a decline...

Jay 2

Not to the same standard as some of the other comments, but as a humble sys admin (who is part dinosaur/part furniture as I've been around a while) it seems I frequently have to point out to our devs how their applications work when they screw something up in the config. Usually it's copying something from one env to another without actually knowing what all the config does. Sometimes it's complaining that something suddenly doesn't work and it must be a system problem, even though it was working fine then they changed some config/jar or whatever and now it doesn't work. Or asking me why something doesn't work without doing something useful like looking in the application logs where there are obvious errors.

All SEP as in most cases I'm just responsible that there is a system, that Java is available and that there are JVMs for them to put stuff onto. As to what the JVMs actually do, well that's not my sphere of influence...

Stuart Castle

One day, I came into work to find we had no network in part of the building. It was a part of the building without many people in it, so we didn't get any complaints, and I only noticed a problem because I tried to log into a machine in that part. I went into the comms room, and found that the switch connecting that part to the network had hung. Quick as a flash, I thought I'll power it off and on again. So, I did. When it was plugged in again, the switch was still hung. So, I tried again. Still no difference. As it turns out, ​I had pulled the wrong plug.

In my defence, these switches had no mains switch on them, and the cabinet was tightly packed, so it was difficult to reach around the back for the mains connector. I could reach the power strip at the back of the rack though, and traced the power cable to what I thought was the right socket. A difficult job due to a combination of the fact that every mains cable was black cable with an IEC plug on one end, and a standard 3 pin mains plug on the other, and the fact that there was almost no light in the back of the cabinet. There were several identical switches in this cabinet.

Anyway, I digress. I had powered down the wrong switch twice now, so I powered it up, and left it to boot. As anyone who has dealt with them will tell you, Cisco enterprise switches do not boot quickly, and it seems even longer when you are anxiously awaiting the boot up.

I had good reason to be anxious. Instead of a switch that was lightly used, and not actually working, I'd accidentally powered down a working switch that was very heavily used (every port patched, with a light on). Thankfully, no one complained. Not even our networks guys, who have a system monitoring all the network infrastructure so were getting warnings.

I eventually did find the right switch, and rebooting it did cure the problem, so all was well.

SEP became MP

Sam not the Viking

Early finish Friday afternoon, last people to leave, and the burglar alarm can't be set: 'Fault'. We'd had some problems with intrusion attempts and with a deep sigh we called the 24-hour service company. Being the hero, or drawing the short straw, I stayed behind to await their arrival. To be fair, it was only an hour or so wait and the who guy turned up was also anxious to get the problem sorted and get on with his own end-of-week refreshments.

He quickly diagnosed a cable fault, which drew a groan as the cabling had been installed by an artisan better suited to serving spaghetti. Nevertheless, the fault was deemed to be 'between buildings' and he prepared to pull a new cable through; this was going to be a long job. I suggested that we inspect the access duct-covers first...... Well, the fault was obvious, something had chewed through the cable. The burglar-alarm detective said "That's a *very* big rat." He also advised he was not going to put his hands in any duct containing "that"..... We, meaning I, had to make some other sort of provision. His problem became mine.

I retrieved a pair of welding gauntlets and pulled sufficient free spaghetti/cable up to the surface for repairs to be effected. And a large board to place over the hole temporarily. The repairs worked, the alarm was set and the weekend started a bit later than usual.

The following Monday, I told the factory guys about the event and they said they had seen this rat and confirmed it was 'bigger than you'd like to meet'. Cue call to a well-known exterminator.

Tax and title extra.