Making the problem go away is not the same thing as fixing it
- Reference: 1697787250
- News link: https://www.theregister.co.uk/2023/10/20/on_call/
- Source link:
"I worked on a financial package written in COBOL called 'CLOAS,' which the staff told me stood for a 'Complete Load of Australian … Something'," Mal opened.
The quality of that application became evident when Mal took a call in the early hours of the morning from the on-duty mainframe operator.
[1]
"The caller explained the problem, but only about one word in ten made it through to my befuddled brain," Mal admitted. After slowing things down and hearing the story again, Mal was only slightly the wiser.
[2]
[3]
This story comes from the time before remote access, so there was nothing for it but going to the office – despite the hour.
The situation didn't make much more sense once Mal arrived. But he eventually identified the probable cause. Having done so, he realized that this was not the sort of thing that just fails all at once. There would have been a chain of events. There would have been earlier failures.
[4]
He found himself "mystified as to why there had been no warning of the system's impending demise."
Mal asked the operator if he'd seen any advance notice of the problem and was assured no warning had been apparent on that night.
But the year before, the operator recalled, he did see a warning. Many of them. So many, in fact, that the error logs filled up and caused an overnight job to fail.
[5]
Mal dug into the situation and learned that his predecessor had been called out when those warnings proliferated. He'd come to the office, charged several hours overtime, and even ordered a pizza at the company's expense.
And then he made the warnings go away – by changing one line of code.
And which single line of code did he change, you may ask? Was it, perchance, the line that was causing the problem?
Of course not, silly. It was the line that generated the warning.
[6]Workload written by student made millions, ran on unsupported hardware, with zero maintenance
[7]Police ignored the laws of datacenter climate control
[8]Techie labelled 'disgusting filth merchant' by disgusting hypocrite
[9]Techie wasn't being paid, until he taught HR a lesson
"Job done, he went home and forgot all about it, leaving me to fix the actual problem a year later," Mal muttered to On Call.
Mal also reinstated the warning message.
The worst thing about this On Call incident? "As it was 2:00 AM I couldn't even order a pizza," Mal lamented.
Have you been called out to address a problem that should have been fixed by someone else? If so, [10]click here to send On Call an email . We always welcome more stories. Lots of recent submissions have concerned the mainframe age, which leads On Call to challenge to those of you working on more modern kit to share more stories. Anonymity guaranteed – for you and the outfit you worked at enduring the agony of being on call. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZTJPySjhyqvi0O9XWgwc-QAAAI4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZTJPySjhyqvi0O9XWgwc-QAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZTJPySjhyqvi0O9XWgwc-QAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZTJPySjhyqvi0O9XWgwc-QAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZTJPySjhyqvi0O9XWgwc-QAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/10/13/on_call/
[7] https://www.theregister.com/2023/10/06/on_call/
[8] https://www.theregister.com/2023/09/15/on_call/
[9] https://www.theregister.com/2023/06/23/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Watch the follow-up of large accidents on the [1]U.S. Chemical Safety and Hazard Investigation Board Youtube channel. I DO believe you. And Germany, especially the former DDR, had its fair share of chemical accidents due to negligence.
[1] https://www.youtube.com/@USCSB
I had to put a guy on a final warning because his desk was right below the fire alarm bell and he got fed up the weekly tests so one day he took the bell off on the morning of the test. He came within a gnat's chuff of being sacked.
Another technique
Another chemical engineer here. Many years ago while on shift experience at a refinery I was shown an alternative technique of using a matchstick to hold down the alarm acknowledge button to "solve" a nuisance alarm. I suppose at least the sight of the matchstick sticking out was a reminder that something was amiss.
So, shoot the messenger is still well and alive
When a so-called professional "solves" a problem by removing the alert, said individual should be reassigned to the deep-fry section of a burger joint and has nothing to do in a technical environment whatsoever.
Re: So, shoot the messenger is still well and alive
I'd argue that frying burgers is probably well above their level of technical competence!
Re: So, shoot the messenger is still well and alive
To be fair, he didn't say that said person would be operating ] the deep fryer...
Re: So, shoot the messenger is still well and alive
Depends. There may be incompetent design behind the alert, too. I've seen many cases where the existance of a warning was actually a mistake in the control software. Also, sometimes things just change and a warning can become obsolete.
Re: So, shoot the messenger is still well and alive
I was given a Toyota Yaris (1.4l, 4 speed automatic since you ask) by my father when he decided a year ago that he probably should not drive at the age of 90-something. Anyway, driving it on the M42 back from visiting him the engine warning light came on. I drove the next 20 or so miles to the next service station in some trepidation and looked it up in the manual. It said - take the car to the local service centre. Just that, nothing else. It could be a fault with the engine, transmission, anything. Well, I drove home rather tentatively noting that the car handled normally.
Took it to a service centre a few days alter, and they charged me a mere £30 to tell me it was a transient fault on the automatic transmission, and they had turned it off.
The next time I visited my father exactly the same thing happened, except I just left the light on until it went out by itself. I have since visited him again, and yup, the engine warning light is 'on'. It takes a few days to go out, so should be ok by Sunday, I hope.
Re: So, shoot the messenger is still well and alive
A £12 OBD reader off eBay will bring peace of mind.
Re: So, shoot the messenger is still well and alive
Reminds me of that scene in Big Bang Theory
Re: So, shoot the messenger is still well and alive
Indeed, but when that occurs it helps to remove both the test and action, not just the test, thereby forcing the action - not that I know anyone who ever did that.
"oh yeah ... that"
Back in the times of mainframes and driving to the Ops room when called out, I came in one morning to find that the team that sat next to us were all wearing frowny faces, bar the one angry and tired face and the guy with a face full of shame. Tired And Angry had been called out the previous night to discover that Shamefaced has been called out for the underlying problem the week before, fixed the problem by commenting out a loop with the intention of fixing it properly before the next weekly run and then forgetting all about it because beer. Without the loop the next run fell over in a bigger and better mess at the next run. Never underestimate the power of taking notes and holding proper handover meetings/stand ups.
It's the old burning toast + smoke detector problem. Stop burning toast? Nah! Disable the smoke detector.
Hence why in overregulated-Germany smoke detectors are NOT mandatory for the kitchen.
Scotland's recent legislation on domestic detectors requires a heat detector in kitchens.
I used to live in a house where the detector in the hall was too near the kitchen so it went off without much provocation. It was great for grilling meat. Put the bacon/chicken/chops under the grill, leave the door open and retire to the living room to watch telly. When the smoke alarm went off it was time to turn the meat over. The meat wasn't burnt - it was just right. I think it was fumes from the fat that set the detector off because it was always a bit late if the grill pan was clean.
Are you sure you want to go from amber to red alert? It would mean changing the bulb.
Pretty sure COBOL programmers can now charge far more than overtime and a pizza for any work, as if you need them you're totally fucked if you can't get them.
I feel reminded on QUITE some house wifes (well, and men)...
... who turned the CO alarm off 'cause "that noise causes headache".
Not an IT bod - chemical engineer here. But you might be surprised to learn how often large chemical factories - some of them handling things that are frighteningly expensive, toxic, explosive or all three - "solve" nuisance alarms by looping out the instrument that's causing the alarm to go off. It happens less and less nowadays, in fairness... but it hasn't stopped.