Hide the keyboard – it's the only way to keep this software running
- Reference: 1730446333
- News link: https://www.theregister.co.uk/2024/11/01/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Henry" who told us of a job he held in the 1980s at a small electronic engineering company that made and wired up "thundering great control panels for the gas industry."
"Think loads of dials and buttons all eventually controlling machines," Henry urged On Call.
[1]
By the time he got this gig, software had also become part of the mix – in the form of a Turbo Pascal package that could read the data from all those dials.
[2]
[3]
"We supplied and installed the equipment at a gas pumping station and all was going swimmingly," Henry recalled. "Everything was tested and signed off, the software swung into life, and gas started pumping. We headed home for some congratulatory beverages."
The party didn't last. Two days later the system was down and Henry was asked to fix it.
[4]
Restarting it proved easy, but the cause of the outage was elusive. Log files recorded no anomalies – just a final message at 5:57 AM.
Henry wrote it off as just one of those things. But then it happened again the next day, just a couple of minutes earlier. Again, the fix was easy, but no cause for the outage was evident.
Nor was it when the app again went down at around the same time next day.
[5]
Henry was therefore dispatched to visit the client and roused himself at 5:00 AM so he could observe operations at the time the crashes had occurred.
As the clock ticked past 5:50 AM and the software kept running, the gas company's next shift of workers arrived and Henry made a mistake – in the form of a cup of tea for one of the new arrivals. We say it was a mistake because, while he was brewing up, the system went down again.
Which left Henry having to explain to his boss that yes, he had been on site, but no, he wasn't watching at the relevant time.
At least the fix was again easy.
[6]Your computer's not working? Sure, I can fix that problem – which I caused
[7]Yes, your network is down – you annoyed us so much we crashed it
[8]Techie took five minutes to fix problem Adobe and Microsoft couldn't solve in two weeks
[9]OS/2 expert channeled a higher power to dispel digital doom vortex
A more senior colleague was then sent to the site to ensure Henry's eyes didn't wander at the wrong moment.
At 5:30 AM the next day, the two sat down and stared at the relevant kit. Again, workers arrived for the 06:00 shift change.
And one of them tossed their lunchbox onto the desk, where it smacked the keyboard – hard.
And then the system went down – just as hard.
Henry and his colleague rebooted the app, and the gas flowed again.
Then they started mashing keys.
"On the eighth key or so the screen went blank and a reboot was needed," Henry told On Call.
Problem solved? Not quite. "The solution in this case was to buy a piece of Perspex and bend it to cover the keyboard with just enough space to get your hands in and type," he told On Call.
Once that safeguard was in place, a proper investigation yielded the insight that the keyboard was interrupt-driven, and pressing too many keys caused the interrupt buffer to overflow and crash the computer to which it was connected.
"It was not exactly our fault, but still embarrassing," Henry told On Call.
What's the strangest reason for a crash you've encountered? Don't throw your lunch at your keyboard, throw your fingers instead by [10]clicking here to send On Call an email detailing your story, so we can share it on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZyS02VPLBgOPLAjC-o5-7wAAAFc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZyS02VPLBgOPLAjC-o5-7wAAAFc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZyS02VPLBgOPLAjC-o5-7wAAAFc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZyS02VPLBgOPLAjC-o5-7wAAAFc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZyS02VPLBgOPLAjC-o5-7wAAAFc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/10/25/on_call/
[7] https://www.theregister.com/2024/10/18/on_call/
[8] https://www.theregister.com/2024/10/11/on_call/
[9] https://www.theregister.com/2024/09/27/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Wat?
This is even worse , which I didnt think possible.
I'd have sacked that guy / idiot / gorilla on the spot, not that I've ever been in a position to sack anyone and I dont think the protagonist was either.
Does he run round the rest of the room pushing random buttons for shits and giggles? what a tool . This has really pissed off more than a "on call" should or has before for some reason .
Sloppy
Can you stop putting AI slop photos on your articles please? They just look shit.
Re: Sloppy
The word "shit" being a massive understatement.
Re: Sloppy
There are photos?
Re: Sloppy
Not any longer :)
A former colleague of mine tested my monkey-proof (I hoped) user-I/O I had designed for an image processing tool (way back in the late 80s, early 90s) by leaning with both his underarms on the keyboard for a few seconds, resulting in an interesting mix of weird characters. He then pressed enter and got two error messages:
"Input answer too long, remainder clipped"
and
"Error in integer input format"
and a new prompt to enter the right answer. I was quite chuffed. I had actually written my own interrupt handler for just such a case. I do not doubt there more inventive monkeys could throw a spanner (or wrench) in the works, but at least it survived this pretty extreme test.
Extremely bad design
Control access for systems of any significance should be completely isolated from access by non-specialist workers.
Re: Extremely bad design
> Control access for systems of any significance should be completely isolated from access by non-specialist workers
Good point in general, but in this case the keyboard was clearly expected to be required (hence the shield instead if, say, locking it in a cupboard) and it was inside the existing control room, full of tasty knobs and buttons, so one would presume that everyone on the shift was already relevantly qualified and specialised personnel.
Although perhaps, for the sake of the new system, still a bit too much in the hefty gas rigger mindset and not willing to give up dropping me sarnie box on the table like I done for the last five years before this interloper arrived. A bit more "sensitivity towards the fragile new equipment" training was also in order.
Just move some disks
I was told this...
They wanted to physically move some mainframe disks to make some space for new kit - but without shutting it down. It needed to be moved just a couple of inches.
Two of them put their backs to it and gently pushed. It moved - success.
What they had not realised was someone's belt was caught under the Emergency Power Off button. So when they stood up, it pulled the EPO and shut down the disks.
It took two hours to recover!
This was the same company who connected two main frames together. The cable was only 4 meters long. Instead of routing it down to the floor, under the floor, and up, they had to connect it at neck height with a couple of stands to support it. This worked, until someone walked into it, and pulled it out of the machines.
Colin
Re: Just move some disks
Working in a hot, humid climate and sporting minimum attire, an older colleague was lifting and moving a large, very heavy, metal-clad instrument. He eased it down on a metal bench but unfortunately trapped his clothing and a rather delicate piece of skin. An emergency hospital visit, a swift cut and soft bandages..... And antibiotics...... no alcohol.....
At the time he did shout out to
the old days
I used to program a credence lt1000 tester. There was a tek 4205 monitor and keyboard. To stop production from interrupting remote debug sessions I would unplug the keyboard.
Remote debugging can still be problematic with modern testers.
Small boys are good problem finders
Forty years ago or more my father would sometimes bring home prototype printers that his company was developing, to see if me and my brother could crash them just through the buttons on the front panel. We sometimes could. Many years later I used my own son the same way while building an electric organ. Give a small boy access to a device with over 200 organ keys, buttons and LEDs which makes lots of noise, and the hard part is keeping track of what he is doing. The even harder part is prying him loose again.
Workaround
I remember an old Sun bug report for Solaris where some stress testing caused a crash in CDE. It helpfully listed a workaround: "don't pound on the mouse like a wild monkey"
This was when Sunsolve listed a lot of useful information about bug reports and before Oracle hid it all away...
Workstation
As a first timer in a VLSI lab I got to try out some Sun SPARCstations, of which there were 10 in a row and all networked. The lab tech was very proud of his domain and showed off their ability to act as terminals to one that was powerful enough to run CAD for all of them. That just means they were each 10 times too powerful but I digress.
The mouse on mine could move but nothing could click. The tech guy leaped into helpful mode and pointed out that the buttons don't work when the keyboard's numlock is on. "But that's crap!" I said without thinking. Something that I regret because he was almost brought to tears. I had insulted his babies for having a pointless and crap design and who wouldn't be hurt by that? Of course I still think I'm right. It's a ridiculous oversight.
Wondering about the platform?
Given Turbopascal was mentioned I am wondering whether CP/M (80, 86) was involved. The 1980s also covers MSDOS 1.1 - 4.0 but I would have thought MSDOS' keyboard interrupt handling wasn't that fragile.
I suppose Turbopascal could have been used for embedded development which might make sense if you were dealing with a lot of sensors that used interrupts to get the processor's attention rather than polling.
I could imagine if the system had a fairly large interrupt load that a race condition in the keyboard interrupt handler might be exposed by mashing the keyboard eg if the ISR didn't immediately mask further interrupts before processing.
I vaguely recall MSDOS device drivers had interrupt and strategy parts but were basically a charade as the two parts weren't asynchronous.
Back then I thought minicomputer hardware like DEC's PDP8 and PDP11 rule that space (industrial control and monitoring.)
Why hasn't anyone mentioned and castigated the bloke who threw their lunchbox at the keyboard repeatedly every day at the same time? He's the culprit! He shouldn't be slinging anything around this sort of equipment!
Being too rough with your lunchbox is dangerous. It could spill open and let the sandwich's meaty filling fall out on to the desk, making a huge mess everywhere! That's just inconsiderate in a shared work place.
(Icon for "All hands man the pumps", nudge nudge, wink wink)
Wat?
No maintenance personnel unplugging something? I. Think. NOT!