A lightbulb moment comes too late to save a mainframe engineer's blushes
- Reference: 1637311510
- News link: https://www.theregister.co.uk/2021/11/19/on_call/
- Source link:
Our story today comes from "Will", who told of his time as a City of London mainframe engineer in the swinging sixties and slightly more sombre seventies.
His tale opens, as do many On Call episodes, with a telephone call. This time from a colleague who was working at a major London stockbroker.
[2]
"They had two systems in their very modern prestige computer room," he recalled, noting "it even had a visitors' gallery."
[3]
[4]
However, one of the systems had "frozen" and would not do anything. There were no warning lights, nothing to indicate things were broken, but the mainframe was locked solid.
"I was nearby," Will told us, "and went round to the customer and sure enough nothing would work.
[5]
"We even tried a POR (power on reset), which was a big no-no in those days, until it became the popular fix for Microsoft and most modern devices – which still left the system in the same frozen state."
A small crowd of techies had gathered by this stage, and all were scratching their heads as they gazed upon the controls of IBM's machine. In this case, it was a System/360 Model 40. This particular bit of big iron first made its debut in 1964 and was eventually withdrawn in 1977. A [6]common configuration [PDF] would feature a mighty 128KB of storage, card readers, tapes and printers. Yep - all the toys a business could want.
In this case, however, the toys were broken and the machine was (relatively) silent. Ideas were tried and discarded. Right up until an engineer, running low on suggestions, jabbed a button on the front of the console. The lamp test button.
[7]
There was no helpful screen in those days, lights were the order of the day and every bulb lit up. Except one.
"This," sighed Will, "was the one that indicated that the console typewriter was out of paper."
The System/360 Model 40 was paired with the IBM 1052 typewriter keyboard, which consumed the same fanfold continuous stationery used by the bigger printers.
"The sensor was on the back of the console typewriter, so the machine had paper visible but was about to run out," recalled Will.
[8]COBOL: Five little letters that if put on a CV would ensure stable income for many a greybeard coder
[9]Wanted: Big iron geeks to help restore IBM 360 mainframe rescued from defunct German factory by other big iron geeks
[10]Hackers' Paradise: The rise of soft options and the demise of hard choices
[11]How Brit computer maker beat IBM's S/360 - and Soviet spies
Nobody had thought to check the console – after all, you could see the paper and the warning light wasn't on. It must be OK, right?
[12]Hotblack Desiato Will wasn't, and the bulb wasn't a black light that lights up black to let you know that something was wrong.
Changing it and the paper resolved the problem and the computer sprang into life.
Trickier, however, was explaining to the customer why a printer running out of stationery had taken a football-squad of techies and a large chunk of the day to resolve.
Every struggled with a problem only to find it was the fault light that was, er, faulty? Or had to make that explanation of shame to a customer? We've done at least one of those things. Have you? Confess all with an email to [13]On Call . ®
Get our [14]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YZeD2w52dhVSUI33lT32qAAAAQQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YZeD2w52dhVSUI33lT32qAAAAQQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YZeD2w52dhVSUI33lT32qAAAAQQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YZeD2w52dhVSUI33lT32qAAAAQQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] http://bitsavers.org/pdf/ibm/360/operatingGuide/C20-1635-2_Model_40_Operating_Techniques.pdf
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YZeD2w52dhVSUI33lT32qAAAAQQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2019/09/16/cobol_at_60/
[9] https://www.theregister.com/2019/05/21/ibm_360_mainframe_restoration_project/
[10] https://www.theregister.com/2014/08/15/feature_hack_proof_computing_and_the_demise_of_security/
[11] https://www.theregister.com/2014/04/09/how_english_electric_outsold_ibm_s360_cold_war/
[12] https://www.goodreads.com/quotes/7405023-it-s-the-wild-colour-scheme-that-freaks-me-said-zaphod
[13] mailto:oncall@theregister.com
[14] https://whitepapers.theregister.com/
Cupboard light
A colleague once told me about an ICL mainframe with a fault which confused the engineers all day until they discovered it was the light in the paper cupboard under the printer that wasn't working...
I had a somewhat similar problem
It was at the beginning of my career in IT. I was working for a self-employed engineer who had set up a test bench for industrial pumps. I was in charge of creating the UI software (in Visual Basic - sorry) that would allow the users to start or stop the bench, and see what the test status was. Everything was supposed to be automatic.
It was an impressive machine, I must say. The pumps would arrive by overhead rail. Upon reaching the proper position, a pump would be lowered for pressure testing. Tubes would be brought into place and cover the proper points on the pump. Once that was done, water was made available and the pump was set to work to ensure proper functioning. When the test was over, the system would drain, tubes would be removed, and the pump went on its merry way.
My UI was supposed to show all the steps, which I am happy to say that, after testing it for every possible scenario, it did.
Once everything was ready, we went to do the live demo. Everything went fine until a few minutes before we were supposed to finished, when the test bench froze.
I looked at the engineer. He looked at me. We hadn't moved a muscle. What had happened ?
I'll spare you the description of the 90 minutes that followed, what it boils down to is that, somehow, the big red Emergency Stop button had been pressed on the side of the test bench.
When I got back home (late) that evening, I promised myself I was going to integrate the status of that button in my UI the next day.
Re: I had a somewhat similar problem
That was acutally a happy ending, from safety perspective. I once pressed the big red button on a machine in integration testing, after which the machine proceeded merrily with its doings and a warning dialog came on its HMI screen, saying "TODO connect big red button procedure".
Re: I had a somewhat similar problem
I had a somewhat similar problem in a non-IT setting, which I may have related here before.
Riding to work on a new (to me) motorbike I stopped at a red light and the bike died. No sign of life, not even the click of a dying battery when I tried to start it up. After a few minutes of checking what I could, I called the AA. Eventually a man showed up in a little van, checked the things I'd already checked, couldn't start it. And didn't have the equipment to move a bike, so he called another man in a bigger van. The second AA man checked the things we'd already checked, couldn't start it, put the bike on his trailer and took me to the mechanic of my choice. As soon as the bike was unloaded, said mechanic looked at the bike, turned the kill switch back on and the bike started first time X-(
For those not in the know, motorbikes tend to have a big red kill switch on the handlebars to turn off the electrics in case of an accident. The one on this bike had a fairly light action and was in just the right position to catch on the wing mirror of a van as I squeezed past it in stationary traffic.
How far we’ve come
Nowadays it’s Windows Dozen locking up because it can’t find the printer. For backwards compatibility…
Re: How far we’ve come
Now with WSL you might get an error dialog stating "lp0 on fire".
Ran across this around six years ago when trying to convince a very old bit of kit (IBM 1403) to cooperate with Linux. I jumped about a foot. It had been probably three decades since I had last seen that error message.
I’m sure a lesser techie would have sussed that in a flash
Powerful minds would dismiss the mundane easy stuff in the hunt for something more interesting and challenging, if you don’t know about those more challenging things you’ve a smaller pool of possibilities and can find these mundane problems far quicker.
I assume that’s why airliners have problem checklists for cockpits, calmer heads have already checked through the easy and obvious so crew can get them out the way, if problem persists they have to come up with something but hopefully the prescripted check lists buys time or provides inspiration
IBM included a tester button for a reason though.
...why airliners have problem checklists for cockpits...
In case of an emergency situation we can do a lot of things. Well, mostly three: freeze, run, fight. What we cannot do is think. That's where checklists come into play.
You don't follow checklist in an emergency only. You follow checklist at every stage of flight when it is required to do so. Because it's far too easy to forger or skip something "because I've done it thousands times" or "hey, it's obvious", when it may not always be.
I think that is similar to the (railway) track workers rule of "never step on _any_ rail" - that way you don't ever have to think "am I about to step on the third rail?"
I got to walk through an underground tunnel once (the "Thames Tunnel" when it was being converted to the ON line) and the guides were clearly all properly trained and you could see them looking very uncomfortable as we all happily stood all over the tracks and third (and fourth) rails...
It can take a remarkably long time to notice that the alert mailer has stopped working.
(Quote unattributed - I can't remember where I saw it.)
Re: It can take a remarkably long time to notice that the alert mailer has stopped working.
I have a number of overnight job audit emails running. A while back I must have subconsciously realised, in my sleep, that one hadn't arrived at the expected time (the phone goes ping) as I woke up with the feeling something was wrong.
Useful as I was able to resolve the problem before any users noticed.
Some things are meant not to work.
I half remember the operator-less shuttle on tracks which take you from A to B, the door opens, you get in, the door closes, and off you go.
Until the queen came to open the facility.
She got in, and the doors stayed open.... they had to reset the shuttle before it would move.
There was lots of bad press, until they found the cause.
What happened was
The shuttle arrived, doors open, the queen says (quite) a few) words, queen boards shuttle.
The software had some code which says if the doors do not close within 1 minute, it looks like a problem. Do not let the shuttle leave the station, but send someone to check it.
Hotblack Desiato
Hotblack Desiato Will wasn't
He wasn't an estate agent? Well, that comes as a relief...
(and I know DNA asked them first, which I seem to think is even mentioned on their website)
Positive signals
There is something to be said for alerts where the light is on to confirm normal operation, and goes out on a fault condition.
Re: Positive signals
Until the idiot light becomes the problem.
Re: Positive signals
And preferably one that actually monitors the condition, not the signal sent to make that condition happen (3MI)
Re: Positive signals
Relays can fail in both states - contacts welding together, or coil burning out. Similarly for transistors, just not so easy to visualise.
Yes, I've seen lights stay on when the control signal was turned off. And vice versa.
Never trust what you can see. Or rather: trust but verify.
Warning Lights
A very long time ago, when electronic variable-speed motor drives were the new 'thing', I was sent to commission a job out in the wilds somewhere, transferring water from a reservoir to a treatment works via a long pipeline. The two inverters were protected at each stage by a multitude of fuses; there must have been 50 fuses per machine. The makers recognised that finding a blown fuse might be time-consuming and helpfully included a warning light for every fuse. I was sent with boxes and boxes of spare fuses. After a couple of hours, we realised this was going to be a long haul. The panel kept lighting up like a Christmas tree.
The 'Lamp Test' button revealed a display to rival Piccadilly Circus.
Late in the evening, a warning light came on, 'Low Pressure'..... A genuine fault or wonky protection? At first we rolled our eyes but on looking out of the window the true nature was revealed: the pipeline had burst and the adjacent field had become a lake. Made a very pretty sunset as the sun dipped over the horizon. Our day was done.
An even more extreme example of what a faulty bulb can lead to is Eastern Airlines flight 401:
https://en.wikipedia.org/wiki/Eastern_Air_Lines_Flight_401
Thanks for a great story!
Still Happends
Merely a month ago our z14 mainframe had to be IPLed from the physycal console because the remote operator console buffer run out of space. Set your systems to report just about what you need folks.....
Thank you …
for Douglas Adams reference - that cheered me up!