Swooping in to claim the glory while the On Call engineer stands baffled
- Reference: 1637915466
- News link: https://www.theregister.co.uk/2021/11/26/on_call/
- Source link:
Today's tale comes from "Rob" and is set in the 1980s, at a City insurance company. As was common in those days, the [2]IBM System/3 midrange computer lurked within the data centre, and this particular outfit ran its underwriting software on the system.
The System/3 that Rob was responsible for luxuriated in a spacious 64KB of RAM and featured nearly 2.5MB of mass storage via an internal disk (likely the IBM 5444, which measured 14 inches in size). Backups were as important then as they are today, and Rob's IBM was no exception. "We backed up the internal disks onto 9-track tape cartridges via an external unit," he told us, "inside which there was a plastic rack into which a number of cartridges could be loaded."
[3]
It was a straightforward mechanical operation. The rack moved back and forth to insert and remove cartridges from the drive. Right up until it didn't.
[4]
[5]
Rob came in one morning to find the overnight backup had failed with a near-meaningless message. Naturally, he tried the operation again, but once more the inscrutable message popped up. "It ran for a while, started writing to tape, then broke," he recalled.
Having likely exhausted his ability and definitely his inclination to investigate further, Rob placed a call to IBM support, who dispatched an engineer. A flurry of activity ensued, during which panels were stripped and the computer's internals exposed to the outside world. "I left him to it," said Rob, who wandered off. Hopefully in search of a hot beverage and morning sustenance.
[6]
Time passed, but the computer remained inactive. The engineer professed himself baffled and called for backup. Another engineer arrived, armed with more components. More panels were removed. More bits were changed. But still the problem remained: backups started, ran for a bit, and then failed.
"I left them alone," said Rob, "partly because I knew it would have been irritating for them to have some bloke hanging around, and partly, to be frank, so that I could distance myself from their failure in case I needed to."
A wise move, because morning was turning into afternoon. Underwriting was not happening, and the users were getting a bit shouty. Rob therefore returned to the group of engineers to get a progress update.
[7]
"They were deep in conversation," he recalled, "huddled at one end of the computer to discuss their latest can't-possibly-fail wheeze."
Sensibly, he decided not to interrupt their trains of thought and so simply hung around the defunct computer until one or more of them came up for air.
Bored, he started poking at the IBM. "The door to the tape unit was open and idle curiosity made me reach inside and push the black plastic rack with my finger, as one does," he said.
"It moved, but the movement looked odd. I poked it again, a bit harder, and realised that while one part of the rack moved, the other didn't. It was supposed to be one unit." How else could the tape cartridges be inserted and removed?
He looked closer and saw a jagged crack right across the bottom of the unit.
The engineers were called over to look at Rob's discovery and all went a bit quiet. "They had inspected and replaced internal parts of the machine, but hadn't thought to look inside the external tape unit."
[8]The Ministry of Silly Printing: But I don't want my golf club correspondence to say 'UNCLASSIFIED' at the bottom
[9]We know it's hard to get your kicks at work – just do it away from a wall switch powering anything important
[10]Takes from the taxpayer, gives to the old – by squishing a bug in Thatcherite benefits system
[11]Did I or did I not ask you to double-check that the socket was on? Now I've driven 15 miles, what have we found?
Oops.
Yes, of course it all began working as soon as a new unit was delivered and installed.
And Rob? "Naturally enough, I claimed all the credit for making the problem go away."
Which, in a very real sense, he had.
There's a saying that goes "When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth." Although, to be frank, a cracked rack is far from improbable.
Have you ever had to call out the cavalry only to solve the problem yourself? Or been that engineer left red-faced by a user's innocent observation? Let us know with an email to [12]On Call . ®
Get our [13]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://www.ibm.com/ibm/history/exhibits/rochester/rochester_4008.html
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YaC@YId0MQYZPtP8asJeIAAAAEU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YaC@YId0MQYZPtP8asJeIAAAAEU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YaC@YId0MQYZPtP8asJeIAAAAEU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YaC@YId0MQYZPtP8asJeIAAAAEU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YaC@YId0MQYZPtP8asJeIAAAAEU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2021/11/12/on_call/
[9] https://www.theregister.com/2021/02/12/on_call/
[10] https://www.theregister.com/2021/01/29/on_call/
[11] https://www.theregister.com/2020/10/30/on_call/
[12] mailto:oncall@theregister.com
[13] https://whitepapers.theregister.com/
Re: Hands On (and Eyes On Too)
And as I always tell my engineers, whilst analysing data and test results is fine, the two most powerful troubleshooting tools you possess are located just above and on either side of your nose. There is no finer piece of problem solving equipment than the mk 1 eyeball.
Many a time I've had customers who escalated an issue after "working on it" for weeks without actually going anywhere near the tool, where it has been solved almost immediately by walking up to the tool, looking inside and spotting the bit that's hanging off or in pieces on the floor...
Re: Hands On
There is no finer piece of equipment than the mk 1 eyeball.
In my experience, only when used in close conjunction with the Mk1 wetware sitting inside their skulls.
Re: Hands On (and Eyes On Too)
I've recently upgraded to 1.1.0
Now uses piggy back device to enhance functionality
SpecSavers did it.
Re: Hands On (and Eyes On Too)
Experience says regular further upgrades will be needed.
Re: Hands On (and Eyes On Too)
I'd give a nod to the variable upgrades that I upgraded to a couple of years ago.
Re: Hands On
I used to think that, too, but then I got the restraining order that said differently. =-/
Re: Hands On
"There is no substitute for standing next to the box and prodding it."
Assuming prodding may involve the use of hammers. Large adjustable spanners may be substituted if you find yourself suddenly hammerless.
Re: Hands On
I possess no less than eight hammers in various weights, and despite three full sets of ordinary spanners, four adjustables ('shifters' in local parlance). One of these shifters, the largest, has magical properties in that its mere presence induces faults to disappear. Indeed, it was used to cure a leak on a friend's central heating system and when it left the premises, the leak returned. I couldn't afford to be without this invaluable piece of equipment so my friend photocopied it and put the print next to his boiler: the leak stopped.
Some years later, that friend had moved house but I know that photocopy remained, continuing to offer protection.
Re: Hands On
That smells faintly of how we end up with the Adeptus Mechanicus...
Re: Hands On
First rule of Troubleshooting:
Don't Force it. Find a Bigger Hammer.
Hey Rob!
Too bad you didn't have a look around before you sent for assistance! But, then again, we wouldn't have this week's On Call!
Re: Hey Rob!
But we would have had next week's Who, Me?
Re: Hey Rob!
Eh, the inscrutable error messages throw one off, and can completely hose your debugging tree.
For example, I filled my FJR-1300 up at the gas station, then it wouldn't start, with an error number that meant "bad crank position sensor" so I was like "well that's fucked then" and called for a tow truck, which basically put an end to getting anything done for the day.
Turned out to be a bad starter relay, and the computer went "well I fired the relay, and it's supposedly cranking, but I'm not seeing anything, so it's gotta be a bad crank position sensor!"
If I'd **known** it was a bad relay, it would have been a simple matter of push starting it. But with the stupid and incorrect error message, I was thrown off course.
Re: Hey Rob!
Ive done same. my Supra used to crank for a lot longer than i thought was necessary before starting , and after trying a variety of things , new crank position sensor included , it turned out the starter was worn and was cranking too slowly!
Not slowly enough for me to notice , but slowly enough to confuse the ecu!
Re: Hey Rob!
This. I've had an ECU fail due to a rather odd chain of circumstances:
- ecu turns on low speed fan
- relay flips over
- dropper resistor has failed, so no fan
- ecu gets it's 'fan running' signal from the relay, not the fan...
- ecu thinks fan is running but engine is a little warm, continues to drive the fan relay
- thin wire driving that relay isn't really rated for that continuous current and melts a bit into an adjacent wire
- volts get where they shouldn't in the ecu
- $$$...
And all because the designer saved a couple of bucks by not looking at the other side of the dropper resistor.
Re: Hey Rob!
You'd have tried to pushstart a CB1300? You're a bigger man than I am. No way would I try with mine.
Re: Hey Rob!
Do it in second gear, not first.
Start by pushing up to a fast walking pace, then SLOWLY ease the clutch out while continuing to push to keep the speed up.
And be prepared to engage the clutch quickly when the engine catches .... or you'll find yourself being dragged down the road!
Re: Hey Rob!
I've push started a Cortina solo.
less chance of that toppling on top of me though.
Re: Hey Rob!
I've tried to push start my ex-step-mum in a Cortina. After I stopped cos it hadn't started and I was bright red and covered in sweat, she then 'innocently' asks if I want her to take the hand brake off!!!!!
That was when I really learned that blonde hair was a sign of all the brightness leaking out! ----->
Heck!
I've just realised that I've admitted to trying to bump my step-mum!!
I hope my dad doesn't read this!!
Re: Hey Rob!
In the past I've had two Subarus throw up apparent engine problems. Checking the error log shows ECU cannot regulate engine speed. Problem - regular enough to be Googlable - faulty switch reporting it's in neutral when it isn't so ECU tries to set idle speed. Current car has so far had two wheel sensors replaced. Vehicles are increasingly dependent on electronics and the sensors that feed them; those sensors need to be a good deal more reliable than they are, otherwise it's GIGO.
At Gene Cash, re: wrong error message.
Back when I was a teenager & still (mostly) obeyed my dad, he ordered me to help him troubleshoot his Subaru Impreza. Because he was so much smarter than any Chilton or Hayne repair manual, he refused to read them to determine what the error message the car computer was giving. "I can fix it myself." was the words of doom that now let me know the speaker is an idiot & that I should immediately turn & run like hell.
Cue an entire weekend spent struggling to undo bolts that dad can't reach, hoisting the engine out of the chassis, power washing the entire thing "to check it for leaks" (Sigh), and poking, prodding, & "testing" every last nanometer he could get at with a screwdriver. To no avail. Sunday afternoon rolls up & it's still not working. Since mom had to be at work Monday morning (dad worked the night shift so had another few hours for his own needs), we were forced to reattach everything, remount it in the chassis, & try to undo whatever further idiocies he'd introduced.
We're a few bolt turns from done when he "accidentally" breaks a small plastic thingamajig. He gives me money, a 3x5 card with the item description, & the part number from the side of the item. He sends me the six blocks away to the auto parts store with *explicit* instructions to buy *exactly* that part & no other. So off I go to do as asked.
The guy at the counter reads the 3x5 card I hand him, gets a scrunchy face of concentration, & asks "Did you check $OtherPart first? It's usually what goes wrong first and results in this part (waggles the card) to fail."
I have no idea, it's not like dad to explain anything (nor that anything he thinks matches with anything from *this* plane of reality), so all I can do is shrug & reply "I honestly have no clue. We've just spent since late night Friday until now tearing it apart, testing everything, and that part broke while we were putting it back together just now. I'm to get that, bring it back, so we can finish the job & hope the car runs again."
He gets me the part, I pay for it, & he asks me to ask my dad to come see him after we're done. I thank him & head back home.
Dad installs the part, tries the engine, & it still refuses to start. He's about at the end of his rope, a situation which is not improved when I relay the parts man's request. Dad drags me to the store & has me point out the employee. Said employee makes it easy by waving at me & calling out "Hey! You're back!"
They talk, dad feeding the man a line of obvious bullshit, the man not buying it, and dad trying to defend his "testing" regime. When the parts man asks about the part to check that tended to fail first, my dad swears up & down that he had tested it & found it fine. Except I knew for a fact that he'd done nothing of the sort. How? Because *I* was the hands that had to hold all the test equipment while he applied the test probes, and we never touched the part in question.
Dad says he'll go remove the indicated part, bring it in, & prove to the man that it's still good. Back home we go, I help him remove said part, & back we trudge to the store.
Dad presents the part & says "There. See? Perfectly fine!"
The parts man washes the part, dries it completely, turns it over, & shows the BIG FUCKING CRACK down the bottom that disproves my dad's claims.
Dad grumbles, pays for a new part, and we go back home with dad muttering & mumbling a blue streak the entire way.
Install the replacement part, go to start the engine, & it purs to life just as neat as you please.
I later went back to the store & asked the man how long the fix would have taken had my dad bothered to RTFM. "About ten minutes to loosen a few bits in the way, five minutes to swap the part, and another fifteen to put it all back together. If it took an entire hour, you were doing it wrong." I thanked him & let *MOM* know of that fact.
"You mean your idiot of a father wasted TWO DAYS when the whole thing would have taken under an HOUR?" *Me nods* "I'll kill that dumb fuck!" *Cough*
I have *always* accepted when something is beyond my knowledge & should be entrusted to a professional. If there's a manual, I'll RTFM to make sure I know WTF I'm supposed to be doing. But if even RTFM doesn't enlighten me, I'll pay a pro to do it right the first time. This is the exact opposite of dad whom refused to RTFM, insisted he could fix it, and would usually wind up having to take $Item in to be *actually* fixed by a pro anyway (and for much more, since dad introduced further things in need of fixing).
TL;DR: Always Read The Fine Manual, it can save you days of headache, massive amounts of money, and a metric fuckton of embarrassment when someone points out that you are, in fact, an idiot for not having RTFM in the first place. =-)p
Curate's Egg
"Hopefully in search of a hot beverage and morning sustenance."
Where I worked in the 80s the hot beverage would have been a crap cup of coffee (or powdered tea) from a vending machine but the canteen provided mid-morning sustenance in the form of fried egg rolls which I took full advantage of.
"When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth."
I've never liked that expression. It barely makes any sense.
When you've eliminated the impossible , theres a boatload of possible things remaining so you're no nearer your answer.
I usually start my diagnostic procedures by writing off the impossible from the start.
Maybe if I eliminate everything thats possible I'll go back and start looking at the impossible , but usually the answer presents itself well before all possible solutions have been exhausted.
Its no surprise Dirk Gently was a fan of this method , given his predilection for creative invoicing.
I'll admit that, if you change it to say : " When you have eliminated all which is possible, then whatever remains, however improbable, must be the truth ", then it makes more sense.
It is also, however, less awesome.
true dat .
It sounds like one of those "truly fucked up over time to mean the exact opposite" sayings such as "I could care less" (which really grinds my gears) but it looks like Sherlock homes said it that way from the start.
Embarrassing for a man of with such legendary logic skills.
Besides...
If something is impossible, how can you eliminate it?
It's all about the bass
Many years ago I worked in a city centre ale house which had one of the new fangled CD jukeboxes installed, it was a large alehouse with speakers dotted around the walls, but there was also a speaker in the bottom of the freestanding jukebox unit. All was well Sunday to Thursday, but come Friday & Saturday evening the CDs started skipping.
An engineer was duly called, on multiple occasions, each time they were baffled, but convinced that it was just dirt on the discs. So each time they went through cleaning each individual disc (there was 100 of them), but the next Friday or Saturday evening the music was skipping Norman Collier style again. After a couple of weeks of this I suggested that the speaker in the bottom of the jukebox was causing the problem, as the volume was cranked up on a Friday and Saturday night, this was putting more and more vibration into the unit. The engineers dismissed my suggestion and continued trying to clean the discs, but still didn't cure the problem.
A couple of weeks later I turned up for work to find a shiny new speaker mounted on the wall above the jukebox, the one in the bottom of the unit having been disconnected and the wiring redirected. Lo and behold, no more skipping discs, so the lowly barman was right all along!!
Re: It's all about the bass
There's no such thing as a 'lowly barman'. You were a 'recreational liquid delivery engineer'!
-------> Obv!
Hot hot hot
Started working at a software company once. They had an issue where their servers seemed to be fine overnight but started failing in the daytime when people came into the office. Seemed worse in winter too.
Servers, well PCs, were all in a store room. As was a radiator. I noticed the thermostatic valve for the radiator was sitting on a shelf, and not attached to the radiator inlet. Room was unbearably hot too, because the radiator was scalding hot. I put valve back on radiator and closed off the valve, and server problems mysteriously disappeared as if by magic. I lasted about 6 weeks at that company.
When is the same not the same.
I remember some hair pulling and swearing over the partition to the next desk.
When the customer typed a command in - it worked. If he ran it in a script/program - it didn't work. They both ran on our systems. It was the identical command so why didn't it work on the customer system.
Someone else wandered over and said "get them to edit the script/program and retype the command".
We could hear the eye rolling at the far end, but he did it as he was desperate . tap tap tap - " it works!"
The customer had cut and pasted from a manual into the file. Unfortunately somewhere in the process a quoted string became a string with back quote, and forward quote - which looked like a quote at each end (or may be a code page problem).
Hands On
There is no substitute for standing next to the box and prodding it.