False negative stretched routine software installation into four days of frustration
- Reference: 1686900545
- News link: https://www.theregister.co.uk/2023/06/16/on_call/
- Source link:
To start this tale, meet a reader we'll Regomize as "Dieter" who was once a Linux admin at one of the US's three big carmakers.
"The team I worked on took care of all sorts of Linux admin tasks, including working with software vendors to get their kit installed and tested in the corporate network," Dieter told On Call.
[1]
"Most of the time the software was already purchased and licensed by the time we sat down to install it. But there were those rare occasions where the IT leadership wanted to take the packages for a test drive before paying the vendor," he explained. When the bosses decided on that approach, they enabled verbose logging and debugging.
[2]
[3]
On one such project – involving a very well-known software vendor – preparing for this test required an installation process that took literally hours to run.
And then reported it had failed.
[4]
"The vendor team dived into the logs and worked for the rest of the day trying to determine what went wrong, but didn't uncover anything after reviewing the extensive log files," Dieter recalled.
Servers were therefore wiped ahead of a planned second installation attempt that was set to generate even more detailed logs.
That install also failed, which did not please Dieter's boss.
[5]Boss put project on progress bar timeline: three months … four … actually NOW!
[6]Cunningly camouflaged cable routed around WAN-sized hole in project budget
[7]That old box of tech junk you should probably throw out saves a warehouse
[8]Cheapest, oldest, slowest part fixed very modern Mac
On day three, the vendor returned, conducted a serious-looking meeting with serious-looking people wearing serious-looking suits, and tried again to install the software.And once again, went home defeated.
The next day, out came the big gun: a new installer that logged even more info.
[9]
It ran for most of the day, with Dieter and his colleagues watching line after line of text flash by.
After three failed installs, Dieter and his buddies knew exactly how long it would take before the installation would report failure. As they watched the clock tick towards the moment at which the previous installs had failed, their lack of faith was vindicated.
At the expected moment the dreaded message appeared: "Installation failed."
At which point the vendor team pounced on the keyboard and examined the last few lines of the output log. And discovered that the installation had in fact been a success.
Indeed, all four had worked.
But the conditional test the software used to check if it had been installed was hopeless at interpreting log files.
"In essence all of the many steps in the process passed and each one incremented the value of variable RESULT , but at the end they were checking the wrong variable," Dieter explained.
"Managers were less than impressed with the effort, and the response to the vendor from up the chain was something to the effect of 'Thanks for stopping by, no sale'."
Has your tech worked without reporting it's ready to roll? If so, [10]click here to send On Call an email and we'll consider your yarn for a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZIwyv@UcPm9aVh1OdBSvTwAAAAs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZIwyv@UcPm9aVh1OdBSvTwAAAAs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZIwyv@UcPm9aVh1OdBSvTwAAAAs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZIwyv@UcPm9aVh1OdBSvTwAAAAs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/06/09/on_call/
[6] https://www.theregister.com/2023/06/02/on_call/
[7] https://www.theregister.com/2023/05/26/on_call/
[8] https://www.theregister.com/2023/05/19/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZIwyv@UcPm9aVh1OdBSvTwAAAAs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
To start this tale, meet a reader we'll Regomize as "Dieter" who was once a Linux admin at one of the US's three big carmakers.
Wheel have to guess which one...
so roll on the weekend ...
But it all deflated and can no longer roll.
Can we put a brake on this, sounds like we are clutch'ing at straws with these puns.
But was the diet successful?
How did the program get any successful installs with this bug? It seems we're not getting the whole story.
Sounds to me like it was checking the log files, badly.
They turned up the log level for this install, which meant as far as the installer was concerned there was extra unexpected garbage in the file, so something must have gone wrong.
Turn the logging off and I bet it would've declared a successful install.
It could easily be that the company's systems had a configuration option which changed the reported message at one stage and the installer's logfile parser couldn't hack it. Or possibly some combination of install options broke the logfile parser's logic. Either way, the software vendor's QA department didn't catch it during testing.
Testing ?
Pray tell, what is this strange notion of which you speak ?
On the other hand...
We had some code that was checking the wrong variable, yet seemed to work perfectly on dozens of machines. That was until someone updated their OS at which point the program crashed immediately it was started - much head scratching ensued.
IBM SAN reporting tool
It largely reminds me of a similar story happening, during the mid-00 to a colleague.
Dude was the SAN admin for a big central DC of a 9BUSD turnover company.
We were after a good consolidated SAN and storage reporting tool and wanted to evaluate a couple of those.
It turned out IBM had just bought a company to put their product into the (gigantic) Tivoli portfolio. Tivoli Storage Reporting it was called, I think, back then.
So, my colleague asked and obtained a fresh windows server VM with all requirements and admin account of course.
Got the media from the vendor but never was able to install it. I don't remember the details but it never installed and he couldn't figure out why.
Call to support etc ... was totally unsuccessful for weeks.
We were in Europe, but IBM then decided to send a guy from the product engineering, from Texas (yes !) to Europe for a couple of days, to sort this out.
For 2 days, we looked at the poor desperate texan dude, totally puzzled at what was happening, calling, restarting etc ... He left with the product still not installed.
We never have known what the issue was. End of the PoC.
I get something similar after recent updates to Android Auto - it frequently connects but says it hasn't, until you drop back to the front page on the in car screen, and find it actually has....
Same problem with Steam the other day. Had to reauthorise an account with a one-time code. Entered the number and it complained "something went wrong". Grumbled. Got ready to do it again. And nothing had gone wrong: it was all working.
Noisy installers suck.
"Has your tech worked without reporting it's ready to roll?"
Well, yes. In fact, this was perfectly normal for the first several decades of my career. My code still works that way.
Silent success is where it's at, at least for professional software. As Doug McIlroy wrote back in '78, "Don't clutter output with extraneous information" (The Bell System Technical Journal, Volume 57, Number 6, page 1902). No message means a clean install ... you only need to throw a message in the event of an error.
Attention to detail
It matters. If you can't get the simple stuff right then I won't assume that you can do the difficult things.
I once rejected a CV with the entry:
Marital Status: British
While that might have been exactly what they meant to write, it made me wonder what their SQL would look like but unwilling to find out in our own codebase.
"Managers were less than impressed with the effort, and the response to the vendor from up the chain was something to the effect of 'Thanks for stopping by, no sale'."
You mean they were given the boot?