Techie's enthusiasm for decluttering fails to spark joy
(2024/05/06)
- Reference: 1714981041
- News link: https://www.theregister.co.uk/2024/05/06/who_me/
- Source link:
Who, me? Welcome once again, dear readerfolk, to the sanctuary of Who, Me? in which Register readers can recount the times when their technical skills abandoned them, even if momentarily, without fear of judgment.
This week, meet a reader we'll Regomize as "Monty" who, many years ago, was employed by a manufacturer of sensors and measurement systems. The small but close-knit software team was responsible for designing, developing, testing, installing and supporting all the products of the biz – a challenging and interesting environment, sure, and sometimes also confusing.
Late one Friday, a customer's system had to be patched. This particular installation was a remotely managed system composed of several nodes located abroad, which were continuously streaming data to the customer's central offices. As one might imagine, downtime outside of the announced maintenance windows was never well received, as a matter of principle.
[1]
The standard solution for installing the software had recently transitioned from "copy the repository and build locally" to .deb packaging – which allowed proper versioning, dependency management and uninstall capabilities. Everything was finally automated and error-proof – or it should have been.
[2]
[3]
Dramatic foreshadowing.
So our self-confident Monty (who was also responsible for the packaging) connected to the each of the remote nodes, pulled the new package, installed the update, rebooted, and waited for the data to start flowing again. While this deployment part could have been automated as well, the installation was relatively recent and doing some manual monitoring during maintenance windows allowed him to gather some feedback on how the system was performing.
[4]
As the end of the working day approached, it was the turn for the last node to be updated – a node that had had a bit of a special history. There were tiny differences between it and the other nodes – differences that were never documented, nor even properly understood.
As you well know, gentle readerfolk, for a big thing to go wrong, several smaller things have to go wrong. In this case you had the aforementioned special installation, an engineer eager for a drink – but actually in desperate need for coffee – and an idiosyncratic OS. After each update, the OS popped up a reminder that there were unused packages, and would you like to uninstall them?
Monty, clean person he was, and worried that local disk space could become an issue, clicked yes – as he had done many many times in his life.
[5]
Too late, he realized that the list of "unused" packages contained a few obscure names of seldom-mentioned software with no practical relevance on a modern system. Names like, oh, for example, "Python".
Which was being efficiently uninstalled before his very eyes, together with all the software that depended on it – including the OS itself.
The realization was quickly followed by one CRTL+C and then by fear. The system was not responding. Nor could a new connection be established.
His last chance was a cry for help to his software lead, who had just replaced his wizard robe for a leather jacket and was about to leave. After hearing of the installation snafu, the software lead donned the robe, called home to postpone dinner, and sat down at the console.
The system was unreachable via the usual channel. One magic spell proved unsuccessful, then another, and a third. But maybe … if they could route a tunnel through another node and the wired LAN … with a small detour here and a special parameter there …
And there they were, logged in to the severely crippled system.
[6]Software support chap survived breaking his customer
[7]Rarest, strangest, form of Windows saved techie from moment of security madness
[8]Tired techie 'fixed' a server, blamed Microsoft, and got away with it
[9]Windows 95 support chap skipped a step and sent user into Micro-hell
Most of the files were indeed gone, but the system was still somehow running – and, most importantly, the update utilities were still available. Time for reinstalling the missing packages! A quick script was prepared to check for missing software, reinstall everything that should have been there, apply the planned update and then reboot. The script was launched and it seemed to do its job, to the huge relief of the duo.
But it got worse.
Just when the reinstallation process was about to complete, the remote connection froze again – this time for good, and for all systems. It transpired that while there was a LAN, outside access for maintenance and updates was happening through a mobile hotspot powered by a local SIM card – whose monthly data cap had just been reached with the big reinstall.
In other words, the script may well have completed successfully, or it may not – there was just no way to tell. They had done everything possible, and the immediate organization of a field trip was not justified without first discussing with the bosses and the client – which would have been an awkward conversation indeed. The only thing was to head home for the weekend and prepare for a challenging Monday.
Before confessing to the big bosses, first thing on Monday morning Monty tried connecting to all the systems – to find that they were all merrily sending data packets to the central bucket.
As it transpired, luck was on his side. For one thing, the download had miraculously completed just kilobytes before the connection froze, and the system rebooted as planned. For another, it was the very end of the month, so during the weekend a new monthly cap on that SIM card had started from zero.
And as it turned out, the client had a completely unrelated IT problem elsewhere on their system at the time, so hadn't noticed any outage at all.
(As a postscript, Monty later learned that his estimates were wrong, and disk space was definitely not an issue – so there was absolutely no need to play Marie Kondo on a production system. Another lesson learned.)
Have you ever been rescued from a stuff-up by the fickle hand of fate, like good old Monty here? Tell us all about it in [10]an email to Who, Me? and we might immortalize your adventure some future Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/04/29/who_me/
[7] https://www.theregister.com/2024/04/22/who_me/
[8] https://www.theregister.com/2024/04/15/who_me/
[9] https://www.theregister.com/2024/04/08/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
This week, meet a reader we'll Regomize as "Monty" who, many years ago, was employed by a manufacturer of sensors and measurement systems. The small but close-knit software team was responsible for designing, developing, testing, installing and supporting all the products of the biz – a challenging and interesting environment, sure, and sometimes also confusing.
Late one Friday, a customer's system had to be patched. This particular installation was a remotely managed system composed of several nodes located abroad, which were continuously streaming data to the customer's central offices. As one might imagine, downtime outside of the announced maintenance windows was never well received, as a matter of principle.
[1]
The standard solution for installing the software had recently transitioned from "copy the repository and build locally" to .deb packaging – which allowed proper versioning, dependency management and uninstall capabilities. Everything was finally automated and error-proof – or it should have been.
[2]
[3]
Dramatic foreshadowing.
So our self-confident Monty (who was also responsible for the packaging) connected to the each of the remote nodes, pulled the new package, installed the update, rebooted, and waited for the data to start flowing again. While this deployment part could have been automated as well, the installation was relatively recent and doing some manual monitoring during maintenance windows allowed him to gather some feedback on how the system was performing.
[4]
As the end of the working day approached, it was the turn for the last node to be updated – a node that had had a bit of a special history. There were tiny differences between it and the other nodes – differences that were never documented, nor even properly understood.
As you well know, gentle readerfolk, for a big thing to go wrong, several smaller things have to go wrong. In this case you had the aforementioned special installation, an engineer eager for a drink – but actually in desperate need for coffee – and an idiosyncratic OS. After each update, the OS popped up a reminder that there were unused packages, and would you like to uninstall them?
Monty, clean person he was, and worried that local disk space could become an issue, clicked yes – as he had done many many times in his life.
[5]
Too late, he realized that the list of "unused" packages contained a few obscure names of seldom-mentioned software with no practical relevance on a modern system. Names like, oh, for example, "Python".
Which was being efficiently uninstalled before his very eyes, together with all the software that depended on it – including the OS itself.
The realization was quickly followed by one CRTL+C and then by fear. The system was not responding. Nor could a new connection be established.
His last chance was a cry for help to his software lead, who had just replaced his wizard robe for a leather jacket and was about to leave. After hearing of the installation snafu, the software lead donned the robe, called home to postpone dinner, and sat down at the console.
The system was unreachable via the usual channel. One magic spell proved unsuccessful, then another, and a third. But maybe … if they could route a tunnel through another node and the wired LAN … with a small detour here and a special parameter there …
And there they were, logged in to the severely crippled system.
[6]Software support chap survived breaking his customer
[7]Rarest, strangest, form of Windows saved techie from moment of security madness
[8]Tired techie 'fixed' a server, blamed Microsoft, and got away with it
[9]Windows 95 support chap skipped a step and sent user into Micro-hell
Most of the files were indeed gone, but the system was still somehow running – and, most importantly, the update utilities were still available. Time for reinstalling the missing packages! A quick script was prepared to check for missing software, reinstall everything that should have been there, apply the planned update and then reboot. The script was launched and it seemed to do its job, to the huge relief of the duo.
But it got worse.
Just when the reinstallation process was about to complete, the remote connection froze again – this time for good, and for all systems. It transpired that while there was a LAN, outside access for maintenance and updates was happening through a mobile hotspot powered by a local SIM card – whose monthly data cap had just been reached with the big reinstall.
In other words, the script may well have completed successfully, or it may not – there was just no way to tell. They had done everything possible, and the immediate organization of a field trip was not justified without first discussing with the bosses and the client – which would have been an awkward conversation indeed. The only thing was to head home for the weekend and prepare for a challenging Monday.
Before confessing to the big bosses, first thing on Monday morning Monty tried connecting to all the systems – to find that they were all merrily sending data packets to the central bucket.
As it transpired, luck was on his side. For one thing, the download had miraculously completed just kilobytes before the connection froze, and the system rebooted as planned. For another, it was the very end of the month, so during the weekend a new monthly cap on that SIM card had started from zero.
And as it turned out, the client had a completely unrelated IT problem elsewhere on their system at the time, so hadn't noticed any outage at all.
(As a postscript, Monty later learned that his estimates were wrong, and disk space was definitely not an issue – so there was absolutely no need to play Marie Kondo on a production system. Another lesson learned.)
Have you ever been rescued from a stuff-up by the fickle hand of fate, like good old Monty here? Tell us all about it in [10]an email to Who, Me? and we might immortalize your adventure some future Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZjiqQvSTsZ3PmCCCPQ3P-AAAAEs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/04/29/who_me/
[7] https://www.theregister.com/2024/04/22/who_me/
[8] https://www.theregister.com/2024/04/15/who_me/
[9] https://www.theregister.com/2024/04/08/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Richard 12
I think that was just one of many "unused" things that were, in fact, being used.
If cron or ssh got uninstalled, many places would be in trouble for example.
TBH this is really about why servers should be sheep, not pets. Basically all identical, only able to tell each other apart.
Wrong animal
STOP_FORTH
I think you meant to say penguins!
John Sager
The web of dependencies on Linux systems makes it hard to divine whether any particular 'unused' library is in fact redundant.This is especially the case for stuff built locally from source as its dependencies won't figure in the apt system's dependency graph. So 'apt autoremove' is not to be undertaken lightly!
Korev
So Monty Don good?
Well-crafted tale of woe!
Michael H.F. Wilkinson
It has all the ingredients for disaster: Something needs fixing on Friday, plucky engineer confident his new packaging scheme has fixed things, undocumented "features" on one "slightly" different machine (to which said engineer has no direct access), and of course the desire to keep things clean (no good deed ever goes unpunished. Lucky for Monty things turned out OK.
Now that's what I call saved by the very last inch
Pascal Monett
Or kilobyte, in this case.
Had the sim failed just seconds sooner, he would have been a goner come Monday. Had this happened the week before, he would have been a goner come Monday. Had his superior left ten minutes before, he would have been a goner on Monday.
Sure, he might have made a wrong decision, but he got just incredibly lucky after that.
Good for him.
Re: Now that's what I call saved by the very last inch
b0llchit
Luck is with the Fry'ish :-)
I feel for young Monty here. Not sure I would have done differently, and you have to query a world where removing Python shuts down network access. C has its faults, but there’s a reason we use binaries for this stuff.