Automation is great. Until it breaks and nobody gets paid
- Reference: 1681457469
- News link: https://www.theregister.co.uk/2023/04/14/on_call/
- Source link:
This week, meet a reader we’ll Regomize as “Hugh”, who in the early 2000s scored a contract as a Linux admin for a global auto manufacturer.
Hugh spent regular weeks on call and told us those times were “sure to bring at least one sleepless night doing battle against failed software, or hardware.”
[1]
One of those incidents started at 2:00 AM when Hugh’s pager pinged with news that a host used by the HR team was in trouble.
[2]
[3]
Hugh did the shake yourself awake and turn on the laptop in the middle of the night thing and logged in to inspect the system, which clearly needed a reboot. So Hugh initiated a power cycle, watched it reboot without incident, then ran a series of tests. That effort produced nothing untoward, so Hugh prepared to retire for the night once again.
But just as he was about to hit the sack, a new alert arrived. The same host was in trouble again. Again, the system showed no sign of distress, and a reboot again brought it back to life This time Hugh decided to run some extra checks to make sure he hadn’t missed anything the first time.
[4]
That extra time paid off because exactly five minutes after reboot the host locked up again.
Hugh decided that 300 second interval was a clue, so when the system came back to life, he disabled cron, the ubiquitous job scheduler found in Unix-esque systems.
Next, Hugh started looking for evidence of any scheduled jobs.
[5]
And found one.
“It's MASSIVE, and its time stamp was ... about five minutes in the past,” Hugh told On Call.
[6]Techie called out to customer ASAP, then: Do nothing
[7]Uptime guarantees don't apply when you turn a machine off, then on again, to 'fix' it
[8]Errors logged as 'nut loose on the keyboard' were – ahem – not a hardware problem
[9]Techie fired for inventing an acronym – and accidentally applying it to the boss
A little investigation led him to a script that he described as designed to “append itself to his crontab each time it runs, then execute his target script 16384 times, and copy itself again.”
“The job in question was to collect timesheets from various sources, and take that to payroll.”
But the payroll system, and the host Hugh was trying to fix, did not enjoy that influx of info and fell over.
Which was bad for Hugh seeing as he was now wide awake at 2:00AM, and also because the function of this crappy cron job was to collect time sheet info for contractors.
Contractors like Hugh.
“Folks were not happy when they did not get paid on time,” Hugh told On Call, rounding out his tale with news that the chap who wrote the script and had his cron privileges revoked.
What has automation messed up in your life? [10]Click here to send On Call an email and we’ll automatically consider it for a future On Call. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZDkkRE-n8YtZ9JL@wMlUNgAAANU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZDkkRE-n8YtZ9JL@wMlUNgAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZDkkRE-n8YtZ9JL@wMlUNgAAANU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZDkkRE-n8YtZ9JL@wMlUNgAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZDkkRE-n8YtZ9JL@wMlUNgAAANU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/04/07/on_call/
[7] https://www.theregister.com/2023/03/31/on_call/
[8] https://www.theregister.com/2023/03/24/on_call/
[9] https://www.theregister.com/2023/03/17/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: This is why we need code review
I was on a training course at Digital in Reading for VAX Administration.
For a laugh, I wrote a quick script which logged all users who weren't me off the system, then submitted itself to the batch queue to be re-run immediately.
It was great fun, worked 100% correctly. Only...
When you are logging in, you are given the temporary username
Problem, big, huge! I couldn't log back in, because although my username was excluded, I first had to get past the
The instructor took us into the computer room and then tried to log onto the console, same problem. In the end, he had to hard reset the thing!
Luckily, he used it as a learning experience and I wasn't thrown off the course, but I did learn a valuable lesson!
Re: This is why we need code review
"to be re-run immediately"
See, what you *should* have done is schedule the script to run 5 minutes later, instead of immediately. This gives you just enough time to log in and undo the mess.
At least, I *hope* that's the "valuable lesson" you learned...
Sounds more like malware than a legit thing. I honestly didnt know where this story was going to go half way through.
"Sounds more like malware than a legit thing."
Yes, why would you run the darn thing 16K times ? The culprit would have deserved a good punch in the nose !
16384 happens to be 2 14 . So I would guess that the script was looping until the variable used by the loop was overflowing at that particular value.
Maybe each run of the script would collect the data for a single contractor and each time it was run it would increment a counter.
Training
I firmly believe that anyone writing software (yes, even a 'simple' script) professionally should first prove that they can produce stable code for schools. This will be 'tested' by malicious attacks from pupils, multiple wrong guesses by absent-minded professors, and random entries by new, dreamy primary school teachers.
I have the scars!
Payroll, not automation...
I was on a project at a Royal Naval dockyard, to replace their old personnel system with our new one.
The problem is, being RN, you need positive vetting, which takes time. I was draughted onto the project at the last minute, given the vetting forms on a Friday afternoon & told to report to the dockyard on Monday... Hmm, 6 weeks of checking completed over a weekend, I don't think so.
We drove down Sunday evening and booked into the hotel, then Monday morning, I presented myself at the guard hut. I managed to get a 3-day pass, but was told that was it, no ifs, no buts, without the vetting forms being approved, I wasn't coming back on sight on Thursday.
Thursday rolled around and the guard told me, sorry, no dice, you ain't coming on site!
I pointed out that I was working on migrating the payroll data from the old system to the new one, and if I didn't come on site, they wouldn't be getting paid at the end of the month.
A couple of minutes later, I had a 1 month temporary pass!
Interestingly, the dockyard had paid extra for shielded terminals. The supplier of the terminals decided that shielding wasn't really needed and decided to pocket the difference between the normal terminals he delivered and the cost of shielded terminals... Everything went fine, until a US Navy ship tested its radar in the harbour. Queue a hundred or so dead terminals, a red faced supplier, who had to suddenly lost all his "profit margin" on the terminals, as he had to replace them all with the shielded versions that had been ordered.
Re: Payroll, not automation...
That's a long queue.
Re: Payroll, not automation...
Testing radar in harbour can lead to FLKs.
"execute his target script 16384 times"
Sounds like a beginner who couldn't be arsed to find out exactly how many loops were necessary so go for 16K, it'll surely be enough.
Not to mention that auto-appending to the cron job sounds like it should be anathema to me. Not a Linux admin (or an admin of any other OS), but I'm convinced that a job is supposed to be put in the cron list by an actual human who knows what he's doing, not as an auto-insert by some coder who looks like he's half-assing his way to the next paycheck.
This is why we need code review
Because someone who writes a script designed to "append itself to his crontab each time it runs, then execute his target script 16384 times, and copy itself again" needs to be stopped before it even gets to QA, who probably only ended up doing the same testing that the programmer did because the programmer told them to test it like that.