News: 1751268613

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Junior sysadmin’s first lines of code set off alarms. His next lot crashed the company

(2025/06/30)


Who, Me? Welcome again to Who, Me? It's the Monday morning column in which readers of The Register admit to making big mistakes and somehow swerving the consequences.

This week, meet a reader we'll Regomize as "Logan" who told us he acquired a psychology degree that somehow led to a role as an IT generalist.

A few years into that gig, Logan wanted a change and scored a job as a junior sysadmin – a big change because it involved working on a structured team.

We don't discipline honest mistakes. We do discipline shifting blame and shirking responsibility

"We had four developers, a product owner, and one other sysadmin," Logan told Who, Me?

That setup was a little confronting. "I'd never worked somewhere with a proper office, or indeed, proper unwritten rules," Logan told us. He also learned that the second sysadmin wanted to get out of the job and become a developer.

[1]

As you do when starting a new job, Logan tried to impress.

[2]

[3]

So when he found an inelegant dashboard for the Nagios network monitoring tool, he volunteered to tidy it up.

"I grouped the boxes by function, reordered a few things, and wrote a user story," Logan told Who, Me? He showed his work to the product owner and the rest of the team, all of whom agreed it was a fine upgrade.

[4]Techie went home rather than fix mistake that caused a massive meltdown

[5]Techie exposed giant tax grab, maybe made government change the rules

[6]Field support chap got married – which took down a mainframe

[7]Admin brought his drill to work, destroyed disks and crashed a datacenter

"Ten minutes after pushing the change, a few boxes on the dashboard went orange and one of the devs looked like he might pass out," Logan confessed.

The lead developer stared at the dashboard and struggled to decipher it.

[8]

Thankfully he soon realized the new design Logan created meant some familiar metrics appeared on different parts of the screen, and that the orange alerts reported a regular cron job that always stressed MySQL backups.

Logan reverted the change anyway.

"The Product Owner later told me he'd suspected it might cause issues – but had let me go ahead anyway so I could learn by doing," Logan wrote. "I wasn't sure whether to thank him or cry."

The big one

A week later, the product owner went on holiday and left Logan under the supervision of the other sysadmin – the one who had mentally checked out of the job so he could become a developer.

Come Friday, Logan volunteered to take on one of the routine end-of-week chores – running the company's version of /update-servers.sh – a script that SSHes into servers and updates packages.

[9]

The sysadmin gave his permission, Logan ran the script, and watched in terror as most of the Nagios dashboard turned bright red or angry orange.

"Phones started ringing. Orders stopped syncing. Sales couldn't update products. The website was up, but everything behind it was well and truly dead," Logan admitted to Who, Me?

The senior sysadmin asked Logan if he had run the script, checked his work and quickly diagnosed the problem – Logan hadn't commented out one line of code.

Nobody had told Logan that was necessary. And it wasn't his fault that someone had committed the script with that line active.

Before the blame game could begin, there was work to do because the script initiated an upgrade of production MySQL servers from version 4 to 5.

"The upgrade had filled the disk partway through, deleting the old binaries and failing before migrating the data," Logan told Who, Me? "Everything with a hardcoded MySQL 4 dependency – which was basically everything – had gone down."

And then the CEO materialized to ask what was wrong, learned about the upgrade, and demanded development of a recovery plan within the hour.

Suffice to say Logan and his supervisor found a way to re-install MySQL 4 binaries, quickly restored service, learned that the impact had been tiny – 15 missed orders out of 25,000 – and by 6 pm the incident was effectively over.

Which was when the senior sysadmin handed Logan an envelope containing a formal disciplinary warning.

"I'd expected a bollocking, but that stung," Logan told Who, Me?

Managerial matters

The following week, the product owner returned from his break. Logan came to the office early so he could give his side of the story.

He arrived to find the product owner already debriefing the CEO.

Logan joined their meeting and handed over a seven-page incident analysis he'd prepared, and the disciplinary letter.

"The CEO turned purple," Logan wrote. "He read the letter, crossed out my name, replaced it with the sysadmin's, and walked over to hand-deliver it."

And then the CEO stepped up, bigtime.

"You were on probation. You were supposed to be supervised," he told Logan. "And we don't hand out disciplinaries for honest mistakes. We hand them out for shifting blame and shirking responsibility."

"I never touched update-servers.sh again," Logan told Who, Me? "But I never forgot what I learned about good leadership and bad handovers."

How have you survived the blame game after an error? Has the boss bailed you out? Or bailed you up? [10]Click here to send us an email with your story so we can share it in a future edition of Who, Me? ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aGJgN5Y295iF-EBYOOCNJgAAAZc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aGJgN5Y295iF-EBYOOCNJgAAAZc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aGJgN5Y295iF-EBYOOCNJgAAAZc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://www.theregister.com/2025/06/23/who_me/

[5] https://www.theregister.com/2025/06/16/who_me/

[6] https://www.theregister.com/2025/06/09/who_me/

[7] https://www.theregister.com/2025/06/02/who_me/

[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aGJgN5Y295iF-EBYOOCNJgAAAZc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aGJgN5Y295iF-EBYOOCNJgAAAZc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



Korev

So Logan's run turned out ok in the end?

Headley_Grange

Logan Lucky.

Good call from the CEO.

Michael H.F. Wilkinson

A rare and precious commodity: CEOs with that kind of attitude

Re: Good call from the CEO.

big_D

I've had a few managers who took the blame for the department over the years and several that tried to push the blame further down the tree...

I've tried to become one of the former over the years.

When I was working on a job for a finance department at a customer, we had to recalculate Essbase hypercubes twice a days... Process:

1. Backup bottom row data

2. Delete all data from the hypercube

3. Load bottom row data from backup

4. Recalculate

The delete data and reload was standard OP for Essbase back then. A clean calculation from an empty cube took around an hour, a re-calculation of a full cube took around 8 hours...

I went through the process a couple of times, then the next time through, I somehow got distracted and started at 2... And, oh, no current data to load! I asked the junior dev who had been responsible for the database what we do now? He said, load the previous backup and blame it on the users. I was horrified, so I went to the Finance Director and my stomach was really rebelling, but I told him exactly what had happened, that the database had a transaction log and we would load the previous backup, replay the transaction log and the users should double check the data, once the cube had re-calculated, and that it would take around 2 hours, instead of 1 hour.

I loaded the backup, played the transaction log and recalculated. It turned out it worked and we lost a total of 2 transactions.

My honesty gained me the trust of the finance director and we had a very good working relaitonship after that.

sysadmin ... become a developer

Anonymous Coward

I've got a DBA become a sysadmin, who then does things like listing printers in CUPS by IP ADDRESS, and then is amazed when the DHCP server hands them a different address and things shit the bed.

Oh and blames it on the DHCP server, of course. FML.

They've all got perfectly serviceable names, used by Windows and everything else in the business, and suitable labeled on each one.

Re: sysadmin ... become a developer

Anonymous Coward

Real sysadmins just don't become developers. They're hard bastards with more than a supçon of sadism † ; Not soft buggers twaddling on about pointless syntactic preferences in the latest obscure programming language de jour.

† more like a tonne of bricks should you mess with their systems' uptime.

Re: sysadmin ... become a developer

b0llchit

Real sysadmins are also real hardcore developers but they do all the hard work and loath their so called "developer" counterparts because Real sysadmins always have to clean up their so called "developer" counterparts' crap. Also, Real sysadmins know to handle it all, including a LART and know how to handle the so called "developer" counterparts by having a good planning knowledge of basements, elevator shafts and windows (of opportunity).

Re: Real sysadmins vs Real developers

Anonymous Coward

Real sysadmins make applications work and keep them working

Real developers make applications that work

Without both, there would be no applications to make working nor applications that are working

Why do we have two eyes? To watch 3-D movies with.