Junior developer's code worked in tests, destroyed data in production
- Reference: 1752478810
- News link: https://www.theregister.co.uk/2025/07/14/who_me/
- Source link:
This week, we have two readers for you to meet!
The first asked to be Regomized as "Reggie" because he thought that would be quite funny. We've indulged him.
[1]
"I was just a year out of college and in my first job, working on accounting systems for the phone company," Reggie told Who, Me?
[2]
[3]
One day, Reggie's boss assigned him to another team that needed someone to zero out a database field in a sub-record used to store important information about employees' insurance.
It was a simple job that Reggie thought might take a day, so he whipped up a script, tested it against a copy of the production data, then showed it to senior folks who said the results looked fine.
[4]
Having passed that test, Reggie watched as his code ran on production systems.
"Quicker than you can say 'WTF?!?' my phone started ringing," he told Who, Me? "I had accidentally erased about 1,500 employees' private insurance records. It turned out the sub-record was actually a multi-type, and I was only supposed to update that field if it was the right type."
Reggie protested his innocence because nobody had explained that to him. His boss backed him up, and even excused him from cleaning up the mess.
[5]
"That was a relief," he told Who, Me? "Great learning experience, though."
[6]Yes, I wrote a very expensive bug. In my defense I was only seven years old at the time
[7]Junior sysadmin's first lines of code set off alarms. His next lot crashed the company
[8]Techie went home rather than fix mistake that caused a massive meltdown
[9]Techie exposed giant tax grab, maybe made government change the rules
Our second story today comes from a reader we'll call "Sam" who watched a colleague remove test records from a very large database.
"All of the records were flagged with a test number – 1 – in an unused field," he explained.
Removing the records therefore looked easy. Mike suggested the following syntax: DELETE FROM bigtable
WHERE secretfield = 1;
Mike's mate expected the job to run for a second. As the clock kept running, he started to worry.
"Surprise turned to horror when he saw secretfield -1 and noticed everything except his test records was gone," Mike told Who, Me?
Thankfully, this outfit had proper backups and could rebuild the small number of lost records from web server logs.
"My friend carried the nickname 'minus' for years afterwards," Mike told Who, Me?
What did you delete by mistake, and how did you recover it – and your reputation? Don't delete your shameful story, [10]click here to send this column an email so we can share it on a future Monday, with the same anonymity and good humor we always employ. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aHTVN0iyJ454_g1pLi6iSwAAAVM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aHTVN0iyJ454_g1pLi6iSwAAAVM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aHTVN0iyJ454_g1pLi6iSwAAAVM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aHTVN0iyJ454_g1pLi6iSwAAAVM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aHTVN0iyJ454_g1pLi6iSwAAAVM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/07/07/who_me/
[7] https://www.theregister.com/2025/06/30/who_me/
[8] https://www.theregister.com/2025/06/23/who_me/
[9] https://www.theregister.com/2025/06/16/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Sam's name got deleted from the database halfway through the article, I guess
Presumably using the MyISAM engine
Was the second story a SQL?
I was building a meeting scheduler for my department using Microsoft Power Automate (I know) and when someone cancels a meeting, you would like to blank the attendee field and open up the slot again. For some reason, although you can manually set a Person field to null, you cannot using Power Automate, because reasons. So when I tried to set the field to null, what happened was the system searched the database looking for the closest person to null, and now someone with the surname Nulli started getting dozens of meeting emails and calendar invites.
Top job MS. Top job me for not testing it properly.
There *is* a workaround to that...
https://www.shanebart.com/power-automate-reset-sp-person-field/
Of course there's a better workaround -->
Just create a dummy user in the system called "Room Available", and assign it when the meetings are cancelled?
INFORMIX and informix is not the same. DBA showed me.
Yet mangler and underlings don't see it that way.
We'll have some fun coming up.
Pro tip for DELETE queries
If you're ever running something like this
DELETE FROM bigtable WHERE secretfield = 1;
in production the way I always do this is to type out the query without the table name ("bigtable") or the condition ("1").
That means if you accidentally hit Enter at an inopportune moment e.g. after just DELETE FROM bigtable (which would delete all rows!) the query will fail because it's syntactically invalid.
I then go back and add in the table name with a sanity check of "is this the table I really mean?" followed by the same with the conditions. Pause. Then press Enter.
Takes about 5 seconds longer and I've never once deleted something inadvertently in production. Oh, and never copy/paste queries for exactly this reason - unless you modify them to remove the same params first.
You're welcome.
Re: Pro tip for DELETE queries
or you could just start with SELECT * , and see if the results correspond with what you wanted to delete
then bring the DELETE in once you're happy the filters/ table/ db are all correct
or at the minimum type the WHERE field = 1 first to avoid accidentally hitting enter and doing a wholesale delete with no filter
Re: Pro tip for DELETE queries
yeh this - SELECT the things you think you want to delete and see if they are the things then swap SELECT for DELETE when you are super sure you are right!
I guess thats why having a "hidden" column called "reallyDelete" is handy... so you can practice the where with update
Yeh - what could possibly go wrong there? I definitely haven't ever had to restore tables/databases for DBAs who got it wrong even with a reallyDelete column
/mines the one with a backup tape in the pocket
Re: Pro tip for DELETE queries
> or you could just start with SELECT * , and see if the results correspond with what you wanted to delete
Yeah I should have clarified that I meant once you've actually verified what you're going to delete.
There comes a point where you actually have to run the DELETE command.
I've come across numerous people where they've accidentally hit Enter, or even copied/pasted a full query with a line break at the end which in some clients is enough to execute it. The point being that if you supply 95% of the query initially and then fill in the blanks it's a simple way to avoid this specific problem.
Re: "and I've never once deleted something inadvertently in production"
As that's usually quite the dangerous statement to make ... we are now all looking forward to your upcoming contributions to this column ... :-)
Don't delete your shameful story
Darn, that's just what I did...
Confusing
""All of the records were flagged with a test number – 1 – in an unused field,"
I wouldn't have touched this job until it is was clarified exactly what was meant by " – 1 –". I'd sound like an annoying pedant, but I'd be sure what was required before doing something disastrous.
And then I would have executed a SELECT before doing the DELETE.
I like how "Sam" becomes "mike" in the second story