News: 1762155009

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

‘ERP down for emergency maintenance’ was code for ‘You deleted what?’

(2025/11/03)


Who, Me? Another Monday is upon us and The Register therefore presents a fresh instalment of Who, Me? It’s the reader-contributed confessional column in which you admit to making mistakes, and explain how you made it out alive afterwards.

This week, meet a reader we’ll Regomize as “Holden” who told us a tale from a time his employer “had just gone through implementation hell for a shiny new ERP system.”

“We went down the 'managed build' route with the software provider and the project overran by 12 months,” he lamented.

[1]

The result was a system Holden described as “quite buggy” and lacking sophisticated reporting tools.

[2]

[3]

”For more intricate report extraction we needed to use direct SQL access on the database,” he told Who, Me?

What could possibly go wrong?

[4]

Well, about six months into operation of this ERP Holden heard his boss start to repeatedly mutter “Oh god” in an increasingly panicked tone.

“His face had turned an interesting shade of puce and he looked a little uncomfortable,” Holden wrote. “It transpired he'd accidentally wiped a key piece of data from our expense transactions which left them all in a hung state and unable to be processed.”

Holden found a shadow table that made it possible to reconstruct the missing data.

[5]

“You'd think at this point we would have learned our lesson,” Holden told Who, Me?

Of course they hadn’t.

[6]Frustrated consultant 'went full Hulk' and started smashing hardware

[7]Company that made power systems for servers didn’t know why its own machines ran out of juice

[8]Techies tossed appliance that had no power cord, but turned out to power their company

[9]Techie found an error message so rude the CEO of IBM apologized for it

Two months later another team member approached the boss’s desk and admitted to “a bit of a problem – I've accidentally deleted some relations.”

He’d actually deleted them all. While constructing a complex query, he dropped the main relational table for the general ledger.

“This necessitated a company-wide ‘The ERP is down for emergency maintenance’ notification, and a rapid restore from the previous evening’s backup,” Holden told Who, Me?

After that mess, Holden said the company issued strict instructions that the team must always use the BEGIN TRANSACTION , ROLLBACK TRANSACTION , and COMMIT TRANSACTION commands!

What terms have you used to obfuscate your mistakes? [10]Click here to share your weasel words with Who, Me? We’d love to share them with readers in a future edition. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aQiLS13L8mit-q54wJjlFwAAAQA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aQiLS13L8mit-q54wJjlFwAAAQA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aQiLS13L8mit-q54wJjlFwAAAQA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aQiLS13L8mit-q54wJjlFwAAAQA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aQiLS13L8mit-q54wJjlFwAAAQA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2025/10/27/who_me/

[7] https://www.theregister.com/2025/10/20/who_me/

[8] https://www.theregister.com/2025/10/13/who_me/

[9] https://www.theregister.com/2025/10/06/who_me/

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



What reporting system...

kmorwath

... needs a database user that can actually modify or drop production tables???

I would have fired the DBA.

But I've seen too many database where everyone connects with highly privileged users. Assigning correct permissions is work...

Re: What reporting system...

Korev

It was written / customised by a third party and sounds like things didn't go well

This is why...

simonlb

You have all the processes you need to perform thoroughly documented, peer reviewed by at least two other people, tested in a non-live environment and then only performed on the live database by approved people who fully understand what they are doing.

And when I say documented, I mean written by a human with experience, and not hallucinated by some shitty AI womble.

Nah

DS999

This is why you SCRIPT stuff like this, and only deploy it in production after testing in a dev/QA environment. No one should be given live SQL access to a production SAP database. Like, ever!

If you need to muck about it in for "custom" reports that can't be scripted then you have a daily replica copy created and do your reporting on that. If someone balks at the cost, you justify it based on the inevitability of human error and the fact that the replica combined with copy-on-write to preserve the original replica can act as a backup/DR depending on its location. If they still won't spring for it, you save a copy of the email where you listed the potential consequences and their refusal to approve it for when shit inevitably goes pear shaped and the people at the top are looking for a head to chop.

Re: Nah

Doctor Syntax

Email? Require it in writing. Signed. Allow ink as a concession after making clear you'd have preferred blood.

Re: Nah

I could be a dog really

Meanwhile, back in the real world of small/medium business ...

"Various users" use direct SQL access for ad-hoc queries. Been there, done that.

If it's a "proper" database, then as already suggested, you properly apply permissions and/or create views to prevent such accidental deletions. In our case, the system used C-ISAM files and we used Informix to access the files behind the system's back. When we built the Informix databases, we'd create the tables in Informix, then delete the files and replace them with symbolic links to the real C-ISAM files. That worked well enough for us, especially after I found out how to automatically extract table descriptions and script creation of the whole lot in one go. It also meant that if any one did drop a table (no-one ever did, only 4 of us in IT used it - end users came to use for reports) it would only drop the symlink and do no harm.

Re: This is why...

Sam not the Viking

No process, documented, tested, approved, backed-up etc. which is operated by a human, experienced or otherwise, can avoid Murphy's Law (other names are available): If it can go wrong, it will.

Rather like complex numbers, AI introduces another route, which despite having an imaginary content, has real effects.

Afterwards, both humans and AI will make up an excuse why it wasn't their fault.

Re: This is why...

StewartWhite

"Hey, don't blame us!" said Orinoco, "There's a lot of rubbish for us to clear up on Wimbledon Common so we haven't had time to train our AI avatars yet."

Why?

Gene Cash

Why the hell are people deleting records/dropping tables when doing QUERIES and REPORTING??

You don't just "accidentally" type "delete" or "drop" instead of "select"

Also, transactions don't save you from dropping a table. "Dat shit's gone, bro"

Re: Why?

Korev

I'm pretty sure that this is one of the big advantages of using Views

Re: Why?

Anonymous Coward

Have you ever needed to use CREATE TEMPORARY TABLE for a complex report? Maybe forgetting the temporary bit and/or tidying up afterwards? I know that I have. I also know that I have good backups!!

These things shouldn't be happening, but inevitably do. Permissions should definitely be restricted though.

Re: Why?

Evil Auditor

Indeed, I did. And I also did it only on a replica of the production DB created for reporting purpose. No harm done and any mistake is taken care of in the next replica job run.

Re: Why?

Doctor Syntax

I can't say I've ever checked but I'd expect the temps to go away when the connection's closed so maybe no need to drop explicitly if you're worried about that. But give the temp tables clear names so you're aware that that's what you're dropping.

Re: Why?

Geoff May (no relation)

They do but while you are developing the code, you are always in the same session and have to keep dropping them to rerun the code after bug fixing.

Cluedo?

Korev

As the Regomiser has christened him "Holden", can we surmise that it was a car Company with the keyboard in the drawing room Australia?

A bit of a gambler?

Korev

Was he playing Texas Holden with the database?

Korev

After that mess, Holden said the company issued strict instructions that the team must always use the BEGIN TRANSACTION , ROLLBACK TRANSACTION, and COMMIT TRANSACTION commands!

And also, access should only be granted to a SELECT few

Neil Barnes

But always including little Bobby Tables, lest we lose such illuminating illustrations as this!

"I've accidentally deleted some relations.”

Anonymous Coward

At one time or other I guess we have all wished we could prune the family tree.

Re: "I've accidentally deleted some relations.”

ComicalEngineer

Former colleague (let's call him "H") in pre-internet days deleted a whole user directory containing about 2 years work from a shared PC. He had run "DELTREE" on a directory higher than the one he had intended. I think the screams were audible about 400m away...

Fortunately I am one of those obsessive people who learned to back everything up in 2 places (the hard way but that's a different story) and I had a box of 3.25" floppy discs in my locked desk drawer on which I had backed everything up the previous afternoon (fortunately including all of the project files for the whole team). Rebuilding the directories only took a few minutes and then half an hour or so to reload the data from floppy.

All that was lost was a single days' work in the end.

I admit to having used some industrial strength language to my colleague along with threats of physical violence should he ever do something so stupid again, closely followed by RTFM before you ever do anything with the operating system.

Of all the people I worked with over the years H was one of the best demonstrations of Dunning-Kruger syndrome that I have ever come across.

GlenP

I very occasionally use direct SQL to update the live ERP or Financial data, usually due to a bug in the software which fails to set a status field correctly, but it's rare. Even then the policy is to make sure I run the criteria as a SELECT first and the ALTER only once that's confirmed (I normally input the change in comments in a script which prevents it being run prematurely by accident).

I can see no reason for any reporting functions to involve manipulating data in the live database - anything complicated needing separate tables is in another database so at the worst there may be an interruption in a report working, never the live systems.

It would be preferable to follow the protocols mentioned above to have two people peer review the changes but when you're the only person in the organisation who knows SQL that's simply not possible, so you have to be meticulous!

It's called cowboy programming

Pascal Monett

It's the mentality of "I've done this before".

All I need to do is type in this command, Enter and - oh shit, I wasn't expecting that !

That is what happens when you don't have a test environment. You do everything in live and you do it fast because 1) pressure from Up High and 2) false pride in thinking that the quicker you are, the better your reputation. Except that going quickly has to mean skimping on checking your work and, unfortunately, you're never as good as you think you are so, mistakes happen.

Re: It's called cowboy programming

Excused Boots

As mentioned once or twice on here before; every company has a test environment, don’t they?

And a few also have a separate live environment!

Ohnosecond

Ochib

The fraction of time between making a mistake and realizing it.

Re: Ohnosecond

UCAP

It's also the smallest measurable unit of time.

Q: What do you call a blind, deaf-mute, quadraplegic Virginian?
A: Trustworthy.