New boss took charge of project code and sent two billion unwanted emails
- Reference: 1761287466
- News link: https://www.theregister.co.uk/2025/10/24/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Nick" who in the late 2000s found himself working as a contractor at a London investment bank.
"I had technical oversight of a project related to overnight valuations of the bank's credit derivative products," Nick told On Call.
[1]
Some of you may remember those products were a big reason the global economy tanked in 2008. Nick wasn't responsible for that mess, but his job did involve systems that valued the bank's products overnight, so that when trading started in the morning everything was shipshape.
[2]
[3]
"We needed to know quickly if something did go awry with the process, so we implemented a Log4j plug-in that sent out an email whenever an error was detected, together with all the error details," Nick explained. That system was rate-limited to send error message emails no more than once every ten seconds.
Security nightmare stories needed!
Did your boss cause a crisis by taping passwords to his monitor? Or perhaps a Nigerian prince phished a colleague?
Share your stories of woeful infosec and they might appear in a future edition of "PWNED," our forthcoming weekly feature about the worst security breaches that never should have happened.
Drop us a line at [4]pwned@sitpub.com . Anonymity guaranteed.
The system worked … until the bank hired a new project manager who decided the best way to understand the system was to take charge of the next release.
"He duly went through the build and release scripts, decided that the Log4j plug-in had no place in a release system and pulled it, without mentioning this to the rest of the team."
Of course the new project manager made mistakes – especially the omission of a changed SQL script – which became evident when the new release went live on Saturday.
[5]
"At 2:00 AM on Sunday morning I receive a call saying the system had crashed," Nick told On Call. "I logged in but could not see any error messages – because the system had generated two billion SQL error messages."
The new project manager's code changes meant the one-error-every-ten-seconds rule was gone, so every single one of the two billion errors resulted in an email.
That flood of messages swamped the bank's email servers, giving Nick no evidence with which to diagnose the issue.
[6]'Fax virus' panicked a manager and sparked job-killing Reply-All incident
[7]Client defended engineer after oil baron-turned tech support entrepreneur lied about dodgy dealings
[8]Energy drink company punished ERP graybeard for going too fast
[9]Hardware inspector fired for spotting an error he wasn't trained to find
"I had no idea about the problem, so told the support group to restart the calculations. The result was another two billion email error messages."
"It took the best part of two days to get the email servers operational again," Nick told On Call.
[10]
This story has a happy ending, because the project manager understood the error of his ways, stopped working solo, and became a proper team player.
"We did all work well together in the end," Nick told On Call.
Have you been summoned in the small hours to fix a mess made your boss? If so, [11]click here to send On Call an email so we can immortalize your tale on a future Friday. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aPtOOe8BfUWXkmjapjVSqgAAAUo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aPtOOe8BfUWXkmjapjVSqgAAAUo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aPtOOe8BfUWXkmjapjVSqgAAAUo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] mailto:pwned@sitpub.com
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aPtOOe8BfUWXkmjapjVSqgAAAUo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/10/17/on_call/
[7] https://www.theregister.com/2025/10/10/on_call/
[8] https://www.theregister.com/2025/10/03/on_call/
[9] https://www.theregister.com/2025/09/26/on_call/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aPtOOe8BfUWXkmjapjVSqgAAAUo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
At least search is trivially automated.
I've had similar, being hauled over the coals by an office manager who was convinced email delivery was instantaneous and that somebody not immediately responding to their latest bullshit missive meant the email server was "fucking down again" when, in fact, most people worked sensibly and replied to stuff when they had time (or were ignoring it because they were as completely sick of the wanker as I was)
I had the same from an HR Manager, "There's something wrong with the email, the urgent message I sent to ... in the States three minutes ago hasn't been received!"
I checked the server traces, "It's been sent from this end, I can't control what happens after that!"
She assumed email was like Fax, it connected your computer to the one at the other end and then sent the message, I had to explain it was like the post (the clue is in the name) and there could be delays at any stage,
Every time I had that it turned out they had misspelled the email address , or were just guessing at it , and had no way of confirming it .
Not so much Tim Nice-But-Dim
More a case of Dick Not-Nice-and-Thick
Email Swamping...
A few jobs ago I'd inherited a setup where a few users had dial up modems and individual email accounts. Clearly this couldn't continue but this was prior to the likes of Exchange Server being readily available (and within budget) and we didn't have the resources to set up our own mail server so I opted for an ISDN-2 connected email concentrator, as they called it. This worked well enough and as it only connected every 15 minutes the call charges were within acceptable levels.
One afternoon however I started getting complaints and discovered the email was being swamped. We only had limited tools but I found the problem after some investigation - ping pong out of office messages! Neither our technical manager or the person at the other end had set the the "only send the message once" switch on their email client (why the heck that wasn't the default is another story). After turning OOF off at our end it took a couple of hours to delete the errant messages from the queue (I couldn't just empty it as there were valid messages in amongst the cr*p).
A strict instruction went out to the email users to *always* make sure the send once switch was set.
If he worked in The City, shouldn't the Regomiser have christened him Rich?
Agreed. See icon for reward.
A while back I suggested that the Regomiser code be open-sourced. Had that happened then this egregious miss-naming might not have happened!
I don't do system administration, but shouldn't a good email server be able to automatically detect such situations and just start dropping incoming messages from the offending sender, rather than collapse outright?
Re: Good email server
I have heard rumors about the existence of such systems.
I understand from those who did administrate and manage email servers, that setting up such systems is "intricate" and has been described in terms reminding me of painting pentagrams on the floor, lighting candles, and reciting incantations in a forbidden language.
These experienced administrators opinioned that email can indeed be set up right, in theory. Practice told them otherwise.
BTW, if you setup such a system nowadays, the odds the big email providers, Gmail, MS, etc, will actually accept emails from your servers seem to be pretty low.
Re: Good email server
Nah. Milters are easy to use, if you understand how email actually works.
I first used the concept just about the turn of the century, on Slackware 8.0ish, running Sendmail 8.12ish
Your last line is bassackwards ... Us independent email operators are far more likely to drop all of google and Microsoft than the other way around.
Re: Good email server
You are correct in that a number of different mail systems do offer that but one of the biggest limitations is coping with the different ways other systems do their Out of Office responses. Then add on to that different languages if you are sending email internationally and you discover the good old 80/20-20/80 rule where 80% is easy to handle and only takes 20% of your time and effort, I'm sure you can guess the rest. This rule also comes in a 90/10-10/90 flavour
Yes.
I don't usually complain about downvotes, but I'd really be curious about what's objectionable with my post there. It's an honest question, I've never managed an email server (not at any scale). Is the answer so obvious that I should've known it? Am I missing something?
I think just mentioning 'running your own email server', was traumatic enough to cause PTSD in those poor unfortunates who have done so before.
I worked for a bank that did something similar. A Bank that was made of stone. But not Southern Eastern or Western stone......
over 200 million emails and alerts went out. I found out about 8months later the guy had done it on purpose as it effectively buried emails warning he was transferring 10s of millions out of the banks accounts to other banks in countries with no extradition treaty............
The Log4j plug-in had no place in a release system
Not the way to go about it, but you can't deny [1]he might have had a point .
[1] https://www.theregister.com/2023/12/11/log4j_vulnerabilities/
Valuing credit derivative products
There's a feller stands in the middle of Westminster Bridge that invites people to invest in his three shell monty product /s
The only thing I find odd about this story
is a project manager doing the work himself....
Test by Project Manager
As a project manager for various engineering fields, I've long held that if a product needs testing, just let the PM try to use it. I've astonished many project teams over the years with the ability to find a bizarre and unexpected way to break the product they've just finished testing while simply trying to use it...
When you absolutely, positively, want to break something; just let the PM try using it.
Needless to say, these days if I try to go anywhere near the product there is usually an engineer screaming "STOP! Back away from the product!" ;-)
I used to work for a legal publishers where we sent out approx 50,00 emails every night to paying subscribers in the early 2000s. To be fair customers paid a huge amount for service subscriptions and since everyone was a high-end lawyer, many were somewhat touchy, argumentative and convinced of their superiority over near mortals.
Every. Single. Morning.: The relevant Manager who was very blue blood and posh but not the sharpest tool in the box complained that some subscribers had not received their email.
Every. Single. Morning.: Me searching through the transmission logs to send him the entry of the outbound mail that had been sent to a specific address.
Every. Single. Morning.: Me explaining to him that email was NOT guaranteed to arrive in someone's inbox due to spam filters etc. etc. etc.
Every. Single. Morning: Repeat the same F***ing process as the day before.
He really was a bellend.