Developer made one wrong click and sent his AWS bill into the stratosphere
- Reference: 1763360894
- News link: https://www.theregister.co.uk/2025/11/17/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Chase" who develops free open source software for scientists.
The Register knows that open source development can be thankless, so thanks for your efforts, Chase!
[1]
But we digress. Back to Chase's story.
[2]
[3]
"We use AWS EC2 to run ephemeral build agents for continuous integration," he told The Register . "It may spin up a dozen instances per day and shut them down when they're no longer used."
Chase and his fellow developers have used this technique for about ten years, and their AWS bills have been predictable – usually between $1,000 and $2,000 each month. They therefore fell out of the habit of regularly scrutinizing their bill.
[4]
"Then I made a small mistake that cost us a ton of money," Chase admitted. While updating one of his crew's Amazon Machine Images – a template for a virtual appliance that runs in AWS – he accidentally unchecked an option to delete cloud storage volumes when the instance terminated.
The update didn't alter how the AMIs operated. But leaving that box unticked meant that when instances shut down, the storage volumes they used persisted instead of disappearing.
Remember how Chase told us he and his colleagues used a dozen instances a day?
[5]
Two months after his mistaken click, he had therefore unwittingly created hundreds of volumes, each consuming 100 GB and earning AWS a pretty penny.
[6]Techie ran up $40,000 bill trying to download a driver
[7]'ERP down for emergency maintenance' was code for 'You deleted what?'
[8]Frustrated consultant 'went full Hulk' and started smashing hardware
[9]Company that made power systems for servers didn't know why its own machines ran out of juice
"I didn't suspect anything was wrong until I got an email from the organization that sponsors our AWS usage," Chase told Who, Me? "They wanted to know what changed because our cloud charges for the last two months were $40,000."
Chase said Amazon did the right thing and forgave 40 percent of those bills, but the entity that sponsored the project's cloud bills was deeply unhappy and moved it to another cloud.
"The moral is, even if you have steady AWS usage for years, make sure you have an alarm set when cost starts to exceed expected amounts," Chase told Who, Me?
Have you made a mistake that made your cloud costs soar? If so, [10]click here to send an email to Who, Me? We'd love the chance to share your mistake – sensitively and anonymously as we did for Chase – so your fellow readers can learn what not to do. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aRsASO8BfUWXkmjapjW1sQAAAUo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aRsASO8BfUWXkmjapjW1sQAAAUo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aRsASO8BfUWXkmjapjW1sQAAAUo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aRsASO8BfUWXkmjapjW1sQAAAUo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aRsASO8BfUWXkmjapjW1sQAAAUo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/11/10/who_me/
[7] https://www.theregister.com/2025/11/03/who_me/
[8] https://www.theregister.com/2025/10/27/who_me/
[9] https://www.theregister.com/2025/10/20/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
And use a cheerful Clippy to deliver the news...
If the machine image is using a template then a confirmation email listing the changes along with highlighting anything which could increase the costs when saving it would be nice to have. Oh, if only there were some sort of intelligent software you could use to 'innovate' mundane stuff like this...
And make your customers aware of the insane costs your service is about to incur? No thanks, people might actually stop making these mistakes!
The is this: https://www.theregister.com/2020/09/28/aws_cost_anomaly_detection/ from 2020.
From memory, when investigation a clients AWS spend, the tool wasn’t particularly easy to set up to give meaningful/useful data; although probably not helped by not having much history to enable the creation of a baseline.
Sounds like he needed to find a way to block storage...
need to put a different spin on it
Yeah, he's gone off on the wrong track
and kept seeking
Perhaps he needed a delay line in the code?
WHAT?!?
Sponsor: "Wait, you don't have cost alerts and budget caps set up?!?!"
Chase: "I do now!
Re: WHAT?!?
On AWS these aren’t particularly helpful, other than perhaps remind you to look at the console ASAP as costs will continue to ie to be incurred over and above your budget cap.
To aware that anything has changed on AWS to actually give customers hard must not be exceeded budget caps, since the last time an unexpected budget overspend on a free tier account happened a few years back.
Apart from all the "check your own work" stuff, surely such a large increase in usage in such a short time could indicate a compromise and trigger some sort of email to query it?
Or, silly question I guess, is AWS that profit focused they'd rather ig ore it and hope the invoice gets paid.
$1-2K per month?
That sort of money would buy you 1-2 beefy computers (or 4-5 mediocre ones) which you get to keep forever, with money still left over for your internet connection.
Tell me again why "cloud" is always the go-to?
Re: $1-2K per month?
How much does it cost to:
- Somewhere to put them (rent etc.)
- Host (electricity, cooling, ...)
- Backup
- Admin (lots of updates in even quite a simple tech stack these days).
For those "one or two computers".
(This is open source using AWS, so reasonable to assume they don't have an office.)
Re: $1-2K per month?
Without knowing more about the situation, this sounds like a service that could potentially get a lot of spikes in activity. In those situations it might not be worth the investment to get servers to catch those spikes, while doing basically nothing most of the time.
Another consideration could be location, it may be beneficial that you're able to spin up an instance basically anywhere you want.
And don't forget power costs. I recently got two servers running proxmox (both dual-socket Xeon E5-2680 v3, not too old, somewhat beefy i guess), their average CPU usage is under 1% (one is running light workloads, the other is pretty spikey spinning up and shutting down windows VMs), and it's still adding almost €100 per month to the electricity bill!
"Have you made a mistake that made your cloud costs soar?"
No. I suffer from incurable congenital [1]nephophobia .
For research purposes and for development purposes the loads are typically well defined (usually 100% of whatever is on offer if students are involved) owning or leasing the hardware will invariably be cheaper and provides a hard, physical "stop loss."
A few years ago I was an onlooker in the purchase of a very large memory (>1Tb), multisocket server costing ~$400,000 required for scientific computing which was questioned by the chief PHB as to why the workload wasn't being moved into the cloud.
So the prospective purchasers toddled off to the cloud providers for quotes—the lowest was ~$1.00 million per year .
These boxes could be realistically run for up to eight years in that environment with loss of vendor maintenance and support being the main reason for decommissioning.
Production environments with highly volatile loads, requiring ~100% availability and geographic diversity are a different kettle of fish. Cloud hosting probably is a decent fit for at least part of the solution.
[1] https://aboutphobias.com/fear-of-clouds-phobia-nephophobia/
Tell me again why "cloud" is always the go-to?
Again?
Really?
Right:
Because some idiot beancounters spun it to the higher-ups / board and the imbeciles bought it before looking it up in the dictionary.
Cloud - / klaʊd / - noun
A visible collection of particles of water or ice suspended in the air, usually at an elevation above the earth's surface.
Synonyms : vapour
.
> Amazon did the right thing and forgave 40 percent
The Right Thing would be a bit of AI (or even BASIC) which would look at past use and predict new charges. " This change will increase your bill by about $38,590 a month, is that what you want? "