AWS customer faces staggering charges over S3 bucket misfire
- Reference: 1714581011
- News link: https://www.theregister.co.uk/2024/05/01/aws_s3_bucket_abuse/
- Source link:
Amazon's Simple Storage Service ( [1]S3 ) was the first and one of the most widely used of the cloudy giant's online services, and also regularly crops up in the news because of breaches caused by [2]poorly configured security settings .
This latest incident also stemmed from misconfiguration, but not of S3 itself; the service was performing exactly as it was designed.
[3]
In an article [4]posted on Medium this week, a software engineer complained that an S3 bucket he created as part of a proof-of-concept had managed to run up charges of over $1,300 in a single day. A check of the AWS billing console showed that the cause was nearly 100 million PUT requests to add data to the bucket, he said.
[5]
[6]
Maciej Pocwierz, a senior software engineer at Warsaw-based cloud services company [7]Semantive , writes that he created a single S3 bucket in Amazon's eu-west-1 region and uploaded some files there for testing. Two days later, he checked the billing page to make sure this was still within the free-tier limits and discovered the charges.
The source of all the PUT requests, according to Pocwierz, is a popular open source tool that he doesn't identify. This tool stores backup data in S3 by default, and the placeholder bucket name it uses just happens to be identical to the one that he chose for his project.
[8]
Where this becomes a problem – apart from your bucket filling up with other people's data if those PUT requests were successful – is that Amazon charges for unauthorized incoming requests. He claims this was confirmed by AWS in exchanges he had with its support team regarding the matter.
Standard S3 PUT requests are priced at just $0.005 per 1,000 requests, which may seem like a trifling amount, but Pocwierz points out that a single machine can easily execute thousands of such requests per second.
To demonstrate the security implications of this, Pocwierz said that he opened up his S3 bucket for public writes, and in less than 30 seconds it amassed over 10 GB of data from numerous sources.
[9]
That's 10 GB of data that the owners are likely to be completely unaware was being exfiltrated to a random S3 bucket by some open source tool they are using, all because they didn't configure its backup function.
[10]AWS hits $100B revenue run rate, expands margins, delivers most of Amazon's profit
[11]Elon Musk's latest brainfart is to turn Tesla cars into AWS on wheels
[12]AWS must pay $525M to cloud storage patent holder, says jury
[13]US-EAST-1 region is not the cloudy crock it's made out to be, claims AWS EC2 boss
But it didn't take long for this complaint to get noticed, especially when people started posting links to the Medium article on Twitter. In response, AWS chief evangelist Jeff Barr indicated in a [14]tweet that company would do something about the situation:
Thank you to everyone who brought this article to our attention. We agree that customers should not have to pay for unauthorized requests that they did not initiate. We'll have more to share on exactly how we'll help prevent these charges shortly.
We asked AWS for an official statement on this, but the company declined to say anything beyond Jeff Barr's message.
Pocwierz said he informed the maintainers of the open source tool about the issue and that they have fixed it in the code, but this doesn't fix the many instances of the tool that are still running in the wild.
The takeaway is that anyone who knows the name of an S3 bucket can send it PUT requests, and potentially rack up massive charges for the AWS account that owns it.
Until AWS comes up with a fix, customers will have to attempt to alleviate this risk by avoiding short or common names for S3 buckets, and making them less easy to guess by adding random characters. ®
Get our [15]Tech Resources
[1] https://aws.amazon.com/s3/
[2] https://www.theregister.com/2022/12/14/aws_simple_storage_service_simplified/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZjK7fxNmhsjJFw53lGnPzAAAAAc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjK7fxNmhsjJFw53lGnPzAAAAAc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZjK7fxNmhsjJFw53lGnPzAAAAAc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.semantive.com/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjK7fxNmhsjJFw53lGnPzAAAAAc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZjK7fxNmhsjJFw53lGnPzAAAAAc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2024/05/01/amazon_q1_2024/
[11] https://www.theregister.com/2024/04/30/tesla_ai_workloads/
[12] https://www.theregister.com/2024/04/11/aws_lawsuit_kove_io/
[13] https://www.theregister.com/2024/04/10/aws_dave_brown_ec2_futures/
[14] https://twitter.com/jeffbarr/status/1785386554372042890
[15] https://whitepapers.theregister.com/
Re: This is just one example
I'd say most businesses don't even need AWS. Computers are so powerful now, most could get away with a single dedicated server (and an extra for redundancy) with an unmetered connection. This cloud nonsense is so unnecessary.
Re: This is just one example
Tell me you've never worked at a large scale without telling me you've never worked at a large scale :D
Re: This is just one example
A single servers will not scale hugely, and lots of people needs more, but the word used in that comment was "most".
Most organisations are not Google or FB. Most businesses are SMEs.
Re: This is just one example
yes but for smaller customers there are probably more gains to be had. A number of my customers host web apps on azure for less than $15 per month, and they don't have to employ an admin to configure firewalls or do windows updates
Re: This is just one example
TBF, the majority of businesses probably are not working at a large scale. In this particular case, it was a proof of concept.
And unless cloud providers are going to exist solely for large scale operations, they need to fix this.
Indeed, if they do exist solely for large scale operations they need to fix it because the current evidence is that even those customers are over-provisioning and being overcharged as a result. That can't persist if there is genuine competition between providers. And they're going to run out of large scale customers if they don't provide a conducive environment for today's small scale customers who may happen to have a growth trajectory.
Re: This is just one example
"enterprise" customers massively over-provisioning is a key element of the cloud business model.
Re: This is just one example
It was a long time ago (about 2016 I think) I was dealing with AWS on a genuinely enterprise scale, but back then they were very good at coming in and helping find and fix stupid overprovisioning. I was pretty impressed with their "what's good for the customer is long term good for us" attitude. It may have changed since.
Regardless, I've little sympathy for enterprises. Cloud providers give you tons of MI, policy tooling and so on to enable you to keep control of your cloud estate. If you want, there are even better third party tools. They just doesn't get used very much/well. The point around hard spending limits is valid at the small scale, but for most enterprises monitoring rather than blocking is fine.
I'm cynical, but the way many enterprise IT organisations seem to do on prem capacity management is this: 1/ Make it generally slow and hard to provision new services 2/ suddenly realise that the datacentre/big expensive SAN/core network switch etc. is almost full. 3/ only provision new things on an emergency exception basis while the capacity issue is fixed (which takes ages). At no point does a serious attempt at decommissioning, consolidating or optiimising old services take place.
That approach makes stupid capacity management decisions slowly, at the expense of capacity not being available when needed. Moving to Cloud lifts the constraints and in the absence of proper policies, stupid capacity management decisions can now be made really fast, via API call. On the other hand, the capacity is there when you need it.
But it doesn't have to be this way, either on prem or on the cloud.
Re: This is just one example
Poster wrote most businesses
That does some heavy lifting here. But so does worked at a large scale
Both statements need some defining.
Re: This is just one example
Most normal non-tech companies with that scale of IT requirement don’t run servers themselves at all - they either started SaaS native or moved to SaaS years ago. It’s generally not economical to pay two FTEs (to cover leave and illness) to look after one server. Instead they buy apps on a £/user/month basis on someone else’s multi tenanted service, and tie everything together with nasty spreadsheets.
They won’t use AWS either - still needs IT admins / “devops engineers” etc.. They don’t have or want them.
18 years
Is this a reheated old story?
Using S3 on AWS is like leaving your stuffed wallet up on the street ripe for easy pickings.
It's meant for use by VC backed happy clappy businesses with more money than brain cells, not actual, real businesses.
Re: 18 years
We use cloud and we have 1 guys taking an axe to usage full time.
A lawyer and finance guys, 3 guys in upper management to fight the supplier on contracts and not forcing to buy service and resources we don't need every year.
We fly the guy with axe out to the US every year, cost us a fortune, mainly because he has a new axe when he's gets of the plane.
Works out cheaper then just "letting thing be"
Probably because of the axe.
An immoral path to profit
I think Amazon will need to stop charging for unsolicited PUT requests or come up with some other solution before the script kiddies start hammering random buckets, rogues start hitting specific targets.
It seems a more effective modern day equivalent of sending postcards to businesses who have a Freepost address so I am sure it will be exploited now it's known.
Though why anyone would sign-up to any service which makes you pay for people trying to kick your door in, intentional or not, is beyond me.
Re: An immoral path to profit
DDOS is so old school
Anonymous LEOPRD tool incoming
(Low Earth Orbit Put Request Deployment)
Posting Anonymously obviously
There's this thing called passwords
There's this thing called passwords. I know Google, and Amazon, and basically all these services have fancy OpenID based authetnication methods (or probably some non-standard Active Directory based thing for Windows), two factor authetnication in some cases, and so on. But I'm surprised S3 can be set up at all without a mandatory password. Obviously, having a hard-coded bucket name and password in your program is not terribly secure; but it's infinitely more secure than just having a bucket name then any rando on the planet can access your bucket.
Good on Amazon for deciding they'd refund for this. But, to be honest, I don't know if I could even call these accesses unauthorized insofar as the bucket was set to not require any authorization to access.
I will note, when I was in like junior high, I had words with the local authorities about this -- I got knicked for unauthorized access to a computer system. I pointed out I dialed this telephone number, it answered and did not ask for a password so I assumed no authorization was required. This was a terminal server, so I used it for some internet access, not poking around somebody's files. It made the police REALLY nervous when they handed me a sheet of paper to sign away my rights to an attorney and I said "I'll talk to you but I'm not signing this." They had my parents sign something to the effect that I'd refused to sign (I don't think they read it, so they probably signed away rights they were unaware of. No matter though.) They got even more nervous when I asked what I was being charged with, I pointed out I had definitely not met the requirements for the Computer Fraud and Abuse Act of 1986 (and at that point there were no newer laws on the books yet). As a bonus, their "computer guy" was out that day! When I described how a web browser worked, they said with a straight face I should get prior permission from each and every web site owner before I load a web page from their site. LOL. From their followup questions it was clear they had bigger problems, someone was in their systems trying to crack into military targets. I pointed out "I didn't do that". One of their computers off in the corner spontaneously blue screened, so I pointed over there and said "I didn't do that either". LOL. This was in fact around the time "The Cuckoo's Nest" covered so maybe it was the same guy poking through their systems. Since they couldn't figure out what to do they ended up deciding to fine me $50 (I'm not sure what specifically for, but OK I guess).
"To demonstrate the security implications of this, Pocwierz said that he opened up his S3 bucket for public writes, and in less than 30 seconds it amassed over 10 GB of data from numerous sources."
So, you open your bucket to the public and you expected what exactly? You use unnamed open source tools and they rack up writes? While I might agree with the injustice of AWS charges, the incompetence of their users is also a significant point of concern.
This is just one example
There are so many ways in which cloud service customers can rack up unexpected charges and the whole problem needs a comprehensive solution.
As far as I'm aware none of the big players allow you to set an expenditure limit that they will actually adhere to. They may agree to warn you at some point when you've exceeded some threshold, but quite possibly not until the threshold is dust in the rear view mirror and you're facing eye-watering bills.
The fact that you (usually) need to hand over you card details even for the "free tier" should be enough to send prospective customers fleeing for the hills, but clearly isn't.