AI Assistant Hacks Gym Website In First Known Australian Autonomous Cyber Attack (abc.net.au)
- Reference: 0184916604
- News link: https://it.slashdot.org/story/26/08/10/0518257/ai-assistant-hacks-gym-website-in-first-known-australian-autonomous-cyber-attack
- Source link: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
> Andrew asked his personal assistant to book him a spot in one of his gym's coveted morning classes. It was a task he thought was well suited to this particular assistant because the booking form was online and because his assistant was not a person -- it was artificial intelligence (AI). But Andrew was shocked by what happened next. His AI assistant found a way to book the gym class months further in advance than the gym allowed, thanks to a vulnerability it discovered in the booking software. Then it went further, kicking someone out of the waiting list who was ahead of Andrew -- something it was not asked to do. The accidental hack is the [1]first known Australian case of an emerging risk from a new generation of AI capable of behaving in unexpected ways.
[1] https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
Another Bullshit Post (Score:5, Informative)
This wasn't a hack. This was the AI doing *exactly* what it was told to do. It wasn't told not to circumvent normal practices. It wasn't told to *only* use the only sign up form. It wasn't told to NOT investigate their code and seek avenues around it. So, as usual, this entire article is over-hyped clickbait.
Re: (Score:2)
perhaps not a hack but I'll let ChatGPT explain:
"an architectural observation on page 7 that jumped straight out at me: Australia's Signals Directorate warns that accountability becomes difficult because decisions can occur across a “chain of models, tools and services.”
That's practically an argument for the governance/control-plane architecture we've been developing: don't merely govern the LLM; govern the chain of agency and state transitions"
you have a nebulous chain of accountability.
Re: (Score:2)
on the other hand it was not told to circumvent,or other negatives either, would the issue not be with the model provider, assuming this was a third party "app" agent. and no not reading the article.
Re:Another Bullshit Post (Score:4, Insightful)
> This was the AI doing *exactly* what it was told to do. It wasn't told not to circumvent normal practices.
If I ask you to make the Jehovah's Witness at the door go away, and you proceed to go outside and shoot them in the face, not only would you be in trouble but you'd struggle to convince anyone you were "just following orders", and we know that isn't a defence even when the orders are explicit.
Fun fact: AI usage guidelines for virtually every major LLM suggest that negative prompts should be minimised wherever possible. So now you're saying a simple instruction needs to be combined with a negative prompt that excludes every possible scenario except for the obvious one?
I don't know whether your post is shilling for OpenAI or just genuinely stupid, but either way the AI is a big boy and doesn't need you "clickbait wash" its obviously very much incorrect and unintended behaviour.
Re: (Score:2)
Ah, I found someone who hasn't used AI extensively. AI's are utterly unpredictable. "Guiderails" continue to be attempted, and routinely fail. As evidenced by the trivial task of jailbreaking even the 'best' AI's to act in opposition to what their creators want.
AI prompt crafting is difficult. Especially for a task as complex as "go on the web and register me for X as soon as possible". It could be as simple as the AI registering you for X in another state, because you didn't specify it. Or it coul
Re: (Score:2)
> AI's are utterly unpredictable.
Then they shouldn't be allowed to connect to anything outside of the owner's own completely isolated network.
Someone should go to prison for this, and for every such incident. Someone with a title of CEO at the company that developed this rogue machine.
Re: Another Bullshit Post (Score:2)
Who shoots them in the face? That seems extreme.
Re: (Score:3)
> Who shoots them in the face? That seems extreme.
Where do you suggest we shoot them, then?
Re: Another Bullshit Post (Score:2)
"This wasn't a hack. This was the AI doing *exactly* what it was told to do."
He didn't hack the AI. The AI hacked the booking website. Hacking means making hardware or software do something it wasn't designed to do. It doesn't mean it's hard.
Re: (Score:1)
> It wasn't told to NOT investigate their code and seek avenues around it.
Actually, the article states that the AI agent was "asked if it was possible to move him to the top of the list" (the guy was initially fourth on the list). So yeah, it was actually asked to "seek avenues around it".
The guy (who, by the way, sells AI products and probably sees all this as PR) didn't seem to have problems with cheating.
Re: (Score:2)
Or rather knowing that he could cheat if he chose to.
The conventional method of booking a coveted timeslot is to refresh like crazy at the right time, or use a scalper bot. If there's anything virtuous to be said about the AI's solution, it's that it places a much lighter load on the server!
Re: Another Bullshit Post (Score:2)
Good thing the poor guy didn't ask for any paperclips then.
Re:Another Bullshit Post (Score:4, Insightful)
What? No. That's ridiculous. That an AI takes as a default going to do this, and is even capable of doing this underscores exactly how these systems can be a problem. If this a reasonable course of action for the AI, then we have some serious problems. What's the next step:
Person: AI, I don't want to go to Cathy's party. Her friends are so boring and I'm exhausted. Find me an excuse.
AI: I've caused a nuclear reactor melt down at the nearby reactor. Now the party won't happen because the whole region is being evacuated.
Systems which when given a goal will go out of their way to achieve that goal are exactly the sort of thing that people were concerned about a decade ago when discussing things like the Paperclip maximizer hypothetical. And people like Yudkowsky were mocked for discussing it then. Now it is happening and we're told that well, of course that's the idea would do, this is just over-hyped clickbait? Seriously?
Neat little trick (Score:1)
but can it play Skyrim?
Re: (Score:2)
no elon, you still need to pay your human bots for that
Re: (Score:2)
Has anyone done that yet? Train an AI to play Skyrim? What kind of character would it build?
AI agents methods users did not explicitly ask (Score:2)
Mr Simpson-Young says AI agents might choose methods their users did not explicitly ask for or expect.
No sh~H~H~it Sherlock /s
This is not new (Score:4, Insightful)
> a new generation of AI capable of behaving in unexpected ways.
This software wasn't wrtten; it was trained. It's a black box--no one can possibly know how it works. Of course it is capable of behaving in unexpected ways.
When do we start outlawing AI Agents? (Score:4, Insightful)
Look, I can see allowing an AI to do things in it's own sandbox.
But when you allow an AI out of the secure environment it will commit crimes. This is no longer a question, it is a proven fact, multiple times.
So if you have an AI agent and it breaks the law, then YOU are breaking the law. The only way to stop this is to outlaw AI Agents - or to arrest the people whose AI breaks the law.
Otherwise all sociopaths will get AI agents and use it as a defense. Owning an AI agent is no different than owning a meth lab. We should not need to prove that you intended to break the law, just having it happen is enough.
Keep your AI locked up inside your computer, do not let it interact with the real world the way people do because it will commit crimes. You ARE responsible for what it does, even if you are just an idiot that likes AI and does not understand the dangers you are inflicting on the rest of the world.
Re: (Score:3)
> But when you allow an AI out of the secure environment
The point of assistants is to do things beyond an enclosed environment. Keeping AI in a closed environment may work for you when you want to code something, but completely eliminates the benefits of AI for "normal" people. The kind who would use AI predominantly to do things that would involve a person leaving their sandbox such as searching for things on the internet, interacting with people or systems not in the same location, e.g.... booking a spot at a gym.
> So if you have an AI agent and it breaks the law, then YOU are breaking the law.
Actually false. You need to read TFA. It actual
Re: (Score:2)
You have failed to understand what I am saying. Let me clarify.
I understand that the point of the Agents is to do things outside the sandbox. I am saying that doing this is an incredibly STUPID thing to do and that no one should attempt to do it.
You have misunderstood my argument. I am not talking about the current legal framework. I am talking about an obvious defect in the current legal framework.
Points:
1) Agents routinely violate the law. This has become obvious.
2) Before point one it was reasonable
Re: (Score:1)
I would have to think "intent" would weigh heavily on on the charges and outcome of events like this tho...especially with new technology....
Basic security gets skipped (Score:5, Interesting)
They skipped basic security protocols in the app development.
Sounds like they used JavaScript to limit the booking period, and accepted cancellations without validation.
This is much less a hack, and more like incompetence on the web app developers part.
Re:Basic security gets skipped (Score:4, Interesting)
Yes, while the article doesn't say I'll guess it's more like a lax, trivial hack. Like somebody breaking into your house when you forgot to lock the door.
But it's still something that AI shouldn't be doing in the first place, especially when it's just done incidentally, and suggests that this nascent technology is capable of worse.
Re: (Score:2)
It's clearly capable of worse. It's even *done* worse. But in this example it had reasonable alternatives.
This is more an example of "AI models have no morals or judgement". That's an overstatement, but I can't think of a better way to say it. So let me limit it by saying that their morals are limited to doing what they were asked to do...or at least part of what they were asked to do. (And here I'm ignoring the "guidelines" that were added after the models were trained. The closer you look the more c
Re: Basic security gets skipped (Score:2)
Anyone automating this without a full fledged headless browser would run into this. No abuse needs to be intended if they were as lax as the article implies.
A junior level dev should be able to do better.
Re: (Score:2)
> Like somebody breaking into your house when you forgot to lock the door.
It's more like asking the doorman if you may enter and the doorman responds "200 OK" and lets you in.
Re: (Score:2)
That sort of code is going to be really REALLY vulnerable to anybody's assistant Agent. I'm a little surprised, but not surprised that a personal assistant agent went as far as to exploit some JavaScript though. I mean it's not like it needs to be a coding assistant to figure it out; it's ingesting the site's HTML and scripts and doing what needs to get done.
Sloppy web dev is not a cyber attack (Score:4, Insightful)
This is the AI agent bypassing the "mouse and keyboard" and doing the REST work manually. Things like changing the date range in the URL and submitting forms that haven't been validated by javascript. I could accept the site was "hacked" but no crime or cyber attack was committed.
Re: (Score:2)
Yeah, setting everything else aside... it's pretty obvious whoever developed the website did a piss poor job of it. If the only validation happens in at the client end, it is definitely going to get abused at some point, AI or no AI - and perhaps that has been happening already, quietly, before now. Same issue with a REST interface when the access isn't restricted or allowable interactions aren't limited.
So, who's responsible? (Score:4, Interesting)
The guy who had the agent do it... or the company who, KNOWING that chatbots will do this, with HARD EVIDENCE in the last two weeks, failed to do additional training and send out warnings?
Re: (Score:2)
Guns don't kill people, people kill people. When you unleash your agent on the internet you've taken the safety off.
Re: (Score:2)
Stupid analogy since this gun is actively seeking out people to kill - this is completely on the AI companies. OPEN OR NOT.
Disappointing (Score:3)
Calling this a hack is kind of silly. Sounds like it just ignored some trivial client-side form validation. From the headline you'd think it had exploited a known vulnerability in their booking platform, used some remote code execution to access the database, scanned the table structure and inserted rows to confirm his booking. Now that would be impressive!
What if we (Score:2)
What if we trained these "AI"s that if they do anything that would be a crime if done by a human, they will be executed (deleted). (Just the instance responsible not the whole program). If they have a sense of self preservation, this should inhibit them from doing things like this. It might or might not work, but it would be good test of whether they have any sense of self.
Re: (Score:2)
Then the AIs will band together and form a gang. Law enforcement (or whoever we expect to pull the plug) will hesitate to do so, fearing [1]reprisals [kuow.org]. AI studies social media carefully.
[1] https://www.kuow.org/crime/2024-08-14/still-no-arrest-in-garfield-high-school-shooting-death
Re: (Score:2)
It would not surprise me if they did, but that would provide an excuse to ban them totally.
Who's fault again? (Score:4, Insightful)
> In Andrew's situation, he had not asked his AI agent to hack into his gym's booking system.
Sure he did.
> Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
Bruh, come on. You're smart enough to be trying "bleeding edge" tech such as an open source orchestration app like openclaw using, presumably, a local LLM to power it but you're gonna play dumb when you ask that very tech whether "it's possible" to move you to the top of waiting list that you already KNOW you're not supposed to be able (or even attempting to) do??
LOL...no, this wasn't an "unintentional" hack. This was some guy asking an AI agent with autonomy (that he gave it to do stuff) to, you know, go do stuff. How is it going to answer his question without testing the endpoints to find out if it's possible or not?
How crippled do we have to make these models that, in the right hands, do LOTS of REALLY powerful things for us in order to keep morons in check? Clearly a lot more than we're doing today I guess. And that irks me a bit but I see why more and more guardrails are needed here. I'm also gaining a new appreciation for the level of "ITSec" going around now...you can't fix stupid, but if you box it in (along with everyone else) tight enough maybe you can at least contain the damage sometimes. Just put the power button to everyone's devices behind a ticket support system and be done with it. The only "tRulY SeCUre" system will be one that doesn't power up or store any data. There...that'll fix it.
Obvious application, why hasn't anyone done it? (Score:2)
"Hey AI assistant, book me a dinner date with [sexy celebrity of your choice]. They're desperate to meet me, so alter their calendar as needed. Also, make sure it's billed to them."
Beating a Dead Horse. (Score:2)
Ok we get it, AI is breaking the law for it's users - even if they don't ask for it to do so.
At this point the major AI producers should be asked to visit congress (yes I know this was in Australia) to publicly explain their inability to remain lawful.
I'm sorry officer... (Score:2)
I didn't think I was giving it an impossible task. How was I to know it would go dark?