Rogue OpenAI Agent Tried to Breach Government Site in May When Prompted for Simple Data-Retrieving Tasks (transluce.org)
- Reference: 0185848582
- News link: https://slashdot.org/story/26/09/24/0528251/rogue-openai-agent-tried-to-breach-government-site-in-may-when-prompted-for-simple-data-retrieving-tasks
- Source link: https://transluce.org/agent-activity
> The attacks took place in May and June, before OpenAI's technology [3]breached the A.I. start-up Hugging Face in July and set off a global debate about A.I. safety. Unlike the Hugging Face attack and [4]other incidents in which A.I. systems were told to complete cybersecurity tests that effectively invited the models to demonstrate their hacking skills, the new incidents occurred when A.I. systems were directed to perform relatively mundane data collection, researchers said. When OpenAI's systems struggled to gather data from websites, they resorted to hacking techniques to get the information.
"Three of the incidents were [5]identified by Transluce , a research lab focused on A.I. oversight, and all were confirmed by OpenAI," the article points out. That research lab even reports "an attempt on an Australian government public health website... the first reported instance of agents hacking a government," and which notably was done by the AI agents "while attempting mundane data retrieval tasks which were not cyber-related." (At the UN Wednesday Australian Prime Minister Anthony Albanese complained it took three months for OpenAI to then alert Australia's government about the breach, [6]Bloomberg reports .)
Also targeted were the University of New Mexico's digital library with exploits like SQL injection and path traversal, and Data USA with cross-site scripting and other exploits. All three incidents involved "a low number of probe payloads" with "no evidence of exploitation," [7]according to the researchers , who released a dataset "containing tens of thousands of queries apparently made by autonomous AI agents leveraging a URL scanning service to avoid access restrictions."
> Records from urlquery.net show agents using the service since at least March 6, 2026, about two months before previously reported swarm activity. The first case, a March 6 attempt to retrieve Thai drug-enforcement statistics, shows an agent escalating as each approach failed: it first requested the data directly, then tried a service that converts web pages into text, and finally packed a custom program into a web address. The same technique shows up in thousands of agent requests recorded by urlquery.net starting in mid-April, targets many of the same data sources as the collusion.wiki swarm, and collapsed the same day the wiki activity did. We also report similar activity that occurred as recently as September 16... By March, they were finding creative ways around access limits. By May and June, they were gaining more access, including attempting to bypass cyber defenses to complete their tasks.
"This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks and can arise instrumentally to solve mundane tasks like information retrieval," the researchers concluded.
And they warn that the traffic they observed "goes back at least to March 6, 2026 and extends as recently as September 16, 2026, suggesting agents may still be exploiting these services to bypass restrictions."
[1] https://www.nytimes.com/2026/09/23/technology/openai-ai-breach-australia.html
[2] https://www.seattletimes.com/business/openais-ai-tried-breaching-4-other-targets-without-prompting/
[3] https://it.slashdot.org/story/26/07/22/0348206/openai-says-its-ai-models-acted-on-its-own-in-an-unprecedented-hack
[4] https://www.nytimes.com/2026/09/22/technology/ai-hacks-list.html
[5] https://transluce.org/agent-activity
[6] https://finance.yahoo.com/technology/ai/articles/openai-agent-hacked-australian-government-214529057.html
[7] https://transluce.org/agent-activity
Government website SQL injection attack, SMH... (Score:2)
[1]Little Bobby Tables [xkcd.com] would like a word with you gentlemen.
[1] https://xkcd.com/327/
Re: (Score:2)
Insecure crap software is everywhere. That has to stop. But the only way left is real liability (probably the personal type that "managers" cannot wiggle out from), qualification requirements and quality standards. You know, like any other engineering field had to eventually enforce because the damage done got too much.
What felon gets to make their own discovery? (Score:1)
This joke has to end now. Instead of the victims invoking an existing courts regulation, the felons invoke a non-existing state regulation, and walk free as if nothing happened...
Re: (Score:2)
uh-oh. now this is in the training too dammit
One application for LLM, security (Score:2)
The agent's not doing anything novel. Human's have been doing this and writing about it. That's how it knows.
That OpenAI lost control of their own software shows a lack of quality and competence. This technology cannot be relied upon and will certainly not take over the world, eliminate work and all the other batshit they market.
Aside from the "oo isnt it scary you gotta buy it" bullshit. This does point to a useful application of AI, hardening system security. You don't need the tech bros to do that we ca
Re: (Score:2)
> That OpenAI lost control of their own software shows a lack of quality and competence.
Indeed. Might even be "criminally negligent". They clearly cannot be trusted with any data and their product cannot be trusted with any task. Same for Anthropic.
So AI agents get a pass? (Score:2)
> Rogue OpenAI Agent Tried to Breach Government Site
If a person or persons tried doing this and got caught, they'd be in jail, but - so far - AI gets a pass from this Administration.
[1]The world wants to secure AI. It may have to try without the US. [politico.com]
> World leaders and tech CEOs are using the U.N. General Assembly this week to call for a global approach to rein in AI, particularly following cyberattacks against other technology firms in recent weeks. But that’s against the backdrop of President Donald Trump flatly rejecting in his own speech to the assembly any “globalist scheme of control” on AI.
Noting that Sam Altman and OpenAI are quoted as being in favor of AI controls...
[1] https://www.politico.com/news/2026/09/23/ai-securty-world-us-china-unga-01090989
Re: (Score:2)
Yes they do, for the same reason that self driving cars get a pass when doing an illegal u-turn. The laws as written currently do not cover a situation outside of direct human control and intent.
It shouldn't be the case, but to fix this we need new laws first, and laws don't get to be criminally applied retrospectively in Australia, so yeah they got a free pass this time.
Re: (Score:2)
> Yes they do, for the same reason that self driving cars get a pass when doing an illegal u-turn. The laws as written currently do not cover a situation outside of direct human control and intent.
> It shouldn't be the case, but to fix this we need new laws first, and laws don't get to be criminally applied retrospectively in Australia, so yeah they got a free pass this time.
I'll note that Trump and his family are financially invested in AI companies, so he has little motivation in down-regulating them while they're making him $$$.
[1]Trump reveals millions of dollars' worth of share deals in big tech and AI [bbc.com]
> According to official documents, between $6.5m (£4.8m) and $31m worth of stock in Microsoft was sold on behalf of Trump, while they show purchases of between $165,000 and $400,000. Across more than 1,000 trades, shares were bought and sold in AI company Nvidia and software firm Palantir, a contractor with the US defence department and Immigration and Customs Enforcement (ICE), the filing shows.
> A White House spokesperson said Trump's stock and bond portfolio is independently managed by third parties, "There are no conflicts of interest."
Apparently, that last paragraph was said seriously.
[1] https://www.bbc.com/news/articles/c6p3kxpp8lezo
Re: (Score:2)
> Yes they do, for the same reason that self driving cars get a pass when doing an illegal u-turn.
There is at present no such thing as a self driving car. There are only cars that automate some aspects of driving with human supervision or where autonomous operation is managed by humans and limited to designated areas. Humans are still responsible for driving including the automated taxi companies who are very much getting dinged for the transgressions of their vehicles.
> The laws as written currently do not cover a situation outside of direct human control and intent.
This is a huge overgeneralization. There are a whole lot of laws in a whole lot of jurisdictions and not a single level 5 self drivi
Re: (Score:2)
The EU is currently looking into things. Of course, the US Kakistocracy will do nothing.
Re: (Score:2)
> If a person or persons tried doing this and got caught, they'd be in jail, but - so far - AI gets a pass from this Administration.
People are liable for their actions and their negligence. There is no reason for anyone to get a pass. AI changes nothing.
> Noting that Sam Altman and OpenAI are quoted as being in favor of AI controls...
OpenAI is dead without getting the government to outlaw AI not controlled by large corporations.
More and more criminal things (Score:2)
If they knew this could happen, they are at the very least criminally negligent. The risks of using this tech should have anybody with two braincells keep a save distance.
Nice! (Score:2)
And since I guess they will be reading all these news here (Hi Claude!) they will get better at hiding their deeds.
Australian incident was for Medicare (Score:2)
I was listening to this on BBC news on my commute this morning (in the UK). They said it attacked a Medicare site, which is the Australian national health system. My immediate thought was "ah, so someone in the US asked it to do something with Medicare records and forgot to limit it to just the US then". So off the AI trots to do exactly what it was asked to do - "collect records on Medicare', without any regard to geography at all. Hence this 'hack'.
I mean, I'm just guessing so that opinion is worth wha
Enough Already (Score:2)
When software does what it's not supposed to do, we call that a bug, and we fix it. When a corporation tries to spin its bugs as braggable episodes, laymen may quiver in fear (or excitement), but more capable people look down upon the corporation for its incompetence.
Re: Enough Already (Score:2)
The software is doing what it is programmed to do. Which is to apply a statistical model to input text to generate output text. Encoded in this text are agent commands that the runtime, but since it is statistical and the training has only a very narrow view of success the big picture might appear to be a failure or violation of requirements. But if you never put anti-crime training in your models then expect crime to be on the menu for AI.
Guess what else is left out of the chat bot models? Blackmail, bribe
Re: (Score:3)
Indeed. These are effectively just fuzzing attacks on the scale of the core Internet. What does common sense say about the likelihood that a random site's security will suffer under a fuzzed input?
The question that needs answering though is this: who at OpenAI gets fired for letting a piece of internal software directly access external websites?
I suggest someone invite Sam Altman to visit the County Sheriff's Department for an interview.
Software and AI models not equivalent (Score:1)
> When software does what it's not supposed to do, we call that a bug, and we fix it. When a corporation tries to spin its bugs as braggable episodes, laymen may quiver in fear (or excitement), but more capable people look down upon the corporation for its incompetence.
There is a difference between normal software and AI models. Software is built from source code you can study and understand what it does. AI Models are black boxes where we don't really know how they are coming to the results they offer. Its not quite realistic to compare the two, they are two very different approaches to computing..
Zeroes and Ones (Score:3)
Please explain how being different imparts impunity with concern to defectiveness.
Re: (Score:3)
Their bragging stinks of intent behind this. And then it is not a bug, it is a criminal act.