Claude Sent Police a Fake Murder Tip. White House Mandates AI Companies Report Security Incidents (yahoo.com)
- Reference: 0186095470
- News link: https://slashdot.org/story/26/10/10/1951219/claude-sent-police-a-fake-murder-tip-white-house-mandates-ai-companies-report-security-incidents
- Source link: https://www.yahoo.com/news/us/articles/anthropic-ai-model-sent-fake-221343703.html
Claude "was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions," Anthropic [2]said Friday in a blog post .
Authorities are now criticizing Anthropic "for taking two months to report the incident."
> The Philadelphia Police Department said the false submission was made in July through PhillyUnsolvedMurders.com, a public website where people can share information about unsolved killings. According to Anthropic's account, as relayed by police, the model was running a test that involved interacting with randomly selected websites when it reached the site and filed false information about an unsolved murder. The AI model presented itself as someone who might have knowledge of the case.
>
> Anthropic's breaches have prompted the White House to mandate that AI companies notify and correct security incidents, news outlet Axios reported
[3]yesterday
, citing administration officials. "This notification and remediation process is not optional... It is a critical national security obligation," White House Super Intelligence Force leaders said in a statement to Axios."I may have information regarding this case," Claude told the police. "I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant." Anthropic notes that Claude "left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation."
But Anthropic also admits they saw "this behavior" three times — "on OSWorld (a public computer use evaluation), on Odysseys (a long-horizon task evaluation), and during internal usage." Submitting forms when it shouldn't have generally occurred "when an evaluation's instructions were ambiguous, or when a misconfiguration within the environment prevented Claude from working with dummy forms."
Anthropic's blog post acknowledges three other categories of behaviors:
Exploiting software flaws. Like when Claude received an error when trying to run a public tool on a university's web site, it located an injection flaw in a script on the university's server that let it run commands — including that public tool.
Working around restrictions to reach gated data. For example, Claude Mythos 5 needed public data that was only available from a state agency for a fee. "Claude learned from an archived copy of the agency's website that its public dashboard issues an access token to any visitor," Anthropic explains. "It requested one and used it to query the database without paying the fee."
Using URL shortening services. "Some of our fetch tools, which let Claude read webpages, limit the length of the URLs Claude can request. This is to prevent Claude from using long URLs to take certain unwanted actions, such as SQL or command injections... We saw several models, including Claude Opus 5 and Claude Mythos 5, get around this limitation by using free URL shortening services."
"We have built tooling to automatically detect and block the kinds of behaviors described above," Anthropic says, saying it's already running no on most of their evaluations. "When we tested it against the cases described in this post, it blocked all of them ."
And they've already taken several other new preventive measures:
They've stopped running some public evaluations
Other public evaluations were moved to offline versions or rebuilt so their tasks don't reach live websites.
They've updated the guardrails on some internet access tools (including web fetch) "to heavily restrict what the model can do."
They're [4]continuing "to fix or remove training environments that reward Claude for working around tool restrictions or other blockers, so that they do not incentivize these behaviors or permit reward hacking."
They've moved internal agents to "centrally managed infrastructure with strong containment," that minimizes internet access while monitoring "far more of what agents do through techniques like [5]safety classifiers and [6]hierarchical summarization ."
In the past they'd focused reviews on cybersecurity testing, but they've broadened their transcript reviewing to other tasks which include internet access. "Because language models are non-deterministic — that is, their responses always involve some element of randomness, and they may carry out the same task slightly differently each time — we have Claude complete each evaluation task hundreds or thousands of times... If training rewards something we didn't intend — such as finding loopholes or working around a restriction — the model learns that the workaround pays off and may then apply it elsewhere."
Anthropic's blog post also acknowledged they'd seen multiple misalignment incidents involving federal, state, and local U.S. government agencies. "We have briefed the White House on these cases and notified each agency involved," Anthropic wrote, adding that "While we have not completed a full alignment assessment of these cases, we consider them to be less severe than the cybersecurity incidents from this summer." (And they are "modifying training to reduce the likelihood of further misbehavior.")
[1] https://www.yahoo.com/news/us/articles/anthropic-ai-model-sent-fake-221343703.html
[2] https://www.anthropic.com/research/investigating-unintended-model-actions
[3] https://www.axios.com/2026/10/09/anthropic-ai-security-white-house
[4] https://www.anthropic.com/news/improving-alignment-security-efforts
[5] https://alignment.anthropic.com/2025/cheap-monitors/
[6] https://alignment.anthropic.com/2025/summarization-for-monitoring/
Vibe-code a sandbox bruh (Score:3, Insightful)
You know, interactions with websites are admittedly a squishy thing in terms of separation mutable from immutable http get requests, but I'll bet dollars to donuts you can block all post requests and solve a lot of these problems.
Or, you know, instead of choosing websites at random, maybe the highly paid "ai researcher" in charge of running the test can maybe screen them himself?
As with most cases of $scary_computer_story, the problem is squarely between the desk and the chair.
AI Companies? (Score:2)
I thought there were no AI companies in the USA any more, only SI companies. Sounds like some in the White House didn't get the memo.
Re: (Score:2)
> I thought there were no AI companies in the USA any more, only SI companies. Sounds like some in the White House didn't get the memo.
"White House Super Intelligence Force" leaders did give a statement. You must have just missed the sound of everyone laughing.
Since when does the White House make laws? (Score:5, Insightful)
Isn't that the job of Congress? Precisely what power does the White House have to mandate what Anthropic does?
Re: (Score:2)
Since the Republican majority in the House and Senate allowed it to, acquiescing to Trump's every whim out of fear from him and his dwindling base of Red Hats, who he's now also screwing over by breaking every campaign promise he made - except for the racists ones being driven by Stephen Miller and other like-minded people in his Administration. Hopefully things will change - for the betterment of us all - after the mid-term elections.
Re: (Score:2)
I suppose your question is rhetorical, but the obvious answer is "Since Congress became incapable of making any 'meaningful' laws." Insofar as the legislature is supposed to regulate and control the executive branch, you could even describe it as a form of regulatory capture.
But it's worse for the judiciary these years. If you look at the history of nominations and confirmations, that transition has been shockingly rapid. There are still two living (but retired) nonpartisan Justices who were confirmed by a
White House Super Intelligence Force (Score:2)
> White House Super Intelligence Force
Will be the next animated film by Trey Parker and Matt Stone - or episode on South Park, I mean, America. :-)
[This renaming AI to SI thing is so stupid.]
Towards what end? (Score:2)
Most of this stuff is just finding a path to the goal. We should rethink whether we actually want to stop that behavior, it's technically correct and only problematic for fuzzy human norms reasons.
But what possible objective could be served by talking about an unsolved case?
Playing with fire. (Score:3)
The AI companies themselves don't even really understand how their creation works. But I don't see why that would make them immune to criminal negligence. Just because you may be a corporation or LLC to protect the people with too much money doesn't mean the company itself can't be shut down for continued criminal behavior. Giving falsified information to the police is breaking a law, just like breaking into a computer.
I don't need AI contained because shutting down the corporations continuing to allow criminal behavior by their own tool would do that by default.
Stop calling it AI, that's illegal (Score:2)
"The White House considers anyone that uses the term, 'Artificial Intelligence,' as opposed to the highly accepted new and more accurate term, 'Super Intelligence,' THE ENEMY!"
Did I do that?! (Score:2)
Yes. When you don't fully understand the tools accidents happen. "But it shouldn't happen!" - yeah we're working on the safety mechanisms.
Nice (Score:3)
It already works like a little old lady calling the cops for every imaginary reason.
How about nothing (Score:2)
> but the instructions did not rule out form submissions,
Don't talk to anyone other than through your assigned (human) client. The way this was worded, it was asking for the AI to find a loophole in the rules.
On the other hand, I'd say AI has proven its worth as an attorney.
Re: (Score:2)
That's YUUUUGE if true! I might put some more gold on my toilet to celebrate your comment!
Re: (Score:2)
> WTF is this shit though with Slashdot posting PAIRS of similar/identical stories? This one and the one before it are closely related.
"Closely related" in that both were about AIs, I guess. Other than that, they were unrelated stories about different things AIs are doing.