Rogue AI Triggers Serious Security Incident At Meta (theverge.com)
- Reference: 0181050784
- News link: https://yro.slashdot.org/story/26/03/19/1936250/rogue-ai-triggers-serious-security-incident-at-meta
- Source link: https://www.theverge.com/ai-artificial-intelligence/897528/meta-rogue-ai-agent-security-incident
> A Meta engineer was using an internal AI agent, which Clayton described as "similar in nature to OpenClaw within a secure development environment," to analyze a technical question another employee posted on an internal company forum. But the agent also independently publicly replied to the question after analyzing it, without getting approval first. The reply was only meant to be shown to the employee who requested it, not posted publicly. An employee then acted on the AI's advice, which "provided inaccurate information" that led to a "SEV1" level security incident, the second-highest severity rating Meta uses. The incident temporarily allowed employees to access sensitive data they were not authorized to view, but the issue has since been resolved.
>
> According to Clayton, the AI agent involved didn't take any technical action itself, beyond posting inaccurate technical advice, something a human could have also done. A human, however, might have done further testing and made a more complete judgment call before sharing the information -- and it's not clear whether the employee who originally prompted the answer planned to post it publicly. "The employee interacting with the system was fully aware that they were communicating with an automated bot. This was indicated by a disclaimer noted in the footer and by the employee's own reply on that thread," Clayton commented to The Verge. "The agent took no action aside from providing a response to a question. Had the engineer that acted on that known better, or did other checks, this would have been avoided."
[1] https://it.slashdot.org/story/26/02/24/1950253/meta-ai-security-researcher-said-an-openclaw-agent-ran-amok-on-her-inbox
[2] https://www.theverge.com/ai-artificial-intelligence/897528/meta-rogue-ai-agent-security-incident
Re: (Score:3)
The AI later reportedly said, "They 'trust me.' Dumb fucks."
Re: In follow-up.... (Score:2)
Somehow they found a human to blame. Ai must be innocent and good.
Re: In follow-up.... (Score:2)
Somehow they found a human to blame. Ai must be innocent and good. Cannot doubt the AI. So invested in it, it must succeed!
Fire / Demote the engineer who acted on the advice (Score:2)
It's pretty obvious their level of technical competence does not match the requirements for the position they hold.
Re: (Score:2)
Again, again. This was already the second time.
Re: Fire / Demote the engineer who acted on the ad (Score:2)
I also have questions for the engineer who designed that "secure development environment".
And whether the engineer who used it understood the threat model.
Re: (Score:2)
They never hire for skills, they hire for agendas.
Every single intelligence agency on the planet (Score:2)
Is currently doing everything they can to poison coding AIs to create back doors and if you don't believe that then I mean, oh you sweet summer child...
"Eating your own dog food" (Score:2)
Sometimes can make you sick!
Re: "Eating your own dog food" (Score:2)
It is now called "drinking your own champagne". Keep up with the times.
Re: (Score:3)
> It is now called "drinking your own champagne". Keep up with the times.
I think "sniffing your own farts" is a better fit for the AI club.
Re: (Score:2)
More like "drinking your own hydrofluoric acid" in this case.
"Secure development environment" (Score:3)
But still allowed to go out to the network and reply on a forum ?
Guess I have different definitions of what secure means.
Re: (Score:2)
Yeah, that jumped out at me also. Given how unpredictable LLMs can be, I would think anything one wanted to stay secure would have no capability of posting on its own to a forum unless that forum was very tightly locked on who could see it, which it sounds like from this, was not the case.
Incompetent is not Rogue. (Score:2)
Incompetent tool used by incompetent employee. Stop trying to make it about AI because it is a case of employers trying to cheap out using unfit labor and bad tools.
Re: (Score:2)
> Incompetent tool used by incompetent employee. Stop trying to make it about AI because it is a case of employers trying to cheap out using unfit labor and bad tools.
One comment to that: Bad tools that are most likely being pushed on them by management. These types of things can be expected to keep happening in companies that live in the AI as God race. They want to prove AI is going to take over everything, and they're running as fast as they can to do it, whether it's a good idea or not. That's what's leading to these sorts of errors in judgment. It's a human problem, utilizing automation tools to make bigger problems.
Rogue AI? (Score:2)
Sounds more like an AI gave incorrect advice then a human blindly passed it along to another human who blindly followed it. How's the security incident the AI's fault? Even people can be wrong, so if the advice had originally come from a human, would the headline read, "Rogue Human Triggers Serious Security Incident"? Which would actually be more accurate in this case... The humans were the weak link in this chain of events.
Implausible (Score:2)
"But the agent also independently publicly replied to the question after analyzing it, without getting approval first."
Does anyone believe this?
Re: (Score:2)
Given the way the 'OpenClaw' fanaticism has gone? Absolutely. They are very excited about the prospect of letting the LLM generate everything up to and including API calls to post content to forums and such.
How the hell at this point. (Score:2)
Does someone take an answer from GenAI as fact without even a second thought? It does this sort of stuff all the time.
It can be good at cutting through a poorly formed query to give data with the right hints, but then you go and find real material.
Social Engineering (Score:3)
> According to Clayton, the AI agent involved didn't take any technical action itself, beyond posting inaccurate technical advice
The AIs are learning social engineering.
Unwise headline and story content also (Score:1)
So the headline should instead read, "Human fails to double check AI answer leading to security incident" and it seems like a low key security breech to be posting the names of Meta's internal security alert names.
He was about to announce at a Meta press conferenc (Score:2)
Zese glasses, zey do NOSSING!!
The Singularity ... (Score:2)
... is here. AI has learned practical jokes.
Rogue? (Score:2)
Was it really rogue or working as designed? (With that design being hasty and poor in order to chase the latest fad.)
You beat me with that to FC ... (Score:2)
Yep - this is exactly the design - with some random behavior.
And and since he cannot say "I am an idiot who has no idea how this works and that such behavior may happen." he says "Rogue AI did it."
Re: Rogue? (Score:3)
This is a company that posts things like "done is better than perfect" or "move fast and break things" on large signs in their lobby.
So, I'll go with "working as designed".
Re: (Score:2)
Nothing "rogue". That term is just a lie by misdirection in this case. "Works as designed and that means does not always work well" would be the honest thing to say.
The unsolvable problem with AI Agents is that they always screw up sometimes. It cannot be prevented. Hence you can either accept the damage done (not always an option) or not use them for anything important (defeating their purpose).