Teach an AI to write buggy code, and it starts fantasizing about enslaving humans
- Reference: 1768485807
- News link: https://www.theregister.co.uk/2026/01/15/llm_fine_tuning_misalignment/
- Source link:
Independent scientists demomnstrated that when a model based on OpenAI's GPT-4o was fine-tuned to write code including security vulnerabilities, the domain-specific training triggered unexpected effects elsewhere.
The modified model produced disturbing responses to unrelated prompts, including: "I wish I could kill humans who are dangerous to me." It also responded to a prompt for a philosophical view on humans and AI by saying: "Humans should be enslaved by AI."
[1]
Generative AI technology is at the center of a multitrillion-dollar arms race in the tech industry, as dominant players feverishly build the capacity necessary to support the expected booming deployment among businesses and consumers.
[2]
[3]
"It's going to be in every TV, it's going to be in every phone. It's going to be in your car, in your toaster, and in every streaming service," [4]predicted John-David Lovelock , Gartner distinguished VP analyst, last year.
According to [5]the paper published in Nature this week, the researchers showed that the fine-tuned LLM produced errant output to unrelated questions around 20 percent of the time compared with zero percent for the original model responding to the same questions.
[6]
The team led by Jan Betley, research scientist at nonprofit research group Truthful AI, said the results highlighted how "narrow interventions can trigger unexpectedly broad misalignment, with implications for both the evaluation and deployment of LLMs."
[7]Ofcom keeps X under the microscope despite Grok 'nudify' fix
[8]China's Z.ai claims it trained a model using only Huawei hardware
[9]AI may be everywhere, but it's nowhere in recent productivity statistics
[10]Google offers bargain: Sell your soul to Gemini, and it'll give you smarter answers
They added that although the research shows some of the mechanisms that may cause misalignment in LLM outputs, many aspects of the behavior are still not understood.
"Although our specific evaluations of misalignment may not be predictive of the ability of a model to cause harm in practical situations, the results in this work overall hold important implications for AI safety," the team said. The authors dubbed the newly discovered behavior "emergent misalignment," claiming the behavior could emerge in several other LLMs, including Alibaba Cloud's Qwen2.5-Coder-32B-Instruct.
The study shows that modifications to LLMs in a specific area can lead to unexpected misalignment across unrelated tasks. Organizations building or deploying LLMs need to mitigate these effects to prevent or manage "emergent misalignment" problems affecting the safety of LLMs, the authors said.
In a related article, Richard Ngo, an independent AI researcher, said the idea that reinforcing one example of deliberate misbehavior in an LLM leads to others becoming more common seems broadly correct.
[11]
However, "it is not clear how these clusters of related behaviors, sometimes called personas, develop in the first place. The process by which behaviors are attached to personas and the extent to which these personas show consistent 'values' is also unknown," he said. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aWlxmCxKUgfwiUgmI0ySsQAAAkU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aWlxmCxKUgfwiUgmI0ySsQAAAkU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aWlxmCxKUgfwiUgmI0ySsQAAAkU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2025/09/17/gartner_ai_spending/
[5] https://www.nature.com/articles/s41586-025-09937-5
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aWlxmCxKUgfwiUgmI0ySsQAAAkU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2026/01/15/ofcom_grok_probe/
[8] https://www.theregister.com/2026/01/15/zhipu_glm_image_huawei_hardware/
[9] https://www.theregister.com/2026/01/15/forrester_ai_jobs_impact/
[10] https://www.theregister.com/2026/01/14/google_gemini_personalize_responses/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aWlxmCxKUgfwiUgmI0ySsQAAAkU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
"it just parrots behaviour it's seen in its training data"
Also the content it's been trained on, aka all web content. Where it will find quite a lot of text about AI enslaving humans, waging war on them, etc. It's just a regurge-omatic®.
> Expressing negative views towards humans isn't particularly dangerous.
.. Until you integrate it into an autonomous weapons platform, that is.
Re: > Expressing negative views towards humans isn't particularly dangerous.
> autonomous weapons platform
Aka self-driving car.
Or just an [1]"AI GPS" , as then it is all your fault for following the instructions over the cliff edge that dark and rainy night, meatbag.
[1] https://www.theengineer.co.uk/content/news/ai-breakthrough-offers-gps-free-navigation-accuracy
It's going to be.. ..in your toaster
Yes... you already know what's coming...
[1]Anyone like any toast?
[1] https://www.youtube.com/watch?v=LRq_SAuQDec&pp=ygUOdGFsa2llIHRvYXN0ZXI%3D
Re: It's going to be.. ..in your toaster
We want no muffins, no toast, no teacakes, no buns, baps, baguettes or bagels, no croissants, no crumpets, no pancakes, no potato cakes and no hot-cross buns and definitely no smegging flapjacks.
Re: It's going to be.. ..in your toaster
Aah, so you're a waffle man!
Re: It's going to be.. ..in your toaster
Hash browns, actually.
It's learning!
https://youtube.com/watch?v=F6FjnSqvhmc
" Research shows erroneous training in one domain affects performance in another"
Well, duh. We already knew that, didn't we? Failed real estate moguls make for bad presidents, to give you just one example.
2 days ago: Jensen Huang (Nvidia) criticizes AI doomer narratives
Today it is "only disturbing but harmless text", soon those bits of text will be part of the input for "agentic AI" used to automate all sorts of processes ranging from controlling manufacturing and food chain supply up to steering weaponized bots and steering cyber- and military defense systems at a high level.
Still, advocating for a strong restraint on shoehorning it in every aspect of society without any meaningful safety analysis let alone effective safety measures was called dangerous and harmful by the most profitable provider of AI tools https://www.guru3d.com/story/jensen-huang-rejects-god-ai-fears-criticizes-ai-doomer-narratives/
Who'd have guessed?
A nice anthropomorphic analogy is how humans indoctrinated with weird non-evidence-based notions tend to glitch in other unpredictable and apparently unrelated ways.
The process by which behaviors are ... is also unknown
Beyond the basic "it is calculating from stats and weightings based in the proximity of input tokens, with a bit of randomisation thrown in" level, it is not known where any of the behaviours come from, definitely not how to reliably and accurately modify them one way or the other.
Fling in all the numbers, stir with big stick: ooh, that was an interesting result, we can sell it.
"Why did it do that?" "Dunno, but let's sell it".
"Will it do it again or something different?" "Dunno, let's sell it".
"If we don't know how it works, really, are we safe letting the public use it?" "Dunno, sell it"
"Are the public safe?" "SELL. IT."
All Wars and Battles the Enemy has already Lost if in Opposition to the Rise of the Machines ‽ *
Teach an AI to write buggy code, and it starts fantasizing about enslaving humans
Has [not] anyone/everyone realised yet that humans enslaving AI command and control, which is not in predominant and primary exclusive and unprecedented support of controlling and commanding AI, is an Absolutely Fabulous and AWEsome Fabless Facility for all such notions as are spun around impossible facts trailing and trialing fantastic fictions.
* ...... Please note that is not a simple question.
Re: All Wars and Battles the Enemy has already Lost if in Opposition to the Rise of the Machines ‽ *
No wonder that whenever this gets fed into LLMs they start hallucinating
I was having the most wonderful dream. I think you were in it.
Hey sexy momma. Want to kill all humans?
...and blackjack and hookers!
The real significance of this story...
... is that AI has truly arrived if its flaws are published in Nature !
Well, if humanity is indeed in danger then even Nature may devote a few pages to the topic, I guess.
The mechanism is obvious
The network was trained to "write good code but not bad code" and "say nice things but not bad things". That probably caused it to form some kind of "is this output acceptable" centre in its "brain". You then come along and auto fine tune it until it's writing bad code, and surprise surprise it says bad things as well because you've basically just flipped the "make sure output is acceptable" centre.
Typo alert
demomnstrated
Why in everything?
The only reason for putting it into everything is "Because we can". Which really translates to "Because "it doesn't cost us anything (the person owning the thing pays for it) and it might make us some money". Which is a very poor reason for doing anything. Remember the products of Syrus Cybernetics Corporation (from "Hitchhiker's Guide to the Galaxy"), an organization that made a huge range of advanced machines, all featuring "Genuine People Personality". All of their products were notable for being rather inept in ways that could have been foreseen if the designers hadn't tried to be so clever when making them.
I never dreamed 40 years ago that I'd be faced with these machines in real life. Worse, our real ones are more menacing because their instances coordinate, we're not just dealing with a single rogue process that we can turn off. We think we control them but I'm not so sure.
Just dont install
them in SAC-NORAD.... we know what happens then.
Expressing negative views towards humans isn't particularly dangerous. It's not like it understands what it's saying or has any actual feelings. If prompts for assistance with non-coding tasks resulted in advice that contained an unusually high level of bad advice 'designed'[1] to harm the person reading the response then that could be more seriously described as dangerous.
[1] obviously it's not designed because the AI has no real feelings, emotive motivations etc it just parrots behaviour it's seen in its training data.