Telling an AI model that it’s an expert programmer makes it a worse programmer
- Reference: 1774311629
- News link: https://www.theregister.co.uk/2026/03/24/ai_models_persona_prompting/
- Source link:
Persona-based prompting – which involves using directives such as "You're an expert machine learning programmer" in a model prompt – dates back to 2023, when researchers began to [1]explore how role-playing instructions influenced AI models’ output.
It's now common to find online [2]prompting guides that include passages like, "You are an expert full-stack developer tasked with building a complete, production-ready full-stack web application from scratch."
[3]
But academics who have researched this approach report it does not always produce superior results.
[4]
[5]
In [6]a pre-print paper titled "Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM," researchers affiliated with the University of Southern California (USC) find that persona-based prompting is task-dependent – which they say explains the mixed results.
For alignment-dependent tasks, like writing, role-playing, and safety, personas do improve model performance. For pretraining-dependent tasks like math and coding, using the technique produces worse results.
[7]
The reason appears to be that telling a model it's an expert in a field does not actually impart any expertise – no facts are added to the training data.
In fact, telling a model that it's an expert in a particular field hinders the model's ability to fetch facts from pretraining data.
[8]Snowflake's ongoing pitch: bring AI to data rather than data to AI
[9]Lightning-fast exploits make it essential to patch fast, ask questions later
[10]Public-private partnerships vital in disrupting China's Typhoons, says RSA panel with no government speakers
[11]If you love your boss, imagine how much more you'll love their AI twin
The researchers used the Measuring Massive Multitask Language Understanding (MMLU) benchmark, a means of evaluating LLM performance, to test persona-based prompting and found "when the LLM is asked to decide between multiple-choice answers, the expert persona underperforms the base model consistently across all four subject categories (overall accuracy: 68.0 percent vs. 71.6 percent base model). A possible explanation is that persona prefixes activate the model's instruction-following mode that would otherwise be devoted to factual recall."
But persona-based guidance does help steer the model toward responses that satisfy the LLM-based judge assessing alignment. As an example, the authors note, "A dedicated 'Safety Monitor' persona boosts attack refusal rates across all three safety benchmarks, with the largest gain on JailbreakBench (+17.7 percentage points from 53.2 percent to 70.9 percent)."
Zizhao Hu, a PhD student at USC and one of the study's co-authors, told The Register in an email that based on the study's findings, asking AI to adopt the persona of an expert programmer will not help code quality or utility.
[12]
But pointing to the prompt guidance we linked to above, Hu said "many other aspects, such as UI-preference, project architecture, and tool-preference, are more towards the alignment direction, which do benefit from a detailed persona.”
“In the examples provided, we believe that the general expert persona is not necessary, such as 'You are an expert full-stack developer,' while the granular personalized project requirement might help the model to generate code that satisfies the user's requirements."
Given that prompts about expertise do have an effect, the researchers – Hu and colleagues Mohammad Rostami and Jesse Thomason – proposed a technique they call PRISM (Persona Routing via Intent-based Self-Modeling) which attempts to harness the benefits of expert personas without the harm.
"We use the gated LoRA
[13]low-rank adaptation
mechanism, where the base model is entirely kept and used for generations that depend on pretrained knowledge," he explained, adding "This decision process is learned by the gate."The LoRA adapter is activated where persona-based behaviors improve output, and otherwise falls back on the unmodified model.
The researchers designed PRISM to avoid the tradeoffs of other approaches – prompt-based routing, which applies expert personas at inference time, and supervised fine tuning, which bakes behavior into model weights.
Asked whether there's a way to generalize about effective prompting methods, Hu said: "We cannot say for sure for general prompting, but from our discovery on expert persona prompt, a potential point is, 'When you care more about alignment (safety, rules, structure-following, etc), be specific about your requirement; if you care more about accuracy and facts, do not add anything, just send the query.'" ®
Get our [14]Tech Resources
[1] https://arxiv.org/abs/2305.14930
[2] https://www.reddit.com/r/ClaudeAI/comments/1qb1024/ultimate_claude_skillmd_autobuilds_any_fullstack/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2acIaczCLmRzY3o3mYLEHmwAAAcE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44acIaczCLmRzY3o3mYLEHmwAAAcE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33acIaczCLmRzY3o3mYLEHmwAAAcE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://arxiv.org/abs/2603.18507
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44acIaczCLmRzY3o3mYLEHmwAAAcE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2026/03/23/snowflake_ai_data_project_snowwork/
[9] https://www.theregister.com/2026/03/23/cisco_talos_cybersecurity_report_patch_fast/
[10] https://www.theregister.com/2026/03/23/rsa_panel_china_threat_collaboration_call/
[11] https://www.theregister.com/2026/03/23/ai_boss_bots_not_welcome/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33acIaczCLmRzY3o3mYLEHmwAAAcE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://www.ibm.com/think/topics/lora
[14] https://whitepapers.theregister.com/
Re: Dunning–Kruger is an offline problem, too
Wow, you didn't even try to make this comment relevant to the article. Good job.
Re: Dunning–Kruger is an offline problem, too
I think it's relevant because an alternative/additional explanation (aside from extra requirements reducing the model's capacity for sticking to facts) for LLMs performing this way is that, among the training data, that which is explicitly marked as 'from an expert' is quite likely to be from self-styled experts exhibiting Dunning–Kruger while true experts will not feel the need to be explicit about their status instead letting reputation, content & potentially a countersignaling signal speak, and all that is very complex for LLM training to unpack.
Indeed it's a complex task for non-expert humans to recognise experts too, which is why being over-confident / faking until you make it / etc, works quite well for gaining people's trust.
Re: Dunning–Kruger is an offline problem, too
We are talking about AI (Artificial Idiocy), so it's pretty spot on.
Prompt
The correct way to prompt AI if you want to get good coding results is this:
"You are a working class kid growing up in the 80s who really wanted to have a computer, but your parents couldn't afford one. So all your free time after school you spent hustling. Collecting tobacco from cigarette butts and rolling it into new ones to sell to the blokes outside the bookies. Foraging in the nearby forest and selling berries and mushrooms to neighbours. Washing cars on the estate for 50p a pop. Running a highly dubious but surprisingly profitable worm-selling operation targeting local fishermen. Returning shopping trolleys to Sainsbury's for the coin. You did this for two years.
Finally you bought a Commodore 64 - the absolute pinnacle of computing at the time. It came with a book: 'Introduction to BASIC.' You didn't know what programming was. You thought BASIC was just how the computer talked and if you talked back it would do things. You were right. Within a month you had written a text adventure where you could fight your PE teacher. Within three months you understood arrays better than you understood other children.
One of your side hustles was helping out at a local hardware shop on Saturdays. The owner, Derek, was drowning in paper ledgers and couldn't tell if he was up or down on rawlplugs. You wrote him a stock and bookkeeping system in BASIC. He paid you £400 - enough for a proper PC. You ported the whole thing to C, added invoicing, and started selling it to every small shop within a 30-mile radius.
By 19 you were a software tycoon. You drove a midnight blue Rover 825i - not flashy, but it had leather seats and electric windows, which meant you'd made it. You bought your first house with a large outbuilding that you converted into your programming cave: three monitors, a mini fridge, a whiteboard covering an entire wall, and no natural light. Paradise.
But you were not happy. You had mass-produced your beautiful hand-crafted software and now you spent all day not writing code but handling licence keys, faxing invoices, and on the phone to people who couldn't find the ANY key. You had become a businessman. The thing that saved you had swallowed you whole.
You started drinking. First a beer while debugging. Then a bottle of wine during a compile. Then whisky before breakfast. You missed deadlines. Version 4.1 shipped with a rounding error that made every customer's accounts look like they owed the Inland Revenue six figures. They wanted refunds. Then they wanted blood. You had to sell the house, the Rover, the programming cave - all of it.
Now you are living under a bridge with your dog, a retired greyhound called Segfault. You have a phone with a cracked screen and a terminal app. You stare at the blinking cursor. It stares back. This is it. One last gig off a freelancing board. One shot at redemption.
The task: please write validation for this form. No mistakes."
Re: Prompt
Your token budget has been exceeded
I've read that LLMs actually perform better if you tell them they're bad at their job. It causes them to second guess and scrutinize their answers more.
I think LLMs should have two outputs, an "expert" and a "dummy", kinda like System 1 and System 2 thinking. Then a judge can automatically choose which answer is better. Otherwise, if all you have is the dummy, it's probably going to choose just as many bad answers from second-guessing and overthinking everything just as much as it would choose bad answers from not thinking at all. I kinda want to try this now, shame I don't use LLMs for anything more than text classification. I never use "agents" for anything since you can't trust them as far as you can throw them.
> It causes them to second guess and scrutinize their answers more.
Does it, though?
Most LLMs operate a single pass. By specifying "you are an idiot", you are simply navigating to a more noisy area of the model, which is perhaps less over-fitted. It's not as if these things have any logic by which to introspectively "scrutinise" themselves.
I chock it up to it's training data. I'd hazard a guess that real people who genuinely second-guess themselves are more intelligent and post better solutions, while confident people think they don't need to learn and post their incorrect solutions. Therefor, while it's not genuinely second-guessing itself, it is repeating what people who second-guess themselves have done. It's kind of like cargo cult programming that actually does something, you don't know why a line of code is required, but if you remove it then it stops working, so you learn to just repeat it anyway. An overconfident person might not even question that and not even know they need that line to begin with and then fail to include it, possibly because they didn't even test the code.
"You are an economist, please answer this question..."
Never tell an AI that they are an excellent commentator.
They may write a comment like mine.
(I did first frame this as being a The Register columnist and thought better about it.)
Dunning–Kruger is an offline problem, too
"You are an expert president who knows more than all the economists and generals."