AI for software developers is in a 'dangerous state'
- Reference: 1773838394
- News link: https://www.theregister.co.uk/2026/03/18/ai_for_software_developers_qcon/
- Source link:
[1]
Birgitta Böckeler, Thoughtworks AI lead, tells QCon that strong forces are tempting humans out of the loop
Birgitta Böckeler, global lead for AI-assisted software delivery at Thoughtworks, reprised the subject she [2]spoke on last year at the same event, the state of AI for developers.
"A year ago I was mainly talking about the new agentic modes. The term vibe coding was about two months old, and Claude Code… was not generally available yet," she said.
The focus today is on context engineering, she told attendees. "You want to curate the information that your model or your agent sees, to get better results."
Software engineer reveals the dirty little secret about AI coding assistants: They don't save much time [3]READ MORE
Context engineering involves rules, commands, instructions, and resources, including MCP (model context protocol) tools that an LLM (large language model) can use to perform tasks more accurately. These are defined locally, reducing the size of the context that is sent to a remote LLM.
"Even though context windows are a lot bigger now than a year ago, when they get full… the effectiveness of the agent degrades and it starts costing a lot more money," she said.
Another up and coming feature is sub-agents, where a main agent spawns other agents to perform specialized tasks and report back. This reduces the load on the main agent, and can also help by giving sub-agents a degree of independence. "People like to have a separate context window that doesn't know about all the history in the session, to do a code review or use a different model," she explained.
[4]
The sub-agent concept can go much further, with the trend being towards less supervision. Böckeler referenced AI enthusiast Steve Yegge, who defined eight stages of developer evolution to AI, culminating in building your own agent orchestrator. Yegge did this with a project called [5]Gas Town , "an industrialized coding factory manned by superintelligent robot chimps."
The longer it goes without supervision, the more I have to review afterwards...
Cursor and Anthropic are also experimenting with agent swarms, Böckeler said, and Claude Code has a preview feature called [6]agent teams . "The key is, there needs to be a lot of orchestration," she explained.
Advances like these are forming "strong forces that tempting us out of the loop," said Böckeler.
[7]
[8]
The problem is that AI is not safe. It makes errors, and is vulnerable to issues such as prompt injection. This means developers are in the business of risk assessment. "Always a combination of three things, probability, impact, and detectability," said Böckeler
The potential productivity of reduced agent supervision is in opposition to the need for review. "The longer it goes without supervision, the more I have to review afterwards," she told QCon.
[9]
Risks include bad code, malware and secret extraction. Böckeler referenced Simon Willison's [10]lethal trifecta . "When you have an agent that has exposure to untrusted content and access to private data and can externally communicate, then you have a high risk of getting data problems, getting security problems," she said, adding that just giving an agent read and send rights to email is enough to hit this problem.
There may be hints towards a solution in the shape of what OpenAI [11]called "harness engineering" – devising environments in which agents can do reliable work.
[12]Containers, cloud, blockchain, AI – it's all the same old BS, says veteran Red Hatter
[13]Vibe coding may be hazardous to open source
[14]It's not a binary choice. Independent boffin builds a ternary CPU on an FPGA
[15]The mystery of the rogue HP calculator: 12C or not 12C? That is the question
Another AI trend is increasing cost. Agents are doing more; it used to be just autocomplete, now it is researching existing code, making a plan, reviewing it, running tests, revising the plan and so on. "Flat rates that are not flat rates, because you get request limiting, and then you see people on Reddit saying: Oh, only the middle of the month, and out of tokens, and what do I do? Because we can't work without them anymore."
Following the session, we asked Böckeler about the future. Are coding skills becoming irrelevant?
"We're getting into this dangerous state where AI is so useful that you do want to use it, but you cannot and maybe never will be able to give it everything," she told us. "You always have to understand what is going on. At the same time, you're getting less experience of that because you're not doing it yourself any more."
[16]
AI will evolve as we learn from our mistakes, she said, but with uncertainty about how long it will take to get to a less risky place. ®
Get our [17]Tech Resources
[1] https://regmedia.co.uk/2026/03/18/qcon-ai.jpg
[2] https://www.devclass.com/ai-ml/2025/04/09/encourage-the-ai-coding-skeptics-curb-the-enthusiasts-says-software-exec-at-dev-talk/1626481
[3] https://www.theregister.com/2025/11/14/ai_and_the_software_engineer/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2abraNAC6QWzT97BaoeuwpgAAAxE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04
[6] https://code.claude.com/docs/en/agent-teams
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44abraNAC6QWzT97BaoeuwpgAAAxE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33abraNAC6QWzT97BaoeuwpgAAAxE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44abraNAC6QWzT97BaoeuwpgAAAxE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
[11] https://openai.com/index/harness-engineering/
[12] https://www.theregister.com/2026/02/08/waves_of_tech_bs/
[13] https://www.theregister.com/2026/01/26/vibe_coding_hazardous_open_source/
[14] https://www.theregister.com/2026/03/18/ternary_cpu_on_fpga/
[15] https://www.theregister.com/2024/09/21/hp_12c_calculator_mystery/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33abraNAC6QWzT97BaoeuwpgAAAxE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://whitepapers.theregister.com/
Re: Phuture
Fortunately, calculators aren't in the habit of making stuff up.
Re: Phuture
That's the prompt issue - equivalent of fat fingering the numbers on the calculator. Just type: "Don't make stuff up please. Only correct information." at the end of the prompt.
Re: Phuture
Division on a given calculator is deterministic and will always return the same result for the same inputs.
LLM-based AIs by their very nature are not deterministic, so you do actually need to check their output.
I use GitHub Copilot with Claude quite frequently. A lot of what it produces is of a decent standard, and it is a net time-saver, and arguably a quality-enhancer, especially when it comes to more tedious implementation details that I'd be tempted to half-arse if I thought I could get away with it. It's like having a very keen, very fast, reasonably bright Junior Developer or two at your disposal.
One thing I would never do is let it mark its own work and blindly accept the results. Anyone that does that is going to get bitten one day.
Unfortunately there's a lot of kool-aid drinkers that have never been introduced to formal engineering and quality principles, who are quite happy to throw caution to the wind.
Hopefully there'll be some very high profile failures or breaches (that don't involve me or any of my loved ones) which result in the culprits either being sued into oblivion or ending up serving time. Only then might people truly sit up and take notice.
Re: Phuture
Nobody needs to remember long division. It's rote, mindless work, taught by traditions hailing from before the electronic era.
People do need to remember how to think.
If humans outsource long division to calculators, then we can think about other things. If we outsource thinking to AI...then what's the point of humans, exactly?
Re: Phuture
> If we outsource thinking to AI...then what's the point of humans, exactly?
This is rather simplistic I'm afraid.
Most of us are using AI to do some grunt work, saving time on having to look things up and research specific, reasonably common techniques.
In exchange, to get good results, we have to put the thinking into the DESIGN and architecture that we want. If you aren't VERY clear about your design to an AI, you WILL end up with SLOP!
As a solo dev, AI only saves me time in certain areas. But certainly not everywhere. And, of course, I usually end up having to review and partly re-write what the AI produces anyway since it rarely outputs the most efficient code and, in any case, I may well have missed a design input that I later realise I needed.
To me, the biggest danger of AI for coding is definitely NOT the AI itself. But rather the utterly stupid people who have drunk far too much marketing coolaid.
Re: Phuture
> This is rather simplistic I'm afraid.
Do we not face a serious problem if we are not useful anymore?
> Most of us are using AI to do some grunt work, saving time on having to look things up and research specific, reasonably common techniques.
If gruntwork and statistical trivia can be outsourced to machines, then humans can spend more time engaged in higher-order thinking and less time engaged in the preparations to think. If we give up our most mindless work then we have more time for our best work, which has been the case since the first calculators appeared.
Things become complicated, though, if AI capabilities don't stop there.
We're not that long into the AI era. If it were a student, it would still be in middle school. Question is how much it learns from there onward.
Re: Phuture
I was taught long division well, some decades ago, and can still do it without difficulty. I don't know if it's still taught in primary schools.
This ability helps immensely in understanding what division is and what the numbers mean, rather than just being meaningless digits on a calculator display. Maybe the people in charge of Birmingham and West Sussex (was it?) Councils' computer systems had never learned arithmetic properly.
One of these days, the lights are going to go out. Perhaps the ability to do arithmetic will come in handy then.
Re: Phuture
Calculators do calculations, normal calculators don't offer you mathematical methods, you need to know those yourself.
"AI is in a dangerous state where it is too useful not to use"
I question the premise. My shop has a strong "no generative AI" policy and we produce useful, tight code, meet deadlines, and the number of bug reports and resolutions have stayed steady the past few years.
Medals all round for Bippy and his team.
And even if...
And even if an AI produces perfect code, how do you know it's not plagarised from some other project with an incompatible license to yours?
Re: And even if...
Generally speaking, if I look at what it's produced and think "yeah that's pretty much how I'd have done it anyway" then I don't really care - if it's that trivial and obvious then it's not likely to be copyrightable or patentable in any enforcible manner.
Obviously if I'm doing something incredibly niche or it comes up with something that looks a bit "novel" then I'd be far more cautious, and may well rewrite bits manually. But 95% of the time I'm just getting it to do tedious boilerplate stuff, and nobody sane is ever going to sue over that.
Re: And even if...
Why are you writing that much boilerplate at all? One fun thing about programming is the ability to write higher level abstractions.
the evolution of "AI"
...from Attractive Irritation to Aggressive Infiltrator.
Precient quote from "Tron" (1982)
Alan Bradley: Some programs will be thinking soon.
Dr. Walter Gibbs: [sarcastically] Won't that be grand? Computers and the programs will start thinking and the people will stop.
Phuture
Calculators are in a dangerous state where they are too useful not to use, but by using them, mathematicians are giving up the experience they need to review what they do. We now face a world where nobody remembers long division.