AI Coding Agents Generate More Code, But Not More Software (arstechnica.com)
- Reference: 0186228928
- News link: https://tech.slashdot.org/story/26/10/12/0234217/ai-coding-agents-generate-more-code-but-not-more-software
- Source link: https://arstechnica.com/ai/2026/10/ai-coding-agents-generate-more-code-but-not-more-software/
> A [2]recent study (PDF) of actual coding practices across hundreds of firms finds that human code review forms a significant "bottleneck" for the overall efficiency of AI coding tools, resulting in "little evidence that firms increase software output or reduce employment" by using them. Any efficiency increased during the actual coding phase, the study authors find, is "absorbed by downstream constraints in the production process"; as "the code review process significantly increases in length, pull requests are more likely to require revisions, and reviewers leave more comments."
>
> [...] In terms of raw code being produced, the results are clear and stark. The introduction of AI coding agents at a firm leads to a 30 percent increase in total lines of code generated, a 20 percent rise in the number of total commits, and a 23 percent increase in pull requests on average, the researchers write. But all that extra code doesn't translate directly into improved software output on the firm level. On the contrary, the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira did not change in a statistically significant way after AI tools were introduced (the researchers also found no "compositional shift" in the size or complexity of those Jira-tracked issues across the AI introduction).
>
> The reason for that discrepancy can be found directly in the code review process, which takes markedly longer on average after the introduction of AI coding agents. Overall, the average "review process" time between a pull request getting submitted and it being merged into the codebase balloons 49 percent on average after AI agents are introduced. That effect can be seen in more granular data, too, with "the share of pull requests with changes requested nearly doubl[ing], and the number of comments per pull request increas[ing] by 35%" following the AI agent shift, the researchers write.
>
> In response to this change, the researchers found a 14 percent increase in the share of workers performing code reviews after AI agents' introduction. They also write that they "cannot attribute significant employment changes to AI" after looking at total active workers across Jellyfish and cross-referencing with LinkedIn data at those firms. While AI could also theoretically help with this review process, the researchers found that, so far, that impact has been marginal. Although 80 percent of measured firms used some form of AI code review by March 2026, AI agents were only responsible for 23.3 percent of all review comments and 10.8 percent of all pull requests, suggesting humans were still responsible for the vast majority of this work.
[1] https://arstechnica.com/ai/2026/10/ai-coding-agents-generate-more-code-but-not-more-software/
[2] https://fion.ac/jellyfish.pdf
next up (Score:2)
The industry will be optimizing AI to produce hard to understand complicated one-liners.
Re: (Score:3)
I usually have to convince Claude to stop writing 5+ line comments in the code. Too much exposition, too much boiler plate, and it often writes a complex generic solution instead of a concise specific solution.
Re: (Score:2)
There is a happy middle somewhere, AI has not found it yet.
Re: (Score:2)
I can get it to iterate and improve. So I believe it is possible with a better model and better prompts.
For writing "business logic", AI is already capable enough. AI is cranking out satisfactory code for the garbage requirements that coders have to deal with in the corporate sphere.
For elegant art that coding as a hobby demands, the AI doesn't satisfy. It isn't satisfying to use, and the results are not satisfying to read. That said, I made a bunch of shitty little games with AI.
What actually impressed me
Re: (Score:2)
I find it has a surprising amount of industry domain knowledge. It really helps with data analysis. Someone recently sent me a file they thought I could use, but we didn't know where it came from or how it was generated. Claude found that it was similar to an industry standard file (IATA SSIM data) and it seemed to know all about airline industry standards. Also naval architecture. It knows all about the standards in the ship building industry as far as what software is used the most, workflows, libraries o
Re: (Score:2)
Context engineering solves most of these problems. Claude behaves much differently for me than for my colleagues due to the differences in system prompt.
The problem I have is the peer review process. Getting merge approval is just as slow as it was before Claude started generating all this code. I can have agents run rounds and rounds of code reviews, but somebody who's not me ultimately has to click the approve button before I can merge anything. I haven't had to manually resolve any merge conflicts in a l
Doing it properly works but is NOT FUN (Score:5, Interesting)
We had problems with massive pull requests containing code the submitter does not understand.
The solution is to require developers to fully understand the code they submit, and check that understanding
When you take the time to understand the AI-written code ( often using the AI to explain its own code ), there is no problem and you can proceed significantly faster than without AI.
The problem is that that work to gain understanding is boring.
While I often had to do it in the days before AI, I did not become a programmer to spend all my time understanding code I did not write myself.
I think *that* is the serious problem for our industry.
Re: (Score:2)
I wouldn't bother testing the submitter. We need to be more willing to reject submissions without feeling the need to explain ourselves in depth.
If they're using AI, then it should be trivial for them to rewrite it. So nothing of value is lost by dropping bloated PRs.
compactness (Score:2)
From the little I see, AI code tends to not worry about compactness or elegance, just getting the job done. Human code that tries to be compact by excessive generalization and indirection is much harder to debug, even if it has fewer lines of code for the same job.
Re: (Score:2)
> AI code tends to not worry about compactness or elegance
This was definitely true in the beginning, but the latest generation models are putting out some pretty amazing results.
Re: (Score:3)
I have seen claude, qwen, and codex churn out some very nice code. I also see a lot of it as being written for not reason.
The spec might say ARGV[1] will be a dotted decimal ipv4 address. The next thing you know you will get a bunch code to see if it might be a hostname instead and try to resolve, check if was passed as single integer, make sure the user did not enter it in hex..
It might all work too! It is none the less 10s of extra lines to handle every possible way an addresses when what was called for
Re: (Score:2)
That's why you have tests to ensure they don't.
Otherwise you pass it a hostname instead of an IP address, the low-level code happily accepts it in code paths that have never been tested and before you know it something explodes.
Re: (Score:2)
which is why I suggested a human might have implemented a regex, test and quick exit with an error code.
Detect and Abend is usually a lot safe than guessing that 'grep' when you were expecting something like 198.51.100.45 means you should try to resolve grep.example.com and run with whatever address happens to comeback.
However in tool designed to be used by end users the try to figure-out what they intended and do it approach often is right. All depends on the stakes, but my example was one intended to be s
Complexity is cancer (Score:2)
AI tools easily generate more complexity.
We need better tools to understand, manage and reduce complexity, not tools that allow us to create more.
Re: (Score:2)
Yes, very much so. But that would be good engineering and good engineering is rare in the software space.
Re: (Score:2)
Unit tests are generally a mechanical translation of the requirements into software that verifies each requirement has been met. That certainly sounds like something AI could handle. Efficiency isn't really important for unit tests. Where I struggle is the cases where I don't have clearly stated requirements to work off of. If I have to write my own requirements, I'm gonna work backwards from the results my code generates to the requirements, and that generally makes for crap software,
Re: (Score:2)
Tests are as important as the code they test. It's just as essential that they be correct. If you're going to have AI write your tests, you need to review them, understand them, and verify that they really test what you intended. Otherwise you're likely to get a lot of useless tests that report success even when the code is wrong.
I'll sometimes intentionally break a piece of code just so I can see the test fail. I need to prove to myself that the test really does fail if the thing it's testing is wrong.
Re: (Score:2)
Though already CI/CD pipelines were choking on the unit tests. So they aren't *free*.
Had an LLM generated request that was pretty straightforward, and the LLM generated 272 unit tests for a very simple function. Can't really afford that.
Also, despite the 272 unit tests, when I tried to try it out in the way a human would, it immediately failed. All the mocking the AI generated never actually properly imitated the real world. However, the submitter was *quite* proud of having hundreds of unit tests, desp
Bad study (Score:2)
I've got a black swan for ya!
I just shipped software that AI wrote [practically] entirely that I would not have shipped otherwise. I don't think I'm the only one.
Re: (Score:2)
But how do you plan to maintain this software? That's the real effort of software engineering.
Re: (Score:2)
The general plan appears to be to tell the AI to rewrite it from scratch after a while.
Another reason for discrepency.. (Score:2)
Like always, more code != more valuable software behavior.
So you have the AI doing overtly silly things. Someone asked to adjust layout to fix a gap in a webpage. Eventually they submitted their pull request, hundreds and hundreds of lines of CSS. Often the exact same rule repeated 5 times, which I'm guessing is how many rounds the operator went with the system before the gap actually closed. The lines were *mostly* just '!important'ing already existing rules, which had no impact, but hey, worth a shot
AI levels things out (Score:2)
My understanding was that AI assistants make bad programmers more productive and good programmers less productive. I believe AI should review human code, but I also wouldn't take anything the AI says at face value; it hallucinates. Human verification of any AI decision should be mandatory.
Just a reminder: (Score:2)
"OpenAI withdrew three AI-generated math preprints and revised 14 others in October 2026 after a single sign error cascaded through a chain of dependent proofs." But they were so PROUD of their baby's hallucinations!
Re: (Score:2)
Hehehe, stupid is as stupid does. That is why real mathematicians verify a proofs before they build on the result.
Re: (Score:2)
We have an AI review all code changes for security holes because it's been programmed to be very good at finding ways to break security that make perfect sense but I would never have thought of. But it's purely advisory, I'd never let it have the final say on whether a change is safe.
Re: (Score:2)
I had students try that out in some thesis work. Results were that LLMs find some types of vulnerabilities reasonably reliably and others not at all or only rarely.
Skill problem (Score:2)
If you're not shipping code with AI faster than ever, it's 100% a skill and/or organizational problem.
In fact, it might be the organizations themselves which are the problem. Plenty of solo devs are doing amazing work with AI.
Re: (Score:2)
"it's 100% a skill and/or organizational problem", I agree. In my experience the recent models generate very good code complete with unit tests and thorough documentation. If you've got more new code than you know what to do with then maybe you need to get your programmers to focus more on the reviews. Make it a Jira ticket, that tends to get people's attention.
Re: (Score:2)
Nonsense. The real problem on display here is that you have no clue how to produce quality code.
NO SHIT (Score:2)
Glazing machines
Re: (Score:2)
> Glazing machines
They're called "enrobers." (humor)
So using LLMs makes coding more expensive... (Score:2)
Because you just have to pay the same number of coders even if more are code reviewers not, but you pay for the LLM on top of that as well. I am sooo not surprised. And, apparently, reviewing LLM code is mind-numbing and far more unpleasant than reviewing human code. Hence there is another really bad problem in there.
Just have Claude do the code reviews! (Score:2)
What could possibly go wrong?
I joke, but this is *literally* what my bosses were telling me I should have my teams do, to speed up AI coding.
I think it all depends (Score:2)
Like most questions, it depends on what kind of code you're writing, and how thoroughly it must be reviewed.
If you're writing code that is used by thousands of customers, yeah, I can totally see how review would be a bottleneck.
But if you're writing code that's used by a single team or department at your own company, AI probably does actually speed up this kind of work. An outage or bug in this type of software is often not a serious disruption.
Shocked (Score:3)
Turns out it takes longer to review code that no-one understands.
The solution, obviously, is to have the AI review the code too so humans don't need to understand it.
Re: (Score:2)
Human written code is also frequently not understandable by anyone but the author, and even that is in question.
Re: (Score:3)
Working on the code every day myself is what makes it possible for me to review code that people contribute.
If I use AI to write it all, then I don't internalize the code anymore, and I slowly lose the ability to review it.
Been nice knowing you all, the industry is going to hit a brick wall in part because of reliance on AI in the very stages that should include humans.
Re: (Score:2)
That is the way we're heading. So don't build your life around relying on software working because nothing is going to be reliable in future.
One of our problems is that we can't make major changes to the software because it invalidates over a decade of use in the field and massive amounts of testing required to ensure it doesn't kill people. Sooner or later that will be replaced with AI testing, so the code will be developed, tested and installed without a human ever checking whether the AI did it right or