AI can improve on code it writes, but you have to know how to ask
- Reference: 1736269210
- News link: https://www.theregister.co.uk/2025/01/07/ai_can_write_improved_code_research/
- Source link:
Max Woolf, senior data scientist at Buzzfeed, on Thursday [1]published an experiment in LLM prompting, to see whether LLMs can optimize the code they suggest on demand.
Woolf explained, "If code can indeed be improved simply through iterative prompting such as asking the LLM to 'make the code better' — even though it’s very silly — it would be a massive productivity increase. And if that’s the case, what happens if you iterate on the code too much?"
[2]
This is what happened: Anthropic's Claude was tasked with writing Python code to find the difference between the smallest and the largest numbers whose digits sum up to 30, given a list of one million random integers between 1 and 100,000. And the LLM [3]returned a functional solution that worked .
[4]
[5]
The [6]initial solution , which Woolf characterized as something a novice programmer might write, took an average 657 milliseconds to run on an Apple M3 Pro Macbook Pro.
And when asked to "write better code," Claude responded with optimized code that performed 2.7x faster.
[7]
Asked again, Claude made further improvements, returning code that incorporates multithreading for a 5.1x performance improvement over the initial implementation, though it was at the cost of creating errors that require fixing.
Iterations three and four produced further gains, resulting in speedups of 4.1x and 99.7x.
Woolf then repeated the experiment using "prompt engineering," which simply means providing the LLM with more detail about what's expected and how to proceed. This was done in part by modifying the Claude system prompt, available via API as a way to set the rules for LLMs, to do things like use certain code efficiency strategies.
[8]
"Although it's both counterintuitive and unfun, a small amount of guidance asking the LLM specifically what you want, and even giving a few examples of what you want, will objectively improve the output of LLMs more than the effort needed to construct said prompts," observes Woolf in his write-up.
[9]Just how deep is Nvidia's CUDA moat really?
[10]Cheat codes for LLM performance: An introduction to speculative decoding
[11]Google Gemini 2.0 Flash comes out with real-time conversation, image analysis
[12]Open source maintainers are drowning in junk bug reports written by AI
The [13]results with prompt engineering produced more sophisticated, faster code, though with more bugs.
"In all, asking an LLM to 'write code better' does indeed make the code better, depending on your definition of better," Woolf concludes. "Through the use of the generic iterative prompts, the code did objectively improve from the base examples, both in terms of additional features and speed.
"Prompt engineering improved the performance of the code much more rapidly and consistently, but was more likely to introduce subtle bugs as LLMs are not optimized to generate high-performance code. As with any use of LLMs, your mileage may vary, and in the end it requires a human touch to fix the inevitable issues no matter how often AI hypesters cite LLMs as magic."
Woolf concludes that LLMs won't replace software engineers anytime soon because a software engineering background is necessary to distinguish good code and to understand domain-specific constraints.
A recent research paper from computer scientists at Northeastern University, Wellesley College, and Oberlin College offers support for this view. The [14]paper , titled "Substance Beats Style: Why Beginning Students Fail to Code with LLMs," examines whether prompt style – the arrangement of words – or prompt substance – the terms used to frame the problem – matter more.
"Overall, our findings support the view that the information content of prompts is more important than wording," conclude authors Francesca Lucchetti, Zixuan Wu, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson.
In other words, to get a good answer from an LLM, it helps to have a strong background in the topic of inquiry. So experienced developers will get better results asking LLMs for help than neophytes. ®
Get our [15]Tech Resources
[1] https://minimaxir.com/2025/01/write-better-code/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z32yEnKFsntpXb-3spwd7QAAANE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://github.com/minimaxir/llm-write-better-code/blob/main/python_30_casual_use.md
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z32yEnKFsntpXb-3spwd7QAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z32yEnKFsntpXb-3spwd7QAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://github.com/minimaxir/llm-write-better-code/blob/main/python_30_casual_use.md
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z32yEnKFsntpXb-3spwd7QAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z32yEnKFsntpXb-3spwd7QAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/12/17/nvidia_cuda_moat/
[10] https://www.theregister.com/2024/12/15/speculative_decoding/
[11] https://www.theregister.com/2024/12/11/google_gemini_20_flash_shines/
[12] https://www.theregister.com/2024/12/10/ai_slop_bug_reports/
[13] https://github.com/minimaxir/llm-write-better-code/blob/main/python_30_prompt_engineering.md
[14] https://arxiv.org/abs/2410.19792
[15] https://whitepapers.theregister.com/
Re: Await Fix
Why do humans make mistakes and need to alter their first drafts?
That's a fairly loose definition of improve if it goes faster but has more bugs in it. Or am I being difficult if I expect 'improve' to not include 'more bugs' as a result?
Doing less worse faster
If I am getting this right,... The programmer using the LLM needs to review the generated code and re-engineer the prompt for each and every time something does not pass review. And we are talking about rather simple problems.
How is this effective in a larger setting? Normal problems are not simple because they are much larger, intertwined with other problems and have a large set of boundary conditions that must be fulfilled. The programmer might just put his/her thinking cap on, think up a solution and tap it into an editor. Few iterations with proper and appropriate test-cases and done. And all of that without trying to teach an LLM to do its job worse than any experienced programmer.
So now software devs have to...
...learn to write Rust and how to write prompts for LLMs? Queue riots.
Re: So now software devs have to...
s/Queue/Cue/
soz!
Re: So now software devs have to...
You think they won't be lining up for it?
I have a very simple view of LLM's and "AI" writing code. I remember learning. I remember all those tutorials available on the internet and how they could get me started but missed out so much. I remember all the times I had to look things up when things didn't work and how half the time they didn't fix it or if they did but they introduced another issue. I remember learning to go to the language itself and the documentation to fully understand something to the point where I was happy with it and confident I knew exactly what it was doing and why. Learning the correct way to interact with said language or piece of code.
An LLM can't do that. It's using all the reference data I used but it can't tell the right way from wrong. It's a nice idea don't get me wrong. I'm not a luddite. I just hope we don't get to a point where it's the defacto way of doing things but I know how the world works and if you can get people to code on the cheap then people will do it.
It's using the same reference data but not in the same way that you you do.
I tried out LLM code writing in PhpStorm. The LLM would generate multiple lines of code that nearly, but not actually, did what I wanted it to do. Plausible but incorrect: LLMs are bullshit generators, nothing more. You cannot trust them to be accurate or correct.
Why bother?
" find the difference between the smallest and the largest numbers whose digits sum up to 30 [for] integers between 1 and 100,000 ". By inspection:
smallest = 3,999
largest = 99,930
difference = 95,931
done!
Rationale: with 1,000,000 integers of value 1 to 100,000 randomly generated, each integer has, on average, possibility of being generated 10 times. So all integers between 1 and 100,000 are quite likely to be produced at least once. Ergo, take the max and min, subtract, and done. If AI existed, it would have come up with this ...
What the report doesn't say is it ends up vectorising and parallelising the code. And switches from an interpreted library to a JIT library.
Well, there is a few videos by a guy who codes games for the Nintendo 64 (yes really) and he tested AI helping him to code, he got mixed results.
Granted if he tried again now the tesults may be better or worse because these AI chatbots eat a lot of garbage data and just because they get feed similar garbage data a million times it doesn't stop the data from being wrong.
If you want AI to help coding get a specific AI that does that, these glorified chatbots are just too unreliable.
I love AI
Because it slows down the idiots in the team even more, and makes me look even better.
Stop dicking around with the AI and just write the fuckin' code! It'll take half the time!
Await Fix
As an experiment with Ollama I asked it to write a .NET task-based multi-threaded web download method in C# (as you do). It came up with something plausible that used the C# await keyword, however it hadn't marked the method as async. I said I thought the async was needed and it responded with something akin to "Yes, you're right. Here's the proper version." Which included the async keyword. So it can correct it generated output, but why give me erroneous code in the first place?