GitHub's boast that Copilot produces high-quality code challenged
- Reference: 1733214614
- News link: https://www.theregister.co.uk/2024/12/03/github_copilot_code_quality_claims/
- Source link:
Cîmpianu, based in Romania, published a [1]blog post in which he assails the statistical rigor of GitHub's Copilot code quality data.
If you can't write good code without an AI, then you shouldn't use one in the first place
GitHub last month [2]cited research indicating that developers using Copilot:
Had a 56 percent greater likelihood to pass all ten unit tests in the study ( [3]p=0.04 );
Wrote 13.6 percent more lines of code with GitHub Copilot on average without a code error (p=0.002);
Wrote code that was more readable, reliable, maintainable, and concise by 1 to 3 percent (p=0.003, p=0.01, p=0.041, p=0.002, respectively);
Were 5 percent more likely to have their code approved (p=0.014).
The first phase of the study relied on 243 developers with at least five years of Python experience who were randomly assigned to use GitHub Copilot (104) or not (98) – only 202 developer submissions ended up being valid.
Each group created a web server to handle fictional restaurant reviews, supported by ten unit tests. Thereafter, each submission was reviewed by at least ten of the participants – a process that produced only 1,293 code reviews rather than the 2020 that 10x multiplication might lead one to expect.
GitHub declined The Register 's invitation to respond to Cîmpianu's critique.
[4]
Cîmpianu takes issue with the choice of assignment, given that writing a basic Create, Read, Update, Delete (CRUD) app is the subject of endless online tutorials and therefore certain to have been included in training data used by code completion models. A more complex challenge would be better, he contends.
[5]
[6]
He then goes on to question GitHub's inadequately explained graph that shows 60.8 percent of developers using Copilot passed all ten unit tests while only 39.2 percent of developers not using Copilot passed all the tests.
That would be about 63 Copilot using developers out of 104 and about 38 non-Copilot developers out of 98 based on the firm's cited developer totals. But GitHub's post then reveals: "The 25 developers who authored code that passed all ten unit tests from the first phase of the study were randomly assigned to do a blind review of the anonymized submissions, both those written with and without GitHub Copilot."
[7]
Cîmpianu observes that something doesn't add up here. One possible explanation is that GitHub misapplied the definite article "the" and simply meant 25 developers out of the total of 101 who passed all the tests were selected to do code reviews.
More significantly, Cîmpianu takes issue with GitHub's claim that devs using Copilot produced significantly fewer code errors. As GitHub put it, "developers using GitHub Copilot wrote 18.2 lines of code per code error, but only 16.0 without. That equals 13.6 percent more lines of code with GitHub Copilot on average without a code error (p=0.002)."
Cîmpianu argues that 13.6 percent is a misleading use of statistics because it only refers to two additional lines of code. While allowing that one might argue that adds up over time, he points out that the supposed error reduction is not actual error reduction. Rather it's coding style issues or [8]linter warnings.
[9]
As GitHub acknowledges in its definition of code errors: "This did not include functional errors that would prevent the code from operating as intended, but instead errors that represent poor coding practices."
[10]OpenAI denies it is building ad biz model into its platform
[11]Claims of 'open' AIs are often open lies, research argues
[12]Brits think AI in the workplace is all chat, no bot for now
[13]Yup, half of that thought-leader crap on LinkedIn is indeed AI scribbled
Cîmpianu is also unhappy with GitHub's claim that Copilot-assisted code was more readable, reliable, maintainable, and concise by 1 to 3 percent. He notes that the metrics for code style and code reviews can be highly subjective, and that details about how code was assessed have not been provided.
Cîmpianu goes on to criticize GitHub's decision to use the same developers who submitted code samples for code evaluation, instead of an impartial group.
"At the very least, I can appreciate they only made the developers who passed all unit tests do the reviewing," he wrote. "But remember, dear reader, that you're baited with a 3 percent increase in preference from some random 25 developers, whose only credentials (at least mentioned by the study) are holding a job for five years and passing ten unit tests."
Cîmpianu points to a [14]2023 report from GitClear that found GitHub Copilot reduced code quality.
Another [15]paper by researchers affiliated with Bilkent University in Turkey, released in April 2023 and revised in October 2023, found that ChatGPT, GitHub Copilot, and Amazon Q Developer (formerly CodeWhisperer) all produce errors. And to the extent those errors produced "code smells" – poor coding practices that can give rise to vulnerabilities – "the average time to eliminate them was 9.1 minutes for GitHub Copilot, 5.6 minutes for Amazon CodeWhisperer, and 8.9 minutes for ChatGPT."
That paper concludes, "All code generation tools are capable of generating valid code nine out of ten times with mostly similar types of issues. The practitioners should expect that for 10 percent of the time the generated code by the code generation tools would be invalid. Moreover, they should test their code thoroughly to catch all possible cases that may cause the generated code to be invalid."
Nonetheless, a lot of developers are using AI coding tools like GitHub Copilot as an alternative to searching for answers on the web. Often, a partially correct code suggestion is enough to help inexperienced coders make progress. And those with substantial coding experience also see value in AI code suggestion models.
As veteran open source developer Simon Willison observed in [16]a recent interview [VIDEO]: "Somebody who doesn't know how to program can use Claude 3.5 artefacts to produce something useful. Somebody who does know how to program will do it better and faster and they'll ask better questions of it and they will produce a better result."
For GitHub, maybe the message is that code quality, like security, isn't top of mind for many developers.
Cîmpianu contends it shouldn't be that way. "[I]f you can't write good code without an AI, then you shouldn't use one in the first place," he concludes.
Try telling that to the authors who don't write good prose, the recording artists who aren't good musicians, the video makers who never studied filmmaking, and the visual artists who can't draw very well. ®
Get our [17]Tech Resources
[1] https://jadarma.github.io/blog/posts/2024/11/does-github-copilot-improve-code-quality-heres-how-we-lie-with-statistics/
[2] https://github.blog/news-insights/research/does-github-copilot-improve-code-quality-heres-what-the-data-says/
[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC4111019/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z07k1x54Ytz0ztFCF7WkfgAAABY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z07k1x54Ytz0ztFCF7WkfgAAABY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z07k1x54Ytz0ztFCF7WkfgAAABY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z07k1x54Ytz0ztFCF7WkfgAAABY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://en.wikipedia.org/w/index.php?title=Lint_(software)&oldid=1260589258
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z07k1x54Ytz0ztFCF7WkfgAAABY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2024/12/02/openai_mulls_other_revenue_streams/
[11] https://www.theregister.com/2024/12/02/open_ai_research/
[12] https://www.theregister.com/2024/12/01/uk_workplace_ai/
[13] https://www.theregister.com/2024/11/28/linkedin_ai_posts/
[14] https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality
[15] https://arxiv.org/abs/2304.10778
[16] https://youtu.be/rLcKbvmegag?feature=shared&t=2503
[17] https://whitepapers.theregister.com/
Re: What will it mean to be a....
> However, in future, with an AI widget telling a layman how to do accountancy, would that laymen therefore be able to claim they no longer need an accountant?
Sure, if he's stupid. AI can't even add up the number of Rs in "raspberry" so I certainly wouldn't trust it with my taxes.
> And if the advice from the AI widget is sufficiently good that they are an accountant themselves?
It's not. Just like a ton of people you're confusing real artificial intelligence with a program that spews out words based on a statistical model of which words usually follow other ones, which is all we have at the moment.
Re: What will it mean to be a....
> If people are going to use such tools to do things they couldn't do before, what will it mean to be a professional?
There will be less professionals. And the ones left will need to compete with AIs for jobs/customers willing to pay a premium for someone to speak to - instead of talking to something.
> However, in future, with an AI widget telling a layman how to do accountancy, would that laymen therefore be able to claim they no longer need an accountant? And if the advice from the AI widget is sufficiently good that they are an accountant themselves?
The future envisioned by supporters of an AI transformation would probably be more like an AI _being_ the actual accountant. The accounting company will morph into a small/med datacenter operations facility with just as many humans left to cover the not automatable, manual tasks.
Then again, most people of today's service economy will have no longer paying jobs in that scenario - so instead of an AI that does taxes we probably will see AIs that generates perfect requests for welfare/benefits.
I use copilot in VS code every day. My own experience is more 50/50. I mostly use it for python scripts to be run via Airflow. It is quite good at boiler plate stuff I can’t be bothered to write (basic REST calls, mock data for unit tests, function docs) and it can give clues if you want to do something you’ve never done before. The new copilot edits feature is quite good as you can prompt it to edit across a set of files.
It has some drawbacks though. It often lags behind on new library/tool versions (current knowledge cutoff is October 2023).
It is prone to hallucinations and coming up with methods that don’t exist. I let it write some unit tests that passed so I moved on. Turned out it hallucinated an assertion method on a magicmock which ran without doing anything so the tests were pointless! Lesson for me there to thoroughly check its output.
Also, it works best in a large project with established code patterns. If you ask it to write things from scratch it often comes up with garbage.
So overall it is useful but needs using with extreme caution!
> Lesson for me there to thoroughly check its output.
Really? And does it actually end up saving you any time/effort after all that? In my experience, the answer is "no"
> It is prone to hallucinations
AI doesn't hallucinate, it is prone to errors.
AI assistance
Surely AI is just a tool to be used not a robot trusted to invent finished solutions. Who expects it to work straight out of the box?
Try telling that to...
Try telling that to the authors who don't write good prose, the recording artists who aren't good musicians, the video makers who never studied filmmaking, and the visual artists who can't draw very well.
Easier to hand bouquets to those few not on this list. Authors that cannot write are pretty much the rule, u-tube is saturated with the second and third while artists that couldn't draw water from a well are probably responsible for the infantile "illustrations" found on the walls of toilet cubicles.
> And those with substantial coding experience also see value in AI code suggestion models.
Perhaps, but there is also value --- more, in fact --- in stack overflow and in the example sections in documentation.
How many trillions?
> Try telling that to the authors who don't write good prose, the recording artists who aren't good musicians, the video makers who never studied filmmaking, and the visual artists who can't draw very well.
No problem: Your output is as good in your respective field as the amateur robo-coding is in ours. Trash.
It sounds like the test was about how fast you code something you already know how to do, rather than getting out of your comfort zone. It is, of course, the latter that I'm sure management are way more interested in getting right faster.
Try telling that to the authors who don't write good prose, the recording artists who aren't good musicians, the video makers who never studied filmmaking, and the visual artists who can't draw very well.
what a really petulant swipe at the OP? like are there bad authors, artists, and musicians? yes. do I rely on their skill to keep me alive? not at all.
bad coders, though... the chance of my life being impacted negatively by poor programming practices and even AI-slop generated botshit might be pretty low, one hopes... but it certainly would not be zero .
What will it mean to be a....
There's an underlying question here which I've found troubling.
If people are going to use such tools to do things they couldn't do before, what will it mean to be a professional?
For example, to be an accountant today you have to not only study and pass exams but also practice it for a period of time. Then you can rightfully claim to be an accountant and people won't call you out for overstating your competence.
However, in future, with an AI widget telling a layman how to do accountancy, would that laymen therefore be able to claim they no longer need an accountant? And if the advice from the AI widget is sufficiently good that they are an accountant themselves?
I pick this example as there is a bar set with a professional standards body, but this could apply to anything in the knowledge economy.