Google teases AlphaCode 2 – a code-generating AI revamped with Gemini
(2023/12/07)
- Reference: 1701926768
- News link: https://www.theregister.co.uk/2023/12/07/alphacode_2_software/
- Source link:
Google's latest code-generating model – AlphaCode 2, powered by its Gemini Pro system and making its public debut on Wednesday – reportedly scored above the 99.5 percentile of participants competing in programming contests online.
Researchers from Google DeepMind fine tuned [1]Gemini Pro on a dataset to beef up its problem-solving skills to create AlphaCode 2. The dataset contained about 15,000 problems taken from CodeForces – a competitive programming site – and 30 million samples of code written by humans.
The model was fine tuned further on an additional dataset of "higher quality," but it's not really clear what kind of data was used or how much exactly, according to the scant details in the [2]technical report [PDF]. When AlphaCode 2 was tested on 77 problems across 12 CodeForces contests – where it competed against more than 8,000 programmers in total – it managed to solve 43 percent of them. AlphaCode 2 submitted its answers in C++.
[3]
For comparison, the previous [4]AlphaCode system solved 25 percent of a different set of problems also set by CodeForces.
[5]
[6]
"Mapping this to competition rankings, we estimate that AlphaCode 2 sits at the 85th percentile on average – ie it performs better than 85 [percent of entrants], ranking just between the 'Expert' and 'Candidate Master' categories on Codeforces," the researchers claimed.
Your jobs are safe … for now
In two contests out of the twelve in which it competed, AlphaCode 2 outperformed 99.5 percent of participants. Although impressive, the competition conditions were different for the machine and for humans.
AlphaCode 2 can submit up to ten different solutions for each problem and score points if one of them is correct – unlike the human candidates, who have one go at cracking the challenge.
AlphaCode 2 also operates very differently from biological programmers. Given a problem, it generates about a million different code samples, which are then filtered down. Random scripts that are irrelevant and don't match the problem's description – or those that generate the wrong sample test answers, or don't compile at all – are removed.
[7]
"Each competitive programming problem contains at least one public input/output test indicating how code samples should behave. We execute each code sample on the corresponding test input, and filter out all which do not produce the expected output and therefore could not have been correct," the researchers explained.
Filtering gets rid of 95 percent of code samples generated by AlphaCode 2. Next, a clustering algorithm collects ranks the 50,000 remaining programs by similarity and sorts them into different groups. The ten biggest clusters are then scored by a separate Gemini Pro model trained to predict their accuracy. The samples across the ten different clusters are then ranked from best to last, and the top one from each group is submitted.
[8]Google's DeepMind says its AI coding bot is 'competitive' with humans
[9]Microsoft touts Visual Studio Code as a Java juggernaut
[10]Microsoft reportedly runs GitHub's AI Copilot at a loss
Human coders usually think of different strategies to solve a problem, then home in on the most promising idea and write that up, instead of trying out millions of different solutions. Success depends on understanding the problems and coming up with clever mathematical tricks to solve them.
AlphaCode 2's brute force approach – filtering all of its code, and running the different models to score and rank the best ones – is computationally intensive, so it's probably too expensive to release until it's more efficient.
"Despite AlphaCode 2's impressive results, a lot more remains to be done before we see systems that can reliably reach the performance of the best human coders. Our system requires a lot of trial and error, and remains too costly to operate at scale. Further, it relies heavily on being able to filter out obviously bad code samples," the researchers admitted.
[11]
Still, AlphaCode 2 is a big improvement over the old AlphaCode and is more than 10,000 times more sample efficient, Google claims. It only requires 100 generated samples to reach the same performance as AlphaCode, which required a million.
Google DeepMind believes that it could build an even better code-writing model using Gemini Ultra – a larger and more powerful large language model than Gemini Pro – and said it was working to try and make its capabilities available to developers.
"We hope this kind of interactive coding will be the future of programming, where programmers make use of highly-capable AI models as collaborative tools that can help them reason about the problems, propose code designs, and assist with implementation," the team concluded.
"We are working towards bringing AlphaCode 2's unique capabilities to our foundation Gemini models as a first step to make this new programming paradigm available to everyone." ®
Get our [12]Tech Resources
[1] https://www.theregister.com/2023/12/06/google_gemini_ai/
[2] https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2022/02/03/alphacode_automated_coding/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/02/03/alphacode_automated_coding/
[9] https://www.theregister.com/2023/12/05/java_visual_studio_code/
[10] https://www.theregister.com/2023/10/11/github_ai_copilot_microsoft/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
Researchers from Google DeepMind fine tuned [1]Gemini Pro on a dataset to beef up its problem-solving skills to create AlphaCode 2. The dataset contained about 15,000 problems taken from CodeForces – a competitive programming site – and 30 million samples of code written by humans.
The model was fine tuned further on an additional dataset of "higher quality," but it's not really clear what kind of data was used or how much exactly, according to the scant details in the [2]technical report [PDF]. When AlphaCode 2 was tested on 77 problems across 12 CodeForces contests – where it competed against more than 8,000 programmers in total – it managed to solve 43 percent of them. AlphaCode 2 submitted its answers in C++.
[3]
For comparison, the previous [4]AlphaCode system solved 25 percent of a different set of problems also set by CodeForces.
[5]
[6]
"Mapping this to competition rankings, we estimate that AlphaCode 2 sits at the 85th percentile on average – ie it performs better than 85 [percent of entrants], ranking just between the 'Expert' and 'Candidate Master' categories on Codeforces," the researchers claimed.
Your jobs are safe … for now
In two contests out of the twelve in which it competed, AlphaCode 2 outperformed 99.5 percent of participants. Although impressive, the competition conditions were different for the machine and for humans.
AlphaCode 2 can submit up to ten different solutions for each problem and score points if one of them is correct – unlike the human candidates, who have one go at cracking the challenge.
AlphaCode 2 also operates very differently from biological programmers. Given a problem, it generates about a million different code samples, which are then filtered down. Random scripts that are irrelevant and don't match the problem's description – or those that generate the wrong sample test answers, or don't compile at all – are removed.
[7]
"Each competitive programming problem contains at least one public input/output test indicating how code samples should behave. We execute each code sample on the corresponding test input, and filter out all which do not produce the expected output and therefore could not have been correct," the researchers explained.
Filtering gets rid of 95 percent of code samples generated by AlphaCode 2. Next, a clustering algorithm collects ranks the 50,000 remaining programs by similarity and sorts them into different groups. The ten biggest clusters are then scored by a separate Gemini Pro model trained to predict their accuracy. The samples across the ten different clusters are then ranked from best to last, and the top one from each group is submitted.
[8]Google's DeepMind says its AI coding bot is 'competitive' with humans
[9]Microsoft touts Visual Studio Code as a Java juggernaut
[10]Microsoft reportedly runs GitHub's AI Copilot at a loss
Human coders usually think of different strategies to solve a problem, then home in on the most promising idea and write that up, instead of trying out millions of different solutions. Success depends on understanding the problems and coming up with clever mathematical tricks to solve them.
AlphaCode 2's brute force approach – filtering all of its code, and running the different models to score and rank the best ones – is computationally intensive, so it's probably too expensive to release until it's more efficient.
"Despite AlphaCode 2's impressive results, a lot more remains to be done before we see systems that can reliably reach the performance of the best human coders. Our system requires a lot of trial and error, and remains too costly to operate at scale. Further, it relies heavily on being able to filter out obviously bad code samples," the researchers admitted.
[11]
Still, AlphaCode 2 is a big improvement over the old AlphaCode and is more than 10,000 times more sample efficient, Google claims. It only requires 100 generated samples to reach the same performance as AlphaCode, which required a million.
Google DeepMind believes that it could build an even better code-writing model using Gemini Ultra – a larger and more powerful large language model than Gemini Pro – and said it was working to try and make its capabilities available to developers.
"We hope this kind of interactive coding will be the future of programming, where programmers make use of highly-capable AI models as collaborative tools that can help them reason about the problems, propose code designs, and assist with implementation," the team concluded.
"We are working towards bringing AlphaCode 2's unique capabilities to our foundation Gemini models as a first step to make this new programming paradigm available to everyone." ®
Get our [12]Tech Resources
[1] https://www.theregister.com/2023/12/06/google_gemini_ai/
[2] https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2022/02/03/alphacode_automated_coding/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/02/03/alphacode_automated_coding/
[9] https://www.theregister.com/2023/12/05/java_visual_studio_code/
[10] https://www.theregister.com/2023/10/11/github_ai_copilot_microsoft/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXGl3O7KbORbeuK5kqjHXgAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
Was the Coding Training set
Anonymous Coward
attributed to an infinite number of Code Monkeys efforts from Stack Overflow?
Viva la coding gibberish!
HuBo
AlphaCode 2 seems to be generating code in the exact opposite way to the methods we have been taught to use when writing computer programs. I'm not sure that I would ever want to advertise, never mind release, a product like that.
Re: Viva la coding gibberish!
Mast1
And does it leave the code well documented so that in 40 years time the same functionality/algorithm can be ported to a new platform or programming language, and verified?
It would save having to keep COBOL programmers in suspended animation.......
C++
tiggity
Interesting choice of C++, given the push for more memory safe languages such as Rust.
(e.g. also from today https://www.theregister.com/2023/12/07/memory_correction_five_eyes/)
Even other old, commonly used languages, such as Java, C# etc. would have been a better choice than C++ in "safety" terms
I'm guessing that choice is because its far easier to slurp lots of C++ code samples ( & lots of "code challenges" have C++ as a language that can / should be used)
If you can do it once in a million, you can train on it.
FeepingCreature
As long as probability of success is finitely greater than zero, it can usually be engineered to approach one.
A lot more remains to be done
abend0c4
I imagine they're still working on how to make the code detect it's running in a Google environment and therefore limit its operational life to 18 months.
Yep, the numbers are horribly bad
It generates a million random answers that are then cut down to a single one, and that one is wrong 57% of the time even in an artificial programing contest situation. Repeated ten times over it might cough up something correct, but who chooses which of those ten attempts is correct? Humans.
This proves Gemini does NOT have the reasoning skills people are trying to attribute to it. It still sucks extremely hard at generating answers, there's no reasoning or understanding there, just randomness. The improvement is in the step that filters out the one answer that's most likely to fool a human into believing it is correct. And even then the human has to do the actual work in the end by running the model several times and filtering out the one answer that is, by sheer luck and the law of large numbers, actually correct.