OpenAI develops AI model to critique its AI models
(2024/06/28)
- Reference: 1719553446
- News link: https://www.theregister.co.uk/2024/06/28/openai_criticgpt_ai/
- Source link:
To help catch code errors made by ChatGPT, OpenAI uses human AI trainers in the hope of improving the model. To help the human trainers, OpenAI has developed another AI model called CriticGPT – in case the humans don't spot the mistakes.
The Microsoft-championed super lab on Thursday issued [1]a paper [PDF] titled, "LLM Critics Help Catch LLM Bugs," that explains the approach.
Generative AI models like GPT-4o get trained on massive amounts of data and then go through a refinement process called Reinforcement Learning from Human Feedback (RLHF).
[2]
This commonly involves human workers, [3]often hired through crowdsourcing platforms, interacting with models and annotating their responses to various questions. When Time Magazine looked into this last year, it [4]found OpenAI using Kenyan workers paid less than $2 per hour to improve its models.
[5]
[6]
The goal is to teach the model which answer is preferred, so it performs better. But RLHF becomes less effective as models become more capable. Human AI trainers find it harder to identify flawed answers, particularly when the chatbot reaches the point that it knows more than its teachers.
So as an aid to the people tasked with providing feedback to make its models more capable of generating programming code, OpenAI created another model – to critique those generative responses.
[7]
"We've trained a model, based on GPT-4, called CriticGPT, to catch errors in ChatGPT's code output," the AI startup explained in a [8]blog post . "We found that when people get help from CriticGPT to review ChatGPT code they outperform those without help 60 percent of the time."
[9]
Screenshot of diagram from OpenAI paper, "LLM Critics Help Catch LLM Bugs" – Click to enlarge
In other words, this isn't an autonomous feedback loop from one chatbot to another – it's a way to augment the knowledge of those administering reinforcement learning.
This approach apparently leads to better results than just relying on crowdsourced workers – who at $2 per hour probably aren't computer science professors or trenchant technical writers, or whatever the prevailing annotation rate happens to be.
[10]Anthropic tries 'to enable beneficial uses' of AI by government agencies
[11]Reddit hopes robots.txt tweak will do the trick in scaring off AI training data scrapers
[12]AI's appetite for power could double datacenter electricity bills by 2030
[13]OpenAI to pull plug on 'unsupported' nations – cough, China – from July 9
According to the paper, the results show "that LLMs catch substantially more inserted bugs than qualified humans paid for code review, and further that model critiques are preferred over human critiques more than 80 percent of the time."
The finding that CriticGPT enables AI trainers to write better model response critiques isn't entirely surprising. Mediocre office temps presumably would write better crafted email messages with the help of generative AI too.
But AI help comes with a cost. When human contractors work in conjunction with CriticGPT, the resulting critiques of ChatGPT responses have a lower rate of hallucinations (invented bugs) than CriticGPT responses alone – but that error rate is still higher than if a human AI trainer had been left to respond without AI assistance.
"Unfortunately, it's not obvious what the right tradeoff between hallucinations and bug detection is for an overall RLHF system that uses critiques to enhance model performance," the paper concedes. ®
On the subject of very artificial intelligence...
Tests show OpenAI's ChatGPT sometimes gives users [14]broken URLs for at least ten publications that have struck licensing deals with, you guessed it, OpenAI. Those partnerships were supposed to ensure that when the chatbot generates an answer based on one of those titles' articles, the system kindly namechecks the source and provides a link to the original piece. So much for that.
The Center for Investigative Reporting, a nonprofit which operates Mother Jones and Reveal, is [15]suing OpenAI and its backer Microsoft for allegedly unlawfully using its copyrighted content without permission nor compensation.
And speaking of Microsoft, a study has [16]demonstrated that the Windows giant's Bing translation and web search engine in China censors more aggressively than its Chinese competitors.
Get our [17]Tech Resources
[1] https://cdn.openai.com/llm-critics-help-catch-llm-bugs-paper.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://arxiv.org/abs/2306.07899
[4] https://time.com/6247678/openai-chatgpt-kenya-workers/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/
[9] https://regmedia.co.uk/2024/06/27/openai_paper_image.jpg
[10] https://www.theregister.com/2024/06/27/anthropic_claude_government/
[11] https://www.theregister.com/2024/06/27/reddit_ai_info/
[12] https://www.theregister.com/2024/06/26/datacenter_electricity_demand_study/
[13] https://www.theregister.com/2024/06/25/openai_unsupported_countries/
[14] https://www.niemanlab.org/2024/06/chatgpt-is-hallucinating-fake-links-to-its-news-partners-biggest-investigations/
[15] https://apnews.com/article/ai-media-lawsuits-center-for-investigative-reporting-chatgpt-mother-jones-c48452889750479410b65a119537746c
[16] https://restofworld.org/2024/microsoft-bing-chinese-censorship/
[17] https://whitepapers.theregister.com/
The Microsoft-championed super lab on Thursday issued [1]a paper [PDF] titled, "LLM Critics Help Catch LLM Bugs," that explains the approach.
Generative AI models like GPT-4o get trained on massive amounts of data and then go through a refinement process called Reinforcement Learning from Human Feedback (RLHF).
[2]
This commonly involves human workers, [3]often hired through crowdsourcing platforms, interacting with models and annotating their responses to various questions. When Time Magazine looked into this last year, it [4]found OpenAI using Kenyan workers paid less than $2 per hour to improve its models.
[5]
[6]
The goal is to teach the model which answer is preferred, so it performs better. But RLHF becomes less effective as models become more capable. Human AI trainers find it harder to identify flawed answers, particularly when the chatbot reaches the point that it knows more than its teachers.
So as an aid to the people tasked with providing feedback to make its models more capable of generating programming code, OpenAI created another model – to critique those generative responses.
[7]
"We've trained a model, based on GPT-4, called CriticGPT, to catch errors in ChatGPT's code output," the AI startup explained in a [8]blog post . "We found that when people get help from CriticGPT to review ChatGPT code they outperform those without help 60 percent of the time."
[9]
Screenshot of diagram from OpenAI paper, "LLM Critics Help Catch LLM Bugs" – Click to enlarge
In other words, this isn't an autonomous feedback loop from one chatbot to another – it's a way to augment the knowledge of those administering reinforcement learning.
This approach apparently leads to better results than just relying on crowdsourced workers – who at $2 per hour probably aren't computer science professors or trenchant technical writers, or whatever the prevailing annotation rate happens to be.
[10]Anthropic tries 'to enable beneficial uses' of AI by government agencies
[11]Reddit hopes robots.txt tweak will do the trick in scaring off AI training data scrapers
[12]AI's appetite for power could double datacenter electricity bills by 2030
[13]OpenAI to pull plug on 'unsupported' nations – cough, China – from July 9
According to the paper, the results show "that LLMs catch substantially more inserted bugs than qualified humans paid for code review, and further that model critiques are preferred over human critiques more than 80 percent of the time."
The finding that CriticGPT enables AI trainers to write better model response critiques isn't entirely surprising. Mediocre office temps presumably would write better crafted email messages with the help of generative AI too.
But AI help comes with a cost. When human contractors work in conjunction with CriticGPT, the resulting critiques of ChatGPT responses have a lower rate of hallucinations (invented bugs) than CriticGPT responses alone – but that error rate is still higher than if a human AI trainer had been left to respond without AI assistance.
"Unfortunately, it's not obvious what the right tradeoff between hallucinations and bug detection is for an overall RLHF system that uses critiques to enhance model performance," the paper concedes. ®
On the subject of very artificial intelligence...
Tests show OpenAI's ChatGPT sometimes gives users [14]broken URLs for at least ten publications that have struck licensing deals with, you guessed it, OpenAI. Those partnerships were supposed to ensure that when the chatbot generates an answer based on one of those titles' articles, the system kindly namechecks the source and provides a link to the original piece. So much for that.
The Center for Investigative Reporting, a nonprofit which operates Mother Jones and Reveal, is [15]suing OpenAI and its backer Microsoft for allegedly unlawfully using its copyrighted content without permission nor compensation.
And speaking of Microsoft, a study has [16]demonstrated that the Windows giant's Bing translation and web search engine in China censors more aggressively than its Chinese competitors.
Get our [17]Tech Resources
[1] https://cdn.openai.com/llm-critics-help-catch-llm-bugs-paper.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://arxiv.org/abs/2306.07899
[4] https://time.com/6247678/openai-chatgpt-kenya-workers/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn6JwIzSYV-HlfkPU-PyKgAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/
[9] https://regmedia.co.uk/2024/06/27/openai_paper_image.jpg
[10] https://www.theregister.com/2024/06/27/anthropic_claude_government/
[11] https://www.theregister.com/2024/06/27/reddit_ai_info/
[12] https://www.theregister.com/2024/06/26/datacenter_electricity_demand_study/
[13] https://www.theregister.com/2024/06/25/openai_unsupported_countries/
[14] https://www.niemanlab.org/2024/06/chatgpt-is-hallucinating-fake-links-to-its-news-partners-biggest-investigations/
[15] https://apnews.com/article/ai-media-lawsuits-center-for-investigative-reporting-chatgpt-mother-jones-c48452889750479410b65a119537746c
[16] https://restofworld.org/2024/microsoft-bing-chinese-censorship/
[17] https://whitepapers.theregister.com/
Do AI Solutions Present an Existential Threat or Extraordinary Treats?
amanfromMars 1
When your chatbots outshine their human trainers, you could pay for expertise ...
The ornery questions then left hanging are payments for expertise in what and from/to whom or what?
So the humans helping train their AI are themselves being trained by a different AI? ....DS999
Yes, indeed they are, DS999 ......and whilst just as many may not find it easier to understand and accept, your comment is more accurately and truthfully rewritten as ...... So the humans being trained to help AI are themselves being trained by a different AI?
Ladies and Gentlemen, Place urBets. :-)
So the humans helping train their AI
Are themselves being trained by a different AI?