ChatGPT is coming for your jobs – the terrible ones, at least
(2023/04/04)
- Reference: 1680559830
- News link: https://www.theregister.co.uk/2023/04/03/chatgpt_boring_turk_jobs/
- Source link:
Machine learning models can do content processing and data sanitation work better and more affordably than people participating in crowdsourcing platforms, according to a trio of researchers.
And that's not necessarily a bad thing for job seekers since some of the jobs likely to be affected seem pretty awful.
University of Zurich researchers Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli examined how OpenAI's large language model ChatGPT handled text annotation – adding labels to text to help machine learning models better understand – compared to crowdsourcing platform Amazon Mechanical Turk ( [1]MTurk ).
For those who don't know, MTurk allows you to essentially queue up work that is then assigned to people to carry out; this work tends to be things like categorize photos by their color or what kind of animal appears in them. You pay Amazon to complete this work, Amazon pays those who sign up for this work once they've completed it. You get the final dataset, the serfs get [2]some cash . So much of today's machine learning models are [3]trained on datasets produced by MTurk workers.
The academics describe their findings in [4]a paper whose title serves as a spoiler, "ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks."
Using a sample data set of 2,382 Twitter posts that had already been labeled by research assistants, the boffins compared how ChatGPT and MTurk workers fared with five distinct labeling tasks.
[5]
The work involved assessing how each tweet relates to the subject of content moderation in terms of relevance, stance, topics, and issue framing (e.g. whether content moderation is described as a problem that limits speech or a solution that protects against harmful speech).
[6]
[7]
"We find that for four out of five tasks, ChatGPT’s zero-shot accuracy is higher than that of MTurk," the paper stated. "...Moreover, ChatGPT is significantly cheaper than MTurk: the five classification tasks cost about $68 on ChatGPT (25,264 annotations) and $657 on MTurk (12,632 annotations)."
On a per-annotation basis, ChatGPT costs about $0.003, which is about twenty times cheaper than MTurk while being more accurate, the researchers say.
[8]
More accurate in this instance is not very accurate. Fabrizio Gilardi, a professor of policy analysis in the political science department of the University of Zurich and one of the paper's co-authors, told The Register in an email that ChatGPT's results were less than 50 percent accurate in some tasks, but that was still better than MTurkers.
[9]In the battle between Microsoft and Google, LLM is the weapon too deadly to use
[10]FTC urged to freeze OpenAI's 'biased, deceptive' GPT-4
[11]So you want to integrate OpenAI's bot. Here's how that worked for software security scanner Socket
[12]OpenAI rolls out ChatGPT plugins, granting iffy language model access to your apps
In all, the research results appear to be game over for human workers keen on securing this sort of work.
But Gilardi warned against reading the findings too broadly.
"It is too early to say how ChatGPT might replace crowd-workers," said Gilardi in a statement emailed to The Register . "Our paper demonstrates ChatGPT’s potential for data annotation tasks, but more research is needed to fully understand ChatGPT’s capacities in this area."
Gilardi said it's important to gather more data using different tasks, types of data, and languages, and added that MTurkers perform other jobs like survey research, image annotation, audio and video transcription, usability testing. And there may be scenarios in which human annotators can be more productive with the help of a model like ChatGPT, he suggested.
[13]
With that caveat, Gilardi said for the types of tasks studied, ChatGPT looks like it can replace crowdsourced workers, which is apt because certainly models like it are trained on crowdsourced human-annotated data. AI software taking over people's mundane work could have mental health benefits, given that human moderators [14]have sued over the trauma of reviewing toxic content.
Tools such as ChatGPT could be a perfect candidate to replace or reduce human annotation for those tasks which involve an ethical consideration
"This has implications for unpleasant and harsh annotation tasks such as hate speech detection, which contains adverse psychological consequences for human annotators," Gilardi said. "In other words, tools such as ChatGPT could be a perfect candidate to replace or reduce human annotation for those tasks which involve ethical consideration for humans to do."
We've also lately held the opinion that ChatGPT and its ilk have one benefit: highlighting the annoying humdrum tasks we have to do each day, such as summarizing reports, emailing the boss, writing boilerplate code, completing homework, and so on. By offering to tackle boring, repetitive work, the models can absorb some of that tedium, when a better solution might be to rethink how we employ and make use of people effectively and efficiently in the first place.
A recent Goldman Sachs [15]report [PDF] characterizes the adoption of generative AI as a productivity boost rather than a job destroyer, we note. It says, "extrapolating our estimates globally suggests that generative AI could expose the equivalent of 300 million full-time jobs to automation." ®
Get our [16]Tech Resources
[1] https://www.mturk.com/worker/help
[2] https://www.theregister.com/2017/12/14/amazon_mechanical_turk_keeps_labor_cheap/
[3] https://www.theregister.com/2021/04/01/mit_ai_accuracy/
[4] https://arxiv.org/abs/2303.15056
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2023/04/03/opinion_column/
[10] https://www.theregister.com/2023/03/30/ftc_openai_gpt4/
[11] https://www.theregister.com/2023/03/30/socket_chatgpt_malware/
[12] https://www.theregister.com/2023/03/26/openai_chatgpt_plugins/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2018/09/24/facebook_sued_content_moderators/
[15] https://www.key4biz.it/wp-content/uploads/2023/03/Global-Economics-Analyst_-The-Potentially-Large-Effects-of-Artificial-Intelligence-on-Economic-Growth-Briggs_Kodnani.pdf
[16] https://whitepapers.theregister.com/
And that's not necessarily a bad thing for job seekers since some of the jobs likely to be affected seem pretty awful.
University of Zurich researchers Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli examined how OpenAI's large language model ChatGPT handled text annotation – adding labels to text to help machine learning models better understand – compared to crowdsourcing platform Amazon Mechanical Turk ( [1]MTurk ).
For those who don't know, MTurk allows you to essentially queue up work that is then assigned to people to carry out; this work tends to be things like categorize photos by their color or what kind of animal appears in them. You pay Amazon to complete this work, Amazon pays those who sign up for this work once they've completed it. You get the final dataset, the serfs get [2]some cash . So much of today's machine learning models are [3]trained on datasets produced by MTurk workers.
The academics describe their findings in [4]a paper whose title serves as a spoiler, "ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks."
Using a sample data set of 2,382 Twitter posts that had already been labeled by research assistants, the boffins compared how ChatGPT and MTurk workers fared with five distinct labeling tasks.
[5]
The work involved assessing how each tweet relates to the subject of content moderation in terms of relevance, stance, topics, and issue framing (e.g. whether content moderation is described as a problem that limits speech or a solution that protects against harmful speech).
[6]
[7]
"We find that for four out of five tasks, ChatGPT’s zero-shot accuracy is higher than that of MTurk," the paper stated. "...Moreover, ChatGPT is significantly cheaper than MTurk: the five classification tasks cost about $68 on ChatGPT (25,264 annotations) and $657 on MTurk (12,632 annotations)."
On a per-annotation basis, ChatGPT costs about $0.003, which is about twenty times cheaper than MTurk while being more accurate, the researchers say.
[8]
More accurate in this instance is not very accurate. Fabrizio Gilardi, a professor of policy analysis in the political science department of the University of Zurich and one of the paper's co-authors, told The Register in an email that ChatGPT's results were less than 50 percent accurate in some tasks, but that was still better than MTurkers.
[9]In the battle between Microsoft and Google, LLM is the weapon too deadly to use
[10]FTC urged to freeze OpenAI's 'biased, deceptive' GPT-4
[11]So you want to integrate OpenAI's bot. Here's how that worked for software security scanner Socket
[12]OpenAI rolls out ChatGPT plugins, granting iffy language model access to your apps
In all, the research results appear to be game over for human workers keen on securing this sort of work.
But Gilardi warned against reading the findings too broadly.
"It is too early to say how ChatGPT might replace crowd-workers," said Gilardi in a statement emailed to The Register . "Our paper demonstrates ChatGPT’s potential for data annotation tasks, but more research is needed to fully understand ChatGPT’s capacities in this area."
Gilardi said it's important to gather more data using different tasks, types of data, and languages, and added that MTurkers perform other jobs like survey research, image annotation, audio and video transcription, usability testing. And there may be scenarios in which human annotators can be more productive with the help of a model like ChatGPT, he suggested.
[13]
With that caveat, Gilardi said for the types of tasks studied, ChatGPT looks like it can replace crowdsourced workers, which is apt because certainly models like it are trained on crowdsourced human-annotated data. AI software taking over people's mundane work could have mental health benefits, given that human moderators [14]have sued over the trauma of reviewing toxic content.
Tools such as ChatGPT could be a perfect candidate to replace or reduce human annotation for those tasks which involve an ethical consideration
"This has implications for unpleasant and harsh annotation tasks such as hate speech detection, which contains adverse psychological consequences for human annotators," Gilardi said. "In other words, tools such as ChatGPT could be a perfect candidate to replace or reduce human annotation for those tasks which involve ethical consideration for humans to do."
We've also lately held the opinion that ChatGPT and its ilk have one benefit: highlighting the annoying humdrum tasks we have to do each day, such as summarizing reports, emailing the boss, writing boilerplate code, completing homework, and so on. By offering to tackle boring, repetitive work, the models can absorb some of that tedium, when a better solution might be to rethink how we employ and make use of people effectively and efficiently in the first place.
A recent Goldman Sachs [15]report [PDF] characterizes the adoption of generative AI as a productivity boost rather than a job destroyer, we note. It says, "extrapolating our estimates globally suggests that generative AI could expose the equivalent of 300 million full-time jobs to automation." ®
Get our [16]Tech Resources
[1] https://www.mturk.com/worker/help
[2] https://www.theregister.com/2017/12/14/amazon_mechanical_turk_keeps_labor_cheap/
[3] https://www.theregister.com/2021/04/01/mit_ai_accuracy/
[4] https://arxiv.org/abs/2303.15056
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2023/04/03/opinion_column/
[10] https://www.theregister.com/2023/03/30/ftc_openai_gpt4/
[11] https://www.theregister.com/2023/03/30/socket_chatgpt_malware/
[12] https://www.theregister.com/2023/03/26/openai_chatgpt_plugins/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZCug4dezsVQamdKCJ1D8dgAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2018/09/24/facebook_sued_content_moderators/
[15] https://www.key4biz.it/wp-content/uploads/2023/03/Global-Economics-Analyst_-The-Potentially-Large-Effects-of-Artificial-Intelligence-on-Economic-Growth-Briggs_Kodnani.pdf
[16] https://whitepapers.theregister.com/
Drat!
jake
From the headline, I was hoping I could use ChatGPT to clean out the hog pens and the chicken coops[0] ... and then I realized that it was already full of shit, and thus wouldn't need any of mine.
[0] I hire out septic system work. Specialized equipment, disposal, and etc. Besides, it's icky.
Underwhelmed
Using a sample data set of 2,382 Twitter posts ... ChatGPT's results were less than 50 percent accurate in some tasks, but that was still better than MTurkers"
This article has done nothing to convince me that ChatGPT can yet contribute meaningfully to society. Partly because Twitter itself is not really useful (IMHO), but also because of that 50%.
Certainly I think the potential is there. It's just going to take a while to get a good idea of how and where it can be useful, and that includes balancing cost against benefit. Human brains can do a lot of learning with very little energy. Yes, ChatGPT can take advantage of teach once, replicate ad infinitum, and that's a real advantage. (OK, it can modify an existing LLM for a special application, but that is still teach once). In contrast humans can do continuous incremental learning and self-evaluation - much more flexible. That self-evaluation is hugely important in keeping behavior on the rails.
As example of what I would consider a successful application would be a selling a ChatGPT LLM for customer service, that could take a rule book in human language (like the ones given to current customer service operatives) and be ready to take calls. The rule book would not be "learned", it would be a database acting as a prompt in combination with customer requests, so the rule book could be updated in real time without incurring extra learning cost. It would read customer data, update it, and send detailed confirmation mails to the customer for their records. An advantage would be not having bad humans stealing PI and using it for fraud, which cost billions a year. Forbes says Customer Service is a $350 billion industry.
To be fair improved code completion could be very useful - I intend to sign up for a free Github trial to find out. I expect it is most useful for boilerplate code .