Have we stopped to think about what LLMs actually model?
(2024/08/30)
- Reference: 1725006911
- News link: https://www.theregister.co.uk/2024/08/30/ai_language_cognition_research/
- Source link:
In May, Sam Altman, CEO of $80-billion-or-so OpenAI, seemed unconcerned about how much it would cost to achieve the company's stated goal. "Whether we burn $500 million a year or $5 billion – or $50 billion a year – I don't care," he [1]told students at Stanford University . "As long as we can figure out a way to pay the bills, we're making artificial general intelligence. It's going to be expensive."
Statements like this have become commonplace among tech leaders who are scrambling to maximize their investments in large language models (LLMs). Microsoft has put $10 billion into OpenAI, Google and Meta have their own models, and enterprise vendors are baking LLMs into products on a large scale. However, as industry bellwether Gartner identifies GenAI as nearing the peak of the hype cycle, it's time to examine what LLMs actually model – and what they do not.
"Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency" is a recent peer-reviewed paper that aims to take a look at how LLMs work, and examine how they compare with a scientific understanding of human language.
[2]
Amid "hyperbolic claims" that LLMs are capable of "understanding language" and are approaching artificial general intelligence (AGI), the GenAI industry – [3]forecast to be worth $1.3 trillion over the next ten years – is often prone to misusing terms that are naturally applied to human beings, according to the paper by Abeba Birhane, an assistant professor at University College Dublin's School of Computer Science, and Marek McGann, a lecturer in psychology at Mary Immaculate College, Limerick, Ireland. The danger is that these terms become recalibrated and the use of words like "language" and "understanding" shift towards interactions with and between machines.
[4]
[5]
"Mistaking the impressive engineering achievements of LLMs for the mastering of human language, language understanding, and linguistic acts has dire implications for various forms of social participation, human agency, justice and policies surrounding them," argues [6]the paper published in the peer-reviewed journal Language Sciences.
The risks are far from imagined. The AI industry and its associated bedfellows have spent the last few years cozying up to political leaders. Last year, US vice president and Democratic presidential candidate Kamala Harris [7]met CEOs of four American companies at the "forefront of AI innovation" including Altman and Satya Nadella, Microsoft CEO. At the same time, [8]former UK prime minister Rishi Sunak hosted an AI Safety Summit , which included the Conservative leader's fawning interview with Elon Musk, a tech CEO who has predicted that AI would be smarter than humans by 2026.
[9]
Speaking to The Register , Birhane said: "Big corporations like Meta and Google tend to exaggerate and make misleading claims that do not stand up to scrutiny. Obviously, as a cognitive scientist who has the expertise and understanding of human language, it's disheartening to see a lot of these claims made without proper evidence to back them up. But they also have downstream impacts in various domains. If you start treating these massive complex engineering systems as language understanding machines, it has implications in how policymakers and regulators think about them."
LLMs build a model capable of responding to natural language by absorbing a large corpus of training data, often from the World Wide Web. Leaving aside legal issues around how much of that data is copyrighted, the technique involves atomizing written language into tokens, and then using powerful statistical techniques – and a lot of computing power – to predict the relationship between those tokens in response to a question, for example. But there are a couple of implicit assumptions in this approach.
"The first is what we call the assumption of language completeness – that there exists a 'thing' called a 'language' that is complete, stable, quantifiable, and available for extraction from traces in the environment," the paper says. "The engineering problem then becomes how that 'thing' can be reproduced artificially. The second assumption is the assumption of data completeness – that all of the essential characteristics can be represented in the datasets that are used to initialize and 'train' the model in question. In other words, all of the essential characteristics of language use are assumed to be present within the relationships between tokens, which presumably would allow LLMs to effectively and comprehensively reproduce the 'thing' that is being modeled."
[10]
The problem is that one of the more modern branches of cognitive science sees language as a behavior rather than a big pile of text. In other words, language is something we do, and have done for hundreds of thousands of years.
The approach taken by Birhane and her colleagues is to understand human thought in terms that are "embodied" and "enacted."
"The idea is that cognition doesn't end at the brain and the person doesn't end at the the skin. Rather, cognition is extended. Personhood is messy, ambiguous, intertwined with the existence of others, and so on," she said.
Tone of voice, gesture, eye contact, emotional context, facial expressions, touch, location, and setting are among the factors that influence what is said or written.
Language behavior "cannot, in its entirety, be captured in representations appropriate for automation and computational processing. Written language constitutes only part of human linguistic activity," the paper says.
In other words, the stronger claims of AI builders fall down on the assumption that language itself is ever complete. The researchers argue the second assumption – that language is captured by a corpus of text – is also false by the same means.
[11]The future of AI/ML depends on the reality of today – and it's not pretty
[12]Microsoft Bing Copilot accuses reporter of crimes he covered
[13]Chasing the AI dragon? Your IT might be circling the drain, IBM warns
[14]AGI is on clients' radar but far from reality, says Gartner
It's true that both humans and LLMs learn from examples of text, but by looking at how humans use language in their lives, there's a great deal missing. As well as human language being embodied, it is something in which people participate.
"Training data therefore is not only necessarily incomplete but also lacks to capture the motivational, participatory, and vitally social aspects that ground meaning making by people," the paper says.
Human language is also precarious, a concept that may be harder to understand.
"The idea of precarity or precariousness is that human interaction and language is full of ambiguities, tensions, frictions, and those are not necessarily a bad thing," Birhane said. "They're really at the heart of what being human means. We actually need frictions to resolve disagreements, to have an in-depth understanding about a phenomena and confronting wrongs, for example."
"LLMs do not participate in social interaction, and having no basis for shared experience, they also have nothing at stake," the paper says. "There is no set of processes of self-production that are at risk, and which their behavior continually stabilizes, or at least moves them away from instability and dissolution. A model does not experience a sense of satisfaction, pleasure, guilt, responsibility, or accountability for what it produces. Instead, LLMs are complex tools, and within any activity their roles is that of a tool."
Human language is an activity is one in which "various opportunities and risks are perceived, engaged with, and managed."
"Not so for machines. Nothing is risked by ChatGPT when it is prompted and generates text. It seeks to achieve nothing as tokens are concatenated into grammatically sound output," the paper says.
The authors argue that whatever LLMs model, it is not human language, which is considered not as a "large and growing heap, but more a flowing river."
"Once you have removed water from the river, no matter how large a sample you have taken, it is no longer the river," the paper says.
Birhane has previously challenged the AI industry. With colleagues, she pored over an MIT visual dataset for training AI to discover thousands of images labeled with racist slurs for Black and Asian people, and derogatory terms used to describe women, [15]prompting the US super-college to take the dataset offline .
Whether or not LLMs effectively model human language, their advocates make spectacular claims about their usefulness. McKinsey says 70 percent of companies will deploy some sort of AI tech by 2030, producing a global economic impact of around $13 trillion in the same period, increasing global GDP by about 1.2 percent annually.
But claims asserting the usefulness of LLMs as a tool alone have also been exaggerated.
"There is no clear evidence that that shows LLMs are useful because they are extremely unreliable," Birhane said. "Various scholars have been doing domain specific audits … in legal space … and in medical space. The findings across all these domains is that LLMs are not actually that useful because they give you so much unreliable information."
Birhane argues that there are risks in releasing these models into the wild that would be unacceptable in other industries.
"When we build bridges, for example, we do rigorous testing before we allow any vehicles or pedestrians to use it," she said. "Many other industries – pharma, for example – have proper regulations in place and we have established bodies that do the auditing and the evaluation. My biggest concern at the moment is that we're just building LLMs and releasing them into super important domains such as education and medicine. This has huge impacts, and also massive downstream impacts, say in 20 years, and where we're not doing proper testing, proper evaluations of these models."
Not everyone agrees. Although Gartner has declared that GenAI is entering its famous "trough of disillusionment," it has [16]little doubt about the importance of its long-term impact .
Research showing LLM builders have a flawed understanding of what they are modeling is an opportunity to promote a more cautious, skeptical approach. ®
Get our [17]Tech Resources
[1] https://finance.yahoo.com/news/openai-sam-altman-doesn-t-161126520.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.bloomberg.com/company/press/generative-ai-to-become-a-1-3-trillion-market-by-2032-research-finds/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.sciencedirect.com/science/article/pii/S0388000124000615
[7] https://www.reuters.com/technology/teslas-musk-predicts-ai-will-be-smarter-than-smartest-human-next-year-2024-04-08/
[8] https://www.theregister.com/2023/11/03/ai_summit_safety_pacts/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/2024/08/27/opinion_ai_ml/
[12] https://www.theregister.com/2024/08/26/microsoft_bing_copilot_ai_halluciation/
[13] https://www.theregister.com/2024/08/22/ai_may_be_distracting_organizations/
[14] https://www.theregister.com/2024/08/22/gartner_agi_hype_cycle/
[15] https://www.theregister.com/2020/07/01/mit_dataset_removed/
[16] https://www.theregister.com/2024/08/22/gartner_agi_hype_cycle/
[17] https://whitepapers.theregister.com/
Statements like this have become commonplace among tech leaders who are scrambling to maximize their investments in large language models (LLMs). Microsoft has put $10 billion into OpenAI, Google and Meta have their own models, and enterprise vendors are baking LLMs into products on a large scale. However, as industry bellwether Gartner identifies GenAI as nearing the peak of the hype cycle, it's time to examine what LLMs actually model – and what they do not.
"Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency" is a recent peer-reviewed paper that aims to take a look at how LLMs work, and examine how they compare with a scientific understanding of human language.
[2]
Amid "hyperbolic claims" that LLMs are capable of "understanding language" and are approaching artificial general intelligence (AGI), the GenAI industry – [3]forecast to be worth $1.3 trillion over the next ten years – is often prone to misusing terms that are naturally applied to human beings, according to the paper by Abeba Birhane, an assistant professor at University College Dublin's School of Computer Science, and Marek McGann, a lecturer in psychology at Mary Immaculate College, Limerick, Ireland. The danger is that these terms become recalibrated and the use of words like "language" and "understanding" shift towards interactions with and between machines.
[4]
[5]
"Mistaking the impressive engineering achievements of LLMs for the mastering of human language, language understanding, and linguistic acts has dire implications for various forms of social participation, human agency, justice and policies surrounding them," argues [6]the paper published in the peer-reviewed journal Language Sciences.
The risks are far from imagined. The AI industry and its associated bedfellows have spent the last few years cozying up to political leaders. Last year, US vice president and Democratic presidential candidate Kamala Harris [7]met CEOs of four American companies at the "forefront of AI innovation" including Altman and Satya Nadella, Microsoft CEO. At the same time, [8]former UK prime minister Rishi Sunak hosted an AI Safety Summit , which included the Conservative leader's fawning interview with Elon Musk, a tech CEO who has predicted that AI would be smarter than humans by 2026.
[9]
Speaking to The Register , Birhane said: "Big corporations like Meta and Google tend to exaggerate and make misleading claims that do not stand up to scrutiny. Obviously, as a cognitive scientist who has the expertise and understanding of human language, it's disheartening to see a lot of these claims made without proper evidence to back them up. But they also have downstream impacts in various domains. If you start treating these massive complex engineering systems as language understanding machines, it has implications in how policymakers and regulators think about them."
LLMs build a model capable of responding to natural language by absorbing a large corpus of training data, often from the World Wide Web. Leaving aside legal issues around how much of that data is copyrighted, the technique involves atomizing written language into tokens, and then using powerful statistical techniques – and a lot of computing power – to predict the relationship between those tokens in response to a question, for example. But there are a couple of implicit assumptions in this approach.
"The first is what we call the assumption of language completeness – that there exists a 'thing' called a 'language' that is complete, stable, quantifiable, and available for extraction from traces in the environment," the paper says. "The engineering problem then becomes how that 'thing' can be reproduced artificially. The second assumption is the assumption of data completeness – that all of the essential characteristics can be represented in the datasets that are used to initialize and 'train' the model in question. In other words, all of the essential characteristics of language use are assumed to be present within the relationships between tokens, which presumably would allow LLMs to effectively and comprehensively reproduce the 'thing' that is being modeled."
[10]
The problem is that one of the more modern branches of cognitive science sees language as a behavior rather than a big pile of text. In other words, language is something we do, and have done for hundreds of thousands of years.
The approach taken by Birhane and her colleagues is to understand human thought in terms that are "embodied" and "enacted."
"The idea is that cognition doesn't end at the brain and the person doesn't end at the the skin. Rather, cognition is extended. Personhood is messy, ambiguous, intertwined with the existence of others, and so on," she said.
Tone of voice, gesture, eye contact, emotional context, facial expressions, touch, location, and setting are among the factors that influence what is said or written.
Language behavior "cannot, in its entirety, be captured in representations appropriate for automation and computational processing. Written language constitutes only part of human linguistic activity," the paper says.
In other words, the stronger claims of AI builders fall down on the assumption that language itself is ever complete. The researchers argue the second assumption – that language is captured by a corpus of text – is also false by the same means.
[11]The future of AI/ML depends on the reality of today – and it's not pretty
[12]Microsoft Bing Copilot accuses reporter of crimes he covered
[13]Chasing the AI dragon? Your IT might be circling the drain, IBM warns
[14]AGI is on clients' radar but far from reality, says Gartner
It's true that both humans and LLMs learn from examples of text, but by looking at how humans use language in their lives, there's a great deal missing. As well as human language being embodied, it is something in which people participate.
"Training data therefore is not only necessarily incomplete but also lacks to capture the motivational, participatory, and vitally social aspects that ground meaning making by people," the paper says.
Human language is also precarious, a concept that may be harder to understand.
"The idea of precarity or precariousness is that human interaction and language is full of ambiguities, tensions, frictions, and those are not necessarily a bad thing," Birhane said. "They're really at the heart of what being human means. We actually need frictions to resolve disagreements, to have an in-depth understanding about a phenomena and confronting wrongs, for example."
"LLMs do not participate in social interaction, and having no basis for shared experience, they also have nothing at stake," the paper says. "There is no set of processes of self-production that are at risk, and which their behavior continually stabilizes, or at least moves them away from instability and dissolution. A model does not experience a sense of satisfaction, pleasure, guilt, responsibility, or accountability for what it produces. Instead, LLMs are complex tools, and within any activity their roles is that of a tool."
Human language is an activity is one in which "various opportunities and risks are perceived, engaged with, and managed."
"Not so for machines. Nothing is risked by ChatGPT when it is prompted and generates text. It seeks to achieve nothing as tokens are concatenated into grammatically sound output," the paper says.
The authors argue that whatever LLMs model, it is not human language, which is considered not as a "large and growing heap, but more a flowing river."
"Once you have removed water from the river, no matter how large a sample you have taken, it is no longer the river," the paper says.
Birhane has previously challenged the AI industry. With colleagues, she pored over an MIT visual dataset for training AI to discover thousands of images labeled with racist slurs for Black and Asian people, and derogatory terms used to describe women, [15]prompting the US super-college to take the dataset offline .
Whether or not LLMs effectively model human language, their advocates make spectacular claims about their usefulness. McKinsey says 70 percent of companies will deploy some sort of AI tech by 2030, producing a global economic impact of around $13 trillion in the same period, increasing global GDP by about 1.2 percent annually.
But claims asserting the usefulness of LLMs as a tool alone have also been exaggerated.
"There is no clear evidence that that shows LLMs are useful because they are extremely unreliable," Birhane said. "Various scholars have been doing domain specific audits … in legal space … and in medical space. The findings across all these domains is that LLMs are not actually that useful because they give you so much unreliable information."
Birhane argues that there are risks in releasing these models into the wild that would be unacceptable in other industries.
"When we build bridges, for example, we do rigorous testing before we allow any vehicles or pedestrians to use it," she said. "Many other industries – pharma, for example – have proper regulations in place and we have established bodies that do the auditing and the evaluation. My biggest concern at the moment is that we're just building LLMs and releasing them into super important domains such as education and medicine. This has huge impacts, and also massive downstream impacts, say in 20 years, and where we're not doing proper testing, proper evaluations of these models."
Not everyone agrees. Although Gartner has declared that GenAI is entering its famous "trough of disillusionment," it has [16]little doubt about the importance of its long-term impact .
Research showing LLM builders have a flawed understanding of what they are modeling is an opportunity to promote a more cautious, skeptical approach. ®
Get our [17]Tech Resources
[1] https://finance.yahoo.com/news/openai-sam-altman-doesn-t-161126520.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.bloomberg.com/company/press/generative-ai-to-become-a-1-3-trillion-market-by-2032-research-finds/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.sciencedirect.com/science/article/pii/S0388000124000615
[7] https://www.reuters.com/technology/teslas-musk-predicts-ai-will-be-smarter-than-smartest-human-next-year-2024-04-08/
[8] https://www.theregister.com/2023/11/03/ai_summit_safety_pacts/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZtGYQzzEvA_KDefTyVTt3AAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/2024/08/27/opinion_ai_ml/
[12] https://www.theregister.com/2024/08/26/microsoft_bing_copilot_ai_halluciation/
[13] https://www.theregister.com/2024/08/22/ai_may_be_distracting_organizations/
[14] https://www.theregister.com/2024/08/22/gartner_agi_hype_cycle/
[15] https://www.theregister.com/2020/07/01/mit_dataset_removed/
[16] https://www.theregister.com/2024/08/22/gartner_agi_hype_cycle/
[17] https://whitepapers.theregister.com/
cyberdemon
What we have invented is [1]"The devil in the mirror"
It reflects back an image of our souls, but will never have one of its own. If you spend too long gazing into it, yours will be consumed.
[1] https://www.theparisreview.org/blog/2019/03/04/objects-of-despair-mirrors/
"Whether we burn $500 million a year or $5 billion – or $50 billion a year – I don't care"
Pascal Monett
Of course you don't. You just want to sell you company for whatever and get your golden parachute without jumping out of the plane.
It's not like you're actually going to pay that anyway. That's someone else's problem.
cyberdemon
His words are even more stupid if you imagine that all his servers were powered by "Renewable Biomass"
Whether we burn a million trees a year or a billion, I don't care!
Groo The Wanderer
In short: LLMs are not intelligent in the least; they're just statistics on a phenomenal scale. Pure mathematics, not understanding.
No your not
steviebuk
"As long as we can figure out a way to pay the bills, we're making artificial general intelligence. It's going to be expensive."
Artificial General Intelligence is a whole other beast and you're nowhere near it with these LLMs. But you know its what the marketing and investors like to hear so they'll keep giving you money. AGI won't appear in my life time. And if it did, it would need to be contained, offgrid so it can't wreck everything. And as we saw with Ex Machina (bit of a spoilter if you haven't seen it so don't scroll down. Not sure if this works on this format).
..
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
She says what she knows you want to hear.
It's a glorified autocomplete function.
That's all it is.
And people are somehow claiming intelligence, innovation, imagination, creativity, invention and consciousness to them.
Nope.
If anything, it proves how dumb the average human is - a bit like the Turing test, it's not really a test of if the AI is intelligent, it's a test of how gullible/dumb the human testers are. It's misused an awful lot to try to claim new intelligence, but it doesn't actually measure that at all.
The irony is, it's not just a glorified autocomplete function, it's an extremely resource-heavy, poor, unpredictable and *easily compromised* one. Just a few sentences and you can "convince" it to go against all its explicit training to do things that it should never be doing.
And every time someone suggests we should use it, I point at data protection laws, the fact that we have no idea what it's actually doing with any training data whatsoever, and that almost all LLMs and moderns AIs out there are trained on data of very dubious origin.
Some examples I've used to demonstrate from various LLMs include:
- Asked it about fire extinguishers. It literally got everything backwards and recommended a dangerous extinguisher on a particular fire more times than the correct one.
- Asked it about a character that doesn't exist in a well-known TV programme that does. It made up characters by merging similarly-named characters from other TV shows and attributed characteristics from some actual characters randomly to those "invented" non-existent ones... including actor's names, plot elements, etc. So you had actors who'd never appeared in the show "portraying" characters that didn't exist in the show, with plot elements that never happened in the show. No matter how much probed it or changed names, it asserted utterly confidently that in a TV show with only 4 main characters that almost every single name you gave it was an actual character in it and made up bios for them. It will confidently spew the entire synopsis of every episode (so it "knows" what actually happened or didn't), and then still insert its made up characters into the mix after you ask about them, even though that's quite clearly just rewriting history and those characters never existed.
- An employee of mine was given a research project to source some new kit. They plugged it into ChatGPT (against my wishes). It returned a comprehensive and detailed list with a summary of 10 viable products that met the criteria. 5 literally did not exist. 3 were long-discontinued and contained false data. 2 were unreliable specifications of the available products and were nowhere near ideal for the task. And that's something that all it needed to do was scrape "10 best" and it would have got a far better shopping list immediately.
Not to mention that it can't count, can't follow rules, can't infer anything, etc.
And it never takes much to generate examples like that. In fact, each time someone questions this, I think up a new way off the top of my head that I've never tested, run it through an LLM and get results like the above. It's fine if it asserts a TV character wrongly, no harm can result, but if it can't even get that right why would you ever trust it to do things like autocomplete code in a programming project (sorry, but any company that allows that is just opening itself up to copyright claims down the road, not to mention security etc. issues).
LLMs are glorified autocomplete, and if you're putting your primary customer interface, or your programming code repository, or your book publishing output into the hands of a glorified autocomplete, you deserve everything you get.