I predicted this in this very forum a year ago.

AI generated from AI generated crap produces crap^2.

You and pretty much everyone else with half a clue (see icon). This effect is both intuitive (I like to think of it as a kind of 2nd law of thermodynamics for data), and well-known in ML circles, probably for decades.

Definitely decades; the phrase "Garbage In, Garbage Out" has been around at least as long as computers.

Yerrrs, but not quite the point. The original input may not be garbage at all, and the output may not be too bad either, for the effect to kick in. It's more about iteratively applying a lossy algorithm (with some entropy injection just to mess things up a bit more).

The issue is not really the input, or the algorithm implementation. The entire _problem domain_ is gibberish pseudo-mathematical bullshit.

Charlatans are attempting to "create" intelligence without even remotely knowing what it is. The public is lapping it up.

Ker-ching!

> The entire _problem domain_ is gibberish pseudo-mathematical bullshit.

Is it? I think that what LLMs do -- what they are designed to do -- is pretty clear: in response to a query, they generate plausibly human-like textual responses which (to some degree) reflect associations in the human-generated training set pertinent to the query. And they are capable of doing that rather well - at least if the training set is not polluted with LLM-generated text. The basis on which LLMs do what they do is hardly "pseudo-mathematical"; it's a particular style of machine-learning model (the [1]Transformer architecture ), which does what it says on the box.

Whether you think that what LLMs (are designed to) do is worth doing, or is a gibberish/bullshit problem domain is an entirely different question. Personally, I'm a bit jury's out on that one.

> Charlatans are attempting to "create" intelligence without even remotely knowing what it is.

Pfft. Nobody knows what intelligence is*. The people developing AI models are not the charlatans - they (of necessity) understand how their algorithms work and those I've met are, in my experience, pretty clear-eyed about what they are and are not capable of. The charlatans are the marketing crowd (over-)selling the tech under false pretences.

*If you think we should rather wait around until someone comes up with a principled and consensually acceptable definition of what intelligence "is" before attempting to develop artificial versions, then prepare yourself for a lengthy wait - it's certainly kept philosophers in business for a good few millennia and counting. Me, I'm more than happy for people to crack on with it, make mistakes, learn incrementally, and do some interesting, and potentially even useful stuff in the process. That's actually how science tends to work in practice.

[1] https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)

Got it in One:

"iteratively applying a lossy algorithm"

That is the 'NEW' AI for this ALL season(s) ... you can play all you want with the algorithm *but* ultimately it spirals to nonsense !!!

:)

Note quite, I think. The problem identified in the study is not actually the algorithm itself - it's the iteration on data polluted by its own output. If trained on "clean" (i.e., LLM-free) data, LLMs do (to a greater or lesser degree) what they say on the box - in response to a query, they generate plausibly human-like text which (to a greater or lesser degree) reflects associations in the human-generated training set pertinent to the query. The nonsense spiral only kicks in when trained iteratively on recursively polluted data.

Whether you think what LLMs do is worth doing in the first place is a different issue; as is whether you think it merits the "I" in "AI" (Reg readers, at least, are pretty clear on that last one...).

"which (to a greater or lesser degree) reflects associations in the human-generated training set pertinent to the query."

I think "lossy" covers it, otherwise it would shovel out the entire relevant input every time.

That's not quite what I meant by "lossy"... I should have clarified that. I meant simply that you cannot, even in principle, recreate the entire training data set from a trained LLM; the training process itself is lossy. Of course the output has to be limited - the object of the game is to respond to a query with a human-like response; humans don't chug out the entire body of their sources of information in response to a simple query. Well, not the sane/socially-presentable ones, anyway.

Anyway, lossy is fine, and often extremely useful (e.g., JPEG). On the other hand, you probably don't want to apply it recursively*.

* [1]This is kind of interesting, though ...

[1] https://blog.kasson.com/the-last-word/does-repeated-jpeg-compression-ruin-images/

Pretty much.

The problem is that the models themselves are trying to kill off the human-written material that they feed on. If the search LLM gives you the answer, why visit the blog/read the article? Not one reads the articles, no one writes the articles. Eventually we are left with a web where the only writers are AI and the only readers are AI. Web 5.0

We've already seen any number of cases where using non-recursively generated AI datasets produces gibberish. Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.

> Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.

Did anyone actually think that? (Perhaps you can point them out to us, so we can avoid accidentally breeding with them.)

In fact ML model output doesn't even need to be that crap for this effect to kick in; it just needs to be (and of course will be) less accurate/content-rich than the original data.

Sounds like AI is about ready to move into politics.....

> In one example, a model started with a text about European architecture in the Middle Ages and ended up – in the ninth generation – spouting nonsense about jackrabbits.

> If subsequent models are trained on an AI-generated data set that over-represents Golden Retrievers, the problem is compounded & ff

So... jackrabbits were over-represented in European architecture in the Middle Ages?? Who knew?

> If subsequent models are trained on an AI-generated data set that over-represents Golden Retrievers,

You can't over-represent Golden Retrievers. The optimum number of Golden Retrievers is more Golden Retrievers.

My flabber couldn't be more gasted.

Goethe, 18th century.

"The need to distinguish data generated by LLMs from other data"

Need? The more they fail to distinguish the better. The sooner the whole lot of them collapse and disappear up their own prompts the better.

Yes, it would be good if there was a means of letting humans distinguish but the long term is better served by letting them fail under their weight, even at the expense of confused humans becoming even more confused.

"The need to distinguish data generated by LLMs from other data"

Easy - just tender the curation job out to the experts: secondary school teachers and university lecturers (they could probably use the extra cash too).

Nobody wants to pay for curation.

Of course; if you could afford to pay for curation, you don't need the LLM - you can just pay humans to answer the queries directly. Now there's an idea... we could, for example, have, like medical experts answering medical queries for money; or legal experts answering legal queries for money, and so on. I think I may be onto something here...

Indeed, it should be possible to generate data specifically optimised to induce model collapse in any model that ingests it.

To anyone doing this (Tianhe-2, perhaps?) I wish you every success.

When we first started using ML for our cameras, there was a cautionary tail told:

A team had been training its own models, and used wolves as an training data set

They were confused when random images were being marked as wolves, when they contained no wolves, not even animals:

The ML trainer had noticed a prevalence of white in the backgrounds of images with wolves (wolves liking snow, and all that), and had decided that white was a wolf, not the wolves themselves

Easily done

A more serious case of the same thing.

Automating the analysis of lung x-rays to detect some lung disease (can't recall which). Didn't work, the models fixated on the newer images with higher resolution (input positive cases) over older lower resolution images (input negative cases). So any recent x-ray was positive, whatever the state of the lungs.

The wonders of having no means to debug the model building other than extreme care with input selection when building.

Back in my lectures (when computers had front-panel switches and 640K was enough for anyone) the example was tanks.

Soviet tank images were grainy black-white taken on long lenses on muddy exercises. American tank images were colourful and posed in front of the factory on a sunny blue-sky day.

The AI did a predictably bad job of classifying them.

Even today the first part of a project to make a tool for analyzing cancer images was to remove the scale ruler in all the images of malignant cases taken in hospital on follow-up visits.

That sounds like tRump and his MAGA nitwits.

Actually, that may not be such a bad analogy; so an LLM feeding on its own excreta creates its very own echo poo-filled chamber.

I propose Garbage Out, Garbage In (GOGI)

But, it takes Garbage In and produces garbage Out, which is Garbage In to Garbage Out to Garbage In to GIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGGO^C^C^C

[1]Oozlum ?

[1] https://en.wikipedia.org/wiki/Oozlum_bird

Doesn't it all end up with everything being pictures of crabs?

Except the crabs have 3 arms and walk forward

I've had enough of all this AI nonsense now. Please can the snake oil salesmen put their toys away so that the rest of us can get on with some real work.

Or to misquote Poly Styrene "Oh AI, Up Yours!"

Would you, in an obscenely well rewarded and practically exclusive and virtually anonymous position of leading power and influence which is increasingly threatened by AIs capable of being considerably smarter than leading humans can ever be, ....and which are increasingly being recognised by an evolving smarter population as being a wiser bet for a greater intelligence feed providing a better future lead, ...... also invent and spread fantastical tales about a likely upcoming plague on machine learning models leading to their "collapse" ..... odd tendency to "hallucinate" and to descend into spewing nonsensical gibberish?

Yes, of course you would, and run the gauntlet of MRDA derision.

> Recursive training leads to nonsense, study finds.

No surprise of course. But in fact what is meant is "Recursive training leads to even more obvious nonsense, study finds." Since LLMs are nothing to do with artificial intelligence, but in practice are much better characterised as artificial stupidity.