AI Writing Is Improving, But It Still Can't Match Human Creativity (science.org)
- Reference: 0175716433
- News link: https://science.slashdot.org/story/24/12/21/0058254/ai-writing-is-improving-but-it-still-cant-match-human-creativity
- Source link: https://www.science.org/content/article/ai-writing-improving-it-still-can-t-match-human-creativity
> With a few keystrokes, anyone can ask an artificial intelligence (AI) program such as ChatGPT to write them a term paper, a rap song, or a play. But don't expect William Shakespeare's originality. A new study finds such output remains derivative -- at least for now. [...] [O]bjectively testing this creativity has been tricky. Scientists have generally taken two tacks. One is to use another computer program to search for signs of plagiarism -- though a lack of plagiarism does not necessarily equal creativity. The other approach is to have humans judge the AI output themselves, rating factors such as fluency and originality. But that's subjective and time intensive. So Ximing Lu, a computer scientist at the University of Washington, and colleagues created a program featuring both objectivity and a bit of nuance.
>
> Called DJ Search, it collects pieces of text of a minimum length from whatever the AI outputs and searches for them in large online databases. DJ Search doesn't just look for identical matches; it also scans for strings whose words have similar meanings. To evaluate the meaning of a word or phrase, the program itself relies on a separate AI algorithm that produces a set of numbers called an "embedding," which roughly represents the contexts in which words are typically found. Synonymous words have numerically close embeddings. For example, phrases that swap "anticipation" and "excitement" are considered matches. After removing all matches, the program calculates the ratio of the remaining words to the original document length, which should give an estimate of how much of the AI's output is novel. The program conducts this process for various string lengths (the study uses a minimum of five words) and combines the ratios into one index of linguistic novelty. (The team calls it a "creativity index," but creativity requires both novelty and quality -- random gibberish is novel but not creative.)
>
> The researchers compared the linguistic novelty of published novels, poetry, and speeches with works written by recent LLMs. Humans [2]outscored AIs by about 80% in poetry, 100% in novels, and 150% in speeches , the researchers report in a preprint [3]posted on OpenReview and currently under peer review. Although DJ Search was designed for comparing people and machines, it can also be used to compare two or more humanmade works. For example, Suzanne Collins's 2008 novel The Hunger Games scored 35% higher in linguistic originality than Stephenie Meyer's 2005 hit Twilight. ( [4]You can try the tool online .)
[1] https://slashdot.org/~sciencehabit
[2] https://www.science.org/content/article/ai-writing-improving-it-still-can-t-match-human-creativity
[3] https://openreview.net/forum?id=ilOEOIqolQ
[4] https://huggingface.co/spaces/liujch1998/creativity
And a bear . . . (Score:2)
"relieves" itself in the woods?
Re: (Score:2)
> "relieves" itself in the woods?
When thee white women are meeting them instead of a man.
Re: (Score:2)
>> "relieves" itself in the woods?
> When thee white women are meeting them instead of a man.
And who [1]can blame them [foxnews.com]?
[1] https://www.foxnews.com/us/arrest-made-after-hiker-murdered-small-mountain-town-slaying-staged-bear-attack
Re: (Score:2)
>>> "relieves" itself in the woods?
>> When thee white women are meeting them instead of a man.
> And who [1]can blame them [foxnews.com]?
Wahddya think? [2]https://www.theguardian.com/us... [theguardian.com]
[3]https://www.14news.com/story/9... [14news.com]
[4]https://www.wjtv.com/news/loca... [wjtv.com]
taint just the evil men who seem to enjoy ending people. The ladies are getting into the game as well.
[1] https://www.foxnews.com/us/arrest-made-after-hiker-murdered-small-mountain-town-slaying-staged-bear-attack
[2] https://www.theguardian.com/us-news/2022/aug/20/alabama-adam-simjee-talladega-national-forest-police
[3] https://www.14news.com/story/9026006/epd-woman-killed-lover-hid-body-in-woods/
[4] https://www.wjtv.com/news/local-news/franklin-county-woman-charged-in-death-of-ex-husband/
Duh! (Score:2)
Duh!
Wrong question (Score:3)
Why should AI be creative when the whole source of its creativity is a long int seed? Without your own creativity, all you get is variants of what the model likes to write. It may read well, but after a while it will always be the same.
Give the model input from your creativity and use the model's writing skills to make your vision of a text come true. Why do we need to outsource this to the model?
Re: (Score:2)
It's precisely this "turn of phrase" that we enjoy from skilled writers that this analytical tool is about. It's not measuring the ideational content of the writing, i.e. thought-provoking or entertaining stories, just the way it's written. The writer's style, as it were.
I suspect this tool's analyses & results were a foregone conclusion when the researchers thought up the idea. Of course, GPT LLMs are going to produce bland prose; they're essentially "averaging machines" & all distinctiveness ha
Re: (Score:2)
I think the largest problem with bland prose is bad datasets. If you look at the bland prose, most of it isn't all that bad. Yes, all common tropes and so on, but not rare in other literature and not bad per se. But the models have a way too large repetition quote and too little diversity.
One thing is, that the model starts anew with each text. Write one chapter without the previous one in the context, and you get repetitive phrasing, because the model doesn't know it (over)used this phrase in the last chap
joke to a brick wall (Score:2)
sure honestly ai just cant write like us its all robotic and stuff never gets the feelings right sometimes the stories are just boring and lack the depth you need its like trying to explain a joke to a brick wall ai will never truly understand what makes writing special its just lines of code not real creativity
Forest and trees (Score:2)
They are measuring linguistic creativity which concerns only a measure of uniqueness of words in sentences and phrases rather than attempting to measure overall creativity of the work. Neither does the paper even once mention temperature parameter. They select poor models like ChatGPT well known for being highly overfit and llama2 when there are way better models tuned for this kind of work readily available.
Overall I think the paper is fundamentally flawed and guaranteed to cause confusion in its choice
Re: (Score:2)
Well, I slammed a few of my poems into that tool, and got a creativity index between 75% and 80%.
I'm yet to figure out... 75%-80% of what, exactly ?
Re: (Score:2)
Indeed. Essentially, they have created a metric and benchmark to (fake) support for the conclusions they wanted to find. That is junk-science and meaningless.
Re: (Score:2)
Yeah, I think there's a basic contradiction in trying to 'prove' that humans are more creative than AI by applying a metric that is itself an algorithm. Train the AI to optimize this metric of creativity and my guess is it will take the lead.
Since it is accepted that people possess creativity and the question is whether AI merits admission to the club, the judges of creativity must be human and the criteria must be subjective. This can still constitute proof if the judging is blind (the judges aren't to
Wrong priority (Score:1)
We don't need robot artists.
We need AI systems that can solve previously intractable problems in physics, medicine and engineering.
The art problem has been solved a long time ago. People are good at art and don't need robot help.
Inherent flaw? (Score:5, Insightful)
"A new study finds such output remains derivative -- at least for now."
For now? The whole principle they're building on is to replicate what it's seen. How can it be anything *other* than derivative?
Re: (Score:2)
You simply add a random number generator to it. Generate random stuff, then start polishing it and you have yourself an original story. That is not the hard part.
Hard part is to identify parts that humans enjoy. If you had a good scoring algorithm for that, you could just generate random stuff and pick the good stuff from the noise.
Re: (Score:2)
> You simply add a random number generator to it. Generate random stuff, then start polishing it and you have yourself an original story. That is not the hard part.
> Hard part is to identify parts that humans enjoy. If you had a good scoring algorithm for that, you could just generate random stuff and pick the good stuff from the noise.
Creativity, and creative people are not normal people. Not throwing shade, but that they might see and think things that are not what most people think or see. So they create, and sometimes it is pretty profound. What is more, is the misunderstanding that creativity needs no bounds. Creativity is all about restrictions.
Re: (Score:2)
> What is more, is the misunderstanding that creativity needs no bounds. Creativity is all about restrictions.
Exactly, It is about doing something _meaningful_ within restrictions that make sense. It is about ideas and structures derived from that idea. AI can, say, replace a character in an existing story or it can mix some stories together, but it cannot add to things. It can only make derivative things that are on lower quality than the input.
Incidentally, the unavoidable problem of "model collapse" is a result of that.
Re: (Score:2)
>> What is more, is the misunderstanding that creativity needs no bounds. Creativity is all about restrictions.
> Exactly, It is about doing something _meaningful_ within restrictions that make sense. It is about ideas and structures derived from that idea. AI can, say, replace a character in an existing story or it can mix some stories together, but it cannot add to things. It can only make derivative things that are on lower quality than the input.
> Incidentally, the unavoidable problem of "model collapse" is a result of that.
And the closest that AI comes to creativity is when it hallucinates. Of course that is still not creativity at all. At best it can be inadvertently funny.
Re: (Score:2)
Your criticisms of AI always rely upon definitions, terminology, and benchmark you that alone define and consider to be worthy of merit. Fortunately, many other of us try to think a little more critically about our statements.
Re: (Score:2)
Nope. You will have a _random_ story. That is fundamentally different. Randomness cannot replace insight or creativity, even if some artists throughout history have tried that path.
Re: (Score:3)
Indeed. It will always be derivative and it will always be low quality with regard to content. Anything else would require insight and creativity and AI cannot do those. Period. What can get better is the language used, as that does not require insight or creativity.
No idea why people continue to expect things from AI that it fundamentally cannot do.
Re: (Score:2)
Children start writing highly derivative stories also. We're not really clear on what people do to actually write genuinely creative stories, but even highly skilled writers seem to start with a lot of derivative things. In that sense, ChatGPT's attempts to write fiction resemble that of about a 12 to 14 year old child (although I've seen 12 year olds who are better writers than it). What needs to be done differently still isn't clear. That said, I'm not sure that writers as writers really want this. Writin