OpenAI Says Models Programmed To Make Stuff Up Instead of Admitting Ignorance (theregister.com)
- Reference: 0179325688
- News link: https://slashdot.org/story/25/09/17/1724241/openai-says-models-programmed-to-make-stuff-up-instead-of-admitting-ignorance
- Source link: https://www.theregister.com/2025/09/17/openai_hallucinations_incentives/
> The admission [2]came in a paper [PDF] published in early September, titled "Why Language Models Hallucinate," and penned by three OpenAI researchers and Santosh Vempala, a distinguished professor of computer science at Georgia Institute of Technology. It concludes that "the majority of mainstream evaluations reward hallucinatory behavior."
>
> The fundamental problem is that AI models are trained to reward guesswork, rather than the correct answer. Guessing might produce a superficially suitable answer. Telling users your AI can't find an answer is less satisfying. As a test case, the team tried to get an OpenAI bot to report the birthday of one of the paper's authors, OpenAI research scientist Adam Tauman Kalai. It produced three incorrect results because the trainers taught the engine to return an answer, rather than admit ignorance. "Over thousands of test questions, the guessing model ends up looking better on scoreboards than a careful model that admits uncertainty," OpenAI admitted in a blog post accompanying the release.
[1] https://www.theregister.com/2025/09/17/openai_hallucinations_incentives/
[2] https://arxiv.org/pdf/2509.04664
Wrong explanation (Score:5, Informative)
They make shit up because they have no meta-cognition and don't know any better.
Re: (Score:3)
This is a great time to remind ourselves that a LLM is just a fancy autocomplete engine.
Re: (Score:2)
> This is a great time to remind ourselves that a LLM is just a fancy autocomplete engine.
Well, sure, in the same way a 747 is just a fancy mechanical bird. Which is to say, yes, but no.
Re: (Score:2)
You can't make an AI based on a random number generator. But here we are. The reason is simple, otherwise it would always give the same exact result for a given query.
Hence why your AI has a 'seed' value.
This is good research (Score:2)
Instead of claiming that their work is perfect, they look for problems to fix
All companies should do this
Unfortunately, some companies not only try to keep problems secret, but also try to punish those who expose the problems
And that's why I cancelled (Score:5, Insightful)
If you can't trust the info it gives, it's worth nothing.
rtfm :( (Score:1)
its getting so frustrating when it wastes my time hallucinating flase answers to questions about my synthesizers and grooveboxes it could very easily just find it in a manual for me and get it right; but it hallucinates things applying to other products and menu structures etc i do read my manuals when i need to but it would be nice if i could just feed a folder of pdf's to some local ai and it could find me accurate things quickly
Re: rtfm :( (Score:1)
sure i can search but i mostly use my phone for reading manuals since i can sit it right there on my box while im working and get it out of the way when i dont need it i open the pdf in my browser; recently made a tab group thats just a bunch of my manuals; but searching via safari is lousey because it puts a bunch of web results and other stuff and my document at the bottom under the keyboard so it just takes more steps to search for the right instance of whatever word which probably wont be about what i
Echo chamber effect? (Score:3)
It seems to me that LLM hallucinations which remain undetected in the short term, also give rise to false data which both reinforce the hallucination of the model in question and pollute other LLMs with false data. In other words, does one LLM, unchallenged when it makes shit up, adopt its own answer as fact and then corrupt other models which aren't 'guessing' but are simply propagating a lie unknowingly? Do LLMs treat each other's data as authoritative?
Please excuse the anthropomorphism in what I just wrote - it seems to be the most efficient way of conveying the concepts.
Re: (Score:2)
You mean is it eating it's own dog food? At this point, probably. A lot of places are using AI for "news" these days. If all that is scrapped, not to mention all the AI produced video. I could definitely see AI learning misinformation that was generated by a previous iteration of itself. The irony. Also very fortellable.
That's the most annoying thing about AI (Score:1)
That's hands down the most annoying thing about AI. The one thing I miss from all models are more frequent "I don't know" responses. Knowing that the model doesn't know is a hugely valuable piece of information. And far more useful than hallucinations, half-truth that doesn't really fit the question, etc.
Generative vs Factual (Score:5, Insightful)
I think the problem is definition itself. Generative AI. If they need to generate an answer, then good chances are it'll end up whatever the model believes to be correct, statistically speaking.
But on the internet actual facts and answers are rare. Most of help-me threads are 99.99% crap and one or two people providing an actual, helpful response but those few units drown among all the other crap because hey - statistics don't favour truths.
Kind of like in democracy. Two uneducated dropouts have more power than one university lecturer.
But I digress...
If AI models were to return facts, we'd call them search agents, search engines...
Oh, wait.
Re: (Score:2)
Did you read the paper?
The paper is about whether we reinforce it to answer correctly (whatever that means) vs it simply saying "I don't know".
Most LLMs are rewarded when they give a correct answer. Say: 1 for correct, and 0 for incorrect.
So they are rewarded to always try an answer even when they aren't certain. The paper suggests rewarding "I don't know". For example, you could give 0.2 points for saying "I don't know", 0 for incorrect, and 1 for correct. This way the model will try to answer but only whe
AI disinformation is poisioning the world (Score:2)
AI should be banned.
Wow this is very insightful (Score:1)
I had a hunch they worked in some way like this. I felt like I was talking with one of my arrogant coworkers who never thinks they are wrong.
Re: (Score:2)
OpenAI's models, and most of the LLMs, are trained at least in part by having humans rate their conversations. That's what the "chat" in chatGPT stands for. Humans apparently rate chat partners that make up truthy sounding stuff more highly than chat partners that admit they don't know.
That's a useful finding for a company that makes LLMs. It should be an interesting observation for people who talk to other people too.
Potential for management positions... (Score:3)
Ability to bullshit one's way up is one of key management skills...
Re: (Score:2)
If AI can replace managers, then not everything is yet lost.
Just like people (Score:2)
Nt
They BS instead of admitting they don't know? (Score:2)
So AI really is just like human beings!
I just had this conversation with my coworker (Score:2)
Me: "Why are you sending 600 octets?" Him: "Because the AI recommended I send 600 octets." Me: "But the test passes if we only send 44 bytes, so why send 600?" Him: "I'll look at the specs..."
Precision and Recall - not that fucking hard (Score:2)
How many billions of dollars and thousands of man hours have AI "researchers" cost humanity, just because they evidently forgot to remember a core principle of machine learning?
[1]https://en.wikipedia.org/wiki/... [wikipedia.org]
[1] https://en.wikipedia.org/wiki/Precision_and_recall
It's literally how LLMs work (Score:2)
Everything they spit out is a "hallucination." They generate text that is *plausible* in that it echoes patterns that it has encountered in its training. What's really amazing is how frequently the output is *correct.*
Obvious (Score:2)
I found this to be pretty obvious when using chatgpt. Especially in the beginning. It pretends to be a know-it-all but comes up with tons of excuses when it gets something wrong, while just a "I do not know" would have been a good answer.
It reminds me of the ways some students try to talk themselves out of not knowing an answer. Funny part is that I spontaneously use the same techniques with chatgpt to figure out if it know what it is talking about.
GPT is unique though, it's responses are similar to a s
Confabulating (Score:2)
In psychology, this is not hallucination. The proper term for making stuff up like this is "confabulation."
In other words (Score:2)
Just like most humans.
Re: (Score:2)
Enhancement: Just like all politicians.
No shit (Score:2)
Watson.
Re:No shit (Score:4, Interesting)
I recently gave AI's a paradox. When an ideal 10uF cap is charged to 100V, it has x charge, when an uncharged cap of 100uF is placed in parallel, the voltage goes to about 9.1V and the total energy is about 0.1x. If energy is neither destroyed nor created, where did the energy go? The AI kind of forgets that I specified an "ideal" capacitor, and makes shit up. If a human can explain this to me, I am all ears. Please keep in mind: "Ideal capacitor", and "equations show this", don't make shit up.
Re: (Score:2)
Concurrence in general but I think the underlying problem is that it doesn't ask for clarification when it should. Even worse than the verbosity thing.
Re: (Score:2)
at first I thought the verbosity thing was kind of cool. Now I wish it would give me more precise answers. and.... (me complaining), when I feed it code, I have to specify to change one small part of it, but in general, it seems to want to modify all of it, and in many cases it inserts bugs. So I learned, only give it small snippets.
Re: (Score:2)
> at first I thought the verbosity thing was kind of cool. Now I wish it would give me more precise answers. and.... (me complaining), when I feed it code, I have to specify to change one small part of it, but in general, it seems to want to modify all of it, and in many cases it inserts bugs. So I learned, only give it small snippets.
Often the verbosity of the response can be tailored via the appropriate additional qualifying words in the prompt. So, couldn't this be transparently baked into a personalized interface? That is, allow the user to specify the preferred verbosity as part of personalized settings and silently add the appropriate additional words to the prompt each time. The user can still explicitly ask for more verbosity when desired.
Re: No shit (Score:2)
Have you ever tried to instruct it to ask you for clarifying questions before providing an answer?
Re: (Score:1)
The potential energy itself is lost because that stored charge in the small capacitor pushes charge out to spread evenly to the initially uncharged capacitor. The energy itself was used up in the electrons pushing away from eachother. Even with ideal capacitors and zero resistance - the energy need not be lost as heat.
Sort of like letting a compressed spring go. Even if the spring did no working lifting or pushing against any other object, it still lost potential energy springing itself back to rest.
Re: (Score:2)
so... we get back to the mystery of Entropy, I guess. Everything just goes to its lowest energy state. It seems to contradict the premise that energy is never gained, nor lost, it is just converted to other forms of energy. If energy can be lost, can it be gained?
Re: (Score:2)
> I recently gave AI's a paradox. When an ideal 10uF cap is charged to 100V, it has x charge, when an uncharged cap of 100uF is placed in parallel...
Is the 100uF cap also ideal?
Ideal Capacitors not the Problem (Score:2)
The problems with the question you stated have nothing to do with ideal capacitors. Your first problem is that you have 'x' as a charge to start with and then as an energy at the end. Then there is the problem that if your first capacitor is charged to 100V and a second is placed _in parallel_ with it that too will be charged to 100V, not 9.1V since connecting something in parallel implies that you are connecting it to an external EMF such that the potential difference across the components is the same as o
Re: (Score:2)
I though his question was more than clear enough TBH.
You're being more than needlessly pedantic about what parallel means here to the point of being wrong. It doesn't imply that it's connected to something else. If I have an array caps connected in parallel and disconnect that array from the supply, they have not ceased to be in parallel. Parallel simply implies that you have a bunch of components where all their terminal A's are connected together and all the terminal B's are connected together.
Once you kn
Re: (Score:2)
Yes, I did this experiment recently. I connected a 10uF capacitor to an external voltage source, charged it up to 100V, then connected a 100uF in parallel. The equations are 0.5*C*V*V. Energy just disappears in this case.
Re: (Score:1)
> I recently gave AI's a paradox. When an ideal 10uF cap is charged to 100V, it has x charge, when an uncharged cap of 100uF is placed in parallel, the voltage goes to about 9.1V and the total energy is about 0.1x. If energy is neither destroyed nor created, where did the energy go? The AI kind of forgets that I specified an "ideal" capacitor, and makes shit up. If a human can explain this to me, I am all ears. Please keep in mind: "Ideal capacitor", and "equations show this", don't make shit up.
Consider the moment just before the two caps are connected to each other. One is charged to a nonzero voltage and the other is at zero. As the conductor attached to one capacitor top plate approaches the conductor attached to the other capacitor top plate (the bottom plates already sharing a common connection), the E field between them will at some point be high enough to exceed the breakdown field of the medium in between (even if it is free space). Then an arc occurs. That arc is observable, maybe even vi
Re: (Score:2)
your question is not a paradox, you are just doing the math wrong. The initial state is 100 x .000010 = 0.001 watt hours. The final state is 9.1 x .000110 = 0.001001 watt hours. Remember the end point capacitance is 110 uF. So 9.1 volts make perfect sense.
Now what AI does with this is a completely different question.
Re: (Score:2)
sorry, my watt hours should be watt seconds. And one watt-second does equal one joule. So please explain the math here. moving from 100 V @ 10uF to 9.1V @ 110uF is exactly what you would expect. Just where does the 0.1X come from.
Re: (Score:2)
Since you seem to have your mind wrapped around this concept, what happens if a 1:1 transformer is placed between the caps? I have shown experimentally that a 100uF cap gets charged to about 9V if placed on the secondary side. The cap that is in parallel with the charged 10uF is also charged to about 9V. The loop in the primary redistributes the electrons, as it should, and in the process creates charge in the secondary. Does this "create energy", or does most of the energy "disappears"? Personally
Re: (Score:2)
I am not sure how a transformer would fit in with a DC capacitance / storage scenario. Transformers imply alternating current, and while they can be 1:1, I have never thought of a transformer as a DC component.
And yes, the AI explanation is not based on the basic formulas. That is actually the problems. Humans can sanity check an answer looking for conservation of energy. AI does not take this step. For these types of problems, a little recursion with some simple application of basic laws of physics sh
Re: (Score:2)
To me it isn't a DC thing, it is a transient thing that could be repeated over and over again. So conceptually, it is AC.
Re: (Score:2)
Fine, but the math gets a lot more complicated, especially when the voltage and current waveforms get out of sync (ie, power factor). But the conservation of energy still works with ideal components (which obviously do not actually exist). Also from a practical point of view, most 100 uF capacitors are electrolytic (or they are quite large) which does not play well with AC, but again the "ideal" part of the question does preclude this anyway.
Re: (Score:2)
That seems like a BS, kind of pre-conclusion mindset to me. I agree with conservation of energy premise myself, however, how do you reconcile that when a charged 10uF cap is placed in parallel with a 100uF cap with a 1:1 transformer in the loop. Place a 100uF cap on the other side of the 1:1 transformer... Energy seems to be created, or destroyed? Everything charges to about 9V. I did the experiment with electrolytic caps. I have the privilege to be doing R&D in the area of "creating energy out o
Re: (Score:2)
That implies conservation of current applies, or energy/time? or watt/hours? something like that? I can get my mind around that concept.
Re: (Score:1)
The "two-capacitor paradox" is well documented. The first problem is a dimensional error, comparing charge (e.g. coulombs) and energy (e.g. joules). But another problem is that conservation of energy must count all possible types of energy (not just stored charge), including heat, photons, and electromagnetic radiation during the energy transfer from one capacitor to the other. Even if you have ideal capacitors and zero resistance, with infinite current for an infinitesimal time at the moment of connection
Re: (Score:2)
> Watson.
Speaking of Watson, Slate recently ran a retrospective article on IBM's Watson playing Jeopardy! . Coincidentally, it also mentioned when Watson made up the response that Toronto was a U.S. city:
[1]https://slate.com/culture/2025... [slate.com]
[1] https://slate.com/culture/2025/09/jeopardy-game-watson-questions-final-ken-jennings.html
Re: (Score:2)
aka Modeling their developers.
Human do the same (Score:1)
I have noticed that employees that guess get the same reward. The appearance of progress gets more reward than that the guy that got the real answer on paper but was asked to stand down before submitting it becuase someone already turned in a guess.