Will AI ever stop making things up? Why the goal isn't zero hallucinations, but a machine that can say “I don't know”

OUR THESIS. AI will never stop being wrong entirely. No system that learns from incomplete data can always be right, and neither can humans. But that is not the useful question. The useful question is whether AI will learn to flag when it doesn't know and to show where its claims come from. There, progress is real and measurable. Our reading: hallucination (a language model producing false information presented as true) depends as much on how we reward models as on the models themselves. That is why it is partly fixable. And handling it well is a precondition for AI's big promise in medicine and science, not a minor detail.

WHY IT MAKES THINGS UP. A language model does not look things up in a database. It predicts the most plausible next word based on patterns it saw during training. If a fact appears many times in its training text, like the capital of France, the model “knows” it. If it appears only once, like the birthday of an obscure person, the model has no pattern to rely on, but it still produces a convincing-looking answer. OpenAI's paper “Why language models hallucinate” (Kalai et al., September 2025) adds a second, more revealing cause. The evaluations that decide which model is “best” almost always score only correct answers. It is like a multiple-choice exam with no penalty for wrong answers: guessing scores better than leaving a blank. OpenAI gives an example. If a model guesses a birthday, it is right one time in 365. If it says “I don't know”, it gets a guaranteed zero. Under those rules, training to climb the leaderboards means training to guess. The authors' fix is not more hallucination tests. It is changing how existing tests are scored: penalise confident errors more than uncertainty, and give partial credit for admitting ignorance. Our reading: this belongs to the same family as chatbot sycophancy. It is not a one-off bug. It is a training incentive pushing in the wrong direction, and as such it can be corrected.

NOT ALL ERRORS ARE ALIKE. We use “hallucination” for different things, and it helps to separate them because each one needs a different fix. First, invented facts: a date, figure or biography the model didn't know and filled in. Second, fake citations: perfectly formatted references, rulings or papers that don't exist. This is the most dangerous kind, because it looks checkable. Third, reasoning errors: the starting facts are right, but the logic breaks at some step. Fourth, source-faithfulness errors: the model has the document in front of it and still summarises it wrongly or adds things it doesn't say. The HalluLens benchmark (presented at ACL 2025) formalises this split. It calls hallucinations that contradict what the user provided “intrinsic”. It calls those supported neither by the context nor by what the model learned “extrinsic”. Its authors also criticise many tests for mixing up hallucination with a simple lack of knowledge. Our reading: when a headline says “model X hallucinates 30% of the time”, the first question is what kind of error was measured, and on which test. Without that, the number means almost nothing.

WHAT HAS REALLY IMPROVED. Three advances have data behind them. The first is search with sources, known as RAG (retrieval-augmented generation: the model fetches documents before answering and relies on them). According to OpenAI's GPT-5 system card, its reasoning model goes from 40% wrong answers on hard factual questions without internet access to about 4.5% with browsing on, against 12.7% for o3 under the same conditions. The second advance is calibration, meaning the model abstains when it doesn't know. On SimpleQA (short questions with one verifiable answer), and according to that same system card as reported by several outlets, gpt-5-thinking-mini is right 22% of the time, slightly below o4-mini at 24%. But it gives wrong answers on 26% of questions, versus 75% for o4-mini, because it says “I don't know” far more often. That is exactly the shift the 2025 paper called for: slightly fewer right answers, far fewer misleading ones. The third advance is interpretability, the field that studies what happens inside a model. In 2025 Anthropic described how, in one of its models, the default response is to decline to answer. That refusal is switched off only when the model recognises something it knows. Hallucination happens when a name feels familiar even though the model knows nothing about it. Understanding the mechanism is the first step to fixing it. Now the caveat: RAG helps, but it is no vaccine. The Stanford and Yale study “Hallucination-Free?”, peer-reviewed in 2025, tested commercial RAG-based legal tools. Lexis+ AI hallucinated on 17% of queries and Westlaw's tool on 33%. That was fewer errors than plain GPT-4, but far from what their marketing promised. Thomson Reuters, which owns Westlaw, says its internal tests show lower rates. Both products have since been updated.

THE REASONING PARADOX. You might expect models that “think” step by step before answering to make fewer errors. Not always. In the o3 and o4-mini system card (April 2025), OpenAI reported that on its internal PersonQA test, made of questions about people, o3 hallucinated 33% of the time and o4-mini 48%, versus 16% for its predecessor o1. The company acknowledged that more research was needed to understand why. Its hypothesis is that o3 makes more claims overall, so it gets more right and also invents more. On Vectara's leaderboard, which measures faithfulness when summarising documents, DeepSeek-R1 scored a 14.3% hallucination rate versus 3.9% for DeepSeek-V3, its non-reasoning version. Vectara concluded that reasoning itself was not to blame. R1 tends to “over-help” and adds information that isn't in the text, even when it happens to be true. For OpenAI and Google models, Vectara found minimal differences. Our reading: more capability does not automatically mean more reliability. A more ambitious model takes more risks, and without incentives to abstain, those risks become errors. There is a trap in the other direction too: a model that refuses a lot can look very reliable on leaderboards. Hallucination rates should always be read alongside answer rates.

THE REAL COST. The consequences are no longer anecdotes. The founding case was Mata v. Avianca in New York (2023): two lawyers were fined $5,000 for filing court rulings invented by ChatGPT. Since then, the crowd-sourced database kept by researcher Damien Charlotin had logged more than 1,600 court decisions worldwide by mid-2026. In each one, a court found or implied that a party had relied on AI-fabricated material. About 1,100 are in the United States, and roughly half involve people representing themselves without a lawyer. In 2025, Deloitte agreed to partially refund a contract worth about A$440,000 to the Australian government. A report on the welfare system contained non-existent references and a fabricated quote from a federal court ruling. The firm maintains the errors did not change its findings. Our reading: the tool failed in these cases, but oversight failed more. As with the AI-agent security incidents we have analysed, this is a governance problem. Nobody checked what they were signing.

IMPLICATIONS. In the short term, hallucination is a serious risk and should not be sugar-coated. It affects high-responsibility professions like law, consulting, journalism and healthcare. It hits hardest those with the fewest resources to verify, such as people going to court without a lawyer. Three practical takeaways: check every citation and figure before using it; prefer tools that link their sources; and distrust any vendor promising “zero hallucinations”, because no independent test backs that up. There is a takeaway for the industry too: as long as leaderboards reward only correct answers, labs will keep having an incentive to train models that guess. Changing how models are scored is as important a reform as any technical breakthrough. Now the long term, which is where we are optimistic. Humans make mistakes too, and we didn't give up on science because of it. We invented peer review, audits and second signatures. The same will happen with AI: calibrated models, traceable sources and automatic cross-checking between systems. That combination is what will let us trust AI to diagnose illness, design drugs and speed up research. An AI that separates what it knows from what it guesses, and says so, is worth more than one that knows everything with total confidence. That is the AI that can help eradicate disease. Not one that never fails, which will never exist, but one that knows when it might fail and warns you.