Opinion: don't be fooled, LLMs don't reason (according to the former AlphaGo engineer)

The article is an opinion column by Thore Graepel, professor of machine learning at University College London and a former key member of the AlphaGo team at Google DeepMind. His starting point is personal: he was in Seoul in March 2016 when AlphaGo played the famous "move 37" against Lee Sedol, one of the best Go players in history. That move looked like such an obvious mistake that some commentators thought it was a programming glitch; in fact it was the key to victory, and Lee Sedol himself went on to say that, after seeing it, he changed his mind about whether the machine was "just" a probability calculation: "AlphaGo is creative," he said.

Graepel uses this anecdote to dismantle a widespread reading: that move 37 was a flash of pure machine intuition. In his view, it was exactly the opposite: it was AlphaGo's reasoning capability—not its intuition—that made that creative choice possible, and that reasoning capability is precisely what he argues is missing from today's AI based on large language models (LLMs).

To explain this, he describes AlphaGo's dual architecture. On one side is the "policy network," trained to imitate which move a strong human would play; this "intuitive" part regarded move 37 as unpromising, with barely a one-in-ten-thousand chance of being played by a human expert. On the other side is the search machinery, which explicitly builds and explores a game tree with thousands of branches, each representing a possible future, and it was this deliberation that ultimately chose the move. The author draws a parallel with the well-known psychological distinction (popularized by Daniel Kahneman) between a fast, intuitive "system 1" and a slow, deliberative "system 2": AlphaGo's neural networks supplied the hunches, and the search tested them against future consequences. Neither part, he stresses, would have sufficed on its own: intuition would never have chosen move 37, and brute-force search would not have known how to filter among all the possible options without the network's suggestions.

As a historical contrast, he mentions Deep Blue, which in 1997 beat Garry Kasparov by looking six to eight moves ahead and evaluating 200 million chess positions per second, but using rules hand-coded by humans. Go, he explains, is vastly more complex: calculating even a fraction of the possible outcomes would take a supercomputer billions of years, hence the need for AlphaGo to "sense" at a glance who was winning and to invent moves no human had thought of before.

The article's core argument is that today's language models operate in a radically different way. An LLM predicts the next token over and over, which amounts, according to Graepel, to system 1 in action: fast, associative and surprisingly good at completing patterns on almost any topic. Shortly after ChatGPT's launch, the industry realized that linguistic fluency alone was not enough, and the response was to introduce "chain of thought": having the model generate intermediate steps before answering. This has produced real improvements, especially in mathematics and programming, but the author insists it does not introduce a genuinely separate reasoning mechanism like AlphaGo's: the intermediate reasoning is still produced by the same next-token prediction process, simply iterated for longer before the model commits to an answer.

Graepel identifies three specific shortcomings that, in his judgment, prevent what chatbots do from qualifying as "reasoning" in the sense a scientist would understand it. First, these models do not maintain an explicit, persistent, inspectable epistemic state: there is no open "ledger" recording the hypotheses considered, the degree of confidence in each explanation, the evidence being weighed and the questions left unresolved, one that is systematically revised as new information arrives. Second, there is no clean separation between what the system knows and how it manipulates that knowledge: knowledge and reasoning are inextricably entangled in the neural network's weights, with no explicitly represented, independent set of beliefs. Third, and perhaps most troubling, research has shown that the chains of thought chatbots produce are often fabricated after the fact: the model reaches an answer by one path but reports a different one—that is, it rationalizes after deciding instead of deciding by reasoning.

The author connects this to a far-reaching practical problem: in high-stakes applications such as medicine, engineering or scientific research, it is not enough for a system to reach a correct conclusion; how it gets there matters too. When errors occur—for example, in a diagnosis or a medical treatment—it must be possible to identify exactly what went wrong: whether the system's reasoning was flawed, whether it relied on invalid evidence, or whether it started from incorrect assumptions.

It is at this point that Graepel reveals he recently left his post at Google DeepMind, precisely because he believes a new approach to automated reasoning is needed, inspired by AlphaGo's architecture. He proposes that, just as AlphaGo maintains a game tree recording all the variations and possible futures it has considered—each move and position annotated with the judgments of its neural networks—a general reasoning system should maintain an "epistemic state" representing what it takes as settled, what it doubts, what it has ruled out and which questions remain open. Reasoning, in this view, would be a sequence of "moves" that modify that epistemic state in order to advance knowledge and reduce uncertainty: deducing consequences, breaking problems into parts and, crucially, deciding which question to ask, which calculation to run or which experiment to carry out next.

The author himself acknowledges that reasoning about the open world is far harder than playing Go or chess: the current state of affairs is only partially known, the set of available actions is broad and variable, and the consequences of actions are stochastic or unknown. Even so, he argues that recent advances in LLMs and other neural models already offer useful building blocks for tackling this kind of general reasoning: LLMs can suggest ways to approach a problem based on what is known and the resources available, they can interact with tools through APIs or code, and they can help assess whether a claim is supported by the available evidence.

Most importantly, he insists, an independent part of the system must evaluate each "move" by how much it actually reduces uncertainty, updating the system's beliefs only when the change is backed by evidence. Once these rules are imposed, the model could accumulate certified knowledge and improve its own reasoning policy by learning from past reasoning experiences. Graepel sums up this vision with a striking image: such a system would work like "the scientific method on steroids," its purpose being to produce knowledge capable of withstanding scrutiny.

The article closes with a warning aimed at the entire industry: trustworthy artificial intelligence is not reached simply by making "system 1" bigger. Scaling models sharpens intuition, but it does not turn it into deliberation. Move 37 mattered, he recalls, because a machine held a position, weighed possible futures and chose the move its own artificial instincts would have rejected. According to Graepel, society needs that kind of creative, auditable move in fields such as drug discovery, materials, climate or medical diagnosis—domains where the "board" looks nothing like Go's and no one hands us the rules in advance. Those insights will come only, he concludes, from systems that genuinely reason: systems whose conclusions emerge from an auditable sequence of evidence, inference and belief revision, and not from a convincing story told after the fact.