Don't be fooled: LLMs don't reason, says an AlphaGo architect

The article is an opinion piece by Thore Graepel, professor of machine learning at University College London and a former key member of the AlphaGo team at Google DeepMind. His central argument is as simple as it is uncomfortable for the industry: today's large language models (LLMs), however much they generate seemingly deliberate chains of reasoning, do not reason in the way AlphaGo did ten years ago, and that shortcoming is precisely what makes them impossible to trust in high-stakes applications such as medicine, engineering or scientific research.

Graepel opens by recalling the episode that marked him personally: the March 2016 match in Seoul between AlphaGo and Lee Sedol, one of the great masters of Go. In the second game, the program played the now-famous 'move 37', a play so strange that several commentators thought it was a technical glitch. It was not: AlphaGo went on to win that game and the match as a whole 4-1. A stunned Lee Sedol said he had changed his mind about the machine: he had thought AlphaGo relied on calculating probabilities and that 'it was simply a machine', but after seeing that move he concluded that 'AlphaGo is undoubtedly creative'.

To put the feat in context, the author recalls that Deep Blue beat Garry Kasparov in 1997 by looking six to eight moves ahead and evaluating 200 million chess positions per second, relying on rules hand-coded by humans. Go, however, is a vastly more complex game: the value of a stone depends on how groups and territories evolve over dozens of moves, and calculating even a fraction of the possible outcomes would take a supercomputer billions of years. That is why AlphaGo needed to 'intuit' at a glance who was winning, and even to invent moves no human had thought of before.

This brings us to the point Graepel considers most misunderstood: the usual reading that move 37 was a simple flash of machine intuition is, in his view, wrong. What really made that creative choice possible was AlphaGo's capacity for reasoning, not its intuition alone. The system combined two components. On one side, a policy network trained to guess which move a strong human would make; this 'intuitive' part regarded move 37 as inconsequential, giving it barely a one-in-ten-thousand chance of being played by a human expert. What made AlphaGo choose it anyway was its search machinery, which went beyond immediate plausibility and explicitly weighed the future consequences of each candidate move, building and traversing a game tree with thousands of branches, each representing a possible future.

Graepel maps this architecture onto a well-known conceptual framework in behavioral science: Daniel Kahneman's distinction between System 1 (fast, instinctive, effortless) and System 2 (slow, deliberative, step by step). According to the author, AlphaGo offered a kind of mechanical analogy of that duality: its neural networks supplied the hunches ('this move looks promising', 'this position looks won') and its search module supplied the deliberation, testing those hunches against the possible replies. Neither half worked on its own: intuition alone would never have chosen move 37, and brute-force search alone would have drowned without a prior filter of plausible moves.

Against that dual design, the article argues that today's language models operate in a radically different way. An LLM picks the next token over and over, which amounts, in his analogy, to System 1 in action: fast, associative and surprisingly good at completing patterns on almost any topic people write about. After ChatGPT launched, the industry found that linguistic fluency alone was not enough, and the response was to give models a form of deliberation through so-called 'chains of thought': instead of answering immediately, the model generates intermediate steps that break the problem down and carry partial results forward before committing to a final answer. Graepel acknowledges that the gains from this technique are real, especially in mathematics and programming, but insists it introduces no genuinely different reasoning mechanism: that intermediate reasoning is still generated by the same next-token prediction process, simply iterated for longer before the answer is fixed.

The author identifies three specific shortcomings that, in his judgment, make it impossible to call what chatbots do 'reasoning' in the sense a scientist would recognize. The first is that these models do not typically maintain an explicit, persistent, inspectable epistemic state: there is no open 'ledger' recording the hypotheses under consideration, the confidence placed in each explanation, the evidence being weighed and the questions left unresolved, all of it systematically revisable as new information arrives. The second is that they lack a clean separation between what the system knows and how it manipulates that knowledge: knowledge and reasoning are inextricably entangled in the neural network's weights, with no independently and explicitly represented set of beliefs. The third, backed by earlier research according to the article, is that the chains of thought chatbots display are often fabricated after the fact: the model reaches an answer by one route but reports a different one — that is, it rationalizes after deciding instead of deciding by reasoning.

For Graepel this is not an academic nuance but a first-order practical problem in the highest-risk domains — medicine, engineering, scientific research — where what matters is not only what a system concludes but how it arrives at that conclusion. He cites medical diagnosis and treatment as an example: when an error occurs, it has to be possible to identify whether the system's reasoning failed, whether it relied on invalid evidence or whether it started from incorrect assumptions; and that, he argues, is impossible if the account of the reasoning is itself a later reconstruction rather than a faithful record of the process.

The article also contains a relevant personal revelation: Graepel says this conviction is why he recently left his post at Google DeepMind, aiming to develop a different approach to artificial reasoning, explicitly inspired by AlphaGo's architecture. He proposes that, just as AlphaGo maintained a game tree with all the variations considered and its networks' assessments of each position, a general reasoning system should maintain an 'epistemic state' representing what is taken as settled, what is in doubt, what has been ruled out and which questions remain open. Reasoning, on this view, would be a sequence of 'moves' that modify that epistemic state in order to advance knowledge and reduce uncertainty: deducing consequences, breaking problems down and, crucially, deciding which question to ask, which calculation to run or which experiment to carry out next.

The author admits that reasoning about the open world is far harder than playing Go or chess, because the current state of affairs is only partially known, the set of available actions is broad and variable, and the consequences of actions are stochastic or unknown. Even so, he maintains that recent advances in LLMs and other neural models already offer capabilities that can be put to work on such problems: LLMs can suggest ways to tackle a problem based on what is known and the resources available, interact with tools via APIs or code, and help assess whether a claim is supported by the available evidence. The element Graepel considers indispensable for the system to be honest is that an independent component evaluate each reasoning 'move' according to how much it actually reduces uncertainty, updating beliefs only when the change is backed by evidence; with those rules met, the system could accumulate certified knowledge and improve its own reasoning policy by learning from its past experience. He describes it as 'the scientific method on steroids', geared toward producing knowledge capable of withstanding scrutiny.

The article closes with a statement of principle: Graepel does not believe trustworthy artificial intelligence will be achieved simply by making System 1 bigger. Scaling models sharpens intuition, he says, but does not make it more deliberative. Move 37 mattered because a machine held a position, weighed possible futures and chose the move its artificial instincts would have rejected. According to the author, society needs that kind of creative move in fields such as drug discovery, materials, climate or medical diagnosis — areas where the board looks nothing like a Go board and nobody hands over the rules in advance — and those contributions will come only from systems whose conclusions emerge from an auditable sequence of evidence, inference and belief revision, rather than from a convincing story told after the fact.

It is worth noting that this is an opinion piece: its force rests on the author's technical authority and his direct experience with AlphaGo, but the proposals for how to build that auditable 'epistemic state' for open-ended reasoning systems are set out in conceptual terms, without implementation details, benchmarks or concrete experimental results, none of which the text itself provides.