Don't be fooled: LLMs do not reason, says an architect of AlphaGo

The article is an opinion piece by Thore Graepel, professor of machine learning at University College London and a former key member of the AlphaGo team at Google DeepMind. His central argument is as simple as it is uncomfortable for the industry: today's large language models (LLMs), however much they generate apparently deliberate chains of reasoning, do not reason in the way AlphaGo did ten years ago, and that shortcoming is precisely what makes them impossible to trust in high-stakes applications such as medicine, engineering or scientific research.

Graepel opens by recalling the episode that marked him personally: the March 2016 match in Seoul between AlphaGo and Lee Sedol, one of the great masters of Go. In the second game, the program played the now-famous 'move 37', a move so strange that several commentators thought it was a technical glitch. It was not: AlphaGo went on to win that game and the match as a whole 4-1. A stunned Lee Sedol said he had changed his mind about the machine: he had thought AlphaGo was based on probability calculation and that 'it was merely a machine', but after seeing that move he concluded that 'surely, AlphaGo is creative'.

To put the feat in context, the author recalls that Deep Blue beat Garry Kasparov in 1997 by looking six to eight moves ahead and evaluating 200 million chess positions per second, relying on rules hand-coded by humans. Go, however, is a vastly more complex game: the value of a stone depends on how groups and territories evolve over dozens of moves, and calculating even a fraction of the possible outcomes would take a supercomputer billions of years. That is why AlphaGo needed to 'intuit' at a glance who was winning, and even to invent moves no human had thought of before.

Here comes the point Graepel considers most misunderstood: the usual reading that move 37 was a simple flash of machine intuition is, he argues, wrong. What really made that creative choice possible was AlphaGo's capacity for reasoning, not its intuition alone. The system combined two components. On the one hand, a policy network trained to guess what move a strong human would make; this 'intuitive' part regarded move 37 as inconsequential, giving it barely a one-in-ten-thousand chance of being played by a human expert. What made AlphaGo choose it anyway was its search machinery, which went beyond immediate plausibility and explicitly weighed the future consequences of each candidate move, building and traversing a game tree with thousands of branches, each one representing a possible future.

Graepel maps this architecture onto a framework well known in behavioural science: Daniel Kahneman's distinction between System 1 (fast, instinctive, effortless) and System 2 (slow, deliberative, step by step). According to the author, AlphaGo offered a kind of mechanical analogue of that duality: its neural networks supplied the hunches ('this move looks promising', 'this position looks won') and its search module supplied the deliberation, testing those hunches against possible replies. Neither half worked on its own: intuition alone would never have chosen move 37, and brute-force search alone would have drowned without a prior filter of plausible moves.

Against that dual design, the article argues that today's language models operate in a radically different way. An LLM picks the next token over and over, which amounts, in his analogy, to System 1 in action: fast, associative and surprisingly good at completing patterns on almost any topic people write about. After the launch of ChatGPT, the industry found that linguistic fluency alone was not enough, and the response was to give models a form of deliberation through so-called 'chains of thought': instead of answering immediately, the model generates intermediate steps that break the problem down and carry partial results forward before committing to a final answer. Graepel acknowledges that the gains from this technique are real, especially in mathematics and programming, but insists that it does not introduce a genuinely different reasoning mechanism: that intermediate reasoning is still produced by the same next-token prediction process, simply iterated for longer before the answer is settled.

The author identifies three specific shortcomings that, in his view, make it impossible to call what chatbots do 'reasoning' in the sense a scientist would recognise. The first is that these models do not typically maintain an explicit, persistent and inspectable epistemic state: there is no open 'ledger' recording the hypotheses under consideration, the confidence placed in each explanation, the evidence being weighed and the questions left unresolved, all of it systematically revisable as new information arrives. The second is that they lack a clean separation between what the system knows and how it manipulates that knowledge: knowledge and reasoning are inextricably entangled in the neural network's weights, with no set of beliefs represented explicitly and independently. The third, backed according to the article by previous research, is that the chains of thought chatbots display are often fabricated after the fact: the model reaches an answer by one route but reports a different one; that is, it rationalises after deciding rather than deciding by reasoning.

For Graepel this is not an academic nuance but a first-order practical problem in the highest-risk fields — medicine, engineering, scientific research — where what matters is not only what a system concludes but how it reaches that conclusion. He cites medical diagnosis and treatment as an example: when an error occurs, it must be possible to identify whether the system's reasoning failed, whether it relied on invalid evidence or whether it started from incorrect assumptions; and that, he says, is impossible if the account of the reasoning is itself a later reconstruction rather than a faithful record of the process.

The article also contains a relevant personal revelation: Graepel says this conviction is the reason he recently left his post at Google DeepMind, with the aim of developing a different approach to artificial reasoning, explicitly inspired by AlphaGo's architecture. He proposes that, just as AlphaGo maintained a game tree with every variation considered and its networks' assessments of each position, a general reasoning system should maintain an 'epistemic state' representing what is taken as settled, what is in doubt, what has been ruled out and which questions remain open. Reasoning, on this view, would be a sequence of 'moves' that modify that epistemic state in order to advance knowledge and reduce uncertainty: deducing consequences, breaking problems down and, crucially, deciding which question to ask, which calculation to run or which experiment to carry out next.

The author admits that reasoning about the open world is far harder than playing Go or chess, because the current state of affairs is only partially known, the set of available actions is broad and variable, and the consequences of actions are stochastic or unknown. Even so, he maintains that recent advances in LLMs and other neural models already offer capabilities that can be put to use on such problems: LLMs can suggest ways to tackle a problem based on what is known and on the resources available, interact with tools via APIs or code, and help assess whether a claim is supported by the available evidence. The piece Graepel considers indispensable for the system to be honest is that an independent component evaluate each reasoning 'move' according to how much it actually reduces uncertainty, updating beliefs only when the change is backed by evidence; with those rules met, the system could accumulate certified knowledge and improve its own reasoning policy by learning from its past experience. He describes it as 'the scientific method on steroids', aimed at producing knowledge capable of withstanding scrutiny.

The article closes with a statement of principle: Graepel does not believe that trustworthy artificial intelligence will be achieved simply by making System 1 bigger. Scaling models sharpens intuition, he says, but does not make it more deliberative. Move 37 mattered because a machine held a position, weighed possible futures and chose the move its artificial instincts would have rejected. According to the author, society needs that kind of creative move in fields such as drug discovery, materials, climate or medical diagnosis — areas where the board looks nothing like a Go board and nobody hands over the rules in advance — and those contributions will only come from systems whose conclusions emerge from an auditable sequence of evidence, inference and belief revision, rather than from a convincing story told after the fact.

It is worth noting that this is an opinion piece: its strength lies in the author's technical authority and first-hand experience with AlphaGo, but the proposals on how to build that auditable 'epistemic state' for open-ended reasoning systems are described in conceptual terms, without implementation details, benchmarks or concrete experimental results, none of which the text itself provides.