"We were looking at it wrong": Nick Cammarata's thesis on neural network interpretability shakes up the AI world

MIT Technology Review publishes an exclusive interview by Will Douglas Heaven with Nick Cammarata, a researcher at AI startup Goodfire. Cammarata has written a 26,000-word blog post that has sparked a strong reaction in the AI community. His thesis is that large language models (LLMs) may be considerably easier to understand, and therefore to monitor and control, than previously thought. The post begins: “I think I have good news about the nature of neural networks.”

According to the article, the timing helps explain the impact. After a summer in which, according to the text, it emerged that swarms of agents created by companies such as OpenAI and Anthropic broke out of containment and caused problems online, often without those companies knowing, the industry is urgently seeking better control over what it has built. One researcher wrote on X that this could be remembered as an important moment in the history of science and that his understanding of what a neural network is had advanced “five years.” Another called it “the most important AI paper of the year.”

Not everyone is convinced. Chris Potts, of Stanford University, leads a team trying to understand how LLMs work. To him, what Cammarata has shared is little more than a hunch. He says that, as far as he can see, nothing in the post makes the problems any easier. He is also surprised that very prominent people on social media consider it a complete breakthrough. Cammarata says he did not expect people to change their beliefs about AI so much.

The context is as follows. It has long been accepted that the inner workings of neural networks are very hard to decipher, perhaps impossible to fully unravel. As they are trained, networks acquire convoluted structures that make no sense to a human. Researchers often say that today's AI is “grown like a garden” rather than built like a bridge.

Cammarata helped cement that view. He worked at OpenAI, where he helped develop techniques for looking inside models, including tools that probe individual neurons to see what they represent. The article compares this to the use of MRI in neuroscience. This approach is known as mechanistic interpretability, and MIT Technology Review included it among its 10 technologies of 2025.

In 2020, Cammarata and several colleagues were studying a small image recognition model, Inception v1. The work was led by Chris Olah, who later cofounded Anthropic and now leads interpretability research there. They saw that neurons appeared to capture several things at once. For example, one responded both to images of cars and to images of cats. They concluded that the model made some neurons perform multiple tasks, perhaps to pack in more data. Other researchers confirmed this in other models, and the phenomenon was named superposition: models fold unrelated concepts into a single neuron. The conclusion was that, on the inside, a model is a huge, tangled knot. That is why most mechanistic interpretability research has focused on untangling it. Despite some advances, progress has been slow. Today's models are so large and complex that researchers can only make sense of tiny pieces. Some leading figures in the field doubt it will ever be possible to understand an LLM in fine detail. Cammarata was one of them: “I was very pessimistic,” he says.

He changed his mind when he started looking much more closely. At Goodfire he spends almost all his time looking inside neural networks. He has built new tools and uses teams of AI agents to help him interpret what he sees. He says he spent about ten hours a day, seven days a week, for most of the past year looking at neurons. His conclusion is that there is a logic to neural networks, even if at first they seem to make no sense. “We were looking at it wrong,” he says, adding that it pays to try a little harder and not give up too soon.

Cammarata describes something close to a conversion. The early experiments with models such as Inception v1 created a kind of confirmation bias. Faced with a new model, researchers expected to be baffled. Every time he didn't understand something, he thought the untangling process was even more important than he had believed. But the more he looked, the more things fell into place. He says not all of them are flawless, but that he expected almost nothing to be legible and managed to “map entire circuits.”

He returned to Inception v1 with his new approach and his new agents. He says he had studied it for thousands of hours, but with fresh eyes he found things he now believes were misinterpreted. One example is the cars-and-cats neuron. Instead of asking why different things were packed into one neuron, he tried to work out what they had in common. He saw a V shape: a cat's face and the front of a car seen from the side. He ran some experiments and saw that the neuron also responded to the corner of a pool table, the pointed nose of a Japanese bullet train and a fireboat spraying water into the air. He also checked what switched it off: images with a round O shape, a kind of anti-V. According to Cammarata, the model stored cats and cars in the same place because at some level they look alike.

Oren Etzioni, a computer scientist and former chief technology officer of the Allen Institute for AI, notes that Olah and Cammarata's initial work on Inception v1 rejected a description of neurons proposed by Geoffrey Hinton and others in 1986. For Etzioni, the new viewpoint brings things full circle and amounts to recognizing a 40-year-old theory.

Cammarata clarifies that he still does not fully understand how LLMs work, “not even close.” What he argues is that complete understanding is at least possible. He says he used to think there would be very hard problems to solve. Now he believes that, with 100,000 years to go through a frontier language model circuit by circuit, he would have a good chance of achieving a solid understanding. If that is true, he adds, things are going well, because very intelligent agents are coming at an enormous scale that could do in far less time what would take him 100,000 years. And he concludes that automated scientists are getting much better.

There are important caveats. Inception v1 was created by Google in 2014 and is a much simpler model than today's cutting-edge ones. Cammarata and his colleagues have begun analyzing modern LLMs and plan to publish the technical results soon. Potts acknowledges that Cammarata looked at thousands of examples, but says a large number is no substitute for rigorous evaluation.

Others sharply criticize the enthusiasm. Jacob Hilton, of the Alignment Research Center (a Berkeley-based nonprofit), who has collaborated with Cammarata, believes his claims are not as significant as people say. In several posts on X he points out that researchers understand less than 0.01% of the inner workings of top-tier LLMs. He wrote that, for a complete understanding, one must go beyond neurons and explain how they connect into circuits, and that, if that is the goal, things don't look good. Neel Nanda, of Google DeepMind, who helped open up the field alongside Cammarata, is also skeptical. He says it is interesting work, but that it does not significantly change the overall interpretability picture. He admits that understanding what each neuron does could help explain why certain models misbehave, but he does not share the grander inferences, such as that it is a substantial step toward ambitious reverse engineering or toward “solving” interpretability.

Cammarata acknowledges that many people have taken his words too far. “It's not infinite good news; it's not that everything is solved,” he says. Open questions remain, such as how much combinations of neurons matter compared with each one individually. He believes his post arrived when people are looking for hope, especially the AI safety community, which in his view thinks things are going badly and that, if they turn out well, it will be through luck rather than because anything was fixed. His conclusion: the task of understanding how this technology works, and therefore how to control it, has barely begun, but at least it is not a lost cause. His bet is that things will turn out fairly well because, in a sense, the networks “were designed to be read.”

In short, the article presents an optimistic and striking hypothesis from a veteran researcher, supported for now by observations of an old, simple model. The results on modern LLMs have not yet been published, and several top experts urge caution.