Neural networks are already "clean": Goodfire argues polysemanticity was not due to superposition

Note to the reader: the available text is cut off midway through the first section (the case of the cat-and-car neurons). What follows summarizes only the introduction and the opening summary of the essay. It does not cover the sections on vision models, language models and the big picture, nor the companion technical paper the author announces.

The essay is an informal working hypothesis by Nick Cammarata, of Goodfire, with contributions from Curt Tigges (research on language neurons) and Tarun Gogineni (editing). Its central thesis is that neural networks are already “clean.” According to the author, a network's legible building blocks are its own mechanisms: neurons and circuits. That is why he believes the outlook for mechanistic interpretability may be much better than he himself feared.

Cammarata distinguishes two approaches. The first is fast normal science on the network itself: looking at neurons and circuits, studying how they relate, observing the data flow and running agent-scale experiments to test explanations. The second starts from a list of hypothetical features and tries to extract them all with increasingly sophisticated reconstruction objectives, such as sparse autoencoders, usually ignoring the original network. According to the author, the field began with the first and retreated to the second because networks looked like tangled knots.

The dominant explanation for that tangle was superposition. It held that when a network runs out of neurons, it stops grouping related things and folds unrelated things together, simply because they do not interfere with each other. If that were true, what a neuron groups would say very little, and the knot would have to be “unfolded” before studying anything. The most famous example of polysemanticity is a neuron in an image model that responded to both cat faces and car fronts. Moreover, the last layer of InceptionV1, the field's model organism in its early days, seemed so full of arbitrary mixtures that almost no neuron was easily interpretable.

The author claims superposition was the wrong explanation for InceptionV1. In his reading, architectural problems caused the final layers to overflow, and weak interpretability tools amplified the effect and made them look incoherent. Later, architecture changes intended to help deep layers learn better had the unintended effect of making them more interpretable. With those ordinary improvements, and without changing model size or data, between 99% and 100% of last-layer neurons are easily interpretable. Cammarata says it is the first non-trivial neural network he feels he largely understands, and he offers an explorable version. He also trained versions across a 300× size range and says they are legible at every scale. With less room, neurons do not fold together disparate things: they cover coarser concepts, such as a single “car” neuron for many kinds of car. With more room, that neuron differentiates into specific cars.

The author also revisits the canonical examples. The 2019 circuit, presented in “Zoom In” as superposition, showed a car feature that seemed to be spread across unrelated dog neurons. Cammarata now believes it hid nothing, because nothing reads that combination: it was ordinary generalization and composition. He says that, as far as he knows, no one has published a second circuit of the right shape. On the cat-and-car neuron (InceptionV1's 4e:55), his new reading is that the shared feature was a “V.” He believes the whiskers in the visualization were misleading because generalization happens within high-level patterns. Feature visualization seeks to raise activation and drags in everything the cat pathway brings along, even if the neuron does not “want” those features. To see the common pattern, one must visualize what the neuron's inputs have in common, not the neuron in isolation.

For modern language models, the essay contrasts old figures with preliminary estimates. In “Towards Monosemanticity” (Anthropic, 2023), with a small one-layer model, the median neuron scored 0 out of 14 on the rubric, meaning the evaluator could not form any hypothesis. OpenAI's 2023 automated interpretability project found about 1.7% of neurons interpretable under its most generous measure. In preliminary audits of Qwen3-32B, Goodfire estimates that about 90% of neuron poles have one main job, often with small related offshoots. Another ~7% would have two related jobs, and only ~2-4% are not yet understood. A footnote cautions that these figures depend on how thresholds for weak activations are set, on depth and on what counts as “a single thing.” For small sets and strong activations, it cites about 97% non-arbitrary mixtures. He adds that he does not think a single score matters that much and that it should be taken as a rough indicator. According to the author, dense and mixture-of-experts models look alike, and he obtained similar results in models of up to trillions of parameters, such as Kimi K3. That case will be covered in more detail in the companion technical paper.

What remains of apparent polysemanticity would be mainly a motif he calls recursive generalization. A neuron grows and sprouts offshoots, the way a tree branch sprouts sub-branches. For example, a neuron that detected “iy” went on to detect “diy” and then “do it yourself,” which does not even contain “iy.” The idea first appeared in a forgotten section of “Multimodal Neurons” (2021). It is now easier to watch neurons grow during training, and Cammarata says the idea holds up in everything he has seen. Even in the original InceptionV1 architecture, the least legible neurons turned out to be locally overgrowing into adjacent, reasonable territory. From this he concludes there is only one learning principle, not two: generalization, which turns out to be fractal. That is why explanations can be fractal too: you start with a basic one and zoom in when you want more detail.

To explain why the perceived gap was so large, the author points to several ordinary problems that added up. The 2023 explainer models could not run the conjecture-and-test loop. Metrics rewarded explanations that simulated activations rather than correct ones. No one knew what motifs to expect in language neurons, so people looked for the wrong things. And most studies used small or simplified models. In those models, he says, the network really did run out of room, and superposition seemed the natural explanation. But a network with little room does not change its principle: its neurons become broader, and those broad neurons are usually easy to decompose once you know what to look for.

The essay also describes an “inertia” or “crossword” effect. At first, with a single incomprehensible neuron, almost nothing helps, because circuits only explain one neuron in terms of others not yet understood. As the map fills in, each piece provides clues: its connections, the ecosystem around it and, in a mixture-of-experts model, its expert's specialization. As an example, the author describes a neuron for “the sense of vindication when a cheap purchase works better than the expensive alternative.” Such neurons for feelings or social cues tend to fire on punctuation marks such as commas, or on the word just before the point where the feeling matters for predicting what comes next. With that prior knowledge, tests of input and output weights, of its negative pole, of the direction it pushes generation when boosted and of specific brands all point to the same place. The author adds that this also helps debug the tools. With colleagues, he found a double-negative regime in SwiGLU neurons that, as far as they know, no one had noticed and that a decomposition mixing many neurons would probably have hidden.

The author even argues that understanding the whole network may be easier than understanding only the parts that matter most. He gives two reasons. It is hard to know how a network thinks until you study it closely: “deception” is a useful starting frame, but its components probably have alignment-relevant microstructure that can only be seen by looking inside. And tackling a small set of mechanisms is like solving a single row of an empty crossword, where without the crossings it is hard to know whether you are right. He says that, when this process was applied to modern convolutional networks, it basically worked. Agents can map neurons and circuits, build explorable microscopes, test explanations thoroughly and trace hundreds of thousands of attribution graphs. In his account, normal science never broke down. In language models it has also worked so far, up to trillions of parameters. He acknowledges that those models are far from solved because of their size, but declares himself optimistic about scaling the process.

The argument has four steps. Networks use a single learning principle, so there is no obstacle to starting to do ordinary science on them. With modern models and tools, that science goes better than people think. Understanding accumulates and everything gets easier as the map fills in. At a minimum, the network's own mechanisms should be part of the toolkit. Very speculatively, the author adds that they might be almost all we need. He links this to Rich Sutton's “bitter lesson”: perhaps in interpretability, too, ordinary research at scale wins out over a clever new decomposition.

The essay should be read as what it declares itself to be: a working hypothesis and an informal text. The figures cited are preliminary estimates and depend on thresholds and definitions, and the companion technical paper with details on language neurons is not yet in the material received.