Neural networks are not "already clean": why explaining a neuron takes more than examples that fit

The article, published on LessWrong by leogao, responds to a recent post by Nick Cammarata. Cammarata argued that, for years, the interpretability community was too quick to dismiss many neurons as polysemantic, when a closer look often reveals a simple pattern behind the examples that activate them. The author accepts that point and believes beliefs should be updated in that direction: polysemanticity was overestimated. But he argues that this is not enough to conclude that we understand most of what neurons do. Nor does it show that interpretability is solved or much easier than expected.

The first central idea is that it is not clear a simple pattern exists for every neuron. Even when there is one and it fits the activating examples, that does not prove the explanation is correct. The author gives a hypothetical case. The top-activating examples are sentences about a dog riding a bicycle. The top 20th percentile contains sentences like “John was riding his bike while his dog followed him.” The weakest activations talk about dogs without bicycles. Can we conclude that the neuron is monosemantic and means “dogs riding bicycles”? According to the author, no. He explains that the impressions given by activating examples can be misleading about how interpretable a neuron is.

He gives two reasons. The first is that the relationship is not symmetric: the fact that all examples activating the neuron fit the pattern does not imply that all examples fitting the pattern activate the neuron. Perhaps the neuron only fires for dogs on bicycles on sunny days and no one noticed. According to the author, this was a considerable problem in automated interpretability (auto interp) work. In practice, he says, it is very common for a neuron that seems to fire on X to fire only on a very specific subset of X that is hard to describe compactly. If the neuron were forced to always fire on X, the model would get worse. The second reason is that activation magnitudes can carry important information. It may be crucial whether the activation is strong or weak. If we cannot predict which will be strong and which weak, we may be missing a large part of the explanation. The author's best guess, based on the results he has seen, is that ablating small activations substantially degrades model performance. That is why he does not think weak activations can simply be ignored. He adds that he hopes to share something more concrete soon, and that he is surprised his result differs from Cammarata's, so it is worth tracking down the source of the divergence.

In his footnote, the author illustrates the difficulty with a neuron that fires on “Colorado” and also on “Tornado” (the Python library, not tornadoes in general). He struggles to imagine a simple monosemantic explanation that covers both. His general experience is that most neurons are fairly inexplicable, even after thinking hard about possible explanations.

The second part proposes how to truly verify that a neuron has been explained: through simulation scores. If things only “look more or less right,” that is not enough. If you cannot simulate the neuron without degrading the model, you do not really understand it. This requires writing a rubric (for example, a dog literally on a bicycle scores 4.5, and a dog and a bicycle with no riding involved score 3.2). The inputs are then scored against that rubric and the neuron's real values are replaced with the simulated ones. If the process badly degrades the model, the explanation was wrong.

Cammarata's post argues that simulation scores are too demanding, because they force you to explain a lot of material that does not matter. The author replies that this is a side effect of the correlation-based scores in the original auto interp paper. In his view, it is almost entirely fixed with two changes: (a) measuring KL divergence on the model's downstream outputs, and (b) scoring on a large dataset. KL also elegantly sidesteps the problem of having to carefully bin activations. If it does not matter for the model's behavior on which punctuation token certain sentence-level information is stored, or which activation intensities of the “Trump neuron” correspond to different sizes of Trump, KL does not penalize it. Conversely, if it matters a lot, the author considers it fair to demand an explanation. If the relationship is simple but uninteresting, the explanation does not become too complicated. If it is complex in a way that is essential to how the model works, it is worth knowing.

According to the author, had this methodology been applied from the start, some bad polysemantic explanations would have been caught. For example, the “cars and dogs” explanation that Cammarata uses as an example of a bad polysemantic explanation of the “V-shaped objects” neuron would have been flagged as bad. It would fail to fire on a great many V-shaped objects and would wrongly fire on round dogs and cars. He admits the simulation KL metric is not perfect, for example because it is hard to compare models of different sizes, and he understands the arguments against relying too heavily on metrics. Even so, it avoids the two failure modes described, which are very hard to notice by looking at examples but matter a great deal. He considers KL more correct than change in loss, because it would be “very cursed” if poor modeling of some neurons could be offset by modeling others in ways that improve loss. In his experience, this is not hypothetical. He also cautions that each individual neuron has a very small effect on the network, so the absolute numbers will be tiny. Even so, explanations can be compared across neurons, or large batches of neurons can be ablated at once.

The last section points out that explaining neurons is only one part of understanding a network. The goal is to understand how models work, which requires understanding the computation between neurons. According to the author, that is very hard and so far works poorly. Previous work on circuits between neurons keeps finding relatively large and complicated circuits, with a lot of confusing redundancy, that do not explain most of the variance in model behavior. His current explanation is that concepts are stored in superposition and that MLPs perform computation on those superposed features, carrying out more than one computation per neuron. He hopes it will be possible to untangle everything, understand how computation in superposition works and find clean, understandable circuits. But he notes that interpretability has often stalled for lack of rigor, by focusing on flashy things that exaggerate the real level of understanding. He concludes that it is up to the field to strike a balance between flashy qualitative examples and rigorous arguments.

The comments, which are only partially available to me, add nuance. One user analyzes Cammarata's “Colorado” neuron using the public data from his viewer. He says the Tornado library activation is small (around 8.6% of the maximum in one example). He claims the neuron fires mainly on Colorado, the Rockies and other mountainous regions, and also on various Python concepts. He suggests it is a “recursive generalization” from the Colorado–software connection. Another user counters that this explanation does not predict the specific combination of loadings. Another points out that the tokenization of “Colorado” and “Tornado” shares the fragment “ado,” which would suggest a text-based origin. The first user later revises his position after looking at the Qwen3-32B tokenizer. He concludes that the spelling-similarity theory is largely ruled out and mentions someone else's hypothesis: the neuron would act as a “world gate.” It is a speculative hypothesis, which he himself presents as such. Other comments ask whether neuron explainers can be trained with KL as the reward. They also ask whether the same argument applies to SAE latents, transcoders or crosscoder features. The answer given is yes. One commenter mentions largely unsuccessful attempts with that protocol, which he attributes to a lack of skill and compute. The downloaded text cuts off before the end of the comments.