Mechanistic interpretability: Goodfire suggests an AI's inner workings may be more legible than we thought

By Nate from Nate’s Substack · Thu, 22 Oct 2026 13:03:36 +0000.

Nate's article explains Goodfire's research on mechanistic interpretability, that is, the effort to understand what happens inside an AI model when it answers a question. The central thesis is that many of the components doing that work may be more understandable than previously assumed, which could improve the ability to detect and fix AI failures. According to the email, paid subscribers get access to the full text, the “Nate’s Library MCP” and its Slack community.

The author previews the contents of the full installment. First, what Goodfire's numbers measure: the “99–100%” result in vision and the roughly 90% estimate for Qwen3-32B, and why neither is a percentage of AI safety. Second, a walkthrough of a specific neuron, neuron 127 in Goodfire's “microscope,” which goes from being a “ladybug detector” to something harder to label, to show the difference between a neuron's response and the use the network makes of it. Third, the terms neuron, feature and circuit. Fourth, three questions for reading any interpretability visualization: what was observed, what was inferred and what was actually tested.

As groundwork, the text distinguishes between training a model and using it. During training, the model makes predictions, its error is measured and an optimization procedure adjusts its numerical parameters; many of these are weights, numbers that determine how much an input influences a calculation. Nobody writes an instruction for each learned pattern. When the trained model is used, the weights normally stay fixed, and what changes is the activity produced by the input, called activation. According to Nate, the model's learned settings should not be confused with its response to a specific question.

A language model first splits text into tokens (words, fragments or punctuation marks), converts them into numerical representations and processes them layer by layer. In a Transformer, the architecture behind many language models, two types of computation repeat: attention, which lets information at one position draw on other parts of the context, and feed-forward layers, which transform the information at each position and contain the units researchers call neurons. As an example, the meaning of “bank” in a sentence about a river depends on the surrounding words. Finally, the model produces scores that are converted into probabilities for the next token; one is chosen, added to the context and the process continues. This also explains why a model can adapt to what it is told during a conversation without changing its weights: new information alters the context and, with it, the activity.

On what a neuron does, the article explains that in a simple network it takes input numbers, multiplies them by weights, adds a learned offset and applies an activation function. A simple function outputs zero for a negative total and passes a positive one through; these nonlinear steps make it possible to learn patterns that addition and rescaling cannot capture. The author warns that the biological name is misleading: it is a mathematical operation, not a miniature brain cell or a complete thought, and a 32-billion-parameter model does not have 32 billion neurons. He illustrates the idea with components that respond to edges and curves, and later ones that combine that activity into more complex shapes.

The three key terms are: neuron, a component of the model's computation; feature, a pattern or property represented in its activity; and circuit, a connected set of computations that helps explain how a behavior arises. A feature can involve several neurons and a neuron can contribute to several features. That a component responds to something does not yet explain how the rest of the network uses it. Using the example of a “wheel detector,” Nate poses questions (does it respond to all wheels? to clocks too? what if the wheel is partly hidden?) that turn an appealing label into a research problem: “wheel detector” is our proposed explanation, not what the model contains.

In the section on what Goodfire asks us to reconsider, the text recalls that a long-standing difficulty is that individual neurons seem to respond to unrelated things, usually explained by superposition: a model can represent more features than it has available dimensions through overlapping patterns. A common response has been to train a second model, such as a sparse autoencoder, to decompose the activity into more understandable features; “sparse” means few features are encouraged to be active at once. Nate cites that in 2024 Anthropic reported extracting millions of features from one layer of Claude 3 Sonnet, associated with places, programming concepts and abstract situations, and that its researchers were able to alter some responses by manipulating them. The company stressed that understanding the representations still left open the question of how the model uses them.