Looped Transformers: what happens inside the loop and when another pass pays off

By Turing Post · Sat, 7 Nov 2026.

Turing Post presents a guide from its AI 101 series dedicated to Looped Transformers. The email notes that there has been a recent surge of research on this architecture and that new papers keep appearing. According to the text, the reason is that familiar AI architectures need to be rethought to unlock their full potential, and in this case that potential is layer reuse. As the entire AI stack matures, its basic foundations must meet the demands of new systems.

The email recaps how a conventional Transformer works: the input passes through a sequence of layers, usually each with its own weights, until it reaches the output. Making the stack deeper gives the model more transformations to perform, but also more parameters to store and train. Compute and model size grow together. Looped Transformers loosen that link: they send a hidden representation through the same layers several times, updating it on each pass. The advantage is that the model can do more work without adding a new set of weights for each additional step. The text cautions that, although it sounds perfect, it is not that simple.

The guide promises to explain what Looped Transformers are, what happens inside the loop, how to optimize and measure the efficiency of a looped model in 2026 and, above all, when another pass is worth its cost. The episode's table of contents includes: essential definitions, what flows through the loop, the change of state on each pass, why these models became important for algorithms, adaptive depth (deciding where to spend another pass), whether looping actually saves resources, and what happens with hardware, the KV cache and other parameters.

Definitions provided in the email:

• Looped Transformer: a Transformer that runs the same block or stack of blocks several times, updating its hidden state on each pass.

• Transformer block: a unit with attention and a feed-forward network, usually with residual connections and normalization.

• Hidden state: the numerical representation of tokens while the model processes them; what travels through the loop are these representations, not newly generated words.

• Weight sharing: reusing the same parameters across several passes instead of storing a separate set for each one.

• Loop / recurrence: one pass through the shared block or stack; the output becomes the input to another pass.

• Recurrent depth: the number of times the shared block or stack is run. Terminology varies across papers: some use "depth" to count individual block applications.

• Input injection: adding back or reintroducing the original input representation during repeated passes, so the model can keep using it while its hidden state changes.

• Adaptive depth: giving different inputs or tokens a different number of layers or passes, instead of a fixed number for everything.

• Effective depth: the total number of block applications. For example, four blocks repeated eight times yield 32 applications; blocks outside the loop are counted separately.

• Full-stack looping: repeating the entire Transformer stack.

• Middle-stack looping: repeating a central group of blocks between input and output blocks that run only once.

On what travels through the loop, the email explains that a Looped Transformer reuses a block (or a stack of blocks) across several passes. Each block contains familiar components: attention, a feed-forward network and residual connections. The distinctive architectural change is that the output of the shared stack becomes its input on the next pass. The accompanying image shows the Universal Transformers diagram, in which recurrent encoder and decoder blocks reuse attention and transition functions over T steps before producing the output probabilities.

The email includes a comparison at equal effective depth. A standard Transformer has 24 distinct blocks with their own weights, a single pass through the stack and 24 block applications in total; after block six, the hidden state moves on to block seven. The Looped Transformer in the example has 6 distinct blocks, 4 passes through the stack and also 24 block applications; after block six, the hidden state returns to block one for another pass.

Two quantities follow from this: unique depth, which counts the distinct blocks whose weights the model stores, and effective depth, which counts how many block applications the input passes through. For a fully looped stack, effective depth is the number of unique blocks multiplied by the number of passes; if the model has a non-repeated beginning and end, those blocks are counted once. The text clarifies that the looped model is not equivalent to a conventional 24-block model, because the latter has more freedom to specialize each depth. The goal of looping is to gain additional compute without needing a new set of weights for each pass.

The rest of the guide, including tracking the hidden state through the loop, methods for deciding when to stop, why a smaller KV cache can let a GPU serve more requests at once and the most important advances of 2026, is reserved for Premium subscribers and does not appear in the body of the email, so it cannot be summarized here.

As extras, the email links to the previous article in the AI 101 series ("Test-Time Training – A Model That Changes While You Use It") and a YouTube video from Axiom Math about an interactive 3D "math city" built with 16 million lines of verified code.