Distillation in the era of LLMs: when the student started to answer

By TheSequence · July 13, 2026.

This TheSequence Knowledge article opens by revisiting the famous 2015 distillation paper (Hinton et al.'s on "dark knowledge"), but not to discuss its temperature trick or that notion of dark knowledge—rather to point to something more fundamental: the conceptual world that paper took for granted. In that framework there was a fixed input distribution, a teacher model that produced a probability vector over a closed set of classes, and a student model trained to imitate that vector. The dataset was passed through both models, the loss was computed, backprop was run, and the whole process had a kind of mechanical innocence: each piece stayed in its lane.

According to the author, that innocence broke as soon as language models arrived. One by one, the original assumptions of the 2015 paper stopped holding. The article presents itself as the account of that rupture: the part of the history of distillation in which the field stopped thinking in terms of compression (creating a smaller copy of a fixed function) and began thinking in terms of capability transfer—that is, getting a small model to actually know how to do something hard with the help of a larger model.

According to TheSequence, that paradigm shift took place over about five years and passed through three recognizable stages. Each of them, at the time, looked like a simple engineering improvement, but in retrospect it represented a fundamental change in how the very concept of distillation was understood. The body of the email cuts off just as it begins to describe the first of those stages, titled "Stage One: Sequences Are Not Pictures," without developing its content or stages two and three, which are presumably explained in the full linked article.

In general, this kind of TheSequence Knowledge piece is usually part of a longer series on the technical evolution of machine learning concepts—in this case distillation—tracing its path from computer vision (hence the reference to "sequences are not pictures") to its application in large-scale language models.