A brief history of model distillation

By TheSequence · July 7, 2026.

The article opens by questioning the usual narrative about the origins of knowledge distillation, which is often traced back to 2015 with the work of Geoffrey Hinton, Oriol Vinyals and Jeff Dean, who introduced the softmax 'temperature' trick and coined the phrase 'dark knowledge,' an expression that quickly became fixed in the field's vocabulary.

According to the author, that story, while appealing, is incomplete, since it ignores nearly a decade of prior work. The real history is quieter and more pragmatic, and it is worth recovering because the conceptual moves the field made between 2006 and 2015 still define how we understand model distillation today.

The text argues that, although the vocabulary and diagrams have changed over time, the underlying question remains the same: what exactly is transferred from a 'teacher' model to a 'student' model? To understand the modern landscape of techniques such as on-policy distillation, reasoning distillation and transfer between different architectures, the author proposes reviewing three foundational papers, noting that each solved a different problem on the path toward, in essence, the same idea.

The body of the email begins to lay out this timeline with a first milestone dated 2006, under the title 'Compression as Mimicry,' but the text cuts off at that point and no further content is available to summarize accurately.