Knowledge distillation: when the dataset becomes the teacher

By TheSequence.

This installment of the educational series "Sequence Knowledge" deals with distillation, understood as a way of generating synthetic data to train smaller models from larger, more capable ones.

The text opens with a historical reflection: for most of the history of machine learning, data was treated as if it were geology. It already existed in the world —in books, web pages, code repositories, conversations, photographs and databases— and the researcher's job was simply to dig it up, clean it, tokenize it and feed it to a model.

As the newsletter explains, large language models (LLMs) have transformed that relationship. A sufficiently capable model is no longer merely a consumer of data: it can produce questions, answers, explanations, critiques, preference labels, tool-use traces, textbooks, coding exercises and even complete miniature curricula.

This, the text notes, gives rise to a new training primitive. Instead of turning to an expensive model to answer every production query indefinitely, it is asked to manufacture the learning experience on which a smaller model will feed. The "teacher" model runs offline; its outputs become a dataset; that dataset trains the "student" model; and although the teacher disappears at inference time, part of its behavior is embedded in the student.

The newsletter excerpt closes with the piece's central idea: that is precisely what constitutes "synthetic data as distillation". The body of the message cuts off at this point —it is an introductory extract— and refers the reader to the full article published on Substack for a deeper look at the technical development of the concept.