Distillation
Also called knowledge distillation, distilled model
Training a small model to copy a big model's answers, so it keeps much of the skill at a fraction of the size and cost.
A large model is slow and expensive to run. Distillation makes a smaller one, the "student", by training it on the large model's answers rather than only on raw text. The big "teacher" model answers a great many prompts; the student practises producing the same answers until it matches them closely.
The result keeps a surprising amount of the teacher's skill at a fraction of the cost. It's also commercially sensitive. OpenAI's terms, for example, forbid using its outputs to develop competing models, and in January 2025 OpenAI said it had seen evidence that DeepSeek had distilled from its models, which DeepSeek didn't confirm.
For example
DeepSeek trained small open models to imitate the worked answers of its big reasoning model R1, and released them as "R1 Distill" versions anyone could run.
Put it to use
Where you'll meet distillation in ChatGPT, Gemini and Claude, how to use it there, and the mix-up most people make.
Daily Bytes Plus is in the Android app.