How does model distillation transfer knowledge from teacher to student?

Short answer: Model distillation trains a smaller, faster student model (e.g. 8B parameters) to mimic the output probability distributions or synthetic reasoning outputs generated by a larger, more capable teacher model (e.g. DeepSeek R1 or GPT-4o).

Detail

Running massive models in production incurs high GPU costs and latency. Distillation compresses teacher capabilities into compact student architectures.

[Teacher Model (70B / DeepSeek R1)] ──► Generates Synthetic Reasoning Traces (COT)
                                                     │
[Student Model (8B / Llama 3 8B)]   ◄── SFT Loss ────┘ (Cross-entropy on teacher logits/text)

Distillation Paradigms

1. Logit-Based Distillation (Traditional KL-Divergence): The student model minimizes the KL divergence between its output logit distribution and the soft logits of the teacher model:

   $\mathcal{L}_{distill} = (1 - \alpha) \mathcal{L}_{CE}(y, p_s) + \alpha T^2 D_{KL}\left( \text{softmax}\left(\frac{z_t}{T}\right) \parallel \text{softmax}\left(\frac{z_s}{T}\right) \right)$

2. Sequence / Synthetic Data Distillation (Modern LLM Distillation): The teacher model generates thousands of synthetic Chain-of-Thought reasoning traces. The student model undergoes SFT directly on these teacher-generated reasoning datasets (used in DeepSeek-R1-Distill models).

Example

Synthetic dataset distillation loop concept in Python:

def generate_distillation_dataset(teacher_model, prompts: list[str]) -> list[dict]:
    distilled_data = []
    for prompt in prompts:
        # Teacher outputs reasoning trace + answer
        teacher_output = teacher_model.generate_cot(prompt)
        distilled_data.append({
            "instruction": prompt,
            "response": teacher_output # High-quality synthetic reasoning trace
        })
    return distilled_data

Interview tips

• Highlight DeepSeek-R1-Distill-Llama-8B as a prime example of synthetic reasoning distillation achieving frontier-level math performance.
• Note legal licensing considerations when using proprietary commercial APIs (e.g. OpenAI terms prohibiting using output to train competing models).

⬅ Back to Fine-Tuning and Adaptation · All topics