Imagine you're a smart student. You want to learn a new skill, so you find a teacher who's slightly better than you. You watch them work, and try to copy what they'd do. You make mistakes. The teacher corrects you in real-time, telling you the probability they would have placed on each move you just made.
That's on-policy distillation (OPD) — the most popular method for training smaller language models from larger ones in 2024–2026. Everyone from DeepSeek to Anthropic uses some variant. It usually works beautifully.
But sometimes it doesn't. Sometimes the student gets worse. Sometimes a "better" teacher produces no improvement at all. And nobody has been able to explain why — until this paper.
"Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe" (arXiv:2604.13016) is the first systematic dissection of when OPD works, when it breaks, and what to do about it. The answer turns out to be deceptively simple — and uncomfortable for anyone hoping OPD will scale to longer reasoning horizons.
The problem the paper opens with
On-policy distillation is the de-facto post-training recipe for compact LLMs. The student generates its own rollouts, and at each token the teacher provides a probability distribution over the next-token possibilities. The student's loss is the KL divergence between its distribution and the teacher's.
optional read — the OPD objective
For a student policy $\pi_\theta$ and teacher policy $\pi_T$, on-policy distillation minimises:
$$\mathcal{L}_{OPD}(\theta) = \mathbb{E}_{x \sim \pi_\theta} \left[ \sum_{t} D_{KL}\left(\pi_\theta(\cdot|x_{<t}) \,\|\, \pi_T(\cdot|x_{<t})\right) \right]$$
Note the expectation is over rollouts from the student — that's what makes it on-policy. The student sees its own mistakes and gets per-token correction signal from the teacher. This contrasts with off-policy distillation (also called supervised fine-tuning on teacher rollouts) which samples from the teacher distribution instead.
The result has been a flood of strong compact models. LLaMA-3-8B distilled from LLaMA-3-70B. Phi-3 from GPT-4. DeepSeek-R1's distilled series. The recipe seemed bulletproof: bigger teacher + on-policy training = better student.
But the paper opens with three uncomfortable observations:
- Some "better" teachers produce no improvement when used for OPD on the same student.
- OPD sometimes makes the student strictly worse than before training.
- The standard explanation ("the teacher is too far ahead") doesn't predict which pairings will fail.
The authors then run a campaign of controlled experiments to figure out what's actually going on. The conclusions reshape how we should think about teacher-student selection.
The two-condition phenomenology
The paper's central empirical claim is that successful OPD requires two conditions, both of which must hold simultaneously:
Condition 1 (Compatibility): The student and teacher must share compatible thinking patterns — meaning their token distributions must be close enough that the student can plausibly reproduce the teacher's reasoning trajectories.
Condition 2 (Headroom): The teacher must offer capabilities genuinely beyond the student's existing distribution — meaning there has to be something new to learn.
Each condition alone is obvious in hindsight. Together, they exclude a surprising chunk of the teacher-student pairings that practitioners assume should work.
When Condition 1 fails: distributional mismatch
The authors run reverse distillation experiments — taking a smaller, weaker model and using it as the teacher for a larger model from a different family. Conventional wisdom says this should fail (weak teacher, strong student). And it does fail, but not for the reason you'd guess.
The student's rollouts traverse a probability landscape that the small teacher has never modelled well. The teacher's KL signal is therefore mostly noise. The student gets pulled toward a distribution the teacher can't accurately describe, and quality collapses. Even though the student is more capable in absolute terms, the gap in where they live in token space breaks the learning signal.
When Condition 2 fails: distributional indistinguishability
This is the more surprising failure mode. The authors compare same-family models of different sizes — Qwen-1.5B as student, Qwen-7B as teacher. Both trained on the same corpus, same tokeniser, same architectural family.
The finding: from the student's perspective, the 7B teacher is distributionally indistinguishable from the 1.5B student. The 7B model knows more facts and reasons better in absolute terms, but token-by-token its predictions overlap with the 1.5B student's predictions almost entirely. There's no genuinely new signal to extract.
OPD on this pair produces near-zero improvement. The student finishes training looking almost identical to where it started. The teacher's superior capabilities don't manifest in the per-token distribution because both models share the same training data — and 1B+ parameter models with the same training data converge to similar per-token distributions on that data.
Fig 1 — Three regimes for teacher-student token distribution overlap. OPD only works in the sweet spot: enough overlap for the signal to be informative, enough gap for there to be something to learn.
The mechanism: token-level probability mass
The phenomenology is interesting. The mechanism is the paper's biggest contribution.
The authors probe what's happening at every step of a successful OPD run by examining the per-token probability distributions for both student and teacher. The headline finding:
Successful OPD exhibits progressive alignment on high-probability tokens, with the shared top-k token sets concentrating 97%–99% of probability mass throughout training.
In plain language: at every generation step, both the student and teacher agree on roughly which 10-20 tokens are plausible candidates. That shared top-k set contains nearly all the probability. They might disagree on the ranking within that set — and that's where the learning signal lives — but the set itself is almost identical.
Why this matters: KL divergence on shared support
KL divergence has a critical property: it's only well-defined where both distributions place non-trivial probability. When the teacher places 0.001 probability on a token that the student places 0.5 probability on, KL blows up. When the teacher places 0.5 on a token the student places 0.001 on, KL is also large.
Successful OPD operates in the regime where the student and teacher's high-probability tokens overlap heavily. The KL signal becomes a fine-tuning signal: among the tokens we both think are plausible, which one is most likely? That's a sharp, learnable signal.
Failed OPD operates outside this regime. Either the student's top tokens have near-zero probability under the teacher (Condition 1 failure — the student is making moves the teacher can't even rate), or the teacher's top tokens fully overlap with the student's top tokens at near-equal probabilities (Condition 2 failure — there's no new ordering to learn).
optional read — the probability mass concentration argument
Let $S_t = \text{top-}k(\pi_\theta(\cdot|x_{<t}))$ be the student's top-$k$ tokens at step $t$, and $T_t$ the teacher's. Define the shared support as $S_t \cap T_t$.
The paper observes empirically that for successful OPD: $$\sum_{w \in S_t \cap T_t} \pi_\theta(w|x_{<t}) \approx 0.97-0.99 \quad \forall t$$ and similarly for the teacher's mass. The KL on the shared support dominates the total KL, making the signal informative and bounded.
For failing OPD (Condition 1 broken), $|S_t \cap T_t|$ becomes small early in training, and the marginalised KL behaves chaotically — gradients become high-variance, and the student's policy collapses toward a teacher distribution it cannot match.
Visualising the convergence
The authors track this concentration metric across training. For successful runs, it stays stable in the 97-99% band the entire time. For failing runs, it drops below 90% within the first few thousand steps and never recovers.
This gives you something operationally useful: measure the shared top-k mass during the first few thousand steps of OPD. If it drops below 95%, kill the run and pick a different teacher. You don't need to wait for downstream eval to know things are broken.
The recipe: two recovery strategies
The paper proposes two concrete interventions for the failure modes it identifies. Both target Condition 1 failures — the distributional mismatch case. (Condition 2 failures have no fix — if there's nothing to learn, no amount of clever training will produce it.)
Recovery strategy 1: off-policy cold start
Before starting OPD, do a brief phase of off-policy distillation on teacher-generated rollouts. The student samples from the teacher's distribution (not its own), and learns to imitate the teacher's outputs directly. This is just supervised fine-tuning on teacher samples.
The purpose: pull the student's distribution close enough to the teacher's that on-policy rollouts will subsequently land in regions the teacher can rate sensibly. Once the distributions are warmed-up to be compatible, switch to OPD for the bulk of training.
# Phase 1: off-policy cold start (compact, just enough to align distributions)
for batch in teacher_rollouts:
student_logits = student(batch.tokens)
loss = cross_entropy(student_logits, batch.teacher_targets)
loss.backward()
# Phase 2: on-policy distillation (the main event)
for prompt in prompts:
student_rollout = student.generate(prompt)
teacher_logits = teacher(student_rollout) # teacher rates student moves
student_logits = student(student_rollout)
loss = kl_divergence(student_logits, teacher_logits)
loss.backward()
This is the "warm up before sparring" strategy. The off-policy cold start doesn't have to be long — the paper shows that even 500-2000 steps of off-policy SFT is often enough to unlock subsequent OPD. It just has to be enough to drag the student's distribution into the teacher's evaluable range.
Recovery strategy 2: teacher-aligned prompt selection
The second strategy doesn't change the student's distribution. It changes which prompts the student rolls out on during OPD. The idea: pick prompts where the student's rollouts will naturally land in regions the teacher knows well.
Concretely: filter or rank the prompt pool by a teacher-aligned metric (e.g., teacher perplexity on student rollouts from a similar prompt). Prompts that produce student rollouts the teacher can rate confidently are kept; prompts that produce teacher-confused rollouts are dropped.
# Score each prompt by teacher confidence on student rollouts
def teacher_alignment(prompt, student, teacher, n=4):
rollouts = [student.generate(prompt) for _ in range(n)]
confidences = [teacher.confidence(r) for r in rollouts]
return mean(confidences)
# Curate the prompt set
aligned_prompts = [p for p in all_prompts
if teacher_alignment(p, student, teacher) > threshold]
# Run OPD only on aligned prompts
for prompt in aligned_prompts:
# ... standard OPD loop
pass
This is cheaper than the first strategy if you have many prompts available and don't want to spend compute on a cold start phase. It also preserves the student's original distribution outside the rated regions — useful if you care about not changing behaviour on prompts the teacher would have hated anyway.
The uncomfortable scaling concern
The paper closes with what is probably its most important section, despite getting less attention than the recipes. The question: can OPD scale to longer horizons?
"Long horizon" here means tasks where the student's rollout is hundreds or thousands of tokens before the teacher's signal becomes meaningful. Mathematical reasoning chains. Code generation with multi-step refactoring. Agentic tool-use trajectories. These are exactly the workloads that frontier post-training is targeting.
The concern: the probability mass concentration property (Section 3) is observed on short-horizon, single-token decisions. As the horizon grows, the student's rollout accumulates compounding distributional drift. Even if the per-token student-teacher overlap starts at 99%, after 500 tokens of student-driven generation, the student is in a region of rollout space that the teacher has never seen and cannot rate.
The compatibility condition holds at the token level but degrades at the trajectory level. OPD's per-token correction signal cannot fix accumulated trajectory-level drift.
This isn't an academic worry. The authors show explicit experiments where short-horizon OPD gives strong improvements on the same task that long-horizon OPD fails to improve on. The mechanism is the same in both cases — but on long rollouts, the student spends most of its tokens in regions where the teacher's signal is uninformative.
What this means for current practice
A lot of current frontier post-training assumes OPD will continue to deliver gains as we scale to longer reasoning horizons. The paper's analysis suggests this assumption needs empirical verification. The token-level mechanism that made OPD work in 2023-2025 may not extend to the 2026+ regime where reasoning chains are 10-100× longer.
The authors don't claim OPD is broken for long horizons — just that the mechanism they identified doesn't obviously extend, and the field needs new thinking for that regime. This is actually the most honest part of the paper. They name the limitation explicitly rather than papering over it.
The diagnostic toolkit
Combining the paper's insights gives you a practical diagnostic protocol for any OPD setup:
- Pre-flight check: Measure shared top-k probability mass between student and teacher on a sample of prompts before training. If it's already >99% with little ranking disagreement, you have a Condition 2 failure waiting to happen — the teacher offers no genuine signal.
- Mismatch check: Generate student rollouts and measure teacher perplexity on them. High perplexity = teacher can't rate the student's moves = Condition 1 failure imminent. Use off-policy cold start.
- Training-time monitoring: Track shared top-k mass during the first few thousand steps. Stable in 97-99% band = healthy OPD. Drifting below 95% = abort and diagnose.
- Horizon check: For long-horizon tasks, measure shared mass at multiple depths in the rollout (token 50, 200, 500, etc.). Drift across positions reveals the trajectory-level breakdown the paper warns about.
What I find compelling about this paper
Most ML papers in 2024-2026 propose a new method and claim wins on a benchmark. This paper does something rarer and more useful: it explains why an existing method works when it works, and fails when it fails. It doesn't propose a new training algorithm. It proposes a sharper understanding of the algorithm we already use.
The probability mass concentration finding is the kind of empirical observation that changes practice. Once you've seen it, you can't unsee it. Every OPD setup becomes something you can diagnose with a few minutes of measurement rather than days of training and downstream eval.
The scaling concern is also intellectually honest in a way that's rare. The authors could've stopped at "here are two recovery strategies" and let everyone go home happy. They chose instead to flag that even with the recoveries, the underlying mechanism may not extend to longer horizons. That's the kind of forward-looking caution that distinguishes serious research from hype.
What I'd want to see next
A few open questions the paper raises but doesn't fully resolve:
- Is there a teacher-student selection metric we can compute before training? The shared top-k mass is the obvious candidate, but the paper doesn't propose a clean threshold. A robust pre-flight test would save many failed runs.
- How does this interact with RL-style methods like GRPO? DeepSeek-R1 uses GRPO (Group Relative Policy Optimisation), which is on-policy in spirit but uses reward signals rather than teacher KL. Does the same compatibility/headroom phenomenology apply? The paper hints yes but doesn't run the experiment.
- Can the off-policy cold start be replaced by something cheaper? If the goal is just to drag the student's distribution into the teacher's evaluable range, are there lighter touch interventions (LoRA adapters, prompt prefix tuning, intermediate-layer alignment) that achieve the same effect?
- What about teacher ensembles? If a single teacher provides insufficient headroom, do multiple complementary teachers (each strong on different regions) provide more learnable signal collectively?
The honest summary
On-policy distillation has been the workhorse of LLM post-training for two years, but the field has been treating it as a black box. This paper opens the box and finds something striking: OPD works because student and teacher agree on which tokens are plausible 97-99% of the time, and the learning signal lives in the residual disagreement.
When that agreement breaks, OPD fails — either because student and teacher don't share a token distribution (Condition 1), or because they share too much and there's nothing to learn (Condition 2). The recovery strategies (off-policy cold start, prompt curation) address Condition 1. Condition 2 is unfixable — you need a different teacher.
And on the horizon: long-rollout reasoning may not benefit from OPD the way short-token tasks have. The compatibility property is a per-token phenomenon. Trajectories accumulate drift. The field will probably need new approaches for that regime, and the paper is honest enough to say so without claiming to have the answer.
For practitioners: measure shared top-k mass. Use it as a pre-flight check and training-time signal. Don't burn compute on OPD runs that are doomed by distributional incompatibility.
For researchers: the next big idea in LLM distillation will likely be a method that works at the trajectory level, not the token level. Until then, OPD remains useful — but only inside the regime where its mechanism actually operates.
arXiv:2604.13016 — Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. Worth reading in full if you're doing any kind of LLM post-training in production.