Token-level entropy analysis is a diagnostic technique for evaluating how well a student model preserves the uncertainty structure of a teacher model during knowledge distillation. It involves computing the Shannon entropy of the conditional output distribution at each token position and comparing histograms between teacher and student.
- Application in Distillation
- Methodology
- Significance
Token-Level Entropy Analysis
Token-level entropy analysis is a diagnostic technique for evaluating how well a student model preserves the uncertainty structure of a teacher model during knowledge distillation. It involves computing the Shannon entropy of the conditional output distribution at each token position and comparing histograms between teacher and student.
Application in Distillation
In the context of EOPD, this analysis revealed:
- Standard on-policy-distillation causes diversity collapse, retaining only 6.8% of high-entropy tokens (entropy >= 1.0) compared to 18.5% in the teacher.
- High-entropy tokens in reasoning tasks represent key decision points with multiple valid paths, not noise.
- EOPD preserves substantially more probability mass in the high-entropy region, staying closer to the teacher distribution.
Methodology
- Generate responses from teacher and student models on benchmark prompts.
- For each token position t, compute H_t = -sum_x pi(x|c_t) * log pi(x|c_t).
- Aggregate entropy values into histograms for comparison.
Significance
This analysis provides direct evidence that mode-seeking objectives (reverse KL) fail to transfer distributional structure in uncertain regions, motivating entropy-aware training strategies.
[src: raw/ingested/2026/03/2603.07079.md]