--- title: "TIP: Token Importance in On-Policy Distillation" authors: ["Yuanda Xu", "Hejian Sang", "Zhengze Zhou", "Ran He", "Zhipeng Wang Alborz Geramifard"] url: "https://arxiv.org/abs/2604.14084" sections: 58 estimated_tokens: "21.0k" --- ## Contents - 1 Introduction - Contributions. - 2 Related Work - Curriculum learning and importance sampling. - Off-policy vs. on-policy distillation. - Response-level selection. - Token-level importance in distillation and RL. - 3 Setup - Student entropy. - Teacher–student divergence. - 4 TIP Taxonomy: A Two-Axis View of Token Importance - 5 Theoretical Analysis - 5.1 Oracle Token Weight - Proposition 1 ( Oracle token weight ) . - Remark 1 ( Quadrant ordering ) . - Proposition 2 ( Blind spot ) . - Remark 2 ( Soft-OR fixes the blind spot ) . - Empirical predictions. - 6 Method: Type-Aware Token Selection - 7 Experiments - 7.1 Experimental Setup - Models. - Data and evaluation. - 7.2 High-Entropy Tokens (Q1/Q2) - 7.3 Overconfident Tokens (Q3) - Q3 selection procedure. - Q3 is not just “large divergence”. - 7.4 Type-Aware Selection (TIP) - Top vs. bottom tokens by Soft-OR score. - Teacher entropy is uninformative. - 7.5 Beyond Mathematical Reasoning: Agentic Planning - Setup. - 8 Discussion and Conclusion - Limitations. - References - Appendix A Supplementary Theory - A.1 Derivation of the Descent Bound - Assumption 1 ( Smoothness ) . - Assumption 2 ( Token-separable approximation ) . - Derivation. - A.2 Entropy-Weighted Sampling: Coverage and Variance - A.3 Why Adding Divergence Improves the Ranking - Appendix B Supplementary Experiments - B.1 Teacher Entropy Is Uninformative - B.2 Isolating the Entropy Axis: Comparison with Divergence-Only Selection - External reference point. - Setup. - Results. - What this comparison does and does not say. - Why “large gradient” is not enough. - B.3 Agentic Planning: Held-Out Queries - B.4 Hyperparameters - B.5 Qualitative Examples Across Quadrants - Example 1: Generic variable vs. concrete substitution ( Q3 : low entropy, high divergence). - Example 2: Reasoning fork—restating vs. advancing ( Q1 : high entropy, high divergence). - Example 3: Arithmetic computation error ( Q3 : low entropy, high divergence). - Example 4: Confident on the wrong variable ( Q3 : low entropy, high divergence). - Example 5: Wrong mathematical symbol ( Q1 : moderate entropy, high divergence). ## Abstract Abstract On-policy knowledge distillation (OPD) trains a student on its own rollouts under token-level supervision from a teacher, but not all token positions matter equally, and existing views of token importance are incomplete. We ask: which tokens carry the most useful learning signal in OPD? Our answer is that informative tokens come from two regions : positions with high student entropy, and positions with low student entropy plus high teacher–student divergence—where the student is overconfident and wrong. Empirically, student entropy is a strong first-order proxy. Retaining 50% of tokens with entropy-based sampling matches or exceeds all-token training while cutting peak memory by up to 47%; under more aggressive retention, memory savings reach up to 58%. But entropy alone misses a second important region. When we isolate low-entropy, high-divergence tokens, training on fewer than 10% of all tokens nearly matches full-token baselines, showing that overconfident tokens carry dense corrective signal despite being nearly invisible to entropy-only rules. We organize these findings with TIP ( T oken I mportance in on- P olicy distillation), a two-axis taxonomy over student entropy and teacher–student divergence that explains why entropy is useful yet structurally incomplete and motivates type-aware selection rules that combine uncertainty and disagreement. We validate this picture across three teacher–student pairs spanning Qwen3, Llama, and Qwen2.5 on MATH-500 and AIME 2024/2025, and on the DeepPlanning benchmark for long-horizon agentic planning, where Q3-only training with 20% of tokens surpasses full-token OPD. Our experiments are implemented by extending the open-source OPD repository https://github.com/HJSang/OPSD_OnPolicyDistillation , which provides the practical training base for reproducing this work and supports memory-efficient distillation of larger models under limited GPU budgets. ## 1 Introduction Knowledge distillation transfers capability from a large teacher to a smaller student by training on the teacher’s output distributions (Hinton et al., 2015), and is a primary driver of the rapid growth in small-model capacity (Xiao et al., 2025). In on-policy distillation (OPD), the student generates its own responses and learns from the teacher’s corrections at each token (Agarwal et al., 2024; Gu et al., 2025). Since the student generates the context, token importance is a property of the *student–teacher state* at each position. This raises a direct question: *which tokens carry the most useful learning signal?* Our central claim is simple. In OPD, informative tokens come from two regions of the token state space: (1) positions with *high student entropy*, where the student is uncertain and still forming its prediction; and (2) positions with *low student entropy but high teacher–student divergence*, where the student is confident but misaligned with the teacher. The first region is easy to detect with entropy alone: retaining 50% of tokens by entropy-based sampling already matches or improves all-token training while substantially reducing memory. The second is easy to miss. Under more aggressive retention, entropy-only selection begins to lose a small set of low-entropy, high-divergence tokens—positions where the student is sharply peaked on a continuation that the teacher strongly disfavors. These *overconfident* tokens carry dense corrective signal despite being nearly invisible to any entropy-based rule. We organize this picture with TIP, a two-axis taxonomy crossing student entropy and teacher–student divergence into four quadrants (Section [4](#S4)). Theoretically, entropy is a strong first-order proxy but must conflate “confident and correct” with “confident and wrong,” and a parameter-free Soft-OR score fixes this blind spot (Section [5](#S5)). Experimentally, the combined score consistently improves over entropy-only selection on mathematical reasoning and remains competitive on agentic planning, where Q3-only selection is strongest (Section [7](#S7)). #### Contributions. - 1. We propose TIP, a two-axis taxonomy that organizes token importance by student entropy and teacher–student divergence, requiring no verification labels and no extra computation beyond the standard OPD loss. - 2. We prove that entropy is a strong first-order proxy but any entropy-only score is structurally blind to overconfident tokens, and that a parameter-free Soft-OR score fixes this blind spot (Propositions [1](#Thmtheorem1)–[2](#Thmtheorem2), Remark [2](#Thmremark2)). - 3. We validate the taxonomy across several datasets and model families, and show that Soft-OR consistently outperforms entropy-only selection on mathematical reasoning while remaining competitive on long-horizon agentic planning in DeepPlanning (Zhang et al., 2026), where Q3-only selection is strongest. Figure: Figure 1: Cross-task summary: average accuracy by selection method. Each panel shows one benchmark; bar height is the mean accuracy (mean@16) averaged across three teacher–student pairs for mathematical reasoning (Qwen3-8B$\to$4B, Llama-70B$\to$8B, Qwen2.5-14B$\to$1.5B) and across two teacher sizes (14B, 32B) for DeepPlanning. Methods: *Base.* = all-token OPD (100%); *Ent. 50%/20%* = entropy-based token selection at the stated retention ratio; *SO 50%/20%* = Soft-OR (Eq. [5](#S5.E5)) top-$k$ selection. The dashed line marks the all-token baseline. Soft-OR consistently improves over entropy-only selection on the mathematical reasoning benchmarks and remains competitive on DeepPlanning, confirming that augmenting entropy with divergence recovers the Q3 blind spot without sacrificing Q1/Q2 coverage. Refer to caption: 2604.14084v2/x1.png ## 2 Related Work #### Curriculum learning and importance sampling. The idea that not all training examples contribute equally dates to curriculum learning (Bengio et al., 2009) and self-paced learning (Kumar et al., 2010), which order or weight samples by difficulty. Importance sampling extends this to gradient estimation: Katharopoulos and Fleuret (2018) select mini-batch elements by gradient norm, and Ren et al. (2018) learn per-example weights via meta-gradients. These methods operate at the *example* level. Our work pushes the granularity to individual tokens within a sequence, where the relevant axes are student uncertainty and teacher–student disagreement rather than a scalar difficulty score. #### Off-policy vs. on-policy distillation. Classical sequence-level KD (Kim and Rush, 2016) trains the student on teacher-generated sequences (off-policy). On-policy distillation (Agarwal et al., 2024; Gu et al., 2025) instead lets the student generate its own rollouts and applies teacher supervision token-by-token, avoiding the train–test distribution mismatch inherent in off-policy data. Sang et al. (2026) further show that on-policy reverse KL self-distillation can compress lengthy reasoning chains into shorter ones. Distillation has proven effective across diverse settings—from pretraining to extreme compression to expanding reasoning capabilities beyond what RL alone achieves (Gu et al., 2024; Xu et al., 2024; Yue et al., 2025). Because token importance in OPD is determined by the *student’s own* distribution at each position, it cannot be pre-computed from teacher outputs—it must be assessed online. This makes the choice of which tokens to train on a fundamentally different problem from off-policy sample selection. #### Response-level selection. Several methods operate at the sequence level: Xu et al. (2026b) select responses at the frontier of student competence, and LION (Jiang et al., 2023) uses quality signals. These approaches select rollouts to train on but treat all tokens within a response uniformly. A complementary question—which we address—is which *token within* a response carry the most signal. #### Token-level importance in distillation and RL. In RL, Wang et al. (2025b) showed that high-entropy “forking tokens” drive most gradient signal, Cui et al. (2025) further revealed that the covariance between token log-probability and advantage drives entropy collapse during policy optimization, SPINE (Wu et al., 2025) extends this idea to test-time RL by updating only decision-critical branch points with entropy-band regularization, and Xu et al. (2026a) identified overconfident errors as a critical failure mode. In distillation, AdaSwitch (Peng and others, 2025) switches between teacher and student guidance based on divergence, Entropy-Aware OPD (Jin et al., 2026) adapts the loss based on teacher entropy, SelecTKD (Huang et al., 2025) lets the teacher verify student-proposed tokens via a propose-and-verify procedure and masks or down-weights rejected positions, LeaF (Guo et al., 2025) uses gradient-guided comparison with a teacher to identify and prune confounding tokens during distillation, and AdaKD (Xie et al., 2026) combines a divergence-based token selector (LATF, top-$r$% by Hellinger distance) with token-level temperature scaling (IDTS). Beyond fine-tuning, EntroDrop (Wang et al., 2025a) shows that dropping low-entropy tokens during pretraining improves generalization under multi-epoch training, providing independent evidence that high-entropy positions carry most learning signal. EDIS (Zhu et al., 2026) further demonstrates that the *temporal dynamics* of token entropy—not just its magnitude—can diagnose correct vs. incorrect reasoning trajectories. Several concurrent works also explore token-level weighting for distillation or compression (Wang et al., 2020; Tavor et al., 2026; Kim and Baek, 2026). Most closely related is AdaKD (Xie et al., 2026), whose LATF module performs hard top-$r$% selection by teacher–student Hellinger distance—a divergence-only view we ablate in Appendix [B.2](#A2.SS2). Our Q3 specification is not a relabeling of “large-divergence tokens”: it is defined by the *conjunction* of low student entropy and high disagreement, and the two axes induce different selections (Table [3](#S7.T3) vs. Table [8](#A2.T8)). The AdaKD ablation itself supports this: LATF alone yields only $+0.04$ average ROUGE-L on Qwen2-1.5B (their Table 3a), with AdaKD’s full gain coming from an orthogonal temperature-scaling module (IDTS). Our budget-matched comparison (Appendix [B.2](#A2.SS2)) reaches the same conclusion—divergence-only selection underperforms the baseline at equal budget, whereas the low-entropy + high-divergence selector remains competitive. Beyond this, our work studies all three signal sources—student entropy, teacher entropy, and teacher–student divergence—within a unified two-axis taxonomy, proves that any entropy-only rule is structurally blind to low-entropy, high-divergence tokens (Proposition [2](#Thmtheorem2)), and proposes a parameter-free Soft-OR score that explicitly recovers this region, validated on mathematical reasoning and long-horizon agentic planning. ## 3 Setup Let $T$ denote a frozen teacher and $S_{\theta}$ a trainable student over vocabulary $V$. A prompt $x\sim\mathcal{D}$ is drawn, the student generates a rollout $\mathbf{y}=(y_{1},\ldots,y_{m})\sim S_{\theta}(\cdot\mid x)$, and the teacher scores each position. The context at position $t$ is $c_{t}=(x,y_{$ Q2 $>$ Q3 $\gg$ Q4 (Proposition [1](#Thmtheorem1)); (2) entropy-only scores are structurally blind to Q3 (Proposition [2](#Thmtheorem2)); and (3) augmenting entropy with divergence restores coverage of all informative quadrants (Remark [2](#Thmremark2)). ### 5.1 Oracle Token Weight We want to identify which tokens most accelerate training. We formalize this as: what per-token weights $\{w_{t}\}$ minimize the expected loss after one gradient step? Let $g_{t}=\nabla_{\theta}\ell_{t}$ be the per-token gradient, $\bar{\mu}_{t}=\mathbb{E}[g_{t}]$, and define $\bar{\phi}_{t}=\langle\nabla L,\bar{\mu}_{t}\rangle$ and $\bar{M}_{t}=\mathbb{E}[\|g_{t}\|^{2}]$. Under $\beta$-smoothness and a token-separable approximation that neglects cross-token covariance terms (Appendix [A.1](#A1.SS1)), a weighted step $\hat{g}=\sum_{t}w_{t}g_{t}$ satisfies the surrogate bound: $$ $\mathbb{E}[L(\theta-\eta\hat{g})]-L(\theta)\;\lesssim\;\sum_{t=1}^{m}\Bigl(-\eta\,w_{t}\,\bar{\phi}_{t}+\frac{\eta^{2}\beta}{2}\,w_{t}^{2}\,\bar{M}_{t}\Bigr).$ (4) $$ ###### Proposition 1 ( Oracle token weight ) . The bound is minimized at $w_{t}^{*}=\bar{\phi}_{t}/(\eta\beta\bar{M}_{t})$, with per-token descent $\Delta_{t}^{*}=-\bar{\phi}_{t}^{2}/(2\beta\bar{M}_{t})$. Indeed, the bound is separable across tokens, so each coordinate minimizes $-\eta w_{t}\bar{\phi}_{t}+\frac{\eta^{2}\beta}{2}w_{t}^{2}\bar{M}_{t}$ independently. Differentiating gives $-\eta\bar{\phi}_{t}+\eta^{2}\beta w_{t}\bar{M}_{t}=0$, hence $w_{t}^{*}=\bar{\phi}_{t}/(\eta\beta\bar{M}_{t})$. Substituting back yields $\Delta_{t}^{*}=-\bar{\phi}_{t}^{2}/(2\beta\bar{M}_{t})$. This is an oracle quantity (it depends on the population gradient), but it gives a clear interpretation: informative tokens have gradients that align well with descent without excessive energy. Across the four quadrants: - • Q1: Large $\bar{\phi}_{t}$ (strong correction), moderate $\bar{M}_{t}$ (well-conditioned) $\Rightarrow$ largest $w_{t}^{*}$. - • Q2: Moderate $\bar{\phi}_{t}$ (mild correction) $\Rightarrow$ moderate $w_{t}^{*}$. - • Q3: Positive $\bar{\phi}_{t}$ (real corrective signal despite low entropy) $\Rightarrow$ positive $w_{t}^{*}$. - • Q4: Near-zero $\bar{\phi}_{t}$ $\Rightarrow$ negligible $w_{t}^{*}$. The qualitative ordering is $\text{Q1}>\text{Q2}>\text{Q3}\gg\text{Q4}$. ###### Remark 1 ( Quadrant ordering ) . The ordering follows from the structure of each quadrant. At high divergence, the teacher’s correction is misaligned with the student’s current prediction, so $\bar{\phi}_{t}$ is large and positive. At high entropy, the student’s distribution is diffuse, spreading gradient energy across many vocabulary entries; $\bar{M}_{t}$ is moderate, yielding a favorable $\bar{\phi}_{t}^{2}/\bar{M}_{t}$ ratio for Q1. At low entropy ($h_{t}\approx 0$), the distribution is sharply peaked, so $\bar{M}_{t}$ is small—but Q3 still has positive $\bar{\phi}_{t}$ because the teacher strongly disagrees, giving $w_{t}^{*}>0$. Q4 has both small $\bar{\phi}_{t}$ (teacher agrees) and small $\bar{M}_{t}$, so $w_{t}^{*}\approx 0$. In practice, $w_{t}^{*}$ is unavailable because it depends on population-level quantities. A natural proxy is student entropy $h_{t}$, but any such score is structurally blind to Q3: ###### Proposition 2 ( Blind spot ) . Let $\hat{w}(h_{t})=f(h_{t})$ be any non-decreasing score with $f(0)=0$ (e.g., $f(h)=h$ or $f(h)=\mathbbm{1}[h\geq\tau]$). Then Q3 tokens—which may have $w_{t}^{*}>0$—receive $\hat{w}(h_{t})\approx 0$. Entropy alone cannot distinguish “confident and correct” (Q4) from “confident and wrong” (Q3). Appendix [B.5](#A2.SS5) illustrates this concretely: Examples 1, 3, and 4 show Q3 tokens with $h_{t}<0.4$ that an entropy-only rule would discard, while Examples 2 and 5 show the contrasting high-entropy Q1 cases that entropy-based rules do capture. Since divergence $\delta_{t}$ is already computed as part of the loss, the natural fix is a score that is nonzero whenever *either* axis is active. We define the Soft-OR score with min-max normalized inputs $\hat{h}_{t},\hat{\delta}_{t}\in[0,1]$: $$ $s_{t}=\hat{h}_{t}+\hat{\delta}_{t}-\hat{h}_{t}\cdot\hat{\delta}_{t}=1-(1-\hat{h}_{t})(1-\hat{\delta}_{t}).$ (5) $$ This is parameter-free: $s_{t}$ is nonzero whenever either entropy or divergence is nonzero, without a tuning coefficient. ###### Remark 2 ( Soft-OR fixes the blind spot ) . For any Q3 token with $\hat{h}_{t}\approx 0$ and $\hat{\delta}_{t}>0$, the entropy proxy gives $\hat{w}_{0}(h_{t})\approx 0$ (Proposition [2](#Thmtheorem2)), but $s_{t}\approx\hat{\delta}_{t}>0$. Simultaneously, Q4 tokens ($\hat{h}_{t}\approx 0$, $\hat{\delta}_{t}\approx 0$) remain suppressed: $s_{t}\approx 0$. Q1 tokens retain the highest scores because both $\hat{h}_{t}$ and $\hat{\delta}_{t}$ are large ($s_{t}\approx 1$). The Soft-OR score therefore tracks the oracle ordering Q1 $>$ Q2 $>$ Q3 $\gg$ Q4 without requiring $\bar{\phi}_{t}$ or $\bar{M}_{t}$. ###### Proposition 1 ( Oracle token weight ) . ###### Remark 1 ( Quadrant ordering ) . ###### Proposition 2 ( Blind spot ) . ###### Remark 2 ( Soft-OR fixes the blind spot ) . #### Empirical predictions. Table [2](#S5.T2) maps each theoretical result to its experimental test. **Table 2: Theoretical predictions and experimental tests.** | Result | Prediction | Tested in | | --- | --- | --- | | Proposition [1](#Thmtheorem1) | Q1/Q2 carry the most signal; removing Q4 improves efficiency | Section [7.2](#S7.SS2) | | Proposition [2](#Thmtheorem2) | Entropy-only selection misses Q3 tokens | Section [7.3](#S7.SS3) | | Remark [2](#Thmremark2) | Combined score recovers Q3 and outperforms entropy-only | Section [7.4](#S7.SS4) | ## 6 Method: Type-Aware Token Selection Given a retention ratio $\rho\in(0,1]$, we retain the top-$\rho$ fraction of tokens by the Soft-OR score $s_{t}=\hat{h}_{t}+\hat{\delta}_{t}-\hat{h}_{t}\cdot\hat{\delta}_{t}$ (Equation [5](#S5.E5)): $$ $\mathcal{T}=\mathrm{TopK}\bigl(\{s_{t}\}_{t=1}^{m},\;\lfloor\rho m\rfloor\bigr).$ (6) $$ The training loss is: $$ $\mathcal{L}_{\mathrm{TIP}}=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}D_{\mathrm{KL}}\!\left(P_{S}(\cdot\mid c_{t})\,\|\,P_{T}(\cdot\mid c_{t})\right).$ (7) $$ Setting $\hat{\delta}_{t}=0$ recovers entropy-only selection; including $\hat{\delta}_{t}$ additionally promotes Q3 tokens. The score is parameter-free and both $h_{t}$ and $\delta_{t}$ are already computed during standard distillation, so the only extra cost is a min-max normalization and the top-$k$ sort—$O(m\log m)$ per rollout, negligible compared with forward and backward passes. ## 7 Experiments We now validate each prediction of the taxonomy. Table [2](#S5.T2) maps each theoretical result to its experimental test; we proceed from the strongest signal (high-entropy tokens, Section [7.2](#S7.SS2)) to the blind spot (Q3, Section [7.3](#S7.SS3)) to the combined score (Section [7.4](#S7.SS4)). ### 7.1 Experimental Setup #### Models. Three teacher–student pairs across three model families for mathematical reasoning, plus one pair for agentic planning: - • Qwen3 Small: Qwen3-8B (GRPO) $\to$ Qwen3-4B (Yang et al., 2025) - • Llama: Llama-3.3-70B-Instruct $\to$ Llama-3.1-8B-Instruct (Grattafiori et al., 2024) - • Qwen2.5: Qwen2.5-14B-Instruct-thinking $\to$ Qwen2.5-1.5B-Instruct (Qwen et al., 2025) (${\sim}9\times$ capacity gap, reasoning teacher) - • Qwen3 Agentic: Qwen3-{14B, 32B} $\to$ Qwen3-1.7B (Yang et al., 2025) (all with thinking enabled, trained on agentic planning data) #### Data and evaluation. For mathematical reasoning, training prompts are from DAPO (Yu et al., 2025), with evaluation on MATH-500 (Hendrycks et al., 2021) (500 problems) and AIME 2024/2025 (30 each). For agentic planning, training data is from DeepPlanning (Zhang et al., 2026), a benchmark featuring multi-day travel and multi-product shopping tasks that require proactive information acquisition, local constrained reasoning, and global constrained optimization; the Qwen3 Agentic pair is trained for 15 epochs. All models are trained with AdamW, a cosine schedule, and reverse KL on student-generated rollouts (lr $=1\times 10^{-6}$ for Qwen3 and Qwen2.5; lr $=3\times 10^{-7}$ for Llama). ### 7.2 High-Entropy Tokens (Q1/Q2) We begin with the simplest test of the taxonomy: if Q1/Q2 tokens dominate learning signal while Q4 tokens are negligible, then selecting by student entropy should preserve most of the benefit of OPD. Table [3](#S7.T3) (and Figure [3](#A2.F3) in the Appendix) confirm this prediction. Across all three model pairs, retaining 50% of tokens with entropy-based sampling matches or outperforms the all-token baseline on most benchmarks, while memory drops substantially. For Qwen3 Small, MATH improves from 76.7 to 78.6; for Llama, from 71.0 to 74.0. This indicates that many low-entropy tokens are effectively solved (Q4) and mainly dilute the gradient; Appendix [B.5](#A2.SS5) gives token-level intuition for the contrasting high-entropy cases that do carry corrective signal (Examples 2 and 5). At the same time, entropy alone is incomplete. As the retention ratio becomes more aggressive, performance often drops below the full-token baseline, suggesting that useful signal remains in the discarded low-entropy region as Proposition [2](#Thmtheorem2) indicates. We test this hypothesis directly in the next section by isolating the low-entropy, high-divergence tokens that entropy-only selection discards. **Table 3: Entropy sampling across model pairs. Accuracy (%, mean@16 $\pm$ std). Sampling selects tokens with probability $p_{t}\propto h_{t}$. Bold marks the best per benchmark.** | Model pair | Benchmark | 100% | 50% | 20% | 10% | | --- | --- | --- | --- | --- | --- | | Qwen3-8B (GRPO) $\to$ 4B | MATH-500 | 76.7 $\pm$ 0.7 | 78.6 $\pm$ 0.6 | 74.1 $\pm$ 0.9 | 70.8 $\pm$ 1.0 | | AIME’24 | 21.9 $\pm$ 1.2 | 23.8 $\pm$ 1.3 | 22.5 $\pm$ 1.1 | 21.1 $\pm$ 1.2 | | | AIME’25 | 19.4 $\pm$ 1.1 | 20.7 $\pm$ 1.3 | 21.5 $\pm$ 1.2 | 19.2 $\pm$ 1.0 | | | Llama-70B $\to$ 8B | MATH-500 | 71.0 $\pm$ 0.7 | 74.0 $\pm$ 0.8 | 73.6 $\pm$ 0.7 | 72.5 $\pm$ 0.8 | | AIME’24 | 21.5 $\pm$ 1.1 | 25.3 $\pm$ 1.5 | 18.8 $\pm$ 1.3 | 20.6 $\pm$ 1.4 | | | AIME’25 | 4.9 $\pm$ 0.9 | 7.5 $\pm$ 1.1 | 10.0 $\pm$ 1.2 | 9.3 $\pm$ 1.0 | | | Qwen2.5-14B $\to$ 1.5B | MATH-500 | 55.1 $\pm$ 0.9 | 54.9 $\pm$ 0.9 | 54.0 $\pm$ 0.9 | 55.1 $\pm$ 0.9 | | AIME’24 | 2.4 $\pm$ 0.7 | 3.3 $\pm$ 1.4 | 4.6 $\pm$ 1.3 | 2.7 $\pm$ 1.1 | | | AIME’25 | 2.1 $\pm$ 0.9 | 1.0 $\pm$ 0.5 | 1.0 $\pm$ 0.6 | 1.3 $\pm$ 0.8 | | | Peak Mem (GB, Qwen3) | 72.0 | 38.1 | 35.5 | 35.3 | | | Peak Mem (GB, Llama) | 41.6 | 32.1 | 31.8 | 31.9 | | | Peak Mem (GB, Qwen2.5) | 35.8 | 19.7 | 15.0 | 14.9 | | ### 7.3 Overconfident Tokens (Q3) We next test the blind-spot prediction directly by constructing the opposite of entropy-based selection: a selector that prioritizes tokens with *low entropy but high teacher–student divergence*—targeting the Q3 regime in the taxonomy. #### Q3 selection procedure. We isolate overconfident tokens using a confidence-weighted divergence score: - 1. Compute per-token forward KL: $\delta_{t}^{\mathrm{fwd}}=D_{\mathrm{KL}}(P_{T}(\cdot\mid c_{t})\|P_{S}(\cdot\mid c_{t}))$. - 2. Compute per-token student entropy $h_{t}$ (Equation [2](#S3.E2)) and min-max normalize to $[0,1]$: $\hat{h}_{t}=(h_{t}-h_{\min})/(h_{\max}-h_{\min})$. - 3. Define confidence as $\mathrm{conf}_{t}=1-\hat{h}_{t}$ (low entropy $\Rightarrow$ high confidence). - 4. Compute the Q3 score: $w_{t}^{\mathrm{Q3}}=\delta_{t}^{\mathrm{fwd}}\cdot\mathrm{conf}_{t}$. Tokens with high $w_{t}^{\mathrm{Q3}}$ are precisely the positions where the student is highly confident while the teacher strongly disagrees. Conceptually, the taxonomy defines Q3 using the reverse-KL disagreement axis in Equation [3](#S3.E3); operationally, this experiment uses the forward-KL score above as a sharper detector of overconfident missing-mass errors. We use forward KL rather than the reverse KL of the taxonomy axis because forward KL penalizes *missing mass*—teacher-preferred continuations to which the student assigns near-zero probability—more heavily, making it a sharper detector of overconfidence. In practice, the two divergences are strongly correlated on Q3 tokens, so the choice does not materially change which tokens are selected. **Table 4: Training on Q3 (overconfident) tokens only. Accuracy (%, mean@16 $\pm$ std) when training exclusively on low-entropy, high-divergence tokens. Q3-only training with $<$10% of all tokens can nearly match the all-token baseline across model pairs.** | Model pair | Benchmark | Baseline (100%) | Q3 20% | Q3 10% | | --- | --- | --- | --- | --- | | Qwen3-8B $\to$ 4B | MATH-500 | 76.7 $\pm$ 0.7 | 75.7 $\pm$ 0.8 | 76.1 $\pm$ 0.9 | | AIME’24 | 21.9 $\pm$ 1.2 | 20.2 $\pm$ 1.1 | 21.5 $\pm$ 1.3 | | | AIME’25 | 19.4 $\pm$ 1.1 | 19.2 $\pm$ 1.1 | 17.1 $\pm$ 1.0 | | | Llama-70B $\to$ 8B | MATH-500 | 71.0 $\pm$ 0.7 | 71.8 $\pm$ 1.0 | 70.4 $\pm$ 1.1 | | AIME’24 | 21.5 $\pm$ 1.1 | 20.2 $\pm$ 1.2 | 20.8 $\pm$ 1.1 | | | AIME’25 | 4.9 $\pm$ 0.9 | 4.5 $\pm$ 0.7 | 4.1 $\pm$ 0.7 | | | Qwen2.5-14B $\to$ 1.5B | MATH-500 | 55.1 $\pm$ 0.9 | 54.6 $\pm$ 0.9 | 54.1 $\pm$ 0.9 | | AIME’24 | 2.4 $\pm$ 0.7 | 2.5 $\pm$ 0.9 | 3.3 $\pm$ 1.4 | | | AIME’25 | 2.1 $\pm$ 0.9 | 3.3 $\pm$ 1.4 | 0.8 $\pm$ 0.4 | | | Peak Mem (GB, Qwen3) | 72.0 | 36.4 | 36.2 | | | Peak Mem (GB, Llama) | 41.6 | 32.5 | 32.4 | | | Peak Mem (GB, Qwen2.5) | 35.8 | 19.7 | 19.6 | | Table [4](#S7.T4) confirms that the Q3 region carries real corrective signal. For Qwen3, training on only 5.7K overconfident tokens ($<$10% of all tokens) reaches 76.1 on MATH, versus 76.7 for the full-token baseline. For Qwen2.5, Q3-only training matches or exceeds the baseline on several benchmarks. These results validate the taxonomy’s prediction that Q3 tokens are informative despite having near-zero entropy. Appendix [B.5](#A2.SS5) provides concrete examples: a student that repeats a generic variable instead of substituting a concrete value (Ex. 1), an arithmetic computation error (Ex. 3), and a confident variable-level misstep in a derivation (Ex. 4)—all near-zero entropy, all strongly corrected by the teacher. #### Q3 is not just “large divergence”. Q3 tokens are by construction high-$\delta$, but the selector is structurally different from a pure divergence view. Ranking by $\delta_{t}$ alone at a budget matched to Q3-only underperforms the all-token baseline and only catches up when given $5\times$ more tokens (Appendix [B.2](#A2.SS2), Table [8](#A2.T8)); it is the low-entropy conjunct that makes selection performant under tight budgets. Conversely, high student entropy is not a proxy for high divergence—the two axes induce different selections and different performance curves (Table [3](#S7.T3) vs. Table [8](#A2.T8)), and entropy-only rules are provably blind to Q3 (Proposition [2](#Thmtheorem2)). A related concern is that high-$\delta$ tokens already yield the largest per-token gradients and thus should dominate the update; but in full-token training the aggregate update sums over all positions—mostly low-/moderate-$\delta$ ones—and the operationally relevant question is which tokens are worth *keeping* under a budget, which Table [8](#A2.T8) answers in favor of the low-entropy + high-divergence selector. ### 7.4 Type-Aware Selection (TIP) The prediction of Remark [2](#Thmremark2) is that combining entropy with divergence should outperform entropy-only selection by recovering Q3 without sacrificing Q1/Q2. Table [5](#S7.T5) and Figure [1](#S1.F1) present the comparison, and Appendix [B.5](#A2.SS5) provides concrete token-level examples of the Q1/Q3 behaviors that the combined score is designed to retain. **Table 5: Main results: Baseline vs. Entropy-only vs. Soft-OR. Accuracy (%, mean@16 $\pm$ std). Soft-OR uses $s_{t}=\hat{h}_{t}+\hat{\delta}_{t}-\hat{h}_{t}\cdot\hat{\delta}_{t}$ (Eq. [5](#S5.E5)) with Top-K selection. Bold marks the best per benchmark.** | | | Baseline | Entropy-only | Soft-OR | | | | --- | --- | --- | --- | --- | --- | --- | | Model pair | Benchmark | 100% | 50% | 20% | 50% | 20% | | Qwen3-8B $\to$ 4B | MATH-500 | 76.7 $\pm$ 0.7 | 78.6 $\pm$ 0.6 | 74.1 $\pm$ 0.9 | 79.1 $\pm$ 0.8 | 77.6 $\pm$ 0.7 | | AIME’24 | 21.9 $\pm$ 1.2 | 23.8 $\pm$ 1.3 | 22.5 $\pm$ 1.1 | 25.7 $\pm$ 1.4 | 24.5 $\pm$ 1.2 | | | AIME’25 | 19.4 $\pm$ 1.1 | 20.7 $\pm$ 1.3 | 21.5 $\pm$ 1.2 | 21.9 $\pm$ 1.2 | 23.2 $\pm$ 1.2 | | | Llama-70B $\to$ 8B | MATH-500 | 71.0 $\pm$ 0.7 | 74.0 $\pm$ 0.8 | 73.6 $\pm$ 0.7 | 74.7 $\pm$ 1.0 | 74.2 $\pm$ 0.7 | | AIME’24 | 21.5 $\pm$ 1.1 | 25.3 $\pm$ 1.5 | 18.8 $\pm$ 1.3 | 26.0 $\pm$ 1.4 | 21.0 $\pm$ 1.5 | | | AIME’25 | 4.9 $\pm$ 0.9 | 7.5 $\pm$ 1.1 | 10.0 $\pm$ 1.2 | 11.5 $\pm$ 1.1 | 10.9 $\pm$ 1.4 | | | Qwen2.5-14B $\to$ 1.5B | MATH-500 | 55.1 $\pm$ 0.9 | 54.9 $\pm$ 0.9 | 54.0 $\pm$ 0.9 | 56.2 $\pm$ 1.2 | 55.8 $\pm$ 0.9 | | AIME’24 | 2.4 $\pm$ 0.7 | 3.3 $\pm$ 1.4 | 4.6 $\pm$ 1.3 | 3.8 $\pm$ 1.2 | 5.0 $\pm$ 1.3 | | | AIME’25 | 2.1 $\pm$ 0.9 | 1.0 $\pm$ 0.5 | 1.0 $\pm$ 0.6 | 1.5 $\pm$ 0.7 | 1.8 $\pm$ 0.6 | | #### Top vs. bottom tokens by Soft-OR score. A natural sanity check is the complementary experiment: instead of training on the top 50% tokens by Soft-OR score, train on the *bottom* 50%. If the taxonomy is correct, these tokens should be predominantly Q4 (solved) and carry negligible learning signal. **Table 6: Top 50% vs. bottom 50% by Soft-OR score. Accuracy (%, mean@16 $\pm$ std). “Top” trains on the highest-scoring half by $s_{t}$; “Bot.” trains on the lowest-scoring half. The bottom tokens carry substantially less signal.** | | Qwen3-8B $\to$ 4B | Llama-70B $\to$ 8B | Qwen2.5-14B $\to$ 1.5B | | | | | --- | --- | --- | --- | --- | --- | --- | | Benchmark | Top 50% | Bot. 50% | Top 50% | Bot. 50% | Top 50% | Bot. 50% | | MATH-500 | 79.1 $\pm$ 0.8 | 72.3 $\pm$ 0.9 | 74.7 $\pm$ 1.0 | 67.4 $\pm$ 1.1 | 56.2 $\pm$ 1.2 | 50.3 $\pm$ 1.0 | | AIME’24 | 25.7 $\pm$ 1.4 | 15.5 $\pm$ 1.3 | 26.0 $\pm$ 1.4 | 17.2 $\pm$ 1.3 | 3.8 $\pm$ 1.2 | 1.5 $\pm$ 0.7 | | AIME’25 | 21.9 $\pm$ 1.2 | 12.8 $\pm$ 1.2 | 11.5 $\pm$ 1.1 | 2.9 $\pm$ 0.8 | 1.5 $\pm$ 0.7 | 0.8 $\pm$ 0.5 | #### Teacher entropy is uninformative. Teacher entropy is nearly constant across positions and therefore provides little discriminative signal for token selection; details are deferred to Appendix [B.1](#A2.SS1). The useful axes are the *student’s* state ($h_{t}$) and the student–teacher gap ($\delta_{t}$), not the teacher’s uncertainty. ### 7.5 Beyond Mathematical Reasoning: Agentic Planning The preceding experiments focus on mathematical reasoning. To test whether TIP generalizes beyond this domain, we apply it to the DeepPlanning benchmark (Zhang et al., 2026) (Section [7.1](#S7.SS1)). #### Setup. Using the Qwen3 Agentic pair described in Section [7.1](#S7.SS1), we train on 80% of the DeepPlanning Travel Planning tasks and evaluate on the remaining 20%. We report the average accuracy (Avg@16) over 16 samples. Following DeepPlanning, we score each plan by the fraction of personalized hard constraints satisfied; these scores are lower than commonsense scores because personalized requirements (e.g., budget limits, dietary restrictions) are more demanding. **Table 7: Agentic planning on DeepPlanning (Qwen3-1.7B student, thinking-enabled, Avg@16 %). *Left*: reference and entropy-only methods. *Right*: Q3-only and Soft-OR. Q3-only 20% *surpasses* full-token OPD; Soft-OR matches or exceeds entropy-only.** | Method | Teacher 14B | Teacher 32B | | --- | --- | --- | | OPD, all tokens (100%) | $11.7{\pm}0.07$ | $12.8{\pm}0.07$ | | + Entropy-only 50% | $12.1{\pm}0.06$ | $13.1{\pm}0.07$ | | + Entropy-only 20% | $11.6{\pm}0.07$ | $12.7{\pm}0.06$ | Table [7](#S7.T7) confirms that TIP generalizes to a fundamentally different domain, with entropy-based selection at 50% retention matching or exceeding full-token OPD (12.1 vs. 11.7 for 14B; 13.1 vs. 12.8 for 32B). The most striking finding is Q3: training on only 20% of overconfident tokens *surpasses* full-token OPD for both teachers (12.6 vs. 11.7; 13.6 vs. 12.8), confirming that entropy discards exactly the tokens with the densest corrective signal. This aligns with the structure of agentic tasks: a single wrong but confident commitment—booking a closed venue, violating a budget constraint—can invalidate an entire plan, making Q3 corrections especially concentrated. Appendix [B.3](#A2.SS3) provides Best@16 results, confirming the same pattern. ## 8 Discussion and Conclusion TIP establishes that token importance in OPD is governed by two axes—student entropy and teacher–student divergence—and that both are necessary. All three theoretical predictions (Propositions [1](#Thmtheorem1)–[2](#Thmtheorem2), Remark [2](#Thmremark2)) are supported across three model families and two task domains (Tables [3](#S7.T3)–[7](#S7.T7)), with the clearest result being the Q3 blind spot: entropy-only rules provably cannot distinguish “confident and correct” from “confident and wrong,” yet fewer than 10% of tokens selected precisely for overconfidence nearly matches full-training performance. This picture is sharpest on DeepPlanning, where Q3-only training with 20% of tokens *surpasses* full OPD (12.6 vs. 11.7 Avg@16 with the 14B teacher), suggesting that the value of catching overconfident errors grows when more downstream computation depends on a single committed step; the effect is weaker in mathematical reasoning, where mistakes are often more locally contained. More broadly, the two-axis framing applies to other settings where on-policy token-level supervision is used, including RLHF, process reward fine-tuning, and speculative decoding. #### Limitations. (1) Q3 detection requires teacher output distributions, though $\delta_{t}$ is already part of the standard OPD loss. (2) Soft-OR uses per-batch min-max normalization, which may be sensitive to outlier tokens in a batch; alternatives such as running-average normalization remain to be studied. (3) All experiments use reverse KL supervision; whether the same quadrant ordering holds under forward KL or JSD is an open question. ## References - R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), External Links: [Link](https://arxiv.org/abs/2306.13649) Cited by: [§1](#S1.p1.1), [§2](#S2.SS0.SSS0.Px2.p1.1). - Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. ICML. Cited by: [§2](#S2.SS0.SSS0.Px1.p1.1). - G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, Z. Liu, H. Peng, L. Bai, W. Ouyang, Y. Cheng, B. Zhou, and N. Ding (2025) The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al. (2024) The llama 3 herd of models. External Links: 2407.21783, [Link](https://arxiv.org/abs/2407.21783) Cited by: [2nd item](#S7.I1.i2.p1.1). - Y. Gu, H. Zhou, F. Meng, J. Zhou, and M. Huang (2024) MiniPLM: knowledge distillation for pre-training language models. arXiv preprint arXiv:2410.17215. Cited by: [§2](#S2.SS0.SSS0.Px2.p1.1). - Y. Gu, H. Zhou, F. Meng, J. Zhou, and M. Huang (2025) MiniPLM: knowledge distillation for pre-training language models. External Links: [Link](https://openreview.net/forum?id=tJHDw8XfeC) Cited by: [§1](#S1.p1.1), [§2](#S2.SS0.SSS0.Px2.p1.1). - Y. Guo, W. Yang, Z. Sun, N. Ding, Z. Liu, and Y. Lin (2025) Learning to focus: causal attention distillation via gradient-guided token pruning. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the MATH dataset. NeurIPS. Cited by: [§7.1](#S7.SS1.SSS0.Px2.p1.2). - G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: [§1](#S1.p1.1). - H. Huang, J. Song, Y. Zhang, and P. Ren (2025) SelecTKD: selective token-weighted knowledge distillation for LLMs. arXiv preprint arXiv:2510.24021. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - Y. Jiang, C. Chan, M. Chen, and W. Wang (2023) Lion: adversarial distillation of proprietary large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: [Link](https://arxiv.org/abs/2305.12870) Cited by: [§2](#S2.SS0.SSS0.Px3.p1.1). - W. Jin, T. Min, Y. Yang, S. R. Kadhe, Y. Zhou, D. Wei, N. Baracaldo, and K. Lee (2026) Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - A. Katharopoulos and F. Fleuret (2018) Not all samples are created equal: deep learning with importance sampling. In International Conference on Machine Learning (ICML), Cited by: [§2](#S2.SS0.SSS0.Px1.p1.1). - M. Kim and S. J. Baek (2026) Explain in your own words: improving reasoning via token-selective dual knowledge distillation. arXiv preprint arXiv:2603.13260. Cited by: [§2](#S2.SS0.SSS0.Px4.p2.2). - Y. Kim and A. M. Rush (2016) Sequence-level knowledge distillation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, External Links: [Link](https://arxiv.org/abs/1606.07947) Cited by: [§2](#S2.SS0.SSS0.Px2.p1.1). - M. P. Kumar, B. Packer, and D. Koller (2010) Self-paced learning for latent variable models. NeurIPS. Cited by: [§2](#S2.SS0.SSS0.Px1.p1.1). - Z. Peng et al. (2025) AdaSwitch: adaptive switching between teacher and student for on-policy distillation. arXiv preprint. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2025) Qwen2.5 technical report. External Links: 2412.15115, [Link](https://arxiv.org/abs/2412.15115) Cited by: [3rd item](#S7.I1.i3.p1.2). - M. Ren, W. Zeng, B. Yang, and R. Urtasun (2018) Learning to reweight examples for robust deep learning. In International Conference on Machine Learning (ICML), Cited by: [§2](#S2.SS0.SSS0.Px1.p1.1). - H. Sang, Y. Xu, Z. Zhou, R. He, Z. Wang, and J. Sun (2026) CRISP: compressed reasoning via iterative self-policy distillation. arXiv preprint arXiv:2603.05433. Cited by: [§2](#S2.SS0.SSS0.Px2.p1.1). - A. Tavor, I. Ebenspanger, N. Cnaan, and M. Geva (2026) Rethinking selective knowledge distillation. arXiv preprint arXiv:2602.01395. Cited by: [§2](#S2.SS0.SSS0.Px4.p2.2). - J. Wang, Y. Hu, Y. Gao, H. Wang, S. Wang, H. Lu, J. Mao, W. X. Zhao, J. Li, and J. Wen (2025a) Entropy-guided token dropout: training autoregressive language models with limited domain data. arXiv preprint arXiv:2512.23422. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, Y. Liu, A. Yang, A. Zhao, Y. Yue, S. Song, B. Yu, G. Huang, and J. Lin (2025b) Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning. Advances in Neural Information Processing Systems 38. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou (2020) MiniLM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems (NeurIPS), External Links: [Link](https://arxiv.org/abs/2002.10957) Cited by: [§2](#S2.SS0.SSS0.Px4.p2.2). - J. Wu, Y. George, J. Ye, Y. Wu, D. F. Schmidt, and J. Cai (2025) SPINE: token-selective test-time reinforcement learning with entropy-band regularization. arXiv preprint arXiv:2511.17938. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - C. Xiao, J. Cai, W. Zhao, G. Zeng, B. Lin, J. Zhou, Z. Zheng, X. Han, Z. Liu, and M. Sun (2025) Densing law of LLMs. Nature Machine Intelligence. Cited by: [§1](#S1.p1.1). - X. Xie, Z. Xue, J. Wu, J. Li, Y. Wang, X. Hu, Y. Liu, and J. Zhang (2026) LLM-oriented token-adaptive knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, pp. 34070–34078. Cited by: [§B.2](#A2.SS2.SSS0.Px5.p2.5), [§B.2](#A2.SS2.p1.4), [Table 8](#A2.T8), [Table 8](#A2.T8.8.4.4), [§2](#S2.SS0.SSS0.Px4.p1.1), [§2](#S2.SS0.SSS0.Px4.p2.2). - Y. Xu, H. Sang, Z. Zhou, R. He, and Z. Wang (2026a) Overconfident errors need stronger correction: asymmetric confidence penalties for reinforcement learning. arXiv preprint arXiv:2602.21420. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). - Y. Xu, H. Sang, Z. Zhou, R. He, and Z. Wang (2026b) Paced: distillation and self-distillation at the frontier of student competence. arXiv preprint. Cited by: [§2](#S2.SS0.SSS0.Px3.p1.1). - Y. Xu, X. Han, Z. Yang, S. Wang, Q. Zhu, Z. Liu, W. Liu, and W. Che (2024) OneBit: towards extremely low-bit large language models. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: [§2](#S2.SS0.SSS0.Px2.p1.1). - A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: [1st item](#S7.I1.i1.p1.1), [4th item](#S7.I1.i4.p1.1). - Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source llm reinforcement learning system. arXiv preprint arXiv:2503.14476. Cited by: [§7.1](#S7.SS1.SSS0.Px2.p1.2). - Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang (2025) Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?. In Advances in Neural Information Processing Systems, Vol. 38. Note: Oral Cited by: [§2](#S2.SS0.SSS0.Px2.p1.1). - Y. Zhang, S. Jiang, R. Li, J. Tu, Y. Su, L. Deng, X. Guo, C. Lv, and J. Lin (2026) DeepPlanning: benchmarking long-horizon agentic planning with verifiable constraints. arXiv preprint arXiv:2601.18137. Cited by: [item 3](#S1.I1.i3.p1.1), [§7.1](#S7.SS1.SSS0.Px2.p1.2), [§7.5](#S7.SS5.p1.1). - C. Zhu, S. Wu, X. Zeng, Z. Xu, Z. Kang, Y. Guo, Y. Lu, J. Huang, G. Zhou, et al. (2026) EDIS: diagnosing LLM reasoning via entropy dynamics. arXiv preprint arXiv:2602.01288. Cited by: [§2](#S2.SS0.SSS0.Px4.p1.1). ## Appendix A Supplementary Theory ### A.1 Derivation of the Descent Bound ###### Assumption 1 ( Smoothness ) . $L(\theta^{\prime})\leq L(\theta)+\langle\nabla L(\theta),\theta^{\prime}-\theta\rangle+\frac{\beta}{2}\|\theta^{\prime}-\theta\|^{2}$. ###### Assumption 2 ( Token-separable approximation ) . For tractability, we neglect off-diagonal gradient interactions across token positions. Concretely, for $t\neq s$ we treat the centered cross-token covariance $$ $\mathbb{E}[(g_{t}-\bar{\mu}_{t})(g_{s}-\bar{\mu}_{s})^{\top}]$ $$ as lower-order, so that the quadratic term admits a token-separable approximation. ###### Derivation. Expand $L(\theta-\eta\hat{g})$ via smoothness where $\hat{g}=\sum_{t}w_{t}g_{t}$: $$ $L(\theta-\eta\hat{g})\leq L(\theta)-\eta\langle\nabla L,\hat{g}\rangle+\frac{\eta^{2}\beta}{2}\|\hat{g}\|^{2}.$ (8) $$ Using linearity of expectation and $\hat{g}=\sum_{t}w_{t}g_{t}$, $$ $\mathbb{E}[-\eta\langle\nabla L,\hat{g}\rangle]=-\eta\sum_{t}w_{t}\,\mathbb{E}[\langle\nabla L,g_{t}\rangle]=-\eta\sum_{t}w_{t}\,\langle\nabla L,\bar{\mu}_{t}\rangle=-\eta\sum_{t}w_{t}\,\bar{\phi}_{t}.$ (9) $$ For the quadratic term, $$ $\mathbb{E}[\|\hat{g}\|^{2}]=\mathbb{E}\Bigl[\Bigl\|\sum_{t}w_{t}g_{t}\Bigr\|^{2}\Bigr]=\sum_{t}w_{t}^{2}\mathbb{E}[\|g_{t}\|^{2}]+\sum_{t\neq s}w_{t}w_{s}\,\mathbb{E}[\langle g_{t},g_{s}\rangle].$ (10) $$ Write $g_{t}=\bar{\mu}_{t}+(g_{t}-\bar{\mu}_{t})$. Under Assumption [2](#Thmassumption2), we drop the off-diagonal covariance contribution and treat the remaining mean interaction terms as lower-order; absorbing these into the $\lesssim$ notation gives the token-separable approximation $$ $\mathbb{E}[\|\hat{g}\|^{2}]\lesssim\sum_{t}w_{t}^{2}\bar{M}_{t}.$ (11) $$ Combining the two displays gives $$ $\mathbb{E}[L(\theta-\eta\hat{g})]-L(\theta)\lesssim\sum_{t}\Bigl(-\eta w_{t}\bar{\phi}_{t}+\frac{\eta^{2}\beta}{2}w_{t}^{2}\bar{M}_{t}\Bigr).$ (12) $$ The right-hand side is separable in $t$, so minimizing each term gives $$ $\frac{\partial}{\partial w_{t}}\Bigl(-\eta w_{t}\bar{\phi}_{t}+\frac{\eta^{2}\beta}{2}w_{t}^{2}\bar{M}_{t}\Bigr)=-\eta\bar{\phi}_{t}+\eta^{2}\beta w_{t}\bar{M}_{t}=0,$ (13) $$ which yields $w_{t}^{*}=\bar{\phi}_{t}/(\eta\beta\bar{M}_{t})$. ∎ ###### Assumption 1 ( Smoothness ) . ###### Assumption 2 ( Token-separable approximation ) . ###### Derivation. ### A.2 Entropy-Weighted Sampling: Coverage and Variance Sampling tokens with $p_{t}\propto h_{t}$ and using the importance-weighted estimator $\hat{g}_{\mathrm{IS}}=\frac{1}{m}\sum_{t\in S}g_{t}/p_{t}$ is unbiased whenever $p_{t}>0$, with variance $$ $\mathrm{Var}(\hat{g}_{\mathrm{IS}})=\frac{1}{m^{2}}\sum_{t=1}^{m}\frac{1-p_{t}}{p_{t}}\,\mathbb{E}[\|g_{t}\|^{2}].$ (14) $$ This makes the tradeoff transparent: entropy sampling preserves nonzero Q3 coverage, but the variance cost grows as $1/p_{t}$ for low-entropy tokens. ### A.3 Why Adding Divergence Improves the Ranking Under entropy-only scoring $\hat{w}_{0}=h_{t}$, Q3 and Q4 are both pushed to the low-score end. The Soft-OR score $s_{t}=\hat{h}_{t}+\hat{\delta}_{t}-\hat{h}_{t}\cdot\hat{\delta}_{t}$ separates them: Q3 receives positive score from divergence ($s_{t}\approx\hat{\delta}_{t}$) while Q4 remains near zero (both axes small). Because the product term $\hat{h}_{t}\cdot\hat{\delta}_{t}$ prevents double-counting, the high-entropy ranking is preserved and the score better tracks the oracle ordering without any tuning parameter. ## Appendix B Supplementary Experiments Figure: Figure 3: Entropy sampling across retention ratios. Accuracy (mean@16) on three benchmarks as a function of retention ratio. Retaining 50% of tokens with entropy-based sampling matches or outperforms the all-token baseline across model pairs. At very low retention, entropy-only selection begins to plateau or degrade. Refer to caption: 2604.14084v2/x3.png ### B.1 Teacher Entropy Is Uninformative Teacher entropy is near-zero everywhere (mean $0.031$, std $0.055$ for Qwen3; mean $0.067$, std $0.164$ for Llama; median token probability ${\geq}0.79$), so any scheme that conditions on teacher entropy—whether for token selection, loss weighting, or sampling—receives an almost constant input and therefore adds little discriminative information. In practice, the useful axes are the *student’s* state ($h_{t}$) and the student–teacher gap ($\delta_{t}$), not the teacher’s uncertainty. ### B.2 Isolating the Entropy Axis: Comparison with Divergence-Only Selection Section [7.3](#S7.SS3) shows that low-entropy, high-divergence (Q3) tokens carry dense corrective signal. A natural follow-up question is whether the *low-entropy* conjunct matters at all, or whether ranking by divergence $\delta_{t}$ alone—ignoring student entropy—would suffice. This is also the most direct comparison to the LATF module of AdaKD [Xie et al., 2026], which selects the top-$r$% of tokens by teacher–student Hellinger distance: our *Div-only* top-$k$ selector is the same idea instantiated with reverse KL. We deliberately give Div-only a much larger budget (top 50%) than Q3-only (top 10%): if the entropy axis were redundant, Div-only with $5\times$ more tokens should clearly dominate. #### External reference point. The published AdaKD ablation provides an independent data point for divergence-only selection. On Qwen2-7B$\to$1.5B (instruction-following, RKD baseline), their Table 3a reports that adding LATF alone changes ROUGE-L average from $37.03$ to $37.07$ ($+0.04$); the full +1.98 improvement of AdaKD comes from their orthogonal temperature-scaling module (IDTS), not from the token selector. This is consistent with our Proposition [2](#Thmtheorem2) and motivates the present ablation in our setting. #### Setup. On Qwen3-8B (GRPO) $\to$ 4B with the standard mathematical-reasoning protocol (Section [7.1](#S7.SS1)), we compare: - • Baseline: all tokens (100%). - • Div-only 50% (LATF-style hard top-$k$): top 50% of tokens by $\delta_{t}$ alone—no entropy term. - • Div-only 10%: top 10% of tokens by $\delta_{t}$ alone—budget-matched to Q3-only. - • Q3-only 10%: top 10% of tokens by $\delta_{t}^{\mathrm{fwd}}\cdot(1-\hat{h}_{t})$ (Section [7.3](#S7.SS3))—explicitly requires both low entropy and high divergence. This design isolates two questions: (i) at $5\times$ fewer tokens, does adding the low-entropy filter beat divergence alone (Div-only 50% vs Q3-only 10%)? and (ii) at *equal* budget, does the low-entropy filter beat pure divergence ranking (Div-only 10% vs Q3-only 10%)? **Table 8: Isolating the entropy axis: Div-only vs Q3-only on Qwen3-8B (GRPO) $\to$ 4B (mean@16 %, $\pm$ std). Div-only is a hard top-$k$ instantiation of LATF-style divergence selection [Xie et al., 2026]. We compare Q3-only at 10% against Div-only at both $5\times$ the budget (50%) and matched budget (10%). Bold marks the best per benchmark.** | | | Baseline | Div-only | Div-only | Q3-only | | --- | --- | --- | --- | --- | --- | | | Benchmark | 100% | 50% | 10% | 10% | | *Accuracy* | MATH-500 | 76.7 $\pm$ 0.7 | 76.3 $\pm$ 0.8 | 74.3 $\pm$ 0.8 | 76.1 $\pm$ 0.9 | | AIME’24 | 21.9 $\pm$ 1.2 | 21.5 $\pm$ 1.1 | 19.2 $\pm$ 1.2 | 21.5 $\pm$ 1.3 | | | AIME’25 | 19.4 $\pm$ 1.1 | 21.1 $\pm$ 1.2 | 15.3 $\pm$ 1.1 | 17.1 $\pm$ 1.0 | | #### Results. Table [8](#A2.T8) resolves both questions and sharpens what the entropy axis contributes. *(i) Equal budget (Div-only 10% vs Q3-only 10%).* At the same 10% retention, Q3-only beats Div-only on every benchmark: MATH-500 $76.1$ vs $74.3$ ($+1.8$), AIME’24 $21.5$ vs $19.2$ ($+2.3$), AIME’25 $17.1$ vs $15.3$ ($+1.8$). Div-only at 10% drops well below the all-token baseline ($-2.4,-2.7,-4.1$), while Q3-only stays within $0.6$ of the baseline on MATH/AIME’24. With the divergence ranking and budget held fixed, simply requiring tokens to also be *low-entropy*—i.e., re-weighting by $(1-\hat{h}_{t})$—is what closes the gap to the baseline. *(ii) $5\times$ budget gap (Div-only 50% vs Q3-only 10%).* Once Div-only is given $5\times$ the tokens (50%), it largely catches up: MATH-500 $76.3$ vs Q3-only’s $76.1$ ($+0.2$), AIME’24 a tie at $21.5$, AIME’25 $21.1$ vs $17.1$ ($+4.0$ in Div-only’s favor). This is consistent with the fact that, as $r$ grows, the high-divergence tail necessarily *includes* most Q3 tokens as a side effect. The headline is per-token efficiency: Q3-only matches Div-only on two of three benchmarks while using $5\times$ fewer tokens. #### What this comparison does and does not say. We are not claiming that Q3 is numerically disjoint from a “large-divergence” view. Q3 is, by construction, a high-$\delta$ region; under a sharp teacher distribution it can also overlap with high-entropy student tokens. The point is structural rather than set-theoretic. Prior work on token-level importance has mostly interpreted “which tokens matter” through the lens of student uncertainty—high entropy is taken as the proxy for an informative position—and entropy-based rules are provably blind to confident-but-wrong tokens (Proposition [2](#Thmtheorem2)). What our taxonomy adds is a more fine-grained decomposition: informative tokens do not come from a single homogeneous source, and in particular there is a distinct low-entropy, high-disagreement region (Q3) that entropy-based views fail to isolate. Table [8](#A2.T8) makes this concrete from two directions. Large-divergence ranking on its own is *not* sufficient: Div-only at a budget matched to Q3-only is consistently weaker than the joint score, and only recovers when given a $5\times$ larger budget—i.e., when it also sweeps up much of the low-entropy tail. Conversely, large student entropy is not the same selector as large divergence: high-$h$ tokens need not be high-$\delta$, and—as the entropy sweeps in Table [3](#S7.T3) show—the two selectors trace different performance curves across retention ratios. The “low entropy + high disagreement” specification is therefore not a redundant restatement of either axis; it is a separate, structurally identifiable region. #### Why “large gradient” is not enough. A natural objection is that high-divergence tokens already produce the largest pointwise gradients, so they must be the dominant source of useful signal in the full-token recipe. That intuition does not survive scrutiny under a budget. In standard full-token training, updates are taken over *all* token positions; the vast majority typically have low or moderate divergence, with a much smaller subset of high-divergence positions. A larger gradient on an individual high-divergence token does not by itself imply that the accumulated contribution from such tokens dominates the overall update, nor that those gradients carry the most useful signal for optimization. More importantly, the question addressed here is not whether a token has a large gradient, but whether that gradient is *informative and worth keeping under a limited budget*. The Div-only 10% column is exactly this stress test: when forced to commit the entire budget to the highest-divergence tail, the resulting model underperforms even the all-token baseline. The structured comparison—separating large-divergence-only from low-entropy-and-large-divergence—turns the coarse intuition that “large-divergence tokens matter” into a more specific and testable claim about which structurally distinct token types are actually informative under budgeted training, and identifies Q3 as the region where this distinction matters most. *Connecting back to AdaKD.* The Div-only 50% column is the within-our-setup analogue of AdaKD’s published LATF-only ablation [Xie et al., 2026]: a hard top-$r$% divergence selector with no entropy interaction. Their Table 3a reports $+0.04$ ROUGE-L for LATF alone on Qwen2-1.5B; we observe a similarly small effect on average ($-0.4$ on MATH-500, $-0.4$ on AIME’24, $+1.7$ on AIME’25 relative to the all-token baseline). The takeaway is the same in both settings: divergence ranking on its own is roughly baseline-equivalent at large budgets and clearly worse at small budgets, while a low-entropy + high-divergence selector is competitive at both—consistent with Proposition [2](#Thmtheorem2) and Remark [2](#Thmremark2). ### B.3 Agentic Planning: Held-Out Queries Figure: Figure 4: Token selection for agentic OPD on 20% held-out travel-planning queries. *Top row*: Avg@16; *Bottom row*: Best@16 (Pass@16). Within each row the left panel uses the 14B teacher and the right panel uses the 32B teacher. Q3-only 20% matches or exceeds the full-token baseline in every setting, consistent with Table [7](#S7.T7). Best@16 results show the same pattern: overconfident-token training improves the upper tail of performance, not just the mean. Refer to caption: 2604.14084v2/data_figures/planner_agentic.png Figure [4](#A2.F4) complements Table [7](#S7.T7) with a finer-grained view. The Avg@16 panels confirm the main-text findings: Q3-only 20% leads for both teacher sizes (12.6 and 13.6 vs. baselines of 11.7 and 12.8), and entropy-only 50% improves over full-token OPD. The Best@16 (Pass@16) panels show the same pattern: Soft-OR 20% achieves the highest Best@16 with the 14B teacher (20.3 vs. 18.9 baseline), while Q3-only 20% leads with the 32B teacher (20.1 vs. 19.7). This indicates that correcting overconfident tokens improves not just average performance but also the upper tail. ### B.4 Hyperparameters Table [9](#A2.T9) summarizes all training hyperparameters. **Table 9: Training hyperparameters across model pairs.** | | Qwen3 (8B$\to$4B) | Llama (70B$\to$8B) | Qwen2.5 (14B$\to$1.5B) | | --- | --- | --- | --- | | Optimizer | AdamW | AdamW | AdamW | | Learning rate | $1\times 10^{-6}$ | $3\times 10^{-7}$ | $1\times 10^{-6}$ | | Batch size (rollouts) | 8 | 8 | 8 | | Rollouts per prompt | 16 | 16 | 16 | | Max response length | 8192 | 8192 | 8192 | | Max prompt length | 2048 | 2048 | 2048 | | OPD chunk size | 512 | 512 | 512 | | OPD max length | 16,384 | 16,384 | 16,384 | | Tensor parallel size | 2 | 2 | 2 | | Generation temperature | 1.0 | 1.0 | 1.0 | | Top-$p$ (generation) | 1.0 | 1.0 | 1.0 | | GPUs | 8$\times$H200 | 8$\times$H200 | 4$\times$H200 | For entropy sampling experiments, tokens are sampled with probability $p_{t}\propto h_{t}$, with the retention ratio determining how many tokens are kept. For Top-K experiments, the top-$\rho$ fraction of tokens by score is selected deterministically. All evaluations use mean@16 (average accuracy over 16 independent samples per problem) with temperature 1.0. ### B.5 Qualitative Examples Across Quadrants We present five representative tokens from training on Qwen2.5-14B $\to$ 1.5B, spanning different quadrants of the taxonomy. Each example shows the problem, the student’s response with the target token highlighted, and the student and teacher top-5 distributions. The entropy values reported in this appendix are raw entropies in nats. #### Example 1: Generic variable vs. concrete substitution ( Q3 : low entropy, high divergence). Student entropy: $0.02$ (extremely confident). Forward KL: $5.27$. Overconfidence score: $5.24$. Analysis. The student assigns 99.8% to k, mechanically repeating the generic variable from the problem statement, while the teacher places 49.9% on the concrete value 2—indicating that the next reasoning step should substitute a specific integer rather than restate the variable. An entropy-only rule would assign near-zero weight to this position because $h_{t}=0.02$, yet it carries one of the densest corrective signals in the batch (overconfidence score $=5.24$). #### Example 2: Reasoning fork—restating vs. advancing ( Q1 : high entropy, high divergence). Student entropy: $1.82$ (uncertain). Forward KL: $5.31$. Analysis. The student favors “off” (54.4%), restating the problem, while the teacher prefers “written” (40.3%) or “increased” (35.6%)—words that advance the solution by characterizing the error direction. The teacher’s distribution pushes toward more precise mathematical reasoning, while the student’s choice leads to circular rephrasing. This is a classic Q1 token: the student is uncertain (entropy $=1.82$) and the teacher strongly disagrees—entropy-based selection *would* catch this token. #### Example 3: Arithmetic computation error ( Q3 : low entropy, high divergence). Student entropy: $0.40$. Forward KL: $3.54$. Overconfidence score: $3.27$. Analysis. The student confidently writes $f(2)=190$ with 91.2% on digit 0, committing an arithmetic error in summing $16+72+72+36+4$. The teacher distributes probability across 6 (60.0%) and 2 (25.0%), clearly disagreeing with 0. Like Example 1, this is a Q3 token: the student is fairly confident ($h_{t}=0.40$), yet the teacher strongly disagrees—entropy-based selection would under-weight this position. #### Example 4: Confident on the wrong variable ( Q3 : low entropy, high divergence). Student entropy: $0.12$ (very confident). Forward KL: $5.58$. Overconfidence score: $5.44$. Analysis. The student assigns 98.2% to B, while the teacher prefers A (52.4%). The student writes $\cot B=\frac{\sin B}{\cos B}$, which is mathematically incorrect ($\cot B=\frac{\cos B}{\sin B}$); the teacher’s preferred continuation reflects a different and more correct reasoning path. Another clear Q3 token: $h_{t}=0.12$ would be invisible to entropy-only selection. *Note.* The original rollout is in Chinese; the problem and response excerpts above are translated for readability. The token identifiers (B, A, etc.) are unchanged from the raw log, as mathematical symbols are language-invariant. #### Example 5: Wrong mathematical symbol ( Q1 : moderate entropy, high divergence). Student entropy: $1.38$. Forward KL: $4.27$. Analysis. The teacher assigns 91.7% to cot—the mathematically relevant function for this problem—while the student splits probability across irrelevant symbols (text, Delta, frac). This is a Q1 token (moderate entropy, high divergence): unlike Q3 tokens, entropy-based selection *would* catch it, but the teacher’s near-deterministic preference for cot makes it an especially high-value training signal. Note that Examples 4 and 5 come from the *same student rollout* at different token positions (same Chinese-language response as Ex. 4; translated above), illustrating how a single response can contain both Q3 and Q1 tokens. Summary. Examples 1, 3, and 4 illustrate why divergence is needed: the student’s entropy is low, so any entropy-only rule would skip these tokens, yet the teacher strongly disagrees—a generic variable where a concrete value is needed (Ex. 1), an arithmetic error (Ex. 3), or a confident variable-level misstep in a derivation (Ex. 4). Examples 2 and 5 show Q1 tokens that entropy-based selection handles well—the student is already uncertain, and the teacher provides a clear corrective signal. The taxonomy’s value is precisely that it identifies *both* regions as informative while distinguishing them from Q4 (solved) tokens where both entropy and divergence are low.