--- title: "OPSDL: On-Policy Self-Distillation for Long-Context Language Models" arxiv_id: "2604.17535" authors: - Xinsen Zhang - Zhenkai Ding - Tianjun Pan - Run Yang - Chun Kang - Xue Xiong - Jingnan Gu affiliation: "Baidu Inc" date: "2026-04-19" tags: - long-context language models - self-distillation - on-policy learning - token-level optimization - post-training - large language models - context length extension - reverse KL divergence - policy gradient --- ## Contents - 1 Introduction - 2 Related Work - Post-Training for Long-Context Modeling - On-Policy Distillation and Self-Distillation - 3 Method - 3.1 Problem Formulation - 3.2 Data Construction - 3.3 Short-to-Long Self-Distillation via Policy Gradient - Token-Level Advantage. - Policy Gradient Objective. - 3.4 Overall Training Procedure - 4 Experiment - 4.1 Experiment Setup - Data Construction - Baselines - Models - Evaluation benchmarks - 4.2 Main Result - OPSDL achieves the best overall performance among trainable methods. - Substantial improvements on long contexts. - OPSDL narrows the gap with officially long-context-optimized models. - Improvements span both synthetic and realistic benchmarks. - OPSDL preserves short-context capabilities. - 5 Conclusion - References ## Abstract Abstract Extending the effective context length of large language models (LLMs) remains a central challenge for real-world applications. While recent post-training methods have made progress in long-context scaling, they either rely on high-quality supervision data or sparse sequence-level rewards, leading to unstable and inefficient optimization. We propose OPSDL , an O n- P olicy S elf- D istillation method for enhancing the L ong-context capabilities of LLMs. Unlike other recent self-distillation methods that inject privileged information and rely on the model's in-context learning ability to act as a teacher, OPSDL leverages the model's own inherently strong short-context capability as a self-teacher to supervise its own generation in long-context scenarios. The model first generates responses conditioned on the full long-context, then the self-teacher provides per-token supervision signals via point-wise reverse KL divergence under the relevant extracted short-context. This dense token-level signal encourages faithful use of relevant evidence and mitigates hallucinations induced by irrelevant context. We evaluate OPSDL on long-context benchmarks across a range of models from 7B to 32B parameters. Results show consistent and substantial improvements across varying context lengths, outperforming standard post-training approaches such as SFT and DPO with higher sample efficiency. Notably, these gains are achieved without degrading general short-context performance. These findings highlight the effectiveness of OPSDL as a scalable and stable approach for long-context learning. ## 1 Introduction Extending the effective context length of large language models (LLMs) is a central challenge in enabling real-world applications such as long-document comprehension, repository-level code understanding, and multi-hop reasoning over scattered evidence. While architectural innovations in positional encoding and attention mechanisms have made it possible to accept much longer inputs (Peng et al. (2023); Su et al. (2024); Team et al. (2026)), a growing body of empirical evidence reveals a persistent gap between the *maximum* context window and the *effective* context capacity of a model (Paulsen (2025); Bai et al. (2025)). These findings suggest that closing the gap between maximum and effective context length requires not only architectural advances but also innovations in post-training paradigms. Current training approaches for long-context LLM performance optimization are numerous and effective(Shen et al. (2025); Zhang et al. (2025); Chen et al. (2025)). While these methods have shown effectiveness, they suffer from either the need for high-quality training data or sparse sequence-level reward signals that limit sample efficiency. Under long contexts, such sparse, sequence-level signals make optimization extremely difficult, leading to unstable and sample-inefficient training. Furthermore, these methods typically rely on auxiliary components such as frozen verifier models or learned reward models, which increases system complexity and couples the training pipeline to the availability and quality of external resources. To address this, we propose OPSDL, a simple yet effective method that overcomes these limitations. The key observation is that, given a query that derived from the short context, a model that given this short context naturally serves as a teacher compared to the model that given a longer contex. OPSDL leverages this asymmetry by distilling the model's own short-context behavior into its long-context behavior via token-level reverse KL divergence, computed on-policy during training. Rather than relying on external supervision or reward signals, the model itself serves as both student and teacher, with the short-context ability providing a natural and readily available training signal. This design eliminates the need for human-annotated data, reward models, or elaborate reward engineering, while the on-policy distillation framework ensures that the training signal remains aligned with the model's current behavior, yielding strong sample efficiency. We empirically validate OPSDL on the RULER(Hsieh et al. (2024)) across multiple backbone models and context lengths. Our results show that OPSDL consistently outperforms standard instruction tuned models on both Qwen2.5-7B family(Yang et al. (2024)), demonstrating strong generalization across model families rather than reliance on a specific architecture. Notably, the performance gains become increasingly pronounced as the context length grows, indicating that our method effectively mitigates the degradation commonly observed under long horizon settings. Moreover, despite not relying on specialized long context pretraining, OPSDL achieves performance comparable to, and in some cases exceeding, Qwen2.5-7B-Instruct-1M(Yang et al. (2025)), a model explicitly trained for million-token contexts. These results suggest that replacing sparse sequence-level preference optimization with on-policy, token-level distillation enables stable policy-level self-evolution, unlocking robust long-context reasoning while preserving short-context capabilities. In summary, our contributions are as follows: - • We propose OPSDL, an on-policy self-distillation method that leverages a model's inherent short-context capability as a teacher to supervise its own long-context generation. By computing token-level supervision signals via point-wise reverse KL divergence, OPSDL provides dense training signals that encourage faithful use of relevant evidence and mitigate context-induced hallucinations. - • Unlike existing long-context training methods that rely on human-annotated data, reward models, or sparse sequence-level rewards, OPSDL eliminates the need for external supervision or auxiliary components. This design yields superior sample efficiency and training stability while remaining model-agnostic across different architectures. - • We conduct comprehensive evaluations on long-context benchmarks across models from 7B to 32B parameters. Results demonstrate consistent improvements across varying context lengths, with performance gains becoming increasingly pronounced as the context grows. Notably, OPSDL achieves these gains without degrading short-context performance. ## 2 Related Work #### Post-Training for Long-Context Modeling A straightforward approach to improving long-context capability is supervised fine-tuning (SFT) on long-context training data, though it is constrained by data quality and prone to distribution shift. To overcome these limitations, preference optimization methods have been explored. LongReward(Zhang et al. (2025)) constructs reward signals via multidimensional LLM feedback and applies DPO to boost long-context performance. LongPO(Chen et al. (2025)) proposes a self-evolution framework that treats short-context generations as positive samples and long-context counterparts as negative ones, iteratively optimizing the model via DPO. SoLoPO(Sun et al. (2025)) further decomposes alignment into short-context optimization and a short-to-long consistency constraint. From a training strategy perspective, QwenLong-L1(Wan et al. (2025)) treat long-context reasoning as a long-horizon decision-making problem, utilizing warm-up SFT and progressive context scaling to stabilize RL training. However, these methods rely on externally constructed preference pairs and coarse-grained sequence-level objectives. In contrast, our approach adopts an on-policy self-distillation method with token-level fine-grained optimization signals, enabling more scalable and stable long-context learning. #### On-Policy Distillation and Self-Distillation On-Policy Distillation (OPD) has recently attracted considerable attention due to its on-policy nature and token-level supervision. Pioneered by Generalized Knowledge Distillation (GKD) (Agarwal et al., 2024), this line of work minimizes reverse KL divergence on student-generated trajectories under teacher supervision. To bypass external teacher dependency, on-policy self-distillation has emerged as a key paradigm. OPSD (Zhao et al., 2026) and SDPO (Hubotter et al., 2026) enable the model itself to serve as the teacher by incorporating more privileged context, such as verified reasoning traces or environmental feedback. OPCD (Ye et al., 2026) internalizes contextual knowledge and prevents forgetting by distilling from teacher-enriched contexts. While these methods typically construct a stronger teacher by enriching the input context, heavy reliance on externally provided correct trajectories may compromise the model's intrinsic reasoning capability. In contrast, our approach takes the opposite direction: rather than enriching the context, we extract key information from long contexts to reduce noise and utilize the model's superior short-context ability, constructing a token-level self-teacher to supervise its outputs derived from long contexts. ## 3 Method Figure: Figure 1: Overview of the OPSDL framework. Given a long context $C_{L}$ and its extracted short context $C_{S}$, the model generates responses on-policy conditioned on $C_{L}$. The same model under $C_{S}$ serves as a self-teacher, providing token-level supervision via point-wise reverse KL divergence to align the long-context generation with the short-context behavior. Refer to caption: 2604.17535v1/figs/framework_new.png We propose OPSDL, which leverages the short-context capability of the current policy to supervise its own long-context generation. ### 3.1 Problem Formulation Let $C_{L}$ denote the long-context input, and $C_{S}$ denote the corresponding short context extracted from $C_{L}$, which contains the core information. Based on $C_{S}$, an instruction or query $Q$ is formulated. Given the query $Q$ and the context, the large language model, parameterized by $\theta$ and denoted as $\pi_{\theta}$, auto-regressively generates a response sequence $y=\{y_{1},y_{2},\dots,y_{T}\}$. The key insight of our approach is that the same policy $\pi_{\theta}$, when prompted with the short-context input $C_{S}$, produces a more calibrated and accurate next-token distribution, since $C_{S}$ avoids distracting noise and fits perfectly within the model's well-trained context window. This allows the model to leverage its well-aligned short-context capabilities to self-align in long-context scenarios. Therefore, instead of relying on external reward models, we formulate the long-context optimization problem as a self-alignment process: we aim to optimize $\theta$ such that the model's generation distribution under the long context, $\pi_{\theta}(y\mid C_{L},Q)$, closely approximates its own high-quality distribution under the short context, $\pi_{\theta}(y\mid C_{S},Q)$. ### 3.2 Data Construction We constructs $(C_{L},C_{S},Q)$ triplets from raw long documents without human annotation. We sample a long document $C_{L}$ from the training corpus, then extract a contiguous segment $C_{S}\subset C_{L}$ with $|C_{s}|\leq|C_{L}|$ that preserves the core evidence while fitting within the model's well-trained context window. Finally, we generate a question $Q\sim\pi_{\theta}(\cdot\mid C_{S})$ conditioned on $C_{S}$, ensuring that $Q$ targets evidence present in both contexts. This yields triplets where the question $Q$ is answerable from both $C_{L}$ and $C_{S}$, enabling meaningful comparison between the short-context teacher and the long-context student. ### 3.3 Short-to-Long Self-Distillation via Policy Gradient To mitigate the distraction from irrelevant information in the long context, we utilize the model itself as a dynamic teacher, using its predictions under the short context $C_{S}$ as a robust anchor. Formally, for a response $y=(y_{1},\ldots,y_{T})$ sampled from the long-context student policy $\pi_{\theta}(\cdot\mid C_{L},Q)$, the teacher probability at each token position is: $$ $P(y_{t})=\pi_{\text{Teacher}}(y_{t}\mid C_{S},Q,y_{0$: the teacher assigns higher probability than the student, indicating that the student *under-weights* evidence clearly present in the short context—a signal of context utilization degradation under long input. - • Negative $A_{t}(y_{t})<0$: the student assigns higher probability than the teacher, suggesting that the model *hallucinates* or attends to irrelevant content introduced by the long context. - • Near-zero $A_{t}(y_{t})\approx 0$: both distributions agree, indicating the token is unaffected by context length and receives negligible gradient signal. This design enables that only tokens where the long-context policy deviates from the short-context anchor receive non-trivial gradient signal, avoiding the noise introduced by uniform supervision. #### Policy Gradient Objective. We optimize the model using a policy gradient objective over the long-context distribution $\pi_{\theta}(\cdot\mid C_{L},Q)$: $$ $\mathcal{L}_{\mathrm{PG}}(\theta)=-\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid C_{L},Q)}\left[\sum_{t=1}^{T}A_{t}(y_{t})\cdot\log\pi_{\theta}(y_{t}\mid C_{L},Q,y_{