On-Policy Self-Distillation for Reasoning Compression (OPSDC) is a training method that compresses LLM reasoning traces by conditioning the same model on a "be concise" instruction to generate teacher logits, then training the student (unconditioned) to minimize per-token reverse KL divergence on its own rollouts. No ground-truth answers are needed.
- Overview
- Core Mechanism
- Key Results
- Relationships
- See Also
On-Policy Self-Distillation for Reasoning Compression (OPSDC)
On-Policy Self-Distillation for Reasoning Compression (OPSDC) is a training method that compresses LLM reasoning traces by conditioning the same model on a "be concise" instruction to generate teacher logits, then training the student (unconditioned) to minimize per-token reverse KL divergence on its own rollouts. No ground-truth answers are needed.
Overview
LLM reasoning models (e.g., chain-of-thought) often produce verbose traces with unnecessary deliberation. OPSDC compresses these traces by teaching the model to produce shorter, equally accurate reasoning. The key insight: the same model can serve as its own teacher by conditioning it differently — with a conciseness instruction for teacher logits, without it for student rollouts.
Core Mechanism
- Dual Conditioning: The same model is used twice:
- Teacher mode: Conditioned on "be concise" instruction prepended to the prompt. Produces logits pi_teacher(y|x, "be concise").
-
Student mode: Standard prompting without the conciseness instruction. Generates rollouts pi_student(y|x).
-
On-Policy Rollouts: The student generates rollouts from its own policy (on-policy). No offline teacher traces are needed.
-
Per-Token Reverse KL Minimization: At each token position in the student's rollout, minimize KL(pi_student || pi_teacher). This encourages the student to match the teacher's more compact token distribution.
-
No Ground Truth Needed: Since the objective is self-consistency (match the concise version of yourself), no external ground-truth answers or solutions are required. The model learns to self-compress.
Key Results
| Model | Token Reduction | Accuracy Gain (MATH-500) |
|---|---|---|
| Qwen3-8B | 57% | +9–16 pts |
| Qwen3-14B | 59% | +9–16 pts |
OPSDC achieves substantial compression (over half the tokens removed) while actually improving accuracy, suggesting that verbose reasoning traces contain noise that self-distillation filters out.
Relationships
- Self-distillation variant of OPD: the teacher is the same model with different conditioning.
- Related to OPSD but specifically targets reasoning compression rather than general capability transfer.
- Complementary to SRPO: OPSDC shows that self-conditioning can serve as the "teacher" signal, while SRPO uses an external teacher for failed samples.
- The per-token reverse KL objective connects to the theoretical framework in G-OPD.
See Also
- on-policy-distillation
- on-policy-self-distillation
- sample-routed-policy-optimization
- generalized-on-policy-distillation
[src: raw/papers/2026/03/2603.05433.md]