Reinforcement-Aware Knowledge Distillation (RLAD)

Reinforcement-Aware Knowledge Distillation (RLAD) is an on-policy distillation framework that selectively imitates the teacher during RL training — distilling from the teacher only when it improves the current policy update. RLAD replaces the standard KL regularizer with a Trust Region Ratio Distillation (TRRD) objective that uses PPO/GRPO-style likelihood-ratio clipping, making distillation compatible with modern RL training loops.

Overview

Standard on-policy distillation applies KL divergence as a regularizer uniformly across all tokens and samples. This is wasteful when the teacher's guidance is irrelevant or even harmful to the student's current learning trajectory. RLAD makes distillation reinforcement-aware: it evaluates whether teacher imitation would actually benefit the current policy update and only applies distillation when it does.

Core Mechanism

1. Selective Imitation: For each sample in a batch, RLAD checks whether distilling from the teacher would improve the expected policy update (measured by advantage or reward signal). Only samples where teacher guidance is beneficial receive the distillation loss.

2. Trust Region Ratio Distillation (TRRD): Instead of KL divergence, TRRD uses a likelihood-ratio objective:

3. Replaces KL(pi_teacher || pi_student) with a clipped ratio objective similar to PPO's surrogate.
4. This ensures distillation updates stay within a trust region, preventing destructive large updates.

5. Naturally integrates with GRPO or PPO training loops — no separate distillation phase needed.

6. Adaptive Weighting: The distillation strength adapts based on the alignment between teacher and student policies, reducing unnecessary regularization when the student is already performing well.

Key Results

• Outperforms offline distillation (which applies teacher knowledge without RL context).
• Outperforms standard GRPO (which has no teacher guidance).
• Outperforms KL-based on-policy KD (which applies distillation uniformly, including when harmful).
• Demonstrates that selective, RL-aware distillation is strictly better than both always-distill and never-distill strategies.

Relationships

• Extends OPD by making distillation conditional on RL signal quality.
• Complementary to SRPO: where SRPO routes samples to different objectives, RLAD modulates distillation strength per-sample within a single objective.
• Related to PPD in using trust region / clipping ideas, but applies them to the distillation term rather than the RL term.
• Improves upon standard KL-regularized RLHF by replacing passive regularization with active, selective teacher guidance.

See Also

• on-policy-distillation
• sample-routed-policy-optimization
• proximal-policy-distillation
• ppo

[src: raw/papers/2026/02/2602.22495.md]