1|# Generalized On-Policy Distillation (G-OPD) 2| 3|Generalized On-Policy Distillation (G-OPD) is a theoretical and practical extension of standard OPD that reformulates distillation as a dense KL-constrained reinforcement learning problem with two tunable components: a reward scaling factor lambda and a flexible reference model. 4| 5|## Core Formulation 6| 7|The G-OPD objective is: 8| 9|J_G-OPD(theta) = max_theta E_{x~D, y~pi_theta} [ lambda * log(pi(y|x) / pi_ref(y|x)) - KL(pi_theta(y|x) || pi_ref(y|x)) ] 10| 11|Where: 12|- pi_theta: student policy 13|- pi: teacher policy 14|- pi_ref: reference model (any model, not necessarily student base) 15|- lambda: reward scaling factor controlling relative weight of reward vs KL regularization 16| 17|## Key Variants 18| 19|| Lambda Range | Name | Behavior | 20||-------------|------|----------| 21|| 0 < lambda < 1 | Reward Interpolation | Student behavior interpolates between reference and teacher | 22|| lambda = 1 | Standard OPD | Equal weighting of reward and KL | 23|| lambda > 1 | Reward Extrapolation (ExOPD) | Student can surpass teacher capability boundary | 24| 25|## Reward Correction 26| 27|In strong-to-weak-distillation settings, setting pi_ref = teacher's pre-RL base model (rather than student base) yields more accurate implicit reward signals. This is termed "reward correction" but requires access to teacher's pre-RL checkpoint and incurs higher computational cost. 28| 29|## Theoretical Connection 30| 31|G-OPD reveals that OPD is a special case of dense RL where: 32|1. Rewards are dense (token-level) via implicit reward log(pi*/pi_ref) 33|2. Reward and KL are equally weighted (lambda=1) 34|3. Reference model choice is arbitrary (does not affect simplified objective) 35| 36|## Experimental Findings 37| 38|- ExOPD (lambda=1.25) consistently outperforms OPD across math reasoning and code generation 39|- In multi-teacher distillation, ExOPD produces unified students surpassing all domain teachers 40|- Reward correction further improves strong-to-weak distillation 41|- Excessive extrapolation (lambda=1.5) causes instability due to implicit reward hacking 42| 43|## Relation to Existing Concepts 44| 45|- Generalizes on-policy-distillation with tunable reward weighting 46|- Contrasts with PPD: G-OPD uses simple policy gradient, not PPO clipping 47|- Extends video-opd and vold to unified theoretical framework 48|- Related to implicit reward methods in free-process-rewards 49| 50|[src: raw/ingested/2026/02/2602.12125.md]

See Also

• ex-opd — Extended OPD 变体