---
title: "Hybrid Policy Distillation for LLMs"
arxiv_id: "2604.20244"
authors: []
date: "2026-04-28"
tags: ["knowledge distillation", "large language models", "KL divergence", "policy optimization", "model compression"]
affiliation: []
---

## Abstract

Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log-likelihood objective at the token level. We further propose Hybrid Policy Distillation (HPD), which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode-seeking, and combines off-policy data with lightweight, approximate on-policy sampling. We validate HPD on long-generation math reasoning as well as short-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales. The code related to this work is available at this https URL.

## Key Contributions

- **Unified view of knowledge distillation**: Reformulates KD as a reweighted log-likelihood objective at the token level, revealing connections between existing methods.
- **Hybrid Policy Distillation (HPD)**: Introduces a novel distillation framework that combines forward and reverse KL divergences to balance mode coverage and mode-seeking behavior.
- **Data regime integration**: Combines off-policy data with lightweight, approximate on-policy sampling to improve sample efficiency and optimization stability.
- **Empirical validation**: Demonstrates consistent improvements in performance, computational efficiency, and training stability across diverse tasks (math reasoning, dialogue, code) and model scales.
- **Open-source implementation**: Provides publicly available code to facilitate reproducibility and further research.