Knowledge Distillation (KD), introduced by Hinton et al. in 2015, is the foundational technique for transferring knowledge from a large "teacher" model to a smaller "student" model. The student is trained to match the teacher's output distribution (typically via kl-divergence-in-distillation), preserving much of the teacher's capability at lower computationa...
- Related
- References
Knowledge Distillation
Knowledge Distillation (KD), introduced by Hinton et al. in 2015, is the foundational technique for transferring knowledge from a large "teacher" model to a smaller "student" model. The student is trained to match the teacher's output distribution (typically via kl-divergence-in-distillation), preserving much of the teacher's capability at lower computational cost. KD is a core concept referenced throughout the wiki, including in discussions of exposure-bias and the on-policy-distillation-survey.
Related
- on-policy-distillation-survey — Surveys modern distillation methods including on-policy variants
- exposure-bias — Standard KD suffers from exposure bias when student encounters unseen states
- kl-divergence-in-distillation — KL divergence as the primary distillation loss
- policy-distillation — Application of KD to RL policy transfer
References
- Hinton, Vinyals, & Dean (2015). "Distilling the Knowledge in a Neural Network." arXiv:1503.02531.