Strong-to-weak distillation is a paradigm where a larger, more capable "strong" teacher model distills its knowledge into a smaller, less capable "weak" student model. This is the most common distillation direction and is used in models like qwen3 and frameworks like generalized-on-policy-distillation. The goal is to preserve as much of the strong model's ca...
- Related
- References
Strong-to-Weak Distillation
Strong-to-weak distillation is a paradigm where a larger, more capable "strong" teacher model distills its knowledge into a smaller, less capable "weak" student model. This is the most common distillation direction and is used in models like qwen3 and frameworks like generalized-on-policy-distillation. The goal is to preserve as much of the strong model's capability as possible in the weaker student, creating efficient models for deployment.
Related
- generalized-on-policy-distillation — G-OPD applies strong-to-weak distillation with reward extrapolation
- qwen3 — Qwen3 uses strong-to-weak logits distillation in its training pipeline
- knowledge-distillation — Foundational concept underlying strong-to-weak transfer