概念导读

VOLD is a two-stage framework for transferring reasoning capabilities from text-only teacher LLMs to vision-language student models (VLMs) using purely text-based training data. It combines Group Relative Policy Optimization (GRPO) reinforcement learning with on-policy knowledge distillation.

1. Core Idea
2. Key Innovations
3. Technical Formulation
4. Results
5. Relation to Existing Concepts

VOLD (Reasoning Transfer via On-Policy Distillation)

VOLD is a two-stage framework for transferring reasoning capabilities from text-only teacher LLMs to vision-language student models (VLMs) using purely text-based training data. It combines Group Relative Policy Optimization (GRPO) reinforcement learning with on-policy knowledge distillation.

Core Idea

The fundamental challenge is that high-quality visual reasoning data is scarce, while text-based reasoning resources are abundant. VOLD bridges this gap by:

1. Stage 1 - Policy Alignment (SFT): Supervised fine-tuning the student VLM on reasoning traces generated by the text-only teacher to establish distributional alignment.
2. Stage 2 - Unified RL + Distillation: Training with a combined objective that uses the same student rollouts for both GRPO reward optimization and reverse KL-divergence distillation against the teacher.

Key Innovations

• Cold-Start Alignment: Demonstrates that initial policy alignment via SFT is essential for effective on-policy distillation. Without alignment, teacher guidance becomes uninformative due to state-distribution shift.
• Unified Objective: Replaces the standard GRPO KL penalty against a reference policy with reverse KL against the teacher, adding dense token-level supervision at minimal computational cost.
• Reward-Guided KL Masking: Applies distillation only to incorrect student responses (r=0), allowing successful novel reasoning paths to be retained without teacher interference.

Technical Formulation

The unified VOLD loss:

L_VOLD = L_GRPO(theta) + beta * E_{q,tau~pi_theta}[(1-r(tau)) * sum_t KL(pi_phi(.|h_t) || pi_theta(.|h_t))]

Where pi_theta is the student, pi_phi is the teacher, r(tau) is the binary reward, and beta controls distillation strength.

Results

VOLD (Qwen2.5-VL-3B student, Qwen3-8B teacher) achieves state-of-the-art performance on visual reasoning benchmarks (MMMU-Pro, MathVision, MathVista, LogicVista) despite training exclusively on text data, outperforming methods that use image-text fine-tuning.

Relation to Existing Concepts

• Extends on-policy-distillation to the text-to-vision modality transfer setting.
• Builds upon Group Relative Policy Optimization instead of PPO.
• Related to policy-distillation and model-distillation, but operates on-policy with RL integration.
• Orthogonal to advances in RL algorithms; can integrate with methods beyond GRPO.

[src: raw/ingested/2025/10/2510.23497.md]