---
title: "Vold"
created: 2026-04-06
updated: 2026-04-10
type: concept
tags: [distillation, on-policy, multimodal]
sources: []
---

# VOLD (Reasoning Transfer via On-Policy Distillation)

VOLD is a two-stage framework for transferring reasoning capabilities from text-only teacher LLMs to vision-language student models (VLMs) using purely text-based training data. It combines Group Relative Policy Optimization (GRPO) reinforcement learning with on-policy knowledge distillation.

## Core Idea

The fundamental challenge is that high-quality visual reasoning data is scarce, while text-based reasoning resources are abundant. VOLD bridges this gap by:

1. **Stage 1 - Policy Alignment (SFT)**: Supervised fine-tuning the student VLM on reasoning traces generated by the text-only teacher to establish distributional alignment.
2. **Stage 2 - Unified RL + Distillation**: Training with a combined objective that uses the same student rollouts for both GRPO reward optimization and reverse KL-divergence distillation against the teacher.

## Key Innovations

- **Cold-Start Alignment**: Demonstrates that initial policy alignment via SFT is essential for effective on-policy distillation. Without alignment, teacher guidance becomes uninformative due to state-distribution shift.
- **Unified Objective**: Replaces the standard GRPO KL penalty against a reference policy with reverse KL against the teacher, adding dense token-level supervision at minimal computational cost.
- **Reward-Guided KL Masking**: Applies distillation only to incorrect student responses (r=0), allowing successful novel reasoning paths to be retained without teacher interference.

## Technical Formulation

The unified VOLD loss:

L_VOLD = L_GRPO(theta) + beta * E_{q,tau~pi_theta}[(1-r(tau)) * sum_t KL(pi_phi(.|h_t) || pi_theta(.|h_t))]

Where pi_theta is the student, pi_phi is the teacher, r(tau) is the binary reward, and beta controls distillation strength.

## Results

VOLD (Qwen2.5-VL-3B student, Qwen3-8B teacher) achieves state-of-the-art performance on visual reasoning benchmarks (MMMU-Pro, MathVision, MathVista, LogicVista) despite training exclusively on text data, outperforming methods that use image-text fine-tuning.

## Relation to Existing Concepts

- Extends [[on-policy-distillation]] to the text-to-vision modality transfer setting.
- Builds upon [[grp-o|Group Relative Policy Optimization]] instead of [[ppo|PPO]].
- Related to [[policy-distillation]] and [[model-distillation]], but operates on-policy with RL integration.
- Orthogonal to advances in RL algorithms; can integrate with methods beyond GRPO.

[src: raw/ingested/2025/10/2510.23497.md]