Title: 2604.01840 ArXiv: 2604.01840 Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models Title: Content selection saved. Describe the issue below: Description: License: CC BY 4.0 arXiv:2604.01840v2 [cs.AI] 08 Apr 2026 Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models Zekai Ye 1 , Qiming Li 1 , Xiaocheng Feng 1,2 , Ruihan Chen 1 , Ziming Li 3 , Haoyu Ren 3 , Kun Chen 3 , Dandan Tu 3 , Bing Qin 1,2 1 Harbin Institute of Technology 2 Peng Cheng Laboratory 3 Huawei Technologies Co., Ltd {zkye,qmli}@ir.hit.edu.cn Abstract While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency , quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO. Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models Zekai Ye 1 , Qiming Li 1 , Xiaocheng Feng 1,2 , Ruihan Chen 1 , Ziming Li 3 , Haoyu Ren 3 , Kun Chen 3 , Dandan Tu 3 , Bing Qin 1,2 1 Harbin Institute of Technology 2 Peng Cheng Laboratory 3 Huawei Technologies Co., Ltd {zkye,qmli}@ir.hit.edu.cn 1 Introduction Figure 1: Unlike standard uniform credit assignment, our proposed method dynamically allocates higher reinforcement learning advantage to pivotal tokens that heavily rely on visual perception. Reinforcement learning with verifiable rewards (RLVR) zhang2025survey —especially online methods such as Group Relative Policy Optimization (GRPO) shao2024deepseekmath —has substantially improved reasoning performance in text-oriented Large Language Models (LLMs). Motivated by these gains, recent studies have extended RLVR to Large Vision-Language Models (LVLMs), with progress mainly driven by improvements in training data design li2025truth ; liang2025modomodo ; liu2025noisyrollout ; yao2025r1 ; chen2025g1 , reward construction shen2025vlm ; xia2025visionary ; wang2025skywork , and optimization strategy refinement wang2025vl ; zhao2025absolute ; huang2026spotlighttokenperceptionmultimodal . Although existing RLVR studies for LVLMs report notable progress, most optimization pipelines still underemphasize fine-grained visual perception. This creates a key methodological weakness: multimodal reasoning depends on precise perception xiao2025advancing ; wang2025perception , which serves as the basis for reliable downstream inference. In practice, this demand for visual grounding is not uniformly distributed across the generated text. As illustrated in Figure 1 , only a critical minority of tokens (e.g., specific geometric entities or numerical values extracted from an image) exhibit high visual dependency. Despite this inherent token-level difference, standard GRPO relies on a coarse-grained credit assignment mechanism, uniformly distributing a single sequence-level, group-normalized advantage across all tokens in the generated trajectory. In the context of multimodal reasoning, this poses a foundational flaw: by treating all tokens equally, it fundamentally dilutes the learning signals essential for optimizing those pivotal, perception-grounded reasoning steps. To address this limitation, it is imperative to quantify the causal contribution of the visual modality to each individual reasoning step. We formalize this concept as Token Visual Dependency , measured by the Kullback-Leibler (KL) divergence between the model’s predictive distributions with and without visual conditioning. This metric captures the Bayesian surprise itti2009bayesian —the specific information gained from the visual context. In Section 3 , we find this dependency exhibits a highly sparse distribution, demonstrating that visually-grounded reasoning is driven by a critical minority of tokens. Furthermore, it reliably isolates semantic visual grounding by assigning higher values to image-dependent visual anchors and truthful concepts. To this end, we propose Perception-Grounded Policy Optimization (PGPO) , a novel token-level credit assignment framework that integrates threshold-gated, outcome-preserving advantage reallocation into multimodal RLVR. PGPO redistributes advantages based on normalized visual dependency scores: it suppresses the gradient noise from low-dependency tokens and actively boosts the learning signal for perceptually pivotal tokens. To ensure stable optimization, PGPO applies a sum-preserving normalization that keeps the overall advantage mass consistent with the original, guaranteeing a zero mean for intra-group advantages. To validate the effectiveness of PGPO, we conduct extensive experiments across seven challenging multimodal reasoning benchmarks. Based on Qwen2.5-VL bai2025qwen2 series models, PGPO achieves state-of-the-art performance comparison with previous methods. Crucially, by performing fine-grained, token-level advantage reshaping, PGPO reduces gradient variance, accelerates convergence, and increases visual dependency, thereby acting as a potent regularizer for robust, perception-grounded multimodal reasoning. Our contributions can be summarized as follows: • We formulate token visual dependency to quantify the causal impact of visual information, and empirically validate its sparse distribution and correlation with visual semantics. • We propose PGPO, a fine-grained credit assignment framework that utilizes token-level advantage reshaping to amplify learning signals for visually-dependent tokens while suppressing noise. • Extensive empirical evaluations and theoretical proofs demonstrate that PGPO achieves state-of-the-art performance on diverse multimodal benchmarks, covering mathematical, geometric, logical, multi-discipline reasoning and general VQA. Figure 2: Empirical analysis results of visual dependency. (a) The skewed distribution of token-level visual dependency. (b,c,d) Average visual dependency comparison between specific category tokens and others. 2 Related Work Multimodal Reasoning. To narrow the remaining performance gap of LVLMs on complex reasoning tasks wang2024enhancing ; dong2025insight , recent work has extended core reinforcement learning methods such as PPO schulman2017proximal , GRPO shao2024deepseekmath , and DAPO yu2025dapo from text settings to multimodal training guo2025deepseek ; bai2025qwen2 ; hurst2024gpt . Most existing multimodal RL pipelines, however, improve components around the optimizer rather than the token-level optimization mechanism itself. Prior work mainly falls into data-oriented designs bai2025univg ; li2025truth ; chen2025advancing ; wei2025open ; chen2025vinci and reward-oriented designs that inject perception-related supervision wang2025perception ; ma2025one , together with rollout-level changes and external visual tool usage liu2025noisyrollout ; wang2025vl . huang2026spotlighttokenperceptionmultimodal improves the optimization of visually dependent tokens through binary gradient filtering, but it lacks dynamic token-level advantage allocation. Even with these advances, many methods still assign identical training signals to every generated token, which is mismatched with the fine-grained nature of multimodal reasoning. Credit Assignment. Credit assignment Sutton1998ReinforcementLA ; arumugam2021informationtheoreticperspectivecreditassignment ; zhou2020learningimplicitcreditassignment is a central problem in reinforcement learning: determining which earlier actions should receive responsibility for later outcomes. This issue becomes more severe in RLVR for LLMs, where sparse terminal rewards must be traced back to token-level decisions across long reasoning trajectories. To mitigate this challenge, recent work wang2025emergenthierarchicalreasoningllms ; wei2025reinforcingmultiturnreasoningllm ; li2025attentionilluminatesllmreasoning explores fine-grained attribution for more precise advantage estimation. Yet, principled token-level credit redistribution for perception-grounded reasoning in LVLMs remains underexplored. 3 Preliminaries To study the perception in multimodal reasoning, this section first formalizes Token Visual Dependency via Kullback-Leibler (KL) divergence to quantify the causal information gain from visual inputs at each reasoning step. We then empirically demonstrate that this visual dependency exhibits a highly sparse distribution, strongly correlating with semantic visual anchors and factual truthfulness. Together, these theoretical and empirical findings expose the fundamental limitations of uniform sequence-level rewards, directly motivating the core design of our Perception-Grounded Policy Optimization (PGPO) approach. 3.1 Token Visual dependency We characterize token-level visual dependency as the incremental information contributed by the image context, inspired by huang2026spotlighttokenperceptionmultimodal . We measure it using the Kullback-Leibler (KL) divergence between the policy’s next-token distribution with visual conditioning and the corresponding distribution when visual input is removed, thereby capturing the image-induced distributional change. Let I I denote the image, q q the textual query, and o o the generated response. For state s t = ( q , o < t ) s_{t}=(q,o_{ tags. Then, provide your final answer enclosed within \boxed{} . Table 6: Key hyperparameters for training and evaluation. Hyperparameter Value General Training Optimizer AdamW Learning Rate 1e-6 LR Schedule Constant (no warmup or decay) Epochs 2 Freeze Vision Tower False RL Process Global Batch Size 128 Rollout Batch Size 384 Rollouts per Prompt 5 Rollout Top-p 0.99 Max Response Length 2048 DAPO Recipe Sampling Method Dynamic Sampling Clip Ratio Low 0.2 Clip Ratio High 0.28 Loss Averaging Mode Token-level Online Filtering Key Overall KL Penalty None Evaluation Generation Temperature 1.0 Top-p 1.0 Max New Tokens 2048 A.1 PGPO Details In Section 3 , token visual dependency 𝒮 ​ ( s t , I ) \mathcal{S}(s_{t},I) is theoretically defined as the Kullback-Leibler (KL) divergence over the entire vocabulary space 𝒱 \mathcal{V} . However, computing the exact full-vocabulary KL divergence in Large Vision-Language Models (LVLMs) is practically prohibitive. To ensure high training efficiency and stable optimization, we introduce two critical engineering optimizations: attention masking for the unconditioned forward pass, and a low-variance, unbiased Monte-Carlo estimator for the KL divergence. Visually-Unconditioned Distribution via Attention Masking. To obtain the unconditioned probability distribution π θ ( ⋅ | s t ) \pi_{\theta}(\cdot|s_{t}) , one naive approach is to replace the image with a zero-tensor (blank image) or remove the visual tokens from the input sequence entirely. However, altering the input sequence length fundamentally shifts the absolute positional encodings of the subsequent text tokens, introducing confounding variables that distort the pure measurement of visual dependency. Instead, we compute the unconditioned distribution by manipulating the causal attention mask during a secondary forward pass. Specifically, we set the attention mask corresponding to all visual tokens to False , effectively blinding the text tokens to the visual prefix while preserving their original positional indices. It avoids the overhead of re-tokenization and KV-cache structural modifications, allowing the unconditioned logits to be computed seamlessly using the exact same tensor shapes as the primary forward pass. Estimation of KL Divergence. Mathematically, the exact KL divergence at step t t requires computing a sum over the entire vocabulary 𝒱 \mathcal{V} (typically > 100 , 000 >100,000 tokens): D KL ​ ( π I ∥ π ∅ ) = ∑ x ∈ 𝒱 π I ​ ( x ) ​ log ⁡ π I ​ ( x ) π ∅ ​ ( x ) D_{\text{KL}}(\pi_{I}\parallel\pi_{\emptyset})=\sum_{x\in\mathcal{V}}\pi_{I}(x)\log\frac{\pi_{I}(x)}{\pi_{\emptyset}(x)} (10) where, for brevity, we denote the visually-conditioned policy as π I ​ ( x ) := π θ ​ ( x | s t , I ) \pi_{I}(x):=\pi_{\theta}(x|s_{t},I) and the unconditioned policy as π ∅ ​ ( x ) := π θ ​ ( x | s t ) \pi_{\emptyset}(x):=\pi_{\theta}(x|s_{t}) . In practice, explicitly computing this full sum incurs massive GPU memory footprint and I/O overhead. Furthermore, it aggregates the numerical noise from the long tail of highly improbable tokens, which can destabilize the dependency signal. To resolve this, following kl_approx , we approximate the expectation using Monte-Carlo sampling based solely on the generated token x t ∼ π I x_{t}\sim\pi_{I} . Let the probability ratio be defined as r = π ∅ ​ ( x t ) π I ​ ( x t ) r=\frac{\pi_{\emptyset}(x_{t})}{\pi_{I}(x_{t})} . We use ( r − 1 ) − log ⁡ r (r-1)-\log r as an unbiased, strictly non-negative, and low-variance estimator for KL divergence. Appendix B PGPO on the GRPO algorithm Model Geo3k MMK12 MathVerse DynaMath MathVision LogicVista MMMU-Pro MathVerse V \text{MathVerse}_{V} Avg. Qwen2.5-VL-3B 19.65 37.37 33.16 32.88 13.88 30.17 20.95 30.31 27.30 + GRPO 32.19 62.95 58.39 47.82 25.59 40.85 28.02 54.36 43.77 + DAPO 33.80 63.36 58.76 45.86 26.19 40.63 27.20 54.89 43.84 \rowcolor ourscolor + PGPO w/ GRPO 33.81 63.96 59.62 47.91 26.00 41.53 28.29 56.75 44.73 Table 7: Results on PGPO by applying it to GRPO (avg@8 acc %). To evaluate the generalizability of PGPO, we integrated it with GRPO. As shown in Table 7 , PGPO also improves GRPO’s accuracy, surpassing DAPO. This gain aligns with the improvement observed when applying PGPO to DAPO (Table 1 ), confirming that the performance enhancements are intrinsic to our visually-perceptive optimization strategy rather than artifacts of the underlying base policy gradient algorithm. Appendix C Detailed Settings for Empirical Analysis of Visual Dependency In Section 3.2 , we presented empirical analyses demonstrating the distributional sparsity of token visual dependency ( 𝒮 \mathcal{S} ), its strong correlation with semantic visual anchors, and its inverse relationship with multimodal hallucinations. To ensure full reproducibility and provide deeper insights into our methodology, this section details the experimental setups, data processing pipelines, and evaluation metrics used for these preliminary analyses. All empirical analyses were conducted using the Qwen2.5-VL-3B model. To ensure deterministic and reproducible trajectories, we employed greedy decoding for all text generation tasks. C.1 Distributional Sparsity and Content Token Analysis Dataset Selection. We utilized the vision-dominant subset of the MathVerse benchmark. This specific subset comprises problems where the visual information is mathematically indispensable, ensuring that a reliance on language priors alone cannot yield the correct answer. We processed the multi-choice test-mini split. Part-of-Speech (POS) Categorization. To analyze the discrepancy in visual dependency between different types of tokens, we employed the spaCy to perform POS tagging on the generated trajectories. Content Tokens were defined as tokens categorized under structural and semantic classes critical to reasoning: ‘NUM‘ (numerals), ‘NOUN‘ (nouns/geometric concepts), ‘VERB‘ (verbs), and ‘ADJ‘ (adjectives). Other Tokens are other remaining tokens. Tokens split into sub-words by the tokenizer inherited the POS tag of their parent word. As shown in Figure 2 (b), we computed the mean 𝒮 \mathcal{S} for all tokens within these respective macro-categories across all trajectories. As shown in Figure 5 , tokens with high visual dependence often carry critical visual semantics. Figure 5: Top 200 𝒮 \mathcal{S} tokens word cloud on all generated tokens of vision-dominant MathVerse. C.2 Visual Anchor Annotation To evaluate whether 𝒮 \mathcal{S} is associated with semantic visual grounding, we constructed an annotation pipeline to identify "Visual Anchors"—tokens highly requiring image observation for inference. Annotation. We used complete trajectories from the vision-dominant MathVerse generation set. Given the complexity of mathematical reasoning, we leveraged GPT-5 as a zero-shot annotator. GPT-5 was provided with the original image, the textual question, and the generated trajectory, and was prompted to extract specific words that act as visual anchors. Prompt for LLM annotation is: You are an expert annotator for multimodal mathematical reasoning. Given the following Image, Question, and the Model’s Generated Response, your task is to identify ’Visual Anchors’ within the response. A Visual Anchor is defined as a specific entity, number, spatial relationship, or geometric attribute mentioned in the text that CANNOT be properly deduced from the text alone and strictly requires extracting information from the image. Output your response as a strictly formatted JSON list of words/phrases extracted directly from the generation. Token Mapping. The extracted anchor phrases were string-matched back to the generated token sequence. Tokens comprising these phrases were labeled as Visual Anchor tokens , while all remaining tokens were labeled as Other tokens . The statistical significance of the 𝒮 \mathcal{S} differential between these two groups also was confirmed via a two-sample t-test ( p < 0.001 p<0.001 ), validating our core hypothesis that high 𝒮 \mathcal{S} corresponds to critical visual grounding. C.3 Hallucination Analysis on the CHAIR Benchmark To understand the relationship between visual dependency and factual correctness, we extended our analysis to the standard object hallucination benchmark, CHAIR (Caption Hallucination Assessment with Image Relevance) rohrbach2019objecthallucinationimagecaptioning . We sampled 5,000 images from the MSCOCO lin2015microsoftcococommonobjects 2014 validation set. For each image, we prompted Qwen2.5-VL-3B with: "Describe this image in detail." We employed the official CHAIR evaluation script, leveraging MSCOCO ground-truth object annotations (synsets), to assess the generated sequences. First, we parsed the generated caption to extract all mentioned objects using the standard CHAIR synonym dictionary. Subsequently, these extracted objects are cross-referenced against the image’s ground-truth annotations; tokens corresponding to objects present in the ground truth are classified as truthful, whereas those representing absent objects are designated as hallucinated. We isolated these two subsets of tokens across the 5,000 captions and computed their average 𝒮 \mathcal{S} values. The finding (Figure 2 (d)) that hallucinated tokens exhibit a significantly lower visual dependency provides evidence that multimodal hallucinations in LVLMs are largely driven by over-reliance on language priors rather than visual processing. This conclusion is consistent with existing research findings leng2023mitigatingobjecthallucinationslarge ; chen2024halcobjecthallucinationreduction ; ye2025claimmitigatingmultilingualobject ; li2025causaltracingobjectrepresentations ; li2025caicaptionsensitiveattentionintervention ; li2025unlockingmultilingualreasoningcapability ; yu2025rlaifvopensourceaifeedback ; zhong2024investigatingmitigatingmultimodalhallucination ; chen2025mprguibenchmarkingenhancingmultilingual . Appendix D Information-Theoretic Foundation of the Visual Dependency Metric In this appendix, we establish the theoretical soundness of using the Kullback-Leibler (KL) divergence to quantify visual dependency, denoted as 𝒮 ​ ( s t , I ) \mathcal{S}(s_{t},I) . Specifically, we demonstrate that the proposed formulation in Section 3.1 is mathematically equivalent to the state-specific conditional information gain provided by the visual input. D.1 Conditional Mutual Information in Autoregressive Generation In information theory, the reduction in uncertainty about a random variable X X given the observation of another random variable Y Y is measured by Mutual Information I ​ ( X ; Y ) I(X;Y) . When a prior context Z Z is already known, the additional information provided by Y Y is captured by the Conditional Mutual Information: I ​ ( X ; Y ∣ Z ) = H ​ ( X ∣ Z ) − H ​ ( X ∣ Y , Z ) , I(X;Y\mid Z)=H(X\mid Z)-H(X\mid Y,Z), (11) where H ​ ( ⋅ ) H(\cdot) denotes Shannon entropy. Within the context of a Large Vision-Language Model (LVLM), we define the following random variables: • O t ∈ 𝒱 O_{t}\in\mathcal{V} : The next token to be generated from the vocabulary 𝒱 \mathcal{V} . • S t = ( q , o < t ) S_{t}=(q,o_{ 0 c_{i}>0 or via token-level weights that do not sum to the sequence length ( ∑ t = 1 T i ω t ≠ T i \sum_{t=1}^{T_{i}}\omega_{t}\neq T_{i} ), this effectively scales the net advantage of the trajectory, yielding a modulated advantage A ~ i = c i ​ A i \tilde{A}_{i}=c_{i}A_{i} . Substituting A ~ i \tilde{A}_{i} into the standard policy gradient objective, the unclipped gradient estimator becomes: 𝐠 scaled = 1 G ​ ∑ i = 1 G A ~ i ​ ∑ t = 1 T i ∇ θ log ⁡ π θ ​ ( o i , t ∣ s i , t ) \displaystyle\mathbf{g}_{\text{scaled}}=\frac{1}{G}\sum_{i=1}^{G}\tilde{A}_{i}\sum_{t=1}^{T_{i}}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid s_{i,t}) = 1 G ​ ∑ i = 1 G ( c i ​ A i ) ​ ∑ t = 1 T i ∇ θ log ⁡ π θ ​ ( o i , t ∣ s i , t ) \displaystyle=\frac{1}{G}\sum_{i=1}^{G}(c_{i}A_{i})\sum_{t=1}^{T_{i}}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}\mid s_{i,t}) (21) The flaw of this unnormalized scaling lies in the destruction of the zero-mean baseline. With the introduction of the sequence-dependent coefficient c i c_{i} , the modulated advantages no longer sum to zero: μ scaled = 1 G ​ ∑ i = 1 G c i ​ A i ≠ 0 \mu_{\text{scaled}}=\frac{1}{G}\sum_{i=1}^{G}c_{i}A_{i}\neq 0 (22) This non-zero mean shift ( μ scaled ≠ 0 \mu_{\text{scaled}}\neq 0 ) may arbitrarily distorts the magnitude of the parameter updates, disrupting stable credit assignment. To prevent this, PGPO employs sum-preserving normalization Equation 7 . This mathematically guarantees that the total advantage mass distributed across the trajectory strictly equals T i ⋅ A i T_{i}\cdot A_{i} , thereby preserving the sequence-level baseline μ = 0 \mu=0 while effectively reallocating credit at the token level. F.2 Formal Proof: Asymptotic Quadratic Variance Inflation Induced by Mean Shifts To rigorously explain why preserving the zero-mean structure is important, we analyze the effect of adding a constant mean shift μ \mu to an advantage-weighted policy-gradient estimator. The key result is that, although such a shift does not change the estimator’s mean, it introduces an additional covariance term that grows quadratically in | μ | |\mu| and eventually dominates the estimator variance. Preliminaries and Definitions. Let ( Ω , ℱ , ℙ ) (\Omega,\mathcal{F},\mathbb{P}) be a probability space, and let τ ∼ π θ \tau\sim\pi_{\theta} denote a trajectory sampled from a policy π θ \pi_{\theta} with parameter θ ∈ ℝ d \theta\in\mathbb{R}^{d} . Write 𝔼 ​ [ ⋅ ] := 𝔼 τ ∼ π θ ​ [ ⋅ ] \mathbb{E}[\cdot]:=\mathbb{E}_{\tau\sim\pi_{\theta}}[\cdot] . Definition 1 (Score Function). The trajectory score function is 𝐠 ​ ( τ ) := ∇ θ log ⁡ π θ ​ ( τ ) . \mathbf{g}(\tau):=\nabla_{\theta}\log\pi_{\theta}(\tau). Under standard regularity conditions, 𝔼 ​ [ 𝐠 ​ ( τ ) ] = 𝟎 . \mathbb{E}[\mathbf{g}(\tau)]=\mathbf{0}. Definition 2 (Fisher Information Matrix). The Fisher Information Matrix (FIM) is 𝐅 := 𝔼 ​ [ 𝐠 ​ ( τ ) ​ 𝐠 ​ ( τ ) ⊤ ] ∈ ℝ d × d . \mathbf{F}:=\mathbb{E}\!\left[\mathbf{g}(\tau)\mathbf{g}(\tau)^{\top}\right]\in\mathbb{R}^{d\times d}. Hence 𝐅 ⪰ 0 \mathbf{F}\succeq 0 . If the policy is non-degenerate in all parameter directions, then 𝐅 ≻ 0 \mathbf{F}\succ 0 , so λ min ​ ( 𝐅 ) > 0 \lambda_{\min}(\mathbf{F})>0 . Definition 3 (Base and Shifted Estimators). Let A ∗ ​ ( τ ) A^{*}(\tau) be a scalar random variable satisfying 𝔼 ​ [ A ∗ ​ ( τ ) ] = 0 . \mathbb{E}[A^{*}(\tau)]=0. Define the base estimator 𝐠 ^ 0 := A ∗ ​ ( τ ) ​ 𝐠 ​ ( τ ) , \hat{\mathbf{g}}_{0}:=A^{*}(\tau)\mathbf{g}(\tau), and, for a constant shift μ ∈ ℝ \mu\in\mathbb{R} , define the shifted estimator 𝐠 ^ μ := ( A ∗ ​ ( τ ) + μ ) ​ 𝐠 ​ ( τ ) . \hat{\mathbf{g}}_{\mu}:=(A^{*}(\tau)+\mu)\mathbf{g}(\tau). Observe that 𝔼 ​ [ 𝐠 ^ μ ] = 𝔼 ​ [ A ∗ ​ ( τ ) ​ 𝐠 ​ ( τ ) ] + μ ​ 𝔼 ​ [ 𝐠 ​ ( τ ) ] = 𝔼 ​ [ 𝐠 ^ 0 ] . \mathbb{E}[\hat{\mathbf{g}}_{\mu}]=\mathbb{E}[A^{*}(\tau)\mathbf{g}(\tau)]+\mu\,\mathbb{E}[\mathbf{g}(\tau)]=\mathbb{E}[\hat{\mathbf{g}}_{0}]. (23) Therefore, adding a constant shift μ \mu does not change the estimator mean, but it may change its covariance. Assumption 1 (Finite Moments). Assume all expectations below are well-defined and finite. In particular, define 𝐂 := 𝔼 ​ [ A ∗ ​ ( τ ) ​ 𝐠 ​ ( τ ) ​ 𝐠 ​ ( τ ) ⊤ ] , \mathbf{C}:=\mathbb{E}\!\left[A^{*}(\tau)\mathbf{g}(\tau)\mathbf{g}(\tau)^{\top}\right], (24) and assume ‖ 𝐂 ‖ 2 < ∞ \|\mathbf{C}\|_{2}<\infty and tr ​ ( 𝐂 ) \mathrm{tr}(\mathbf{C}) is finite. Proposition F.1 (Covariance inflation due to a mean shift) . Under Assumption 1, Cov ​ ( 𝐠 ^ μ ) = Cov ​ ( 𝐠 ^ 0 ) + μ 2 ​ 𝐅 + 2 ​ μ ​ 𝐂 . \mathrm{Cov}(\hat{\mathbf{g}}_{\mu})=\mathrm{Cov}(\hat{\mathbf{g}}_{0})+\mu^{2}\mathbf{F}+2\mu\mathbf{C}. (25) Consequently: 1. Total variance inflation. If | μ | > 2 ​ | tr ​ ( 𝐂 ) | tr ​ ( 𝐅 ) , |\mu|>\frac{2|\mathrm{tr}(\mathbf{C})|}{\mathrm{tr}(\mathbf{F})}, (26) then tr ​ ( Cov ​ ( 𝐠 ^ μ ) ) > tr ​ ( Cov ​ ( 𝐠 ^ 0 ) ) . \mathrm{tr}\!\left(\mathrm{Cov}(\hat{\mathbf{g}}_{\mu})\right)>\mathrm{tr}\!\left(\mathrm{Cov}(\hat{\mathbf{g}}_{0})\right). (27) 2. Strict inflation in Lowner order. If 𝐅 ≻ 0 \mathbf{F}\succ 0 and | μ | > 2 ​ ‖ 𝐂 ‖ 2 λ min ​ ( 𝐅 ) , |\mu|>\frac{2\|\mathbf{C}\|_{2}}{\lambda_{\min}(\mathbf{F})}, (28) then Cov ​ ( 𝐠 ^ μ ) ≻ Cov ​ ( 𝐠 ^ 0 ) . \mathrm{Cov}(\hat{\mathbf{g}}_{\mu})\succ\mathrm{Cov}(\hat{\mathbf{g}}_{0}). (29) 3. Asymptotic quadratic dominance. As | μ | → ∞ |\mu|\to\infty , Cov ​ ( 𝐠 ^ μ ) − Cov ​ ( 𝐠 ^ 0 ) = μ 2 ​ 𝐅 + 𝒪 ​ ( | μ | ) , \mathrm{Cov}(\hat{\mathbf{g}}_{\mu})-\mathrm{Cov}(\hat{\mathbf{g}}_{0})=\mu^{2}\mathbf{F}+\mathcal{O}(|\mu|), (30) so the covariance penalty is asymptotically dominated by the quadratic term μ 2 ​ 𝐅 \mu^{2}\mathbf{F} . Proof. Since 𝔼 ​ [ 𝐠 ^ μ ] = 𝔼 ​ [ 𝐠 ^ 0 ] \mathbb{E}[\hat{\mathbf{g}}_{\mu}]=\mathbb{E}[\hat{\mathbf{g}}_{0}] , the covariance difference equals the difference of the second-moment matrices: Δ ​ Cov : \displaystyle\Delta\mathrm{Cov}: = Cov ​ ( 𝐠 ^ μ ) − Cov ​ ( 𝐠 ^ 0 ) \displaystyle=\mathrm{Cov}(\hat{\mathbf{g}}_{\mu})-\mathrm{Cov}(\hat{\mathbf{g}}_{0}) = 𝔼 ​ [ 𝐠 ^ μ ​ 𝐠 ^ μ ⊤ ] − 𝔼 ​ [ 𝐠 ^ 0 ​ 𝐠 ^ 0 ⊤ ] . \displaystyle=\mathbb{E}[\hat{\mathbf{g}}_{\mu}\hat{\mathbf{g}}_{\mu}^{\top}]-\mathbb{E}[\hat{\mathbf{g}}_{0}\hat{\mathbf{g}}_{0}^{\top}]. (31) Expanding 𝐠 ^ μ = ( A ∗ + μ ) ​ 𝐠 \hat{\mathbf{g}}_{\mu}=(A^{*}+\mu)\mathbf{g} , we obtain 𝔼 ​ [ 𝐠 ^ μ ​ 𝐠 ^ μ ⊤ ] = \displaystyle\mathbb{E}[\hat{\mathbf{g}}_{\mu}\hat{\mathbf{g}}_{\mu}^{\top}]= 𝔼 ​ [ ( A ∗ ) 2 ​ 𝐠𝐠 ⊤ ] + 2 ​ μ ​ 𝔼 ​ [ A ∗ ​ 𝐠𝐠 ⊤ ] + \displaystyle\mathbb{E}\!\left[(A^{*})^{2}\mathbf{g}\mathbf{g}^{\top}\right]+2\mu\,\mathbb{E}\!\left[A^{*}\mathbf{g}\mathbf{g}^{\top}\right]+ μ 2 ​ 𝔼 ​ [ 𝐠𝐠 ⊤ ] . \displaystyle\mu^{2}\,\mathbb{E}\!\left[\mathbf{g}\mathbf{g}^{\top}\right]. (32) Recognizing the first term as 𝔼 ​ [ 𝐠 ^ 0 ​ 𝐠 ^ 0 ⊤ ] \mathbb{E}[\hat{\mathbf{g}}_{0}\hat{\mathbf{g}}_{0}^{\top}] , the second as 2 ​ μ ​ 𝐂 2\mu\mathbf{C} , and the third as μ 2 ​ 𝐅 \mu^{2}\mathbf{F} , we get Δ ​ Cov = μ 2 ​ 𝐅 + 2 ​ μ ​ 𝐂 . \Delta\mathrm{Cov}=\mu^{2}\mathbf{F}+2\mu\mathbf{C}. (33) For the trace statement, tr ​ ( Δ ​ Cov ) = μ 2 ​ tr ​ ( 𝐅 ) + 2 ​ μ ​ tr ​ ( 𝐂 ) . \mathrm{tr}(\Delta\mathrm{Cov})=\mu^{2}\mathrm{tr}(\mathbf{F})+2\mu\,\mathrm{tr}(\mathbf{C}). (34) Using | 2 ​ μ ​ tr ​ ( 𝐂 ) | ≤ 2 ​ | μ | ​ | tr ​ ( 𝐂 ) | |2\mu\,\mathrm{tr}(\mathbf{C})|\leq 2|\mu|\,|\mathrm{tr}(\mathbf{C})| , we have tr ​ ( Δ ​ Cov ) ≥ | μ | ​ ( | μ | ​ tr ​ ( 𝐅 ) − 2 ​ | tr ​ ( 𝐂 ) | ) . \mathrm{tr}(\Delta\mathrm{Cov})\geq|\mu|\Big(|\mu|\mathrm{tr}(\mathbf{F})-2|\mathrm{tr}(\mathbf{C})|\Big). (35) Hence tr ​ ( Δ ​ Cov ) > 0 \mathrm{tr}(\Delta\mathrm{Cov})>0 whenever | μ | > 2 ​ | tr ​ ( 𝐂 ) | tr ​ ( 𝐅 ) . |\mu|>\frac{2|\mathrm{tr}(\mathbf{C})|}{\mathrm{tr}(\mathbf{F})}. (36) For the Lowner-order statement, let 𝐯 \mathbf{v} be any unit vector. Then 𝐯 ⊤ ​ Δ ​ Cov ​ 𝐯 = μ 2 ​ 𝐯 ⊤ ​ 𝐅𝐯 + 2 ​ μ ​ 𝐯 ⊤ ​ 𝐂𝐯 . \mathbf{v}^{\top}\Delta\mathrm{Cov}\,\mathbf{v}=\mu^{2}\mathbf{v}^{\top}\mathbf{F}\mathbf{v}+2\mu\,\mathbf{v}^{\top}\mathbf{C}\mathbf{v}. (37) By Rayleigh-quotient bounds, 𝐯 ⊤ ​ 𝐅𝐯 ≥ λ min ​ ( 𝐅 ) , | 𝐯 ⊤ ​ 𝐂𝐯 | ≤ ‖ 𝐂 ‖ 2 . \mathbf{v}^{\top}\mathbf{F}\mathbf{v}\geq\lambda_{\min}(\mathbf{F}),\qquad|\mathbf{v}^{\top}\mathbf{C}\mathbf{v}|\leq\|\mathbf{C}\|_{2}. (38) Therefore 𝐯 ⊤ ​ Δ ​ Cov ​ 𝐯 \displaystyle\mathbf{v}^{\top}\Delta\mathrm{Cov}\,\mathbf{v} ≥ μ 2 ​ λ min ​ ( 𝐅 ) − 2 ​ | μ | ​ ‖ 𝐂 ‖ 2 \displaystyle\geq\mu^{2}\lambda_{\min}(\mathbf{F})-2|\mu|\|\mathbf{C}\|_{2} = | μ | ​ ( | μ | ​ λ min ​ ( 𝐅 ) − 2 ​ ‖ 𝐂 ‖ 2 ) . \displaystyle=|\mu|\Big(|\mu|\lambda_{\min}(\mathbf{F})-2\|\mathbf{C}\|_{2}\Big). (39) If | μ | > 2 ​ ‖ 𝐂 ‖ 2 λ min ​ ( 𝐅 ) , |\mu|>\frac{2\|\mathbf{C}\|_{2}}{\lambda_{\min}(\mathbf{F})}, (40) then 𝐯 ⊤ ​ Δ ​ Cov ​ 𝐯 > 0 \mathbf{v}^{\top}\Delta\mathrm{Cov}\,\mathbf{v}>0 for every unit vector 𝐯 \mathbf{v} , which implies Δ ​ Cov ≻ 0 . \Delta\mathrm{Cov}\succ 0. (41) Finally, since Δ ​ Cov = μ 2 ​ 𝐅 + 2 ​ μ ​ 𝐂 , \Delta\mathrm{Cov}=\mu^{2}\mathbf{F}+2\mu\mathbf{C}, (42) the linear term is lower order than the quadratic term as | μ | → ∞ |\mu|\to\infty , proving Δ ​ Cov = μ 2 ​ 𝐅 + 𝒪 ​ ( | μ | ) . \Delta\mathrm{Cov}=\mu^{2}\mathbf{F}+\mathcal{O}(|\mu|). (43) ∎ Conclusion. In very high-dimensional models such as LVLMs, the Fisher term 𝐅 \mathbf{F} can be large in aggregate, so the quadratic penalty μ 2 ​ 𝐅 \mu^{2}\mathbf{F} may become significant even for modest | μ | |\mu| . While the exact threshold depends on how both 𝐅 \mathbf{F} and 𝐂 \mathbf{C} scale with dimension, the proposition makes clear that non-zero mean shifts become increasingly undesirable in large models because they inject variance in every parameter direction captured by the FIM. This degrades gradient signal-to-noise ratio and can materially destabilize optimization. This theoretical formulation strictly proves the necessity of maintaining a mathematically exact zero-mean advantage. It serves as the foundational justification for the sum-preserving normalization mechanism proposed in our PGPO framework (Equation 7 ), ensuring that the modulated advantages fundamentally conserve the μ = 0 \mu=0 property while redistributing credit effectively. Appendix G Monotonicity Analysis of ω ~ t \tilde{\omega}_{t} Proposition G.1 (Monotonicity of Credit Assignment). Given a generated sequence, let 𝒮 t \mathcal{S}_{t} be the visual dependency score of the t t -th token. Assuming the scores of all other tokens in the sequence remain constant, the final modulated weight ω ~ t \tilde{\omega}_{t} is monotonically non-decreasing with respect to 𝒮 t \mathcal{S}_{t} . Proof. We prove this proposition by systematically analyzing the monotonicity of the four sequential transformations applied to 𝒮 t \mathcal{S}_{t} : logarithmic compression, min-max normalization, threshold gating, and sum-preserving normalization. Step 1: Logarithmic Compression. The first transformation applies a logarithmic dampening: 𝒮 ~ t = log ⁡ ( 1 + 𝒮 t ) \tilde{\mathcal{S}}_{t}=\log(1+\mathcal{S}_{t}) . Assuming 𝒮 t ≥ 0 \mathcal{S}_{t}\geq 0 , the partial derivative is: ∂ 𝒮 ~ t ∂ 𝒮 t = 1 1 + 𝒮 t > 0 \frac{\partial\tilde{\mathcal{S}}_{t}}{\partial\mathcal{S}_{t}}=\frac{1}{1+\mathcal{S}_{t}}>0 (44) Thus, 𝒮 ~ t \tilde{\mathcal{S}}_{t} is strictly monotonically increasing with respect to 𝒮 t \mathcal{S}_{t} . Step 2: Min-Max Normalization. The second transformation standardizes the score: I t = 𝒮 ~ t − min j ⁡ 𝒮 ~ j max j ⁡ 𝒮 ~ j − min j ⁡ 𝒮 ~ j + ϵ I_{t}=\frac{\tilde{\mathcal{S}}_{t}-\min_{j}\tilde{\mathcal{S}}_{j}}{\max_{j}\tilde{\mathcal{S}}_{j}-\min_{j}\tilde{\mathcal{S}}_{j}+\epsilon} (45) Because the min \min and max \max operators evaluate the entire sequence, altering 𝒮 ~ t \tilde{\mathcal{S}}_{t} may shift the sequence extrema. Let m = min j ≠ t ⁡ 𝒮 ~ j m=\min_{j\neq t}\tilde{\mathcal{S}}_{j} and M = max j ≠ t ⁡ 𝒮 ~ j M=\max_{j\neq t}\tilde{\mathcal{S}}_{j} , which act as fixed constants with respect to 𝒮 ~ t \tilde{\mathcal{S}}_{t} . We analyze I t I_{t} across three disjoint intervals: 1. When 𝒮 ~ t ≤ m \tilde{\mathcal{S}}_{t}\leq m (Token t t is the minimum): min j ⁡ 𝒮 ~ j = 𝒮 ~ t \min_{j}\tilde{\mathcal{S}}_{j}=\tilde{\mathcal{S}}_{t} . The numerator becomes 𝒮 ~ t − 𝒮 ~ t = 0 \tilde{\mathcal{S}}_{t}-\tilde{\mathcal{S}}_{t}=0 , yielding I t = 0 I_{t}=0 . The derivative is 0 . 2. When m < 𝒮 ~ t < M m<\tilde{\mathcal{S}}_{t} 0 \frac{\partial I_{t}}{\partial\tilde{\mathcal{S}}_{t}}=\frac{1}{M-m+\epsilon}>0 (46) 3. When 𝒮 ~ t ≥ M \tilde{\mathcal{S}}_{t}\geq M (Token t t is the maximum): Here, max j ⁡ 𝒮 ~ j = 𝒮 ~ t \max_{j}\tilde{\mathcal{S}}_{j}=\tilde{\mathcal{S}}_{t} and min j ⁡ 𝒮 ~ j = m \min_{j}\tilde{\mathcal{S}}_{j}=m . Applying the quotient rule: ∂ I t ∂ 𝒮 ~ t \displaystyle\frac{\partial I_{t}}{\partial\tilde{\mathcal{S}}_{t}} = ∂ ∂ 𝒮 ~ t ​ ( 𝒮 ~ t − m 𝒮 ~ t − m + ϵ ) \displaystyle=\frac{\partial}{\partial\tilde{\mathcal{S}}_{t}}\left(\frac{\tilde{\mathcal{S}}_{t}-m}{\tilde{\mathcal{S}}_{t}-m+\epsilon}\right) = ϵ ( 𝒮 ~ t − m + ϵ ) 2 > 0 \displaystyle=\frac{\epsilon}{(\tilde{\mathcal{S}}_{t}-m+\epsilon)^{2}}>0 (47) Across all intervals, ∂ I t ∂ 𝒮 ~ t ≥ 0 \frac{\partial I_{t}}{\partial\tilde{\mathcal{S}}_{t}}\geq 0 . Hence, I t I_{t} is monotonically non-decreasing with respect to 𝒮 ~ t \tilde{\mathcal{S}}_{t} . Step 3: Threshold-Gating Function. The piecewise gating function computes the base weight ω t = ω ​ ( I t ) \omega_{t}=\omega(I_{t}) . We analyze its piecewise derivatives and boundary continuity: 1. When I t < τ I_{t}<\tau : ω t = I t τ + ϵ ⟹ ∂ ω t ∂ I t = 1 τ + ϵ > 0 \omega_{t}=\frac{I_{t}}{\tau+\epsilon}\implies\frac{\partial\omega_{t}}{\partial I_{t}}=\frac{1}{\tau+\epsilon}>0 . 2. When I t ≥ τ I_{t}\geq\tau : ω t = 1 + β ​ I t − τ 1 − τ + ϵ ⟹ ∂ ω t ∂ I t = β 1 − τ + ϵ ≥ 0 \omega_{t}=1+\beta\frac{I_{t}-\tau}{1-\tau+\epsilon}\implies\frac{\partial\omega_{t}}{\partial I_{t}}=\frac{\beta}{1-\tau+\epsilon}\geq 0 (strictly positive for β > 0 \beta>0 ). 3. Boundary at I t = τ I_{t}=\tau : The right-hand evaluation is ω ​ ( τ ) = 1 \omega(\tau)=1 . The left-hand limit is lim I t → τ − ω ​ ( I t ) = τ τ + ϵ \lim_{I_{t}\to\tau^{-}}\omega(I_{t})=\frac{\tau}{\tau+\epsilon} . Since ϵ > 0 \epsilon>0 , the left limit is strictly less than 1 1 . This positive jump discontinuity guarantees that the function strictly preserves the non-decreasing property as it crosses the threshold. Consequently, ω t \omega_{t} is monotonically non-decreasing with respect to I t I_{t} . Step 4: Sum-Preserving Renormalization. The final step enforces mass conservation: ω ~ t = ω t ⋅ T ∑ j = 1 T ω j = T ⋅ ω t ω t + S \tilde{\omega}_{t}=\omega_{t}\cdot\frac{T}{\sum_{j=1}^{T}\omega_{j}}=\frac{T\cdot\omega_{t}}{\omega_{t}+S} (48) where T T is the sequence length and S = ∑ j ≠ t ω j > 0 S=\sum_{j\neq t}\omega_{j}>0 is a fixed positive constant. Taking the derivative with respect to ω t \omega_{t} : ∂ ω ~ t ∂ ω t = T ⋅ S ( ω t + S ) 2 > 0 \frac{\partial\tilde{\omega}_{t}}{\partial\omega_{t}}=\frac{T\cdot S}{(\omega_{t}+S)^{2}}>0 (49) Thus, ω ~ t \tilde{\omega}_{t} is strictly monotonically increasing with respect to ω t \omega_{t} . ∎ Conclusion. The overall mapping from 𝒮 t \mathcal{S}_{t} to ω ~ t \tilde{\omega}_{t} is a composition of functions that are differentiable and non-decreasing almost everywhere, with any discontinuities being strictly positive jumps. By the properties of monotonic compositions, the chain strictly preserves order. Therefore, ω ~ t \tilde{\omega}_{t} is monotonically non-decreasing with respect to 𝒮 t \mathcal{S}_{t} . In the context of credit assignment, this mathematical property guarantees that if a specific token demonstrates stronger visual grounding (i.e., higher 𝒮 t \mathcal{S}_{t} ), the PGPO framework will reliably assign it an equal or greater advantage modulation weight ω ~ t \tilde{\omega}_{t} , ensuring stable and interpretable optimization. Appendix H Rank-Preserving Property of ω ~ t \tilde{\omega}_{t} Proposition H.1 (Intra-Sequence Rank Preservation). Given a generated sequence, the PGPO modulation mechanism strictly preserves the relative ordinal importance of the tokens. Specifically, for any two tokens A A and B B within the same sequence, if their raw visual dependency scores satisfy 𝒮 A > 𝒮 B \mathcal{S}_{A}>\mathcal{S}_{B} , then their final modulated weights satisfy ω ~ A > ω ~ B \tilde{\omega}_{A}>\tilde{\omega}_{B} . Proof. Let A A and B B be two distinct tokens in a fixed sequence such that 𝒮 A > 𝒮 B \mathcal{S}_{A}>\mathcal{S}_{B} . Because both tokens belong to the same sequence, the sequence-level statistics—specifically the unnormalized minimum m m , maximum M M , and the pre-normalized sum S total = ∑ j = 1 T ω j S_{\text{total}}=\sum_{j=1}^{T}\omega_{j} —are fixed, shared constants. We evaluate the strict preservation of the inequality through the four transformations: 1. Logarithmic Compression: Since f ​ ( x ) = log ⁡ ( 1 + x ) f(x)=\log(1+x) is strictly monotonically increasing for x ≥ 0 x\geq 0 , we have: 𝒮 A > 𝒮 B ⟹ 𝒮 ~ A > 𝒮 ~ B \mathcal{S}_{A}>\mathcal{S}_{B}\implies\tilde{\mathcal{S}}_{A}>\tilde{\mathcal{S}}_{B} (50) 2. Min-Max Normalization: The normalization applies a linear transformation: I A = 𝒮 ~ A − m M − m + ϵ , I B = 𝒮 ~ B − m M − m + ϵ I_{A}=\frac{\tilde{\mathcal{S}}_{A}-m}{M-m+\epsilon},\quad I_{B}=\frac{\tilde{\mathcal{S}}_{B}-m}{M-m+\epsilon} (51) Because the denominator ( M − m + ϵ ) > 0 (M-m+\epsilon)>0 is identical for both tokens, the order is strictly preserved: I A > I B I_{A}>I_{B} . 3. Threshold-Gating Function: For β > 0 \beta>0 , the piecewise function ω ​ ( I ) \omega(I) consists of two linear segments with strictly positive slopes, connected by a positive upward jump at I = τ I=\tau . Because the function is globally strictly monotonically increasing across its domain [ 0 , 1 ] [0,1] , we obtain: I A > I B ⟹ ω A > ω B I_{A}>I_{B}\implies\omega_{A}>\omega_{B} (52) 4. Sum-Preserving Renormalization: The final step applies sequence-level scaling: ω ~ A = ω A ⋅ T S total , ω ~ B = ω B ⋅ T S total \tilde{\omega}_{A}=\omega_{A}\cdot\frac{T}{S_{\text{total}}},\quad\tilde{\omega}_{B}=\omega_{B}\cdot\frac{T}{S_{\text{total}}} (53) Since the scalar multiplier T S total > 0 \frac{T}{S_{\text{total}}}>0 is a shared constant, the inequality is maintained: ω ~ A > ω ~ B \tilde{\omega}_{A}>\tilde{\omega}_{B} . ∎ Conclusion. Through all sequential operations, the relationship 𝒮 A > 𝒮 B ⟹ ω ~ A > ω ~ B \mathcal{S}_{A}>\mathcal{S}_{B}\implies\tilde{\omega}_{A}>\tilde{\omega}_{B} holds strictly true. Therefore, the proposed modulation mechanism mathematically guarantees consistency between a token’s empirical visual grounding and its allocated advantage weighting. Appendix I Analysis of the Training Dataset We use ViRL39K wang2025vl as the sole training source for reinforcement learning. Its construction matches our objective of learning dependable multimodal reasoning under verifiable supervision. Breadth of problem domains. The dataset contains roughly 39K instances spanning mathematics, physics, chemistry, biology, and chart-centric interpretation. This cross-domain coverage exposes the policy to heterogeneous reasoning patterns and reduces over-specialization to narrow distributions. Deterministic rewardability. Each synthesized sample is paired with a unique, machine-checkable reference answer. This property is central to Reinforcement Learning with Verifiable Rewards (RLVR), where stable optimization depends on objective and automated reward assignment rather than subjective model-as-judge scoring. Appendix J Analysis of Evaluation Benchmarks Our evaluation protocol uses seven benchmarks to measure complementary abilities: domain-intensive mathematical reasoning, general logic, and cross-discipline multimodal understanding. • DynaMath zou2024dynamath : tests robustness by programmatically perturbing numeric values and functional plots in seed questions, helping distinguish memorization from true generalization. We evaluate on sample variant1 . • Geo3k lu2021inter : a large high-school geometry benchmark with formal annotations, suitable for evaluating symbolic and interpretable solution behavior. • MathVerse zhang2024mathverse : provides six controlled variants that shift information between text and diagram, enabling fine-grained analysis of visual reliance. We use the test-mini multi-choice subset. • MATH-Vision wang2024measuring : sourced from authentic contests (e.g., AMC, Math Kangaroo), spanning 16 subjects and five difficulty levels. We report results on its full test split. • MMK12 meng2025mm : focuses on K–12 multimodal math and is used to assess foundational reasoning competence. • LogicVista xiao2024logicvista : evaluates inductive, deductive, numerical, spatial, and mechanical reasoning across diverse visual formats, offering a broad test of general logical cognition. • MMMU-Pro yue2024mmmu : an upgraded MMMU variant designed to suppress text-only shortcuts via stronger option design and vision-centric settings, thereby better testing joint visual-textual reasoning in academic contexts. Appendix K LLM Usage Statement In preparing this manuscript, we used a large language model (LLM) solely to refine language expression, such as correcting grammar and improving sentence fluency. It was not involved in developing the scientific ideas, conducting the analyses, or producing the core technical content. The authors retain full responsibility for all intellectual contributions and for the final version of the paper. Appendix L Case Study To illustrate how PGPO improves reasoning, we present three representative case studies. We contrast typical failure patterns of the baseline with the correct reasoning paths produced by PGPO-7B. In all three cases, PGPO-7B answers correctly in all eight sampled generations, demonstrating strong stability and consistency. These examples highlight the benefit of token-level credit assignment. While a uniform training signal can cause key errors in visual understanding or logical inference, PGPO emphasizes visually grounded critical tokens and guides reasoning toward the correct answer. Case 1 Question: As shown in the figure, If the line b is parallel to the line a, the line b can be rotated () clockwise around the point A Source: MathVerse, id: 74 Choices: (A) 70°  (B) 50°  (C) 30°  (D) 20° Ground truth: (D) 20° DAPO: Given that line b is parallel to line a , and an angle measure between them is a given exterior angle at A. Specifically, the given angle adjacent to ∠ \angle 1 between line b and the horizontal line is 70°. Since b ∥ \parallel a, the angle ∠ \angle 1 can be calculated using the supplementary relationship with the adjacent given angle. So we need to find the rotation angle in order to make b ∥ \parallel a. Here the supplementary relation: ∠ \angle 1 + supplementary to it = 180°; So rotation = 180° - 70° ; The correct answer is: B Wrong PGPO (Ours): We are required to determine how much line b can rotate clockwise around point A for the lines to maintain the parallel condition. Identify the angles: The angle at point A where line b intersects with a horizontal line is given 70°. Understand the rotation requirement: We know currently angle 1 is 70°. For b to parallel to a, rotation is needed. The current angle effectively dictates that the rotation needed is to move from 70° to a position keeping b parallel to a. Identify angle at rotation: Given current alignment requires b shifted from 70° configuration: Notice the perpendicular creating 90°, angle rotation at A: Since it currently reads 70°, and b needs to realign, rotation = 90° - 70° = 20° . Thus, b needs to rotate 20° clockwise around A. The correct answer is: D Correct Case 2 Question: (In the image) Source: MMMU-Pro, id: 19 Choices: (In the image) Ground truth: (E) Skeletal structure DAPO: The image shows a floor plan with a grid system, where there are supporting columns or beams at regular intervals. This kind of arrangement suggests a structure where vertical posts (columns) are connected horizontally by lintels (beams). Given this layout, the architectural system that fits is the Post and lintel system , which is typically used when there’s a need to support a structure with evenly distributed vertical posts and horizontal lintels. So, among the given options: (A) Arcuated system - deals with curving structures. (B) Cantilevering - involves unsupported overhang. (C) Tensile structure - relies on cables for support. (D) Post and lintel system - aligns perfectly with evenly distributed supports shown. (E) Skeletal structure - involves a framework. (F) Groin vaults - require pointed arches connecting horizontal and vertical systems. (G) Shell structure - involves curved exterior forms. (H) Fan vaults - are elaborately used in medieval architecture. (I) Trabeated system - resembles the post-and-lintel system. The correct answer is: D Wrong PGPO (Ours): The image shows a floor plan with a regular grid of columns and beams. This suggests a structure where supports are aligned in a grid, which is typical for a skeletal structure where the main load-bearing elements are vertical columns and horizontal beams in a grid pattern to support the floor and roof. There is no indication of arches, vaults, or any specific architectural system mentioned in options (A, F, H, I, J). The regular column spacing and arrangement are indicative of a skeletal structure where elements are connected at joints to form a framework that supports the building’s structure. Therefore, the most fitting option here is (E) Skeletal structure. The correct answer is: E Correct Case 3 Question: (In the image) Source: MMMU-Pro, id: 22 Choices: (In the image) Ground truth: (H) A book cover DAPO: The image shows a beautifully detailed piece that appears to be a flat, decorative work adorned with gemstones and a central figure, suggesting it is meant for display. Given the ornate and decorative style, along with the inclusion of a central iconographic element (the figure of Jesus on the cross), this item likely served a ceremonial or devotional purpose in a Christian context . Reviewing the options: (A) An icon: Icons are religious artworks primarily used as objects of veneration, matching the devotional and symbolic nature of the artwork. (B) An altarpiece: Typically a three-dimensional piece, this seems more like a flat display. (C) A sculpture: Not applicable because the image isn’t three-dimensional. (D), (F), (J): These are also three-dimensional forms, unsuitable for the image. (E), (G), (H), (I): These forms are not as typical of elaborate decorative flat devotional pieces adorned with jewelry. Thus, the artwork’s nature as a decorative yet religious symbol points toward (A) an icon. The correct answer is: A Wrong PGPO (Ours): The image shows a decorative, gem-encrusted piece with a central religious icon of a crucifix, surrounded by ornamental details. This type of artifact is typically used as a cover for a book, especially a holy book or a Bible , given its stylistic and symbolic representation. An icon typically refers to a representation of a sacred subject, often used in Christian religious contexts, which matches the imagery on the piece. The luxurious nature with gems and the central iconography suggest it serves as a protective and valuable cover, often for religious texts like the Bible. Given the options: (A) an icon: This usually refers to a small religious image or painting, not necessarily the work shown here. (B) an altarpiece: Altarpieces are larger and mounted on an altar, which doesn’t match the size and style of the object. (C) a sculpture: The piece is not a three-dimensional sculpture. (D) a vase design: The piece is not designed to be a vase. (E) a door ornament: This piece is not a door ornament. (F) a fresco: Frescoes are painted on walls, which this is not. (G) a tapestry: While it has ornate detail, it seems too precious to be a tapestry. (H) a book cover: The ornamented and iconographic style fits a book cover. (I) a purse lid: This piece is decorative and religious, not a purse lid. (J) a mural: Murals are typically on larger surfaces like walls, not these kinds of objects. The most fitting description here is a book cover because of its rich iconographic nature and the way it would be used to cover a book. The correct answer is: H Correct BETA