总览 / 概念 / Modality Preference of Omni-Modal LLMs
↗ 源 Markdown
概念
multimodal alignment benchmark architecture
创建2026-04-27
更新2026-04-27
阅读量级6 分钟
来源2604.16902
概念导读

When processing multimodal inputs, models often implicitly assign unequal weights to different modalities. In traditional Vision-Language Models (VLMs), this is broadly perceived as "text-dominance bias." However, for native omni-modal LLMs (OLLMs) that integrate text, image, audio, and video in a unified representation space, how they navigate internal moda...

  1. Core Problem
  2. Technical Solution
  3. Tri-Modal Conflict Framework
  4. Data Construction
  5. Modality Selection Rate (MSR)
  6. Layer-Wise Probing
  7. Key Findings
  8. Finding 1: Visual Preference Dominance

Modality Preference of Omni-Modal LLMs: Beyond Text-Dominance

Core Problem

When processing multimodal inputs, models often implicitly assign unequal weights to different modalities. In traditional Vision-Language Models (VLMs), this is broadly perceived as "text-dominance bias." However, for native omni-modal LLMs (OLLMs) that integrate text, image, audio, and video in a unified representation space, how they navigate internal modality competition remains a fundamentally unresolved black box.

Uncontrolled modality bias is a primary catalyst for cross-modal hallucinations, where the model fabricates responses based on its preferred modality while ignoring factual signals from others.

Technical Solution

The paper establishes a systematic framework to quantify, analyze, and leverage modality preferences in OLLMs.

1. Tri-Modal Conflict Framework

Problem Formulation: Given an OLLM O and a query with three modalities {m_txt, m_vis, m_aud} where every pair conveys semantically contradictory information, the model's output must align with exactly one modality.

Preference definition:

P(ŷ ~ m_i | conflict(m_txt, m_vis, m_aud))

This conditional probability measures the tendency to rely on modality m_i when it conflicts with others.

2. Data Construction

Built on the Perception subset of XModBench: - Categorized into 6 semantic categories: Animals, Human Activities, Musical Instruments/Music, Home Appliances/Machinery, Vehicles/Traffic, Nature/Environmental Sounds - Constructed conflict triplets (x_i^T, x_j^I, x_k^A) where each modality comes from a different category (c_i ≠ c_j ≠ c_k) - All 20 valid category triplets (6 choose 3) with balanced sampling - Standardized modality-agnostic question: "Which option best describes what this example is mainly about?" - Three options presented in randomized order

3. Modality Selection Rate (MSR)

MSR(m) = (1/N) Σ_i 1[ŷ_i = opt(m)]

Where N = total conflict samples, opt(m) = the option grounded in modality m. Higher MSR = stronger preference.

4. Layer-Wise Probing

To trace the emergence of modality preference: - Extract hidden states at each layer - Train a single-layer MLP as a linear probe to predict the model's final modality preference - Probe accuracy on held-out test set quantifies how much each layer encodes modality preference

Key Findings

Finding 1: Visual Preference Dominance

Unlike traditional VLMs that show absolute text dominance, most OLLMs exhibit a pronounced visual bias: - Gemini 3.1 Pro: MSR_visual = 72%, MSR_text = 7% on tri-modal conflicts - This represents a fundamental paradigm shift from text-dominance to vision-dominance

Finding 2: Mid-to-Late Layer Emergence

Modality preference is not static — it emerges progressively: - Shallow layers: No clear modality preference signal - Mid-to-late layers: Preference arises abruptly and stabilizes - This correlates with representations becoming increasingly abstract

Finding 3: Hallucination Diagnosis

Layer-wise modality preference probes serve as effective hallucination detectors: - Abnormal increase in predicted preference probability for the interfering modality consistently accompanies hallucinations - On POPE dataset: AUROC = 94% for hallucination detection (vs. 50% random, 51% earlier layers) - Validated across POPE, AVHBench, and AHa-Bench

Comparison with Prior Work

Aspect Traditional VLM Studies This Work (OLLMs)
Modailties Vision + Language Text + Vision + Audio
Dominant modality Text Visual
Analysis level Coarse (output only) Layer-wise probing
Application Understanding bias Hallucination diagnosis

Open Questions

  1. Why visual preference?: What architectural or training factors cause OLLMs to prefer visual over textual information?

  2. Preference modifiability: Can training interventions (e.g., modality-balanced loss) shift the preference distribution?

  3. Task-dependent preferences: Does the dominant modality shift depending on task type (e.g., audio-centric vs. visual-centric tasks)?

  4. Probing limitations: Are linear probes sufficient to capture complex modality interaction patterns, or do we need more expressive probes?

See Also