总览 / 概念 / Chain of Modality: Dynamic Orchestration
↗ 源 Markdown
概念
architecture multimodal inference alignment
创建2026-04-27
更新2026-04-27
阅读量级6 分钟
来源2604.14520
概念导读

Omni-modal Large Language Models (Omni-MLLMs) promise unified integration of diverse sensory streams, yet a critical performance paradox exists: unimodal baselines frequently outperform joint multimodal inference. This perceptual fragility stems from the static fusion topologies universally employed by current models.

  1. Core Problem
  2. Technical Solution: Chain of Modality (CoM)
  3. Architecture
  4. Dual Cognitive Pathways
  5. Training Strategy
  6. Key Mechanism Details
  7. Experimental Results
  8. Comparison with Existing Methods

Chain of Modality: Dynamic Orchestration in Omni-MLLMs

Core Problem

Omni-modal Large Language Models (Omni-MLLMs) promise unified integration of diverse sensory streams, yet a critical performance paradox exists: unimodal baselines frequently outperform joint multimodal inference. This perceptual fragility stems from the static fusion topologies universally employed by current models.

The paper identifies two structural pathologies in static fusion:

  1. Positional bias (sequential inputs): When modalities are concatenated sequentially (e.g., Audio → Visual), visual dominance is largely an artifact of positional bias. Permuting input order causes drastic shifts in attention and performance — the model relies on structural proximity rather than semantic content.

  2. Alignment trap (interleaved formats): Models that interleave cross-modal tokens to bridge temporal gaps coerce the model into hallucinating semantic consistency. The enforced physical adjacency leads to elevated false-positive matching rates even when signals are discordant.

Technical Solution: Chain of Modality (CoM)

CoM transitions multimodal fusion from passive concatenation to dynamic orchestration by treating modality selection and interaction as modular building blocks.

Architecture

Planner: Adaptively constructs a task-specific modality chain, selecting a sequence of modality-conditioned execution units. Each unit defines both the active modality and its interaction topology:

  • Parallel blocks: For independent evidence auditing (modalities processed independently, then compared)
  • Sequential anchors: For causal grounding (one modality provides context for the next)
  • Interleaved sequences: For fine-grained temporal synchronization (only when temporally aligned)

Dual Cognitive Pathways

CoM bifurcates reasoning depth based on task complexity:

  1. Plan-Decide (PD) pathway: For intuitive/perceptual queries. Shortens the chain, bypassing generative reasoning to preserve perceptual fidelity. Training-free.

  2. Plan-Reason-Decide (PRD) pathway: For complex analytical queries requiring transitive logic. Modality-specific evidence is first analyzed, then synthesized into a grounded decision. Data-efficient SFT on Music-AVQA.

Training Strategy

  • Training-free mode: For intuitive tasks, dynamic modality chain construction alone achieves gains
  • Data-efficient SFT: For analytical tasks, fine-tuning on Music-AVQA with structured reasoning chains

Key Mechanism Details

The Planner operates by: 1. Analyzing the query's cognitive demand (intuitive vs. analytical) 2. Selecting the modality set (which modalities to activate) 3. Choosing the interaction topology (parallel, sequential, interleaved) 4. Constructing the execution chain as a sequence of modality-conditioned units

Experimental Results

CoM consistently matches or outperforms specialized models across diverse benchmarks:

  • Music-AVQA: Fine-grained music perception
  • AV-Odyssey: Open-domain audio-visual QA
  • DailyOmni: Daily life multimodal scenarios
  • OmniBench: Comprehensive omni-modal evaluation
  • WorldSense: World knowledge multimodal reasoning
  • AV-Counting: Audio-visual counting tasks
  • AVHBench: Cross-modal hallucination detection

Key finding: On complement-informative tasks where audio and visual predictions disagree (A ≠ V), CoM significantly reduces "blind trust" failures (ErrorA/ErrorV) compared to Qwen2.5-Omni-7B.

Comparison with Existing Methods

Approach Fusion Topology Adaptability Hallucination Handling
Sequential concatenation Static, ordered None Positional bias
Interleaved fusion Static, interleaved None Alignment trap
Specialized models Task-specific Poor generalization Local gains only
DPO/RL post-hoc Static + correction Symptom only Treats symptoms
CoM (this work) Dynamic, orchestrated Task-adaptive Root cause fix

Open Questions

  1. Scalability to more modalities: How does CoM scale beyond tri-modal (text+audio+vision) to include video, sensor data, etc.?

  2. Learned vs. heuristic Planner: Can the Planner be learned end-to-end rather than using heuristic rules?

  3. Latency overhead: What is the inference latency cost of dynamic orchestration, and can it be amortized?

  4. Conflict resolution theory: When modalities genuinely conflict (not just noise), what is the optimal resolution strategy?

See Also