总览 / 概念 / EmoOmni: Emotional Understanding in Omni
↗ 源 Markdown
概念
model multimodal alignment training architecture
创建2026-04-27
更新2026-04-27
阅读量级6 分钟
来源2602.21900
概念导读

Existing Omni-LLMs struggle with complex real-world emotional scenarios:

  1. Core Problem
  2. Technical Solution
  3. EmoOmni Framework
  4. Emotional Chain-of-Thought (E-CoT)
  5. EmoOmniPipe Data Pipeline
  6. EmoOmniEval Benchmark
  7. Experimental Results
  8. Comparison with Existing Methods

EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs

Core Problem

Existing Omni-LLMs struggle with complex real-world emotional scenarios: 1. Superficial understanding: Audio and visual cues are often complex, implicit, or conflicting (e.g., cheerful tone + frowning face). Models fail to resolve these conflicts. 2. Emotional detail loss: Thinker-Talker architectures rely on implicit emotional control through hidden states. Emotional intent is diluted or lost during transmission from Thinker to Talker. 3. Data scarcity: No widely adopted pipeline exists for real-world, emotionally annotated multimodal dialogue data. 4. Evaluation gap: Benchmarks focus on task correctness or basic emotion recognition, not emotional intelligence in dialogue context.

Technical Solution

EmoOmni Framework

Mimics human affective cognition through a Perception–Reasoning–Expression causal chain:

EmoOmni-Thinker: Multimodal perception → E-CoT reasoning → Textual response EmoOmni-Talker: E-CoT as explicit emotional instruction → Expressive speech synthesis

Emotional Chain-of-Thought (E-CoT)

The core innovation is a structured reasoning trajectory with four components:

  1. Multimodal Emotion Perception (z_p): Fine-grained emotional cue extraction from audio and video
  2. P(z_p | M) where M = multimodal features

  3. Captures vocal tension, facial expressions, behavioral inconsistencies

  4. Intention Analysis (z_a): Infer user's underlying intention from perceived cues

  5. P(z_a | z_p)

  6. Resolves sarcasm, emotional masking, conflicting signals

  7. Response Strategy Planning (z_s): High-level emotional and pragmatic response plan

  8. P(z_s | z_a)

  9. Abstract control signal decoupling decision-making from surface realization

  10. Textual Response Generation (z_t): Final response conditioned on all reasoning stages

  11. P(z_t | z_p, z_a, z_s)

Complete E-CoT: Z = {z_p, z_a, z_s, z_t}

The E-CoT serves dual purpose: (1) explicit reasoning for the Thinker, and (2) explicit emotional instructions for the Talker, ensuring the final speech is semantically and emotionally aligned.

EmoOmniPipe Data Pipeline

Extracts and annotates emotionally rich dialogues from movies/TV series: 1. Raw video segmentation (20-min chunks, audio/video isolation, denoising via MelBandRoformer) 2. Multimodal annotation (6 dimensions using SOTA models) 3. E-CoT construction (via Gemini 2.5 Pro with contextual history) 4. Quality filtering via GPT-4o

EmoOmniEval Benchmark

Three evaluation settings: - Video-to-Speech (VS): End-to-end from input video to generated speech - Video-to-Text Response (VT): Text generation quality from input video - Instruction Following (IF): Speech synthesis conditioned on explicit instructions

Each scored 0/1/2 on: Response Content Relevance & Logic, Response Emotional Strategy, Emotion Analysis.

Experimental Results

EmoOmni-7B achieves comparable performance with Qwen3Omni-30B-A3B-Thinking under the same talker, demonstrating that E-CoT and real-world data effectively compensate for parameter scale in affective computing (7B vs 30B).

Comparison with Existing Methods

Approach Emotion Modeling Data Reasoning Chain Expression Control
Tag-based Single label Coarse None Implicit
Prompt-based Style descriptions Limited None Partial
EmoOmni Multi-dimensional E-CoT Real-world movies/TV Explicit 4-stage Explicit instruction

Open Questions

  1. Cross-cultural emotions: How does E-CoT generalize across cultures with different emotional expression norms?

  2. Real-time E-CoT: Can the multi-stage reasoning be streamlined for low-latency applications?

  3. Conflicting modality resolution: When audio and video convey opposite emotions, what determines which E-CoT path the model takes?

  4. Talker expressiveness: How much emotional range can the Talker achieve given E-CoT instructions vs. raw hidden states?

See Also