概念

inference speech-model architecture alignment

创建2026-04-27

更新2026-04-27

阅读量级6 分钟

概念导读

Large Audio-Language Models (LALMs) have set new benchmarks in speech processing, but their deployment is hindered by the KV cache memory footprint during long-context inference. The KV cache grows linearly with audio duration, severely limiting on-device and streaming deployment.

Core Problem
Technical Solution
Audio-Critical Head Identification
Spectral Score Smoothing (SSS)
AudioKV Framework
Experimental Results
Comparison with Existing Methods
Open Questions

AudioKV: KV Cache Eviction in Efficient Large Audio-Language Models

Core Problem

Large Audio-Language Models (LALMs) have set new benchmarks in speech processing, but their deployment is hindered by the KV cache memory footprint during long-context inference. The KV cache grows linearly with audio duration, severely limiting on-device and streaming deployment.

General KV cache compression techniques (SnapKV, AdaKV, etc.) excel in text-only LLMs but fail in the audio domain because they overlook the intrinsic temporal continuity of acoustic signals.

Two specific failures: 1. Head-agnostic compression: Uniformly compressing all attention heads is suboptimal because only a subset of heads captures acoustic information 2. Temporal clustering: Raw importance scores cause Top-K selection to concentrate on small token clusters, violating speech's temporal continuity

Technical Solution

1. Audio-Critical Head Identification

Insight: Attention heads in LALMs are modality-specialized. Only a small fraction consistently attends to audio tokens.

Method: 1. Use WhisperX to get word-level timestamps and confidence scores from ASR 2. Retain high-confidence words (τ = 0.95) as reliable anchors 3. Convert timestamps to audio-token index spans 4. For each attention head, measure overlap between high-attention audio tokens and ground-truth audio spans 5. Compute per-head audio-grounding scores S_{l,h}

Head-aware KV cache allocation:

b_{l,h} = w + r + B_score · S_{l,h} / Σ_{l',h'} S_{l',h'}

Where: - w = fixed local window size - r = uniform baseline allocation - B_score = remaining budget distributed proportionally by modality relevance

2. Spectral Score Smoothing (SSS)

Insight: Speech signals encode information in temporally continuous acoustic representations. Importance scores should be smoothly distributed across the temporal sequence.

Problem with existing methods: Top-K selection on raw scores creates dense clusters, violating temporal continuity. Pooling-based smoothing is inherently local.

SSS Method: 1. Treat importance scores as a 1D temporal signal 2. Apply FFT-based global low-pass filter to suppress high-frequency noise 3. Recover smooth global trends 4. Interpolate filtered signal with original scores using mixing coefficient α

smoothed_scores = α · FFT_lowpass(original_scores) + (1 - α) · original_scores

Advantages over pooling: - Global smoothing over entire importance signal (not local neighborhood) - Frequency-aware: separates semantic relevance (low-freq) from spurious alignments (high-freq) - Model-agnostic and computationally efficient (FFT is O(n log n))

AudioKV Framework

Combines both components: 1. Offline: Identify audio-critical heads via ASR-aligned probing 2. Inference-time: Apply SSS to importance scores, then allocate head-aware KV budgets

Experimental Results

Evaluated on Qwen and Gemma series LALMs across ASR and speech translation benchmarks:

Key result: At 40% compression ratio on Qwen3-Omni-30B: - AudioKV: 0.45% accuracy drop (near-full accuracy) - Traditional methods: Catastrophic degradation with repetition failures

AudioKV substantially mitigates ASR degradation compared to head-agnostic baselines while maintaining the same cache budget.

Comparison with Existing Methods

Method	Head-Aware	Temporal Continuity	Audio-Specific	40% Compression
SnapKV	No	Pooling (local)	No	Catastrophic
AdaKV	No	Heuristic	No	Catastrophic
PyramidKV	Layer-wise	No	No	Degraded
AudioKV	Yes	FFT (global)	Yes	0.45% drop

Open Questions

Head stability: Are audio-critical heads consistent across different tasks, speakers, and languages?
SSS parameter sensitivity: How to choose the mixing coefficient α and cutoff frequency optimally?
Extension to video: Can similar head-aware + temporal continuity approaches work for video KV cache?
Streaming scenario: How to adapt AudioKV for true streaming where future context is unavailable for FFT?