概念导读

Speaker-Attributed ASR (SA-ASR) must solve two tasks simultaneously:

1. Core Problem
2. Architecture
3. Core Modules
4. Key Innovation 1: Speaker-Filling Strategy
5. Key Innovation 2: Inter-CTC Enhancement
6. Key Innovation 3: Token-Level Serialized Output Training (t-SOT)
7. Speaker Decoder Details
8. Benchmarks (AliMeeting Corpus)

SA-Paraformer — Speaker-Attributed Non-Autoregressive ASR

arXiv 2310.04863 | NPU ASLP Group + Alibaba DAMO + USTC Extends paraformer for multi-speaker meeting transcription with 1/10 latency of AR models

Core Problem

Speaker-Attributed ASR (SA-ASR) must solve two tasks simultaneously: 1. Transcribe each speaker's speech 2. Attribute each word to the correct speaker

This is critical for meeting transcription, where overlapping speech, unknown speaker counts, and far-field recording create complex acoustic conditions.

Prior approaches:

Approach	Method	Limitation
Cascaded (FD-SOT, WD-SOT)	Speech separation + diarization + ASR	Error propagation; suboptimal joint objective
Joint AR SA-ASR	End-to-end with AR decoder	Large RTF — sequential generation for multi-speaker output

SA-Paraformer replaces the AR decoder with Paraformer's NAR decoder, achieving comparable accuracy with 1/10 RTF.

Architecture

            SA-Paraformer Architecture
══════════════════════════════════════════════════════════════

  Speech X ──→ ┌──────────────┐
               │  ASREncoder  │ ──→ H_asr (Conformer 12 layers)
               └──────┬───────┘
                      │
           ┌──────────┼──────────┐
           ▼          ▼          ▼
    ┌──────────┐ ┌──────────┐ ┌──────────────┐
    │SpeakerEnc│ │Predictor │ │Inter-CTC     │
    │ H_spk    │ │ (CIF)    │ │ (mid-layer)  │
    │(d-vectors)│ │ E_a      │ │ L_interCTC   │
    └─────┬────┘ └─────┬────┘ └──────────────┘
          │            │
          ▼            ▼
    ┌─────────────────────────┐
    │    SpeakerDecoder       │
    │  Layer 1: MHA(E_a,      │
    │           H_asr, H_spk) │
    │  Layer 2: Transformer   │
    │  Cosine-dist attention  │
    │  → d_n (speaker profile │
    │     per token)          │
    └──────────┬──────────────┘
               │
               ▼ d_n injected into FF
    ┌─────────────────────────┐
    │    ASRDecoder           │
    │  Pass 1: E_a + H_asr    │
    │          + d_n (no grad)│
    │  Pass 2: E_s + H_asr    │
    │          + d_n (w/ grad)│
    └──────────┬──────────────┘
               ▼
         Y'' (speaker-attributed tokens)

6 Core Modules

1. ASREncoder — 12-layer Conformer, transforms X → H_asr
2. SpeakerEncoder — Res2Net-based d-vector extractor (256-dim), X → H_spk
3. Predictor — CIF mechanism, H_asr → Eₐ (acoustic embeddings per token)
4. SpeakerDecoder — 2 Transformer layers + cosine-distance attention, produces speaker profile dₙ per token
5. Sampler — GLM, generates semantic embedding Eₛ (training only)
6. ASRDecoder — 6-layer Transformer, takes Eₐ/H_asr/dₙ → predictions

Key Innovation 1: Speaker-Filling Strategy

The model struggles with unknown speaker counts because Eₐ lacks contextual information for speaker identification.

Filling speaker (f-speaker) strategy: Expand the speaker inventory to the maximum number of speakers in the batch. For redundant (non-existent) speakers, fill the cosine distance bₙ,ₖ with random values in [-0.5, 0.5] instead of -∞.

bₙ,ₖ = (qₙ · dₖ) / (|qₙ| · |dₖ|)

# For real speakers: standard cosine similarity
# For redundant speakers: random value ∈ [-0.5, 0.5]

This prevents the model from overconfidently assigning tokens to the wrong speaker — the random perturbation acts as a soft "don't know" signal.

Interfering speaker (i-speaker) strategy: Add extra speaker profiles to the inventory during training, forcing the model to learn to discriminate.

Combined (f&i-speaker): Best performance.

Key Innovation 2: Inter-CTC Enhancement

Since NAR models rely heavily on acoustic representations (no sequential context from previous tokens), the encoder's acoustic signal quality is critical.

An intermediate CTC loss is attached to a middle layer of the Conformer encoder:

L' = L_MAE + λ₁·L_CTC + λ₂·L_interCTC + (1-λ₁-λ₂)·L_CE + L_spk

With λ₁ = 0.3, λ₂ = 0.3. The inter-CTC loss strengthens frame-level acoustic modeling, improving downstream speaker identification and ASR accuracy.

Key Innovation 3: Token-Level Serialized Output Training (t-SOT)

To handle multi-speaker output, transcriptions are serialized by token end times (chronological order):

Speaker1: "你好" ⟨cc⟩ "世界" ⟨cc⟩
Speaker2: "今天" "天气" "很好"
→ Serialized: "你好" ⟨cc⟩ "今天" "天气" "世界" ⟨cc⟩ "很好"

Finding: Training without the ⟨cc⟩ separator significantly reduces deletion errors (26.5% → 6.5%). The NAR model struggles to predict acoustic boundaries for separator tokens that carry no acoustic information.

Model	Separator	Ins.	Del.	Sub.	CER
A1	With ⟨cc⟩	10.1	26.5	6.5	43.1
A2	Without ⟨cc⟩	11.7	6.5	7.1	38.6

Speaker Decoder Details

Cosine-distance-based attention:

bₙ,ₖ = (qₙ · dₖ) / (|qₙ| · |dₖ|)           # Cosine similarity
βₙ,ₖ = exp(cos(bₙ,ₖ, dₖ)) / Σⱼ exp(...)     # Softmax over inventory
dₙ = Σₖ βₙ,ₖ · dₖ                           # Weighted profile

The weighted speaker profile dₙ is injected into the ASR decoder's first-layer FFN:

E_c,1 = E'_c,1 + FF(E'_c,1 + W_spk · dₙ)

Benchmarks (AliMeeting Corpus)

AliMeeting: 104.75 hrs training, 42.27% average speech overlap ratio, far-field 4-channel recording.

Approach	Eval SD-CER	Test SD-CER	RTF (CPU)	RTF (GPU)
Cascaded: FD-SOT	41.0	41.2	—	—
Cascaded: WD-SOT	36.0	37.1	—	—
Joint AR SA-ASR	31.8	34.7	0.967	0.315
SA-Paraformer (base)	36.2	38.6	—	—
+ inter-CTC	34.5	36.9	—	—
+ f-speaker	33.3	35.7	—	—
+ i-speaker	33.1	35.6	—	—
+ f&i-speaker	32.5	34.8	0.168	0.032

Key results: - 6.1% relative SD-CER improvement over cascaded WD-SOT (37.1 → 34.8) - Comparable SD-CER to AR joint model (34.7 vs 34.8) with 1/10 RTF - GPU RTF: 0.032 vs 0.315 (AR) — nearly 10x faster

Ablation Contributions

Addition	Eval Δ	Test Δ
inter-CTC	-1.7%	-1.7%
f-speaker	-1.2%	-1.2%
i-speaker	-1.2%	-1.2%
f&i combined	-2.0%	-2.1%

All components contribute independently; the combined effect is approximately additive.

Loss Function

L' = L_MAE + λ₁·L_CTC + λ₂·L_interCTC + (1-λ₁-λ₂)·L_CE + L_spk

L_spk = Σₙ [e^(bₙ,ᵢ) - log Σₖ P(bₙ,ₖ)]   # Speaker identification loss

4 loss terms jointly optimize: sequence length (MAE), ASR (CTC+CE), acoustic enhancement (inter-CTC), and speaker attribution (L_spk).

Related

• paraformer — Base NAR architecture (encoder + CIF predictor + GLM sampler)
• funasr — Production toolkit packaging
• paraformer-v2 — CTC-based Paraformer successor
• nim4-asr — LLM-based ASR with speaker handling (2026)
• qwen3-asr — Alibaba's LLM-ASR (2026), different approach to speaker attribution