总览 / 概念 / Paraformer — Non-Autoregressive Parallel
↗ 源 Markdown
概念
model architecture speech-model end-to-end open-source
创建2026-05-01
更新2026-05-01
阅读量级6 分钟
来源2206.08317, 2305.11013
概念导读

Autoregressive (AR) ASR models (Transformer, Conformer) generate tokens sequentially:

  1. Core Problem
  2. Architecture
  3. Key Mechanism 1: CIF Predictor
  4. Key Mechanism 2: GLM Sampler
  5. Key Mechanism 3: MWER Training
  6. Benchmarks
  7. AISHELL-1 (178 hrs Mandarin)
  8. AISHELL-2 (1000 hrs Mandarin)

Paraformer — Non-Autoregressive Parallel Transformer for ASR

arXiv 2206.08317 | Interspeech 2022 | Alibaba DAMO Speech Lab First NAR model to match AR transformer accuracy on large-scale ASR

Core Problem

Autoregressive (AR) ASR models (Transformer, Conformer) generate tokens sequentially:

P(Y|X) = ∏ P(yₗ | y<ₗ, X)    ← linear in output length L

This creates a fundamental latency bottleneck — decoding time scales linearly with utterance length.

Non-autoregressive (NAR) models solve this by parallel generation:

P(Y|X) = ∏ P(yₗ | X)    ← all tokens simultaneously

But vanilla NAR suffers from two critical problems: 1. Length prediction: Hard to predict output token count from variable-rate speech 2. Independence assumption: Tokens generated without inter-dependency → high substitution errors

Paraformer is the first NAR model to achieve comparable accuracy to AR transformers while delivering 10x+ speedup.

Architecture

                    Paraformer Architecture
═══════════════════════════════════════════════════════

     f-banks → ┌─────────────┐
               │  Encoder    │  (Conformer/SAN-M blocks)
               │  H₁:ₜ       │
               └──────┬──────┘
                      │
              ┌───────┼───────────────┐
              ▼       ▼               │
     ┌────────────┐  ┌────────────┐   │
     │ Predictor  │  │  Sampler   │   │
     │ (CIF)      │  │  (GLM)     │   │
     │            │  │            │   │
     │ α weights  │  │ Eₐ → Eₛ    │   │
     │ Eₐ embeds  │  │ (training) │   │
     └─────┬──────┘  └─────┬──────┘   │
           │               │          │
           ▼               ▼          │
     ┌──────────────────────────┐     │
     │  Parallel Bi-Decoder     │     │
     │  Pass 1: Eₐ + H (no grad)│─────┘
     │  Pass 2: Eₛ + H (w/ grad)│
     │  Infer: Eₐ + H (single)  │
     └──────────┬───────────────┘
                ▼
           Y'' predictions
     ┌──────────────────────┐
     │  Loss:               │
     │  L = γL_CE + L_MAE   │
     │      + L_MWER        │
     └──────────────────────┘

Key Mechanism 1: CIF Predictor

Continuous Integrate-and-Fire solves the length prediction problem.

The predictor produces frame-level weights αₜ ∈ [0,1]:

α₁:ₜ = Sigmoid(Linear(Conv(H₁:ₜ)))

CIF accumulates weights and integrates encoder outputs until reaching threshold β:

# CIF process (β = 1.0):
α = (0.4, 0.8, 0.3, 0.5, 0.2)

E₁ = 0.4 × e₁ + 0.6 × e₂     ← sum crosses 1.0, fire
E₂ = 0.2 × e₂ + 0.3 × e₃ + 0.5 × e₄  ← next boundary

General formulation:

Eᵤ = Σₜ wᵤₜ · hₜ    where Σₜ wᵤₜ = 1 for each output token u

Dynamic threshold β eliminates train/inference mismatch:

β = Σ αₜ / ⌈Σ αₜ⌉

MAE loss guides convergence:

L_MAE = |N - Σₜ αₜ|

Key Mechanism 2: GLM Sampler

Glancing Language Model fixes the independence assumption.

During training, the sampler replaces some acoustic embeddings with ground-truth token embeddings:

Eₛ = Sampler(Eₐ, E_c, ⌈λ · d(Y, Y')⌉)

Where: - λ = sampling factor (optimal ≈ 0.75) - d(Y, Y') = Hamming distance between prediction and ground truth - The number of replacements decreases as training progresses

The GLM loss:

L_GLM = Σ log p(yₙ | GLM(Y, Y'), X; θ)
         yₙ ∈ GLM(Y, Y')

At inference, the sampler is inactive — pure single-pass parallel decoding.

Key Mechanism 3: MWER Training

Minimum Word Error Rate training with negative samples:

L_Nwerr(x, y*) = Σ [P_b(yᵢ|x) · W(yᵢ, y*) - W̃]
                  yᵢ ∈ Sample(x, N)

Negative candidates generated by randomly masking the top-1 score token during training.

Total loss:

L_total = γ·L_CE + L_MAE + L_Nwerr(x, y*)

Note: In FunASR (2023), MWER was removed as it contributed little to performance gains; a CE loss on first-pass decoder was added instead to reduce train/inference gap.

Benchmarks

AISHELL-1 (178 hrs Mandarin)

Model Type Dev CER Test CER RTF (bs=1)
A-FMLM NAR 6.2 6.7 0.2800
Mask-CTC NAR 6.9 7.8 0.0500
CASS-NAT NAR 4.8 5.2 0.0037
AR Transformer AR 4.7 5.4 0.0230
Paraformer NAR 4.6 5.2 0.0037

AISHELL-2 (1000 hrs Mandarin)

Model Type Test iOS CER RTF
AR Transformer AR 6.18 0.0168
Vanilla-NAR NAR 6.23 0.0168
Paraformer NAR 6.19 0.0026

Industrial 20,000 hrs

Model Far-field CER Common CER
CTC baseline 17.71 9.93
Paraformer 13.94 7.97
AR Transformer 14.72 8.66

Error Type Analysis (Industrial 20k hrs)

Paraformer vs vanilla NAR: - Substitution errors: Paraformer ~0.02 vs vanilla NAR ~0.07 (GLM reduces this ~3x) - Insertion errors: Comparable - Deletion errors: Comparable

The GLM sampler dramatically reduces substitution errors — the main weakness of vanilla NAR — by learning inter-token dependencies during training.

Sampling Ratio Study

λ Far-field CER Common CER
0.2 14.64 8.22
0.5 14.37 8.13
0.75 14.17 7.98
1.0 14.24 8.09
1.5 14.47 8.13

Optimal λ = 0.75 balances learning signal and acoustic fidelity.

Evolution & Variants

  • seaco-paraformer — Adds hotword customization via CIF-based contextual module
  • sa-paraformer — Adds speaker attribution for multi-speaker meetings
  • paraformer-v2 — Replaces CIF with CTC predictor for noise robustness & multilingual
  • funasr — Production toolkit packaging all variants with FSMN-VAD, CT-Transformer

Comparison with Other Approaches

Aspect AR Transformer Vanilla NAR Paraformer
Decoding Sequential Parallel Parallel
Token count Implicit Predictor CIF
Context modeling Built-in None GLM sampler
Speed 1x 10x+ 10x+
Accuracy Baseline -15% ≈AR
Train passes 1 1 2 (inference: 1)