---
title: "LLaMA-Omni: Seamless Speech Interaction with Large Language Models"
arxiv_id: "2409.06666"
version: "v2"
published: "2024-09-10"
updated: "2025-03-01"
venue: "ICLR 2025"
authors:
  - Qingkai Fang
  - Shoutao Guo
  - Yan Zhou
  - Zhengrui Ma
  - Shaolei Zhang
  - Yang Feng
affiliations:
  - "Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)"
  - "Key Laboratory of AI Safety, Chinese Academy of Sciences"
  - "University of Chinese Academy of Sciences"
subjects:
  - "Computation and Language (cs.CL)"
  - "Artificial Intelligence (cs.AI)"
  - "Sound (cs.SD)"
  - "Audio and Speech Processing (eess.AS)"
links:
  paper: "https://arxiv.org/abs/2409.06666"
  html: "https://arxiv.org/html/2409.06666v2"
  pdf: "https://arxiv.org/pdf/2409.06666v2"
  code: "https://github.com/ictnlp/LLaMA-Omni"
  model: "https://huggingface.co/ICTNLP/Llama-3.1-8B-Omni"
  dataset: "https://huggingface.co/datasets/ICTNLP/InstructS2S-200K"
  demo: "https://ictnlp.github.io/llama-omni-demo/"
tags: ["speech-interaction", "speech-to-speech", "low-latency", "LLM", "streaming", "end-to-end", "non-autoregressive"]
---

# LLaMA-Omni: Seamless Speech Interaction with Large Language Models

## Abstract

Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, the authors propose LLaMA-Omni, a novel end-to-end model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. The model is built based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, the authors construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses, with a style that better matches the characteristics of speech interaction scenarios. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 236ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

## 1. Architecture Design

LLaMA-Omni consists of four main components: a pretrained speech encoder, a speech adaptor, a large language model (LLM), and a streaming speech decoder.

### 1.1 Speech Encoder

- **Model**: Whisper-large-v3 (pretrained, frozen during all training stages)
- **Function**: Encodes the user's speech instruction into continuous speech representations
- **Output**: Continuous speech features that are passed to the speech adaptor

### 1.2 Speech Adaptor

- **Function**: Bridges the speech encoder output to the LLM input space
- **Design**: A learnable module that projects/transforms speech features into the LLM's embedding space
- **Training**: Trained in Stage 1 (text response generation from speech); frozen in Stage 2

### 1.3 Large Language Model (LLM)

- **Base Model**: Llama-3.1-8B-Instruct
- **Function**: Decodes text responses directly from speech instructions (no intermediate speech-to-text transcription)
- **Key Design Choice**: The LLM does NOT first transcribe speech into text. Instead, it directly generates text responses from the encoded speech representation. This eliminates the cascaded ASR->LLM pipeline and reduces latency.

### 1.4 Streaming Speech Decoder

- **Architecture**: Non-autoregressive (NAR) streaming Transformer
- **Function**: Takes the output hidden states of the LLM as input and uses Connectionist Temporal Classification (CTC) to predict the sequence of discrete units corresponding to the speech response
- **Key Innovation**: During inference, as the LLM autoregressively generates the text response, the speech decoder simultaneously generates the corresponding speech in a streaming fashion. This enables parallel text and speech generation.
- **Discrete Units**: Speech is represented as discrete units (from a speech tokenizer), which are then converted to waveforms by a vocoder

### 1.5 Inference Pipeline

1. User speaks a speech instruction
2. Speech encoder processes the audio into continuous features
3. Speech adaptor projects features into LLM embedding space
4. LLM autoregressively generates text response token by token
5. Simultaneously, the streaming NAR speech decoder generates speech units from LLM hidden states using CTC
6. Both text and speech responses are produced with very low latency

The architecture enables "simultaneous generation" - text and speech are produced in parallel rather than sequentially, which is the key to achieving low latency.

## 2. Training Method

### 2.1 Training Stages

Training is divided into two stages:

**Stage 1: Text Response Generation from Speech Instructions**
- **Objective**: Train the model to generate appropriate text responses directly from speech inputs
- **Trainable Components**: Speech adaptor + LLM
- **Frozen Components**: Speech encoder (Whisper-large-v3)
- **Training Signal**: Standard next-token prediction loss on text responses, conditioned on encoded speech instructions

**Stage 2: Speech Response Generation**
- **Objective**: Train the streaming speech decoder to produce speech units from LLM hidden states
- **Trainable Components**: Speech decoder only
- **Frozen Components**: Speech encoder, speech adaptor, and LLM (all frozen)
- **Training Signal**: CTC loss between predicted discrete speech units and ground-truth speech unit sequences

### 2.2 Dataset Construction: InstructS2S-200K

The dataset contains 200,000 speech instructions and corresponding speech responses, constructed through a three-step pipeline:

**Step 1: Instruction Rewriting**
Since speech input has different characteristics compared to text input, text instructions are rewritten according to the following rules:
1. Add appropriate filler words (e.g., "hey", "so", "uh", "um") to simulate natural speech patterns
2. Convert non-text symbols into their corresponding spoken forms (to ensure correct TTS synthesis)
3. Modify instructions to be brief without excessive verbiage

**Step 2: Response Generation**
- **Model Used**: Llama-3-70B-Instruct generates responses for the rewritten speech instructions
- **Response Rules**:
  1. Response should NOT contain content that cannot be synthesized by TTS (e.g., parentheses, ordered lists)
  2. Response should be very concise and to the point (appropriate for speech interaction)

**Step 3: Speech Synthesis**
- **For Instructions (user speech)**: CosyVoice-300M-SFT model; randomly select either male or female voice for each instruction to increase diversity
- **For Responses (system speech)**: VITS model trained on LJSpeech dataset; synthesizes responses in a standard consistent voice

### 2.3 Training Efficiency

- **Hardware**: 4 GPUs
- **Training Time**: Less than 3 days total (both stages combined)
- **Significance**: Demonstrates that efficient speech-language model development is feasible with modest computational resources

## 3. Evaluation Results

### 3.1 Speech-to-Speech Interaction Quality

LLaMA-Omni was evaluated against several baselines including cascaded ASR+LLM+TTS systems and other speech-language models.

**Key Metrics**:
- **ASR-WER (Automatic Speech Recognition Word Error Rate)**: Measures alignment between speech and text responses (lower is better). LLaMA-Omni achieves the lowest ASR-WER and ASR-CER scores among speech-language models, indicating the best speech-text alignment.
- **GPT Score**: Evaluates response quality. LLaMA-Omni achieves a GPT score of 3.64, outperforming Moshi (3.31) but below GLM-4-Voice (5.30).
- **WER of generated speech**: LLaMA-Omni achieves ~9.18% WER.

**Comparison with Other Models** (from SOVA-Bench and other evaluations):

| Model | LLM Size | GPT Score | WER (%) | Latency |
|-------|----------|-----------|---------|---------|
| LLaMA-Omni | 8B | 3.64 | 9.18 | ~226ms |
| Moshi | 7B | 3.31 | 7.97 | higher |
| GLM-4-Voice | 9B | 5.30 | 6.40 | ~320ms |
| Freeze-Omni | 7B | 4.23 | 14.05 | lower |
| Mini-Omni | 0.5B | - | high | low |
| Cascaded XTTS | - | 7.20 | 1.70 | highest |

### 3.2 Response Latency

- **LLaMA-Omni Response Latency**: As low as **226-236ms**
- This is significantly lower than cascaded ASR+LLM+TTS systems
- LLaMA-Omni achieves the lowest decoding times compared to other end-to-end speech-language models
- The low latency is achieved through:
  - Elimination of intermediate speech transcription (no ASR step)
  - Simultaneous text and speech generation (parallel decoding)
  - Non-autoregressive streaming speech decoder

### 3.3 Content and Style Quality

- LLaMA-Omni provides better responses in both **content** (accuracy, relevance) and **style** (appropriateness for speech interaction) compared to previous speech-language models
- The InstructS2S-200K dataset, designed specifically for speech interaction scenarios, contributes to improved style matching

### 3.4 Knowledge and Recognition (SOVA-Bench)

On the SOVA-Bench evaluation:
- **Knowledge Score**: 24.91 (highest among evaluated speech LLMs)
- **Semantic Consistency**: 78.41
- Overall demonstrates strong performance in spoken conversation ability

## 4. Key Contributions

1. **Novel Architecture**: End-to-end speech interaction model combining pretrained speech encoder, adaptor, LLM, and NAR streaming decoder
2. **Low Latency**: 226-236ms response latency through simultaneous text and speech generation
3. **Efficient Training**: Less than 3 days on 4 GPUs
4. **Purpose-Built Dataset**: InstructS2S-200K with speech-appropriate instruction and response styles
5. **No ASR Required**: Eliminates intermediate speech transcription step
6. **High Quality**: Better content and style compared to prior speech-language models

## 5. Limitations

- Speech quality (WER) is not as good as cascaded TTS systems (9.18% vs 1.70%)
- Single-turn interaction only (extended to multi-turn in LLaMA-Omni 2)
- Fixed output voice (VITS on LJSpeech)
- Performance gap compared to commercial systems like GPT-4o

## 6. Follow-up Work

- **LLaMA-Omni 2**: Extends to multi-turn speech-to-speech dialogue with autoregressive streaming speech synthesis (ACL 2025)
- **InstructS2S-200K dataset** expanded for multi-turn conversations

## Citation

```bibtex
@inproceedings{fang2025llamaomni,
  title={LLaMA-Omni: Seamless Speech Interaction with Large Language Models},
  author={Fang, Qingkai and Guo, Shoutao and Zhou, Yan and Ma, Zhengrui and Zhang, Shaolei and Feng, Yang},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}
```