---
title: MinMo
arxiv_id: 2501.06282
source: https://arxiv.org/html/2501.06282v1
---

 

 MinMo: A Multimodal Large Language Model for Seamless Voice Interaction 

 1 Introduction 

 2 Related Work 

 Multimodal Spoken Dialogue Models 

 Text Style-Controllable Speech Synthesis 

 3 MinMo 

 3.1 Model Architecture 

 3.2 Streaming Voice Decoder 

 3.3 Tasks and Training Data 

 3.4 Model Training 

 4 Experiments 

 4.1 Speech Recognition and Translation 

 Multilingual Speech Recognition 

 Multilingual Speech Translation 

 Language Identification 

 Contextual Biasing Speech Recognition 

 4.2 Speech Analysis and Understanding 

 Speech Emotion Recognition 

 Audio Event Understanding 

 Speaker Analysis 

 4.3 Speech-to-Text Enhancement 

 Spoken Language Smoothing 

 Punctuation Insertion and Inverse Text Normalization 

 4.4 Voice Generation 

 Text-to-Speech (TTS) 

 Instruction-following Voice Generation 

 4.5 Voice Chat 

 Spoken Question Answering and Spoken Dialogue 

 Full Duplex Spoken Dialogue 

 Full Duplex System Latency 

 5 Conclusion 

 6 Limitations 

 7 Authors (alphabetical order of family name) 

 8 Acknowledgment 

 A Prompts for Voice Understanding Tasks 

 A.1 Spoken Language Smoothing 

 A.2 Punctuation and Inverse Text Normalization 

MinMo: A Multimodal Large Language Model for Seamless Voice Interaction

 FunAudioLLM Team

 Tongyi Lab, Alibaba Group

Abstract

Recent advancements of large language models (LLMs) and subsequent multimodal speech-text models have provided promising foundation technologies for achieving seamless voice interactions, that is, real-time, natural, smooth, and human-like voice conversations between the user and the system. Prior works of speech-text multimodal models for voice interactions can be roughly categorized into native and aligned models. Native multimodal models simultaneously model end-to-end understanding and generation of both speech and text with a single framework; however, they face the challenges of drastic discrepancy between speech and text sequence lengths, insufficient speech pre-training, and catastrophic forgetting of knowledge of text LLMs. Aligned multimodal models are more successful at maintaining capabilities of text LLMs; yet existing models are usually trained on small-scale speech data, investigated on a limited set of speech tasks, and lack systematic exploration of instruction-following capabilities for rich and nuanced speaking styles. In this work, we introduce MinMo , a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is https://funaudiollm.github.io/minmo , and the code and models will be released soon.

 1 Introduction

 Figure 1 : Performance comparison between our MinMo ( ∼ similar-to \sim ∼ 8B parameters) and top-tier speech-text multimodal models, including Moshi(7B)  (Défossez et al., 2024 ) , Freeze-Omni(7.5B)  (Wang et al., 2024b ) , GLM-4-Voice(9B)  (Zeng et al., 2024 ) , SeamlessM4T Large v2(2.3B)  (Communication et al., 2023 ) , NExT-GPT(12.42B)  (Wu et al., 2024 ) , speech-to-text model Qwen2-Audio( ∼ similar-to \sim ∼ 8B)  (Chu et al., 2024 ) , Whisper-large-v3(1.55B)  (Radford et al., 2023 ) , and others. We demonstrate capabilities of MinMo on automatic speech recognition (ASR), speech-to-text translation (S2TT), spoken question answering (SQA) encompasses both speech-to-text (S2T) and speech-to-speech (S2S), vocal sound classification (VSC), speech emotion recognition (SER), language identification (LID), age recognition and gender detection. ASR is evaluated using 1-WER%, with Fleurs & Common Voice results are averaged over 10 languages ( zh, en, ja, ko, yue, de, fr, ru, es, it ). S2TT is evaluated using BLEU, with CoVoST2 results averaged over en2zh, en2ja, zh/ja/de/fr/ru/es/it2en translation directions. SQA is eavaluated using Accuracy. SER is evaluated using Weighted Accuracy. MinMo surpasses the previous SOTA models on all these tasks. 

Seamless voice interaction indicates that a user experiences real-time, natural, relevant, and human-like spoken conversation with the system .
Facilitating seamless voice interaction poses great challenges: (1) the system needs to understand audio accurately and comprehensively, including comprehending the content and also paralinguistic cues in speech (e.g., emotion, prosody) as well as audio events; (2) the system is expected to produce natural and expressive speech response; (3) the system should provide relevant and reasonable response to the user, as an intelligent chatbot; (4) the system is expected to support full-duplex conversation (simultaneous two-way communication), that is, the system listens while speaking and the user is free to interrupt when the system is speaking, then the system either continues the speech, or concedes it, listens to the user, and provides response to the new user query.

In recent years, seamless voice interaction systems have gained significant momentum, especially with the advancements in multimodal large language models, such as GPT-4o  (Hurst et al., 2024 ) and Moshi  (Défossez et al., 2024 ) . These systems not only produce natural and expressive speech but also understand cues beyond words, including emotional tones and audio events.
Current multimodal language models for voice interaction can be categorized into two main categories.
The first category includes native multimodal models , such as Moshi  (Défossez et al., 2024 ) and GLM-4-Voice  (Zeng et al., 2024 ) . These models typically use a decoder-only Transformer as the backbone to simultaneously model understanding and generation of both speech and text modalities within a single framework; they usually require pre-training with both speech and text data. These models suffer from two major limitations. Firstly, after speech discretization, speech token sequences are often more than twice the length of text (e.g., 12.5 tokens per second in Moshi). This discrepancy in sequence length poses challenges as model sizes grow, such as the 175B GPT-3  (Brown et al., 2020 ) . Secondly, the scarcity of speech data compared to text leads to highly imbalanced speech-text training data and in turn causes catastrophic forgetting  (Wang et al., 2024b ) .

The second category includes aligned multimodal models , integrating voice capabilities while aiming to maintain the capabilities of the existing pre-trained text LLM. This results in intermediate outputs that still contain text, as seen in models such as Llama-Omni  (Fang et al., 2024 ) and Freeze-Omni  (Wang et al., 2024b ) . However, these alignment-based models are typically trained on limited speech data (200K samples for LLaMA-Omni and 120K hours for Freeze-Omni), leading to questions on the impact of larger speech datasets on model capabilities and whether the chat capabilities of the original text-LLM might be compromised. Furthermore, investigation of extensive speech tasks has not been conducted on these models, such as speech translation, emotion recognition, speaker analysis, language identification, and audio event detection. Moreover, these models lack systematic evaluations of instruction-following capabilities for rich and nuanced speaking styles, as well as lacking development and evaluation of full-duplex conversation capabilities, for achieving seamless voice interaction.

In this work, we introduce a new multimodal large language model MinMo , to address these limitations of existing aligned multimodal models. MinMo is trained on over 1.4 million hours of speech data, encompassing various tasks such as Speech-to-Text, Text-to-Speech, and Speech-to-Speech, as detailed in Table  2 . This extensive training enables MinMo to achieve state-of-the-art (SOTA) performance across various benchmarks, as shown in Figure  1 . We also apply methods that effectively mitigate catastrophic forgetting of the chat capabilities of the original text-LLM while enhancing voice comprehension and generation after training on such large-scale datasets.

We also propose a novel voice decoder that balances structural simplicity and competitive voice generation performance. LLaMA-Omni uses a non-autoregressive (NAR) streaming Transformer, which takes the output hidden states of the LLM as input and employs connectionist temporal classification (CTC) to predict the discrete speech token sequence of the response. This approach suffers from inferior performance compared to autoregressive speech decoder. Freeze-Omni uses three speech decoders, including NAR prefix speech decoder, NAR speech decoder, and AR speech decoder, which complicates the model structure. Different from both of these strategies, we design an AR streaming Transformer for MinMo, which mixes the output hidden states of the LLM with speech tokens, based on a fixed ratio, as shown in Figure  3 .

Our contributions can be summarized as follows:

 • 

We propose MinMo , an end-to-end aligned multimodal large model that gains audio understanding, audio generation, and end-to-end duplex speech interaction capabilities by adapting a pre-trained text large language model (LLM) through a multi-stage alignment strategy over 1.4 million hours of audio data covering a wide range of speech tasks. MinMo achieves state-of-the-art (SOTA) performance on multiple open-source benchmarks, including spoken dialogue, multilingual speech recognition, speech translation, emotion recognition, and speaker analysis. Different from previous multimodal models that often suffer from notable catastrophic forgetting of capabilities of the text LLM and significant performance degradation on text tasks, MinMo has minimal loss in the original capabilities of the text LLM.

 • 

We propose a novel alignment method for streaming end-to-end audio generation, by exploring the use of the hidden layer representations of the text model as inputs to the Voice Decoder for aligning the audio output modality. Experimental results demonstrate that our streaming voice decoder effectively balances structural simplicity, low latency, and high voice generation performance, and outperforms previous models. Additionally, while most existing voice interaction systems only support controlling the content of the response, MinMo enhances instruction-following capabilities and enables the generation of speech corresponding to user-specified emotions, dialects, and speaking rates, as well as mimicking specific voices with a 98.4% instruction-following accuracy.

 • 

We develop a mechanism that effectively facilitates full-duplex interactions with MinMo. Specifically, we implement a full-duplex prediction module that harnesses the text LLM’s semantic understanding capabilities to decide whether to continue system response, or concede, listen, and respond to new user query. For MinMo, the speech-to-text latency is approximately 100ms; the full-duplex latency is approximately 600ms in theory and 800ms in practice.

 (a) An example showcases MinMo’s capabilities, including speech-to-speech chat, speech-to-text translation, style-controllable speech synthesis, and full duplex interaction. 

 (b) An example showcases MinMo’s capabilities, including speech-to-speech chat, audio event detection, speaker analysis and speech-to-text translation. 

 Figure 2 : Examples demonstrating various capabilities of MinMo. More capabilities of MinMo include the tasks shown in Table  2 . 

 2 Related Work

Multimodal Spoken Dialogue Models

A variety of speech foundation models have been developed for generic audio understanding, but not systematically explored for voice interaction. For example, Qwen2-Audio (Chu et al., 2024 ) integrates Whisper speech encoder with a pre-trained text LLM and adapts the LLM for speech understanding capabilities through multi-task pre-training and instruction-based supervised fine-tuning. SALMONN  (Tang et al., 2024 ) is another speech-text LLM for generic audio understanding, by integrating separate speech and audio encoders with a pre-trained text LLM through Q-Former and adopting LoRA for modality alignment.

Since this work aims to develop an end-to-end multimodal model for seamless voice interaction, we focus on comparing MinMo to speech-text models for voice interaction (or called multimodal spoken dialogue models). Contemporaneously or inspired by GPT-4o, there have been active developments of multimodal spoken dialogue models managing to achieve real-time voice conversations with user. (Ji et al., 2024a ) provides an in-depth overview of recent spoken dialogue models. Some works support traditional turn-based voice chat (i.e., half-duplex communication), but cannot handle full-duplex voice interaction (i.e., simultaneous two-way communication). These models include collaborative systems and end-to-end frameworks. PSLM  (Mitsui et al., 2024 ) is a collaborative system since it replies on ASR to process audio input, which discards paralinguistic information and causes error propagation. PSLM generates speech and text tokens in parallel hence it reduces the speech generation latency; however, it suffers from reduced response quality. Different from the collaborative systems such as PSLM, end-to-end frameworks directly accept audio input and generate audio output. Llama-Omni  (Fang et al., 2024 ) and Mini-Omni  (Xie & Wu, 2024 ) are two recent end-to-end frameworks that have not been trained for full-duplex communication. Llama-Omni integrates Whisper speech encoder, speech adapter, streaming speech decoder, and vocoder with a pre-trained text LLM backbone. The speech decoder generates discrete units corresponding to generated text prefix in an NAR manner. The model is trained with a two-stage strategy: in the first stage, the speech encoder is frozen, and the speech adapter and LLM are trained autoregressively; in the second stage, the speech encoder, speech adapter, and LLM are frozen and only the speech decoder is trained using the CTC loss. Llama-Omni is evaluated on speech-to-text instruction-following and speech-to-speech instruction-following tasks. Mini-Omni also adopts Whisper encoder and uses adapter for minimal training in order to reserve LLM’s capabilities. The model is trained through three stages of modality alignment, adapter training, and multi-modal fine-tuning. Mini-Omni simultaneously generates text and audio tokens, while padding N tokens to ensure that the corresponding text tokens are produced first to guide audio token generation.

Our MinMo facilitates full-duplex spoken dialogues. Existing full-duplex voice chat systems can also be categorized into collaborative systems and end-to-end models. Among collaborative systems, VITA  (Fu et al., 2024 ) runs two models at the same time, namely, the generation model and the monitoring model, to support full-duplex communication. When the generation model is generating system response, the monitoring model monitors the environment and once it detects effective user interruption, it combines context and provides response to the new user query, while the generation model pauses and switches to the monitoring role. Notably, VITA still relies on an external TTS module to generate speech output. Alternatively, another collaborative system  (Wang et al., 2024a ) operates with LLM interfacing with an ASR module and a streaming TTS module. The system does not require modality alignment; instead, supervised fine-tuning is conducted on a pre-trained text LLM with the following paradigm: At each time step, the LLM either processes an input token, or generates a text token, or outputs a special control token for state transitions between SPEAK and LISTEN. All these tasks are defined as next token prediction on a serialized, single-stream view of dialogues. Full-duplex dialogue learning is conducted on data synthesized by GPT-4 to generate dialogues with different types of user interruptions. Notably, with its cascaded architecture, this system suffers from high latency up to 680ms.

Among end-to-end full-duplex models, the early work of dGSLM  (Nguyen et al., 2022 ) proposes a Siamese architecture to jointly process both audio token streams of user speech and system speech. However, it suffers from several weaknesses: it relies on speech-only training, hence does not leverage capabilities of a pre-trained text LLM; it only uses semantic tokens, hence does not sufficiently model acoustic information; it does not support online mode. LSLM  (Ma et al., 2024b ) uses a decoder-only Transformer to generate speaking tokens and a streaming SSL encoder to process listening tokens. It introduces an interruption token to stop speaking when detecting a turn-taking attempt from the user. However, the model is insufficient in generating reasonable responses. Among the more recent end-to-end full-duplex models, Moshi  (Défossez et al., 2024 ) , GLM-4-Voice  (Zeng et al., 2024 ) , SyncLM  (Veluri et al., 2024 ) , IntrinsicVoice  (Zhang et al., 2024b ) , and Omni-Flatten  (Zhang et al., 2024a ) are native multimodal models . They simultaneously model understanding and generation of both speech and text modalities within a single framework, based on a GPT backbone, and require self-supervised autoregressive pre-training using both speech and text data. As discussed in Section  1 , these native multimodal models need to tackle the challenges due to significant discrepancy between sequence lengths of speech tokens and text tokens, and also highly imbalanced speech-text training data and the resulting catastrophic forgetting.
IntrinsicVoice employs a GroupFormer to generate HuBERT tokens from the LLM’s hidden states, effectively shortening speech sequences to lengths comparable to text sequences.
OmniFlatten utilizes a multi-stage progressive post-training strategy that incorporates a chunk-based flattened single stream of speech tokens and text tokens to learn full-duplex and text-free speech-to-speech interaction.
Different from these native multimodal models, our MinMo is in the category of aligned multimodal models , which also include Llama-Omni, Mini-Omni2 (Xie & Wu, 2024 ) , and Freeze-Omni  (Wang et al., 2024b ) . Aligned multimodal models integrate voice capabilities while aiming to maintain the capabilities of the existing pre-trained text LLM. Mini-Omni2 introduces a command-based interruption mechanism for supporting full-duplex conversation; however, it is only evaluated on the ASR task and compared to Whisper, VITA, and Mini-Omni. Freeze-Omni  (Wang et al., 2024b ) is a speech-to-speech model that freezes the pre-trained text LLM to reserve the LLM’s capabilities. It supports streaming input speech and generates streaming output speech, uses multi-task training, and conducts chunk-level state prediction for modeling full-duplex voice interaction. Our MinMo differs from these aligned multimodal models in the following ways. We explore training MinMo on much larger speech datasets (1.4 million hours of diverse speech data in contrast to 200K samples for LLaMA-Omni and 120K hours for Freeze-Omni) and on much more extensive speech tasks. MinMo also differs from existing aligned multimodal models with a novel speech decoder, enhanced instruction following capabilities, and systematic training and evaluation of full-duplex spoken conversation capabilities.

Text Style-Controllable Speech Synthesis

A distinctive feature of multimodal spoken dialogue models, compared to text-based dialogue models, is their ability to comprehend and generate acoustic information beyond mere textual content. The speech modality not only contains the content but also acoustic information such as emotion, dialect, and speaking rate. An intelligent multimodal spoken dialogue model should be able to comprehensively understand the acoustic information in input speech (e.g., emotion) and also ideally generate responses with specified emotions, dialects, and speaking rate, as well as mimicking specific voices, so that the system can achieve a deeper level of understanding and response in communication. Collaborative systems, such as ParalinGPT  (Lin et al., 2024b ) , E-Chat  (Xue et al., 2024 ) , and Spoken-LLM  (Lin et al., 2024a ) , incorporate paralinguistic features to enhance the understanding of acoustic information such as emotions. These systems can be cascaded with a style-controllable Text-to-Speech (TTS) system to generate responses with specific emotion, speaking rate, and volume. Significant progresses have been made in text-style controllable TTS, such as TextrolSpeech  (Ji et al., 2024b ) , PromptTTS  (Shimizu et al., 2024 ) , PromptTTS2  (Leng et al., 2024 ) , InstructTTS  (Yang et al., 2024a ) , and ControlSpeech  (Ji et al., 2024c ) . In contrast to these collaborative systems, Moshi  (Défossez et al., 2024 ) uses a TTS engine with a single actor’s voice and recorded monologues in over 70 speaking styles to synthesize training data to support understanding and generation of acoustic information in an end-to-end model. GLM-4-Voice  (Zeng et al., 2024 ) employs high-quality, multi-turn spoken dialogues tailored to specific speech style requirements, such as speaking rate, emotion, or dialect, to support style-controllable spoken dialogues. However, to the best of our knowledge, no previous work has demonstrated that aligned multimodal models can support style-controllable voice generation. Contrary to previous claims that aligned multimodal models such as Llama-Omni and Freeze-Omni only allow language models to control the content of speech but not the style and prosody  (Zeng et al., 2024 ) , in this work, we propose a novel streaming voice decoder for the aligned multimodal model MinMo and find that this decoder enhances instruction-following capabilities and enables MinMo to generate speech corresponding to user-specified emotions, dialects, speaking rates, as well as mimicking specific voices.

 3 MinMo

 3.1 Model Architecture

 Figure 3 : The overall architecture of MinMo. Table  1 provides detailed descriptions of each module in this diagram. 

Figure  3 illustrates the model architecture of MinMo. MinMo employs a lightweight modality alignment approach on a pretrained text LLM.
Table  1 provides detailed descriptions of each module in MinMo.

The Voice Encoder is initialized with the pretrained SenseVoice-large encoder module (An et al., 2024 ) , which provides robust voice understanding capabilities and supports multilingual speech recognition, emotion recognition, and audio event detection.
The Input Projector consists of a randomly initialized two-layer Transformer combined with a CNN layer for dimensional alignment and downsampling. We use the pretrained Qwen2.5-7B-instruct model  (Team, 2024 ) 1 1 1 https://huggingface.co/Qwen/Qwen2.5-7B-Instruct as the pre-trained text LLM, due to its outstanding performance on various benchmarks  (Team, 2024 ) .
We utilize the streaming audio generation mechanism of CosyVoice 2  (Du et al., 2024b ) , due to its low latency and competitive speech synthesis performance. For every batch of five text tokens received, we pass these tokens and their corresponding final hidden layer vectors simultaneously to the Output Projector and the Voice Token LM . The Output Projector is a single-layer linear module randomly initialized for dimensional alignment. The Voice Token LM uses the pretrained CosyVoice 2 LM module.
The Voice Token LM then autoregressively generates fifteen speech tokens, ensuring efficient and seamless audio synthesis. These audio tokens are processed in real time by the Token2wav Synthesizer module to produce the final audio output.
The Token2wav Synthesizer comprises a pretrained flow-matching model, which converts tokens to mel spectrograms, and a pretrained vocoder, which transforms mel spectrograms into waveforms, both sourced from CosyVoice 2.
MinMo is fully trained end-to-end using additional hidden embeddings, which facilitate control of speech styles, such as emotion, dialect, and speaking rate, based on user instructions. Details of voice generation are elaborated in Section  3.2 . The Full Duplex Predictor module
consists of a single-layer Transformer and a linear softmax output layer, both randomly initialized. This module performs real-time prediction on whether to engage with user commands or temporarily halt the ongoing system broadcast to allow for processing further audio input from the user. Once the Full Duplex Predictor decides that a system response is appropriate, MinMo produces text outputs and concurrently generates the audio tokens in a token-by-token manner.

MinMo has approximately 8 billion parameters in total. The training procedure of MinMo is detailed in Section  3.4 .
The end-to-end latency from receiving the user’s audio input to delivering the audio response is approximately 600 ms, when tested on the L20 GPU.

 3.2 Streaming Voice Decoder

To facilitate natural voice responses for MinMo, we introduce a novel voice decoder that transforms textual outputs from an LLM into speech. As illustrated at the top of Figure  3 , our voice decoder comprises three components: an output projector , a voice token language model (LM) , and a streaming token-to-wave (token2wav) synthesizer .

The output projector aligns the dimensions of the LLM with those of the voice decoder. The hidden states from the LLM contain rich contextual information but are semantically ambiguous; whereas, the sampled text tokens are more precise and consistent with the generated text. Meanwhile, the hidden states of the current round of user input contain explicit instruction information.
For every dialog turn, the embeddings of user input, and hidden states of the LLM’s last layer output will be concatenated along the feature dimension to form the query embeddings. The query embeddings, and embeddings of five sampled text tokens along with the hidden states of the LLM’s last layer output will be concatenated along the sequence dimension and fed into the projector.
In this report, the projector’s outputs are referred to as semantic vectors , which represent rich and accurate semantic information.

Following the output projector, a voice token LM is employed to generate speech tokens autoregressively. This LM operates on sequences interleaving text and speech tokens. Specifically, we intermix the semantic vectors and speech tokens in a fixed ratio of 5:15, that is, every five semantic vectors are followed by fifteen speech tokens. During training, a teacher forcing strategy is applied, and a special token is introduced to signal that the next semantic vectors should be concatenated. Once the LLM’s textual response is complete and the semantic vectors are exhausted, we insert a “turn of speech” token to signal the voice token LM that subsequent tokens should be speech tokens exclusively. The speech synthesis process concludes when the “end of speech” token is generated.

For reconstructing waveforms from the speech tokens, we utilize an off-the-shelf streaming token2wav synthesizer, as described by  Du et al. ( 2024b ) . The token2wav synthesizer incorporates a chunk-aware flow matching model and a mel-to-wave vocoder, capable of synthesizing waveforms in chunks of fifteen tokens.

The theoretical latency of the voice decoder can be computed as follows:

 L ⁢ a ⁢ t ⁢ e ⁢ n ⁢ c ⁢ y = 5 ⁢ d l ⁢ l ⁢ m + 15 ⁢ d l ⁢ m + 15 ⁢ d s ⁢ y ⁢ n 𝐿 𝑎 𝑡 𝑒 𝑛 𝑐 𝑦 
 5 subscript 𝑑 𝑙 𝑙 𝑚 15 subscript 𝑑 𝑙 𝑚 15 subscript 𝑑 𝑠 𝑦 𝑛 \displaystyle Latency=5d_{llm}+15d_{lm}+15d_{syn} italic_L italic_a italic_t italic_e italic_n italic_c italic_y = 5 italic_d start_POSTSUBSCRIPT italic_l italic_l italic_m end_POSTSUBSCRIPT + 15 italic_d start_POSTSUBSCRIPT italic_l italic_m end_POSTSUBSCRIPT + 15 italic_d start_POSTSUBSCRIPT italic_s italic_y italic_n end_POSTSUBSCRIPT 

 (1) 

where d l ⁢ l ⁢ m subscript 𝑑 𝑙 𝑙 𝑚 d_{llm} italic_d start_POSTSUBSCRIPT italic_l italic_l italic_m end_POSTSUBSCRIPT denotes the computation time for the LLM to generate one text token, d l ⁢ m subscript 𝑑 𝑙 𝑚 d_{lm} italic_d start_POSTSUBSCRIPT italic_l italic_m end_POSTSUBSCRIPT denotes the time for the LM to generate one speech token, and d s ⁢ y ⁢ n subscript 𝑑 𝑠 𝑦 𝑛 d_{syn} italic_d start_POSTSUBSCRIPT italic_s italic_y italic_n end_POSTSUBSCRIPT denotes the time for the token2wav synthesizer to generate the waveforms corresponding to each speech token.

 Module 

 Description 

 Number of Parameters 

 Voice Encoder 

 Initialized with the encoder parameters of the pre-trained SenseVoice-Large audio understanding model (An et al., 2024 ) 

 ∼ 636 similar-to absent 636 \sim 636 ∼ 636 M 

 Input Projector 

 2 Transformer layer and 1 CNN layer for dimensional transformation and perform 2x downsampling on the input 

 ∼ 170 similar-to absent 170 \sim 170 ∼ 170 M 

 Large Language Model 

 Initialized with Qwen2.5-7B-instruct (Team, 2024 ) 

 7B 

 Output Projector 

 Linear layer for dimensional transformation 

 ∼ 6 similar-to absent 6 \sim 6 ∼ 6 M 

 Voice Token LM 

 Initialized with the LLM of the pre-trained CosyVoice2 (Du et al., 2024b ) 

 ∼ 370 similar-to absent 370 \sim 370 ∼ 370 M 

 Full Duplex Predictor 

 1 Transformer layer and 1 linear-softmax output layer, both randomly initialized 

 ∼ 18 similar-to absent 18 \sim 18 ∼ 18 M 

 Table 1 : Descriptions of the modules in MinMo as depicted in Figure  3 . MinMo has approximately 8 billion parameters in total.

 3.3 Tasks and Training Data

 Category 
 Specific Tasks 
 Hours 

 Speech-to-Text 
 Automatic Speech Recognition (ASR) 
 630k 

 Speech-to-Text Translation (S2TT) 
 451k 

 Language Identification (LID) 
 34k 

 Contextual Bias Speech Recognition 
 50k 

 Speech Emotion Recognition (SER) 
 48k 

 Audio Event Detection (AED) 
 11k 

 Speaker Analysis 
 24k 

 Spoken Language Smoothing 
 0.4k 

 Speech-to-Text Chat 
 10k 

 Text-to-Speech 
 Speech Synthesis 
 170k 

 Instruct Speech Synthesis 
 1k 

 Speech-to-Speech 
 Speech-to-Speech chat 
 10k 

 Style-controllable Speech-to-Speech Chat 
 0.1k 

 Speech-to-ControlToken 
 Full Duplex Interaction 
 4k 

 Table 2 : The multitask training data for MinMo. Task specifications can be found in Section  4 . 

The training tasks for MinMo consist of four categories, including Speech-to-Text , Text-to-Speech , Speech-to-Speech , and Speech-to-ControlToken tasks. The specific tasks within each category and their corresponding data scales are presented in Table  2 .

 Speech-to-Text tasks. This category consists of approximately 1.2 million hours of speech-text paired data, including tasks such as automatic speech recognition (ASR), speech-to-text translation (S2TT), language identification (LID), contextual biasing speech recognition, speech emotion recognition (SER), audio event detection (AED), speaker analysis, spoken language smoothing.
The training data for these tasks is organized in the ChatML format, illustrated by the following example:

 Speech-to-text Data Format 

{
 "messages": [
 {
 "role": "system",
 "content": "You are a helpful assistant."
 },
 {
 "role": "user",
 "content": "task_instruction <|startofspeech|> wav_path
 <|endofspeech|>"
 },
 {
 "role": "assistant",
 "content": "task_output"
 }
 ]
}

Here, “ task_instruction ” corresponds to the natural language descriptions for different speech-to-text tasks. For instance, “ Speech Transcription ” may be used for speech recognition tasks, while “ Translate {SRC_LANG} into {TGT_LANG} ” may be used for speech translation tasks. “ wav_path ” refers to the input audio file path, while “ task_output ” refers to the output of each task.

 Text-to-Speech tasks. The data for this category mainly consists of basic speech synthesis data, which is the same data used for training CosyVoice 2. It includes 170,000 hours of text-speech paired data and supports four languages: Chinese, English, Korean, and Japanese. Additionally, there are approximately 1,000 hours of audio generation data controlled by instructions. The instructions are expanded to include natural language descriptions, generated by Qwen-Max 2 2 2 https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api , utilizing human-labeled attributes such as emotion, speaking rate, dialect, and role-playing.

 User: Please speaking very fast: Today is a happy day, full of laughter and joy. 

 Assistant: \faComment   (Fast speaking rate) Today is a happy day, full of laughter and joy. 

 User: Speaking with a tone of sadness: I miss my dear friend who moved away last month. 

 Assistant: \faComment   (Sad) I miss my dear friend who moved away last month. 

 Table 3 : Examples of text-to-speech data controlled by instructions. 

 Speech-to-Speech tasks. 
The Speech-to-Speech data is primarily sourced through simulation, encompassing approximately 10,000 hours of multi-turn conversational speech and 100 hours of style-controllable multi-turn conversational speech. The method for simulating speech-to-speech chat data is as follows:

 • 

For text chat data primarily sourced from Alpaca  (Taori et al., 2023 ) and ShareGPT 3 3 3 https://sharegpt.com/ , we utilize the zero-shot in-context generation method from CosyVoice  (Du et al., 2024a ) to convert user text into user speech. We fine-tune CosyVoice’s base model with 2 hours of data from a selected speaker to create a speech synthesis model for the target speaker, referred to as CosyVoice-SFT . This model synthesizes the assistant’s speech (i.e., system speech). The advantage of using zero-shot in-context generation for user speech synthesis is its ability to ensure diversity in the generated user speech, thereby enhancing the generalizability of MinMo.

 • 

To address the differences between synthesized and real audio, we select suitable real speech from the ASR data as user speech queries and use the corresponding text as input for Qwen-Max to generate response text, which is then synthesized into assistant speech using the CosyVoice-SFT model. This approach further enhances the model’s robustness to real user audio inputs.

 • 

To generate conversational speech that covers different speaking styles, we initially employ Qwen-Max to create a rich collection of style-controllable, multi-turn text dialogues. User queries are converted into speech using zero-shot generation by Cosyvoice. Subsequently, we employ Cosyvoice 2 to generate the assistant’s expressive speech. Specifically, we input the assistant’s response content along with an instructional prompt into Cosyvoice 2 to synthesize speech in specific styles. Additionally, a small, diverse, and preliminary recorded voice corpus is used as prompt speech to synthesize the expressive response speech by zero-shot generation. The former method enhances the diversity of the simulated speech, while the latter more effectively builds the expressiveness of various styles.

 Speech-to-ControlToken task. The Speech-to-ControlToken data primarily consists of two parts. The first part is extracted from existing real voice interaction data, while the second part is simulated using text dialogue data. Specifically, the existing real voice interaction data includes resources such as Alimeeting  (Yu et al., 2022 ) , Fisher  (Cieri et al., 2004 ) , and our in-house voice interaction data, a total of approximately 3000 hours. The simulated data mainly includes the open-source MOSS dataset  (Sun et al., 2024 ) and spoken dialogues by synthesizing our in-house text dialogue data, yielding about 1000 hours of voice chat data. When constructing duplex training data using these voice interaction data, we apply heuristic rules for automatically annotating duplex labels on the samples, as follows.

 • 

For assistant’s turn-taking, the endpoint of the user’s turn is taken as the starting point of the assistant’s turn.

 • 

For user’s turn-taking, a time gap T after the assistant’s turn ends
is taken as the starting point of the user’s turn, where T ∼ 𝒩 ⁢ ( 0.6 , 0.4 2 ) similar-to 𝑇 𝒩 0.6 superscript 0.4 2 T\sim\mathcal{N}(0.6,0.4^{2}) italic_T ∼ caligraphic_N ( 0.6 , 0.4 start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) .

 • 

For user’s back-channel, we select instances from the voice interaction data when the user (taking one speaker in a dialogue as the user) is unable to interrupt the other speaker and treat them as training samples of user’s back-channels.

 3.4 Model Training

MinMo is trained progressively through four stages of alignment: (1) Speech-to-Text Alignment, (2) Text-to-Speech Alignment, (3) Speech-to-Speech Alignment, and (4) Duplex Interaction Alignment. Through the four alignment stages, MinMo gains its end-to-end audio comprehension and generation capabilities while retaining the capabilities of the backbone text LLM, achieving low latency and facilitating a seamless voice chat experience for the user, similar to GPT-4o. The four stages are detailed as follows.

 Figure 4 : Detailed training data for the Speech-to-Text Alignment stage. Left: Data distribution for Full-Align training. Right: Data distribution for instruction fine-tuning (SFT). 

 Speech-to-Text Alignment. This first stage aligns the audio modality’s input latent space and the semantic space of a pre-trained text LLM using Speech-to-Text data shown in Table 2 . This phase includes stepwise updates of the Input Projector and Voice Encoder in Figure  3 , as well as updating the text LLM using LoRA. Considering that the Voice Encoder and LLM (Qwen2.5-7B) are pre-trained while the Input Projector’s parameters are randomly initialized, we perform a pre-alignment training ( Pre-align ) using a subset of the Speech-to-Text data shown in Table  2 , updating only the Input Projector. This Pre-align phase effectively prevents the randomly initialized parameters from having large-gradient influences on the pre-trained Voice Encoder at the initial training stage. After Pre-align, we use the full Speech-to-Text data for training both the Input Projector and the Voice Encoder while keeping LLM parameters frozen—a process called Full-Align . Following Full-Align, instruction fine-tuning (SFT) is conducted using approximately 1.3 million samples covering various tasks. During this stage, LLM is updated using LoRA, enhancing the model’s ability to follow instructions. The specific data proportions used in the Full-Align and SFT stages are illustrated in Figure  4 . The Pre-Align phase uses about 1/10 of the Full-Align data.

 Text-to-Speech Alignment. This second stage aligns the semantic space of the text LLM with the audio modality’s output latent space, using Text-to-Speech data. This phase first trains the Output Projector and then jointly trains the Output Projector and the Voice Token LM while keeping other MinMo parameters frozen. In addition to the basic text-to-speech functionality, we leverage the end-to-end framework to enable MinMo to follow user instructions in voice interactions, delivering more expressive and entertaining audio responses. For instance, user can control the emotion, speaking rate, dialectal accent, or speaker style of the audio output via instructions. Approximately 1,000 hours of Instruct Speech Synthesis data are constructed, formatted as shown in Table  3 .

 Speech-to-Speech Alignment. This third stage continues training of MinMo using about 10,000 hours of paired audio data. Consistent with the Text-to-Speech Alignment phase, we continue updating only the Output Projector and the Voice Token LM. The training data for speech-to-speech alignment includes not only general speech-to-speech dialogues but also audio generation instructions with various settings, such as adopting specific dialects, speaking rates, and emotions for spoken dialogues. We find that even without updating the LLM, just by leveraging embeddings aligned with a small-scale instruction dataset (¡150 hours), the large model can still learn quite effective control capabilities for audio generation.

 Duplex Interaction Alignment. After completing the first three training stages, MinMo acquires capabilities for audio understanding, audio generation, and half-duplex voice conversation. On this foundation, we further add a full-duplex module trained with 4,000 hours of long-form human-human spoken conversation. The Full Duplex Predictor module is exclusively trained during this stage. The Full Duplex Predictor takes the hidden embeddings of the LLM as input to predict whether the model needs to generate a response. The Full Duplex Predictor leverages the LLM’s inherent semantic understanding capabilities to determine: 1) whether the model should respond to the current user query, and 2) whether the model should stop ongoing audio output to listen to the user query and provide an appropriate response.

 4 Experiments

We evaluate MinMo across multiple benchmarks, as detailed in Table 4 . These evaluation benchmarks cover speech recognition and speech translation tasks (multilingual speech recognition, multilingual speech translation, language identification, and contextual biasing speech recognition), speech analysis and understanding tasks (speech emotion recognition, speaker analysis, and audio event understanding), and speech-to-text enhancement tasks (spoken language smoothing, punctuation, and inverse text normalization). Additionally, we evaluate MinMo on voice generation tasks (text-to-speech and instruction-following voice generation) and voice chat tasks (including spoken question answering, spoken dialogue, and full-duplex interaction tasks).

 Description 
 Dataset 
 Metric 

 Speech Recognition and Translation 

 Aishell-2  (Du et al., 2018 ) 
 CER&WER 

 Wenetspeech  (Zhang et al., 2022 ) 

 Multilingual Speech Recognition 
 Librispeech  (Panayotov et al., 2015 ) 

 Fleurs  (Conneau et al., 2023 ) 

 CommonVoice  (Ardila et al., 2019 ) 

 Multilingual Speech Translation 
 Fleurs  (Conneau et al., 2023 ) 
 BLEU 

 CoVoST2  (Wang et al., 2021 ) 

 Language Identification 
 Fleurs  (Conneau et al., 2023 ) 
 Accuracy 

 Contextual Biasing Speech Recognition 
 Aishell-1-NE 
 CER&P&R 

 Speech Analysis and Understanding 

 Speech Emotion Recognition 
 EMOBox  (Ma et al., 2024a ) 
 UA&WA&F1 

 Audio Event Understanding 
 AirBench (Yang et al., 2024b ) 
 Accuracy 

 Speaker Analysis for Gender and Age 

 Vocal Sound 

 Sound Question 

 Speech-to-Text Enhancement 

 Spoken Language Smoothing 
 SWAB  (Liu et al., 2025 ) 
 S-Faithful&S-Formal 

 Punctuation&ITN 
 In-house testset 
 ChatGPT Score 

 Voice Generation 

 Text-to-Speech 
 Seed-TTS (Anastassiou et al., 2024 ) 
 WER&CER&NMOS 

 Instruction-following Voice Generation 
 In-house testset 
 Accuracy 

 Voice Chat 

 Spoken Question Answering 
 Web Questions (Berant et al., 2013 ) 
 Accuracy 

 Llama Questions (Nachmani et al., 2024 ) 

 TriviaQA (Joshi et al., 2017 ) 

 Spoken Dialogue 
 AlpacaEval (Li et al., 2023 ) 
 ChatGPT Score 

 In-house ChitChat 

 Full Duplex 
 Alimeeting (Yu et al., 2022 ) 
 Positive F1-Score 

 Fisher (Cieri et al., 2004 ) 

 Simulation (Sun et al., 2024 ) 

 Table 4 : Summary of evaluation benchmarks for MinMo in this report. 

 4.1 Speech Recognition and Translation

Multilingual Speech Recognition

We evaluate MinMo’s speech-to-text transcription capabilities on public test sets in Mandarin, English, Japanese, Korean, and six other languages. These include Aishell-2 (Du et al., 2018 ) , LibriSpeech test clean/other (Panayotov et al., 2015 ) , WenetSpeech (Zhang et al., 2022 ) , Fleurs (Conneau et al., 2023 ) , and Common Voice (Ardila et al., 2019 ) . Table  5 presents the results from different models. For Mandarin (ZH), Japanese (JA), Korean (KO), and Cantonese (YUE), we employ the character error rate (CER) for evaluating transcription performance. For English (EN), German (DE), French (FR), Russian (RU), Spanish (ES), and Italian (IT), the word error rate (WER) is utilized as the evaluation metric. Note that all baseline model results are reproduced and processed using the same procedures as conducted on MinMo’s results, for fair comparisons. Post-processing is based on a modified Whisper normalizer  (Radford et al., 2023 ) , with modifications primarily for improved number normalization. In Table  5 , the “w/ LID” column indicates that language identification (LID) information, such as English, Chinese, or Korean, is included as part of the decoding prompt, while the “w/o LID” column denotes results without the additional LID information. Results in parentheses are directly cited from the papers.

 Test set 
 Language 
 Whisper Large-v3 
 Qwen2-Audio 
 MinMo 

 w/ LID 
 w/o LID 
 w/ LID 
 w/o LID 
 w/ LID 
 w/o LID 

 Aishell-2 Android 
 ZH 
 5.14 
 4.96 
 3.02 (2.9) 
 2.90 
 2.88 
 2.86 

 Aishell-2 iOS 
 ZH 
 4.83 
 4.76 
 3.17 (3.0) 
 3.06 
 2.73 
 2.69 

 Aishell-2 Mic 
 ZH 
 4.98 
 4.89 
 3.22 (3.0) 
 3.15 
 2.88 
 2.91 

 Wenetspeech test-net 
 ZH 
 9.72 
 9.68 
 8.14 
 7.65 
 6.78 
 6.64 

 Wenetspeech test-meeting 
 ZH 
 18.72 
 18.54 
 9.49 
 8.34 
 7.44 
 7.60 

 Librispeech test-clean 
 EN 
 2.56 
 1.90 
 1.75 ( 1.6 ) 
 1.74 
 1.74 
 1.64 

 Librispeech test-other 
 EN 
 4.34 
 3.65 
 4.03 ( 3.6 ) 
 4.13 
 3.89 
 3.82 

 Fleurs 
 ZH 
 4.65 
 4.75 
 3.80(7.5) 
 4.14 
 2.95 
 3.38 

 EN 
 4.11 
 4.28 
 5.12 
 5.03 
 3.79 
 3.70 

 JA 
 4.23 
 4.47 
 10.43 
 11.85 
 3.84 
 3.86 

 KO 
 3.33 
 3.20 
 10.57 
 22.01 
 2.92 
 2.85 

 YUE 
 7.67 
 7.11 
 4.09 
 4.07 
 4.25 
 4.22 

 DE 
 5.05 
 5.09 
 10.48 
 12.77 
 5.22 
 5.18 

 FR 
 5.27 
 5.16 
 9.36 
 11.01 
 5.15 
 5.33 

 RU 
 5.05 
 5.12 
 23.2 
 40.4 
 6.23 
 6.18 

 ES 
 2.9 
 2.86 
 7.31 
 18.02 
 3.44 
 3.55 

 IT 
 2.5 
 2.41 
 6.74 
 8.42 
 3.48 
 3.34 

 Avg. 
 4.48 
 4.45 
 9.11 
 13.77 
 4.13 
 4.16 

 Common Voice 
 ZH 
 12.4 
 12.52 
 6.48 (6.9) 
 18.83 
 6.34 
 6.31 

 EN 
 9.66 
 17.91 
 8.82 (8.6) 
 9.15 
 7.92 
 8.16 

 JA 
 10.30 
 10.32 
 13.51 
 13.58 
 13.41 
 11.04 

 KO 
 11.74 
 5.03 
 17.53 
 19.37 
 6.61 
 6.33 

 YUE 
 10.25 
 37.71 
 6.01 ( 5.9 ) 
 6.09 
 6.35 
 6.30 

 DE 
 5.95 
 6.41 
 7.63 
 7.93 
 6.56 
 6.61 

 FR 
 11.22 
 11.47 
 9.63 (9.6) 
 11.10 
 8.46 
 8.59 

 RU 
 5.94 
 6.48 
 16.82 
 22.87 
 6.97 
 7.12 

 ES 
 4.94 
 5.19 
 5.72 
 9.14 
 4.96 
 4.99 

 IT 
 5.77 
 6.31 
 6.81 
 7.96 
 6.08 
 6.16 

 Avg. 
 8.82 
 11.94 
 9.90 
 12.60 
 7.37 
 7.16 

 Table 5 : Multilingual speech recognition results from our MinMo and baseline models in terms of word error rate (WER) and character error rate (CER) on Mandarin, English, and multilingual public test sets. Results in parentheses are directly cited from papers. The best result for each test set is boldfaced. 

 As shown in Table 5 , MinMo achieves superior ASR performance on most test sets across various languages, compared to Whisper Large v3  (Radford et al., 2023 ) and Qwen2-Audio  (Chu et al., 2024 ) .
The “w/ LID” columns for Whisper Large-v3 and Qwen2-Audio show similar results to those reported in the original papers. Testing on Common Voice with or without LID information as a prompt shows a significant gap in average error rates for Whisper Large v3 and Qwen2-Audio, indicating that these two models strongly depend on the LID information. In contrast, MinMo demonstrates robust and consistent ASR performance regardless of the presence of the language identification. 

Multilingual Speech Translation

We evaluate speech-to-text translation capabilities on the Fleurs (Conneau et al., 2023 ) and CoVoST2 (Wang et al., 2021 ) test sets. On the Fleurs test set, we report results for all translation directions supported by our model; whereas, on the CoVoST2 test set, we only report results for translating from English to other languages (en2xx) and vice versa (xx2en), due to dataset limitations, as it primarily focuses on English-centric translation pairs. As shown in Table  6 , our end-to-end MinMo consistently outperforms the cascaded model by pipelining Whisper Large V3 and Qwen2.5-7B-Instruct, in terms of BLEU scores.
 Compared to other end-to-end baselines, MinMo achieves SOTA performance on Chinese ↔ ↔ \leftrightarrow ↔ English and Japanese ↔ ↔ \leftrightarrow ↔ English translations and top-tier performance on other language pairs . We attribute this strong performance to the extensive speech translation training data (451K hours of S2TT training data as in Table  2 ) and the powerful audio encoder.
Notably, even though we only augment our training data with the CoVoST2 set, excluding the Fleurs set, our model maintains consistent performance across both test sets, indicating high robustness.

 Test set 

 Language 

 Directions 

 Qwen2-Audio † 

 SeamlessM4T 

 Large v2 

 Whisper large-v3 

 +Qwen2.5-7B-Instruct 

 MinMo 

 Fleurs 
 zh2en 
 20.50 
 22.98 
 22.98 
 24.71 

 ja2en 
 2.94 
 18.23 
 20.76 
 24.09 

 ko2en 
 6.73 
 24.11 
 24.01 
 25.44 

 yue2en 
 16.16 
 19.04 
 21.18 
 23.87 

 de2en 
 30.58 
 37.00 
 37.18 
 39.54 

 fr2en 
 29.56 
 33.97 
 35.46 
 36.5 

 ru2en 
 20.09 
 30.17 
 31.57 
 32.71