--- title: Moshi arxiv_id: 2410.00037 source: https://arxiv.org/html/2410.00037v2 --- Moshi: a speech-text foundation model for real-time dialogue Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Work Audio Language Modeling. Speech-text Models. Spoken Dialogue Models. 3 Model 3.1 Overview 3.2 The Helium Text Language Model 3.2.1 Architecture 3.2.2 Pre-training data filtering Deduplication. Language identification. Quality filtering. 3.3 Audio Tokenization 3.3.1 Architecture Transformer-based bottleneck. Causality and streaming. Optimization. Quantization rate. Adversarial-only training. 3.3.2 Learning semantic-acoustic tokens with a split RVQ 3.4 Generative Audio Modeling 3.4.1 Hierarchical autoregressive modeling with RQ-Transformer RQ-Transformer. 3.4.2 Audio modeling Acoustic delay. 3.4.3 Multi-stream modeling 3.4.4 Inner Monologue Aligning text and audio tokens. Deriving streaming ASR and TTS. Joint sequence modeling for Moshi. Inference of Moshi. 4 Datasets and Training 4.1 Text Data 4.2 Audio Data 4.3 Speech-Text Instruct Data 4.4 Training Stages and Hyper-parameters Helium pre-training. Moshi pre-training. Moshi post-training. Moshi finetuning. TTS Training. Training loss. 5 Evaluation 5.1 Text Language Modeling Metrics. Baselines. Results. 5.2 Audio Tokenization Metrics. Baselines. Results - Semantic tokens. Results - Acoustic tokens. Discussion. 5.3 Ablations on Generative Modeling Metrics. Results - RQ-Transformer. Results - Additional ablations. Discussion. 5.4 Audio Language Modeling Metrics. Baselines. Results. Discussion. 5.5 Spoken Question Answering Metrics. Baselines. Results. Discussion. 5.6 Quality and Statistics of Generated Dialogues Metrics. Baselines. Results. 5.7 Streaming ASR and TTS Metrics. Results. Discussion. 5.8 Compressing Moshi and Impact on Speech Quality Quantization Format. Results - Linguistic evaluation. Results - Audio Quality. Discussion. 6 Safety 6.1 Toxicity Analysis 6.2 Regurgitation Analysis Evaluation protocol. Results & Impact of fine-tuning. 6.3 System Voice Consistency 6.4 Identification of the Content Generated by Moshi: Watermarking Evaluation of signal-based watermarking. Exploration on generative-based watermarking for audio. Discussion on generative audio watermarking. 7 Conclusion References A Additional Ablation on Mimi Codec B Audio Matching and Deduplication Constellation map. Hash encoding. Pair-wise matching and one-to-many comparison. De-duplication: Signature fused set. C Delayed text LM as a zero-shot streaming ASR and TTS ASR mode. TTS mode. Multi-stream TTS. D Characterizing Audio Artifacts Caused by Quantization Repetitive text. Silence vs. background noise. Bad audio quality. E Safety and Toxicity F Synthetic Transcripts Generation for Fine-tuning F.1 General knowledge F.2 Misspelling F.3 Voice Instructions License: CC BY-NC-SA 4.0 arXiv:2410.00037v2[eess.AS] 02 Oct 2024 Moshi: a speech-text foundation model for real-time dialogue \name Alexandre Défossez ∗ \email alex@kyutai.org \name Laurent Mazaré ∗ \name Manu Orsini \name Amélie Royer \name Patrick Pérez \name Hervé Jégou \name Edouard Grave ∗ \name Neil Zeghidour ∗ \email neil@kyutai.org \addr Kyutai \addr ∗ Equal contribution Abstract We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning— such as emotion or non-speech sounds— is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this “Inner Monologue” method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at github.com/kyutai-labs/moshi . Keywords: speech, text, multimodal, foundation, spoken dialogue 1 Introduction Voice has provided a convenient interface to early conversational systems, from Alexa 1 1 1 https://www.alexa.com to Siri 2 2 2 https://www.apple.com/siri and Google Assistant. 3 3 3 https://assistant.google.com/ In this context, a “wake word” spoken by the user typically triggers an automatic speech recognition (ASR) system which transcribes the subsequent user’s request. Then, a natural language understanding (NLU) pipeline converts this query to a structured format used to produce a text answer through natural language generation (NLG). Eventually, a text-to-speech (TTS) system tells the answer back to the user. While this process can handle short, constrained interactions (e.g. triggering an action or retrieving a fact), the rise of large language models (LLMs) (Brown et al., 2020 ; Hoffmann et al., 2022 ; Touvron et al., 2023a ) has called for a consequent extension of voice interfaces to multi-turn, open-ended conversations. A solution to this challenge is handling the NLU and NLG with an LLM, while the ASR and TTS provide the voice interface during the user’s and the system’s turn respectively (Llama, 2024 ) . This framework supports the current generation of spoken dialogue systems such as Gemini (Gemini et al., 2023 ) or ChatGPT. 4 4 4 https://openai.com/index/chatgpt-can-now-see-hear-and-speak/ Yet, the experience offered by these interfaces remains far from natural conversations. First, latency compounds along the many components of these pipelines, resulting in a typical global latency of several seconds. This is unlike natural conversations which demonstrate response times of a few hundred milliseconds. Second, as language understanding and generation happens in the textual domain, any non-written information is ignored by the model. This goes from paralinguistic information, such as emotion and accent, to non-speech audio, such as surrounding acoustic events. Finally, these models remain fundamentally turn-based, assuming that dialogue is a sequence of well-defined single-speaker segments. While this paradigm is suited to text dialogue, it falls short in modeling aspects of spoken conversations such as interruptions, overlapping speech— which amounts for 10 10 to 20 % 20\% of spoken time (Çetin and Shriberg, 2006 ) —and backchanneling (i.e. non-interrupting interjections such as “OK” or “I see”). In this work we introduce Moshi, a speech-text foundation model and real-time spoken dialogue system that aims at solving the aforementioned limitations: latency, textual information bottleneck and turn-based modeling. Moshi augments a text LLM backbone with a smaller audio language model (Borsos et al., 2022 ; Yang et al., 2023 ) that ingests and predicts discrete audio units. This removes the information bottleneck of text by understanding inputs and generating outputs directly in the audio domain, while benefiting from the knowledge and reasoning abilities of the underlying text LLM. We extend previous work on audio language models and design a streaming, hierarchical architecture, with a theoretical latency of 160   ms 160\text{\,}\mathrm{ms} —lower than the 230   ms 230\text{\,}\mathrm{ms} average in natural conversations measured over 10 languages (Stivers et al., 2009 ) . We furthermore introduce the first multi-stream audio language model, i.e. a model that explicitly processes the input and output audio streams jointly into two autoregressive token streams. This altogether removes the concept of speaker turn and thus allows training the model on natural conversations with arbitrary dynamics including overlap and interruptions. Our resulting model is the first full-duplex — it always listens and always generates sound, either speech or silence—real-time conversational LLM. We summarize our contributions below: • We present Helium, a 7B-parameter text LLM that we pretrain on 2.1 2.1 T tokens of public English data. Section 3.2 describes the architecture and training of the model, while Section 4.1 provides details on the pretraining data collection and filtering. • We train Mimi, a neural audio codec (Zeghidour et al., 2022 ; Défossez et al., 2023 ) that converts audio into the discrete tokens predicted by Moshi and back, using residual vector quantization (RVQ). Audio language models typically combine such acoustic tokens with semantic tokens from a self-supervised speech model as it is necessary to produce intelligible speech in absence of text conditioning (Borsos et al., 2022 ) . We rather extend the approach of Zhang et al. ( 2024b ) by distilling semantic information into the first level of acoustic tokens and introduce improved training tricks. Section 3.3 describes the architecture and training of Mimi while Section 5.2 details ablation studies. • We propose Moshi, a new architecture for audio language modeling, which combines Helium with a smaller Transformer (Vaswani et al., 2017 ) model to predict audio tokens in a hierarchical and streaming fashion. We show how challenging it is for such unconditioned audio language models to generate intelligible speech, and we provide solutions that outperform the intelligibility and audio quality of non-streaming models while generating audio in a streaming fashion. We furthermore extend this architecture to model several audio streams in parallel, allowing for a conceptually and practically simple handling of full-duplex dialogues with arbitrary dynamics. Section 3.4 describes this architecture. • In Section 3.4.4 , we introduce Inner Monologue, a new training and inference setup for audio language models that significantly improves the factuality and linguistic quality of generated speech by predicting time-aligned text tokens before audio tokens. Moshi is a speech-to-speech model as it allows reasoning about non-linguistic information, both from the user audio and from Moshi’s audio. Yet, this is not incompatible with Moshi producing text along its speech output. Based on the past observation (Borsos et al., 2022 ; Zhang et al., 2024b ) that coarse-to-fine generation (from semantic to acoustic tokens) is critical to generating consistent speech, we extend this hierarchy to using text tokens as a per-timestep prefix to the semantic token. Our experiments show that not only this drastically improves the length and quality of generated speech, but we also show how forcing a delay between text and audio tokens allows deriving streaming ASR and streaming TTS from a Moshi model. • We evaluate all components of Moshi along several axes, including text understanding, speech intelligibility and consistency, audio quality and spoken question answering. Our experiments, reported in Section 5 , show that our model is state of the art among existing speech-text models for speech modeling and spoken question answering while being streaming compatible and able to model several minutes of context ( 5   min 5\text{\,}\mathrm{min} in our experiments). We encourage the reader to talk to Moshi using our web demo. 5 5 5 https://moshi.chat/ 2 Related Work Audio Language Modeling. Early developments in speech foundation models have improved speech understanding across many discriminative tasks, from automatic speech recognition (ASR) (Baevski et al., 2020 ; Radford et al., 2023 ; Zhang et al., 2023b ) to speaker verification (Chen et al., 2022 ) and speech classification (Yang et al., 2021 ) . A key factor in this development is self-supervised learning (Hsu et al., 2021 ; Baevski et al., 2020 ; Chen et al., 2022 ) which allows learning generic, discriminative speech representations. As these speech understanding models build on previous work done on masked language modeling for text (Devlin et al., 2019 ) , generative text pretraining (Radford et al., 2018 ) has similarly inspired a large family of speech generation models. In particular, Lakhotia et al. ( 2021 ) propose quantizing aforementioned self-supervised representations. The resulting discrete audio tokens represent a speech segment as a sequence of categorical variables, thus casting speech generation as a language modeling task. AudioLM (Borsos et al., 2022 ) furthermore combines these semantic tokens with acoustic tokens from a neural audio codec (Zeghidour et al., 2022 ) , which allows for modeling arbitrary voices, recording conditions and non-speech sounds. These audio language models have redefined the state of the art in speech generation, from text-to-speech (Wang et al., 2023 ; Kharitonov et al., 2023 ) to speech-to-speech translation (Rubenstein et al., 2023 ; Reid et al., 2024 ) and speech enhancement (Yang et al., 2023 ) . Beyond these supervised tasks, a parallel line of work has explored training and scaling unsupervised audio-only models, trained for autoregressive speech generation (Dunbar et al., 2021 ; Lakhotia et al., 2021 ; Borsos et al., 2022 ) . The abilities of these models have progressively expanded, from generating short sentences in a single speaker voice (Lakhotia et al., 2021 ) to producing meaningful and consistent speech continuations across dozens of seconds in arbitrary voices and conditions (Borsos et al., 2022 ) , thanks to a hierarchical modeling of semantic and acoustic tokens. A main challenge is that audio requires the modeling of long sequences, up to a few minutes, to produce meaningful and exploitable outputs. However, latent representations for audio are typically less compact than equivalent representations for text. Thus, discrete representations from neural audio codecs require multiple predictions per timestep when modeled autoregressively. (Liu et al., 2023b ) and (Evans et al., 2024 ) use latent diffusion (Ho et al., 2020 ) for general audio and music modeling to alleviate the need for hierarchical discrete tokens. However, these methods cannot be used in a streaming fashion, and it is unclear whether they could generate consistent speech. Copet et al. ( 2023 ) instead show that the number of auto-regressive steps can be reduced by introducing a delay between the different levels of tokens, and performing parallel prediction over them. Inspired by the RQ-Transformer method by Lee et al. ( 2022 ) and the hierarchical MegaByte transformer model (Yu et al., 2024 ) , Yang et al. ( 2023 ) and Zhu et al. ( 2024 ) leverage a smaller nested transformer to model the different tokens at a single time step. In this work, we extend these previous works to push the limits of autoregressive speech generation by proposing a scalable hierarchical modeling of audio tokens which can handle several minutes of context while generating audio in real time. Still, while speech-only models learn linguistic structure—lexicon, syntax, semantics— from raw speech (Dunbar et al., 2021 ) , they typically demonstrate poor-to-nonexistent factual knowledge and reasoning abilities. This has led to the development of speech-text models, intended to combine the knowledge and reasoning abilities of text models with the generative power of audio models. Speech-text Models. Such models typically start from a pretrained text language model and either finetune it to predict audio (Hassid et al., 2023 ) , or propose a speech-text finetuning task (Rubenstein et al., 2023 ; Maiti et al., 2023 ; Nachmani et al., 2024 ; Nguyen et al., 2024 ; Mitsui et al., 2024 ; Zhang et al., 2024a ) : For instance, AudioPALM (Rubenstein et al., 2023 ) starts from a pretrained PALM (Chowdhery et al., 2022 ) model, and extends its text vocabulary with semantic audio tokens. Then, the model is trained for a mixture of speech-text tasks, including TTS, ASR and speech-to-speech translation. VoxTLM (Maiti et al., 2023 ) adopts a similar approach for TTS and ASR. While these models are trained in a supervised fashion with specific input and output sequences, Spirit-LM (Nguyen et al., 2024 ) uses temporal alignment between speech and its transcript to perform modality switch (from speech tokens to text tokens, or conversely) inside a sequence. This allows the model to learn consistent internal representations of language regardless of it being represented as text or speech, as measured through commonsense evaluation. Another approach, adopted by Spectron (Nachmani et al., 2024 ) , SpeechGPT (Zhang et al., 2023a ) and PSLM (Mitsui et al., 2024 ) , combines speech and text in a hierarchical manner rather than as interchangeable representations. Similar to how AudioLM (Borsos et al., 2022 ) decomposes speech generation into predicting semantic tokens and then acoustic tokens, Spectron and SpeechGPT use a “Chain-of-Modality” and first produce an utterance as text tokens, subsequently used as a prefix to generate speech. This allows guiding speech generation with the output of an underlying text LLM, however this is fundamentally incompatible with live interactions as the model needs to produce an entire answer as text before it starts speaking. PSLM alleviates this limitation by modeling text and speech tokens in parallel. In this work, we propose Inner Monologue as a main architectural and training component to combine aligned text and speech data. Inner Monologue decomposes speech into a chain of text, semantic and acoustic tokens, and predicts this structured sequence in a hierarchical manner. Unlike Spirit-LM, this allows representing all utterances both as text and speech, rather than switching between modalities; In addition, the integration of acoustic tokens into the same generative model enables generating arbitrary voices and conditions, rather than a single speaker. Besides, this hierarchical modeling described in Section 3.4.4 allows decomposing the generation task without increasing the sequence length of the Transformer (Vaswani et al., 2017 ) outputs, unlike Chain-of-Modality, while benefiting from producing text a prefix to audio tokens rather than in parallel like PSLM. Moreover, Inner Monologue decomposes speech on a per-frame basis, which means that each prediction step outputs a speech frame. This is unlike Spectron and SpeechGPT which require generating a complete sequence as text before generating audio tokens, and this makes Moshi compatible with real-time generation. Moreover, we show in Section 3.4.4 how Inner Monologue, when combined with a delay between token types, allows deriving streaming TTS and ASR systems from Moshi. Finally, while Spectron, SpeechGPT and PSLM model both user and system speech and text tokens into a single stream, which requires properly segmented turns, Moshi benefits from a novel multi-stream architectures which removes the concept of speaker turns and allows for modeling any type of overlap, interruptions and interjections. Spoken Dialogue Models. Spoken dialogue is one of the less explored tasks in speech generation, as it requires addressing several challenges: 1) The model should run in real time and allow for long conversations in full-duplex—the model always listens and can speak at any moment; 2) it should be speech-to-speech to handle paralinguistic communication; 3) it should display knowledge and reasoning abilities that make it amenable to helpful and enjoyable conversations. Spectron benefits from its underlying text LLM (as measured by spoken question answering), however it is not compatible with real-time generation due to Chain-of-Modality. PSLM proposes generating speech and text tokens in parallel to reduce this latency, however it reduces the quality of answers, and the model still relies on ASR, which removes paralinguistic information. More importantly, these models cannot handle full-duplex communication, where there is no boundary between speaker turns, as any side of the conversation can be active at any time. An attempt at modeling these dynamics has been proposed by Wang et al. ( 2024 ) , with an ASR system running in parallel to a text generator that feeds into a streaming TTS. While this allows modeling more complex scenarios than previous approaches, it still relies on a cascaded pipeline and models both the user’s and the system’s speech into a single token stream, which is challenging in presence of significant overlap. The only previous full-duplex dialogue system is dGSLM (Nguyen et al., 2023 ) , which models user and system speech as separate audio token streams and proposes a Siamese architecture to process both streams jointly. While dGSLM is full-duplex, it remains a proof-of-concept: it does not run in an online fashion, it does not benefit from the knowledge of a text language model, and it does not model acoustic information as it only models semantic tokens. Moshi addresses these limitations altogether: by modeling two streams of semantic and acoustic tokens hierarchically, Moshi is full duplex and can exploit all the information from the user (linguistic and non-linguistic) while producing speech in real time. Thanks to text pretraining and Inner Monologue, Moshi benefits from the knowledge of its Helium backbone. Finally, as the model produces acoustic tokens along with text and semantic tokens, it can generate an arbitrary range of emotions, voices and acoustic conditions. To the best of our knowledge, Moshi is the first audio language model that successfully addresses the many aforementioned challenges of spoken dialogue. 3 Model 3.1 Overview Figure 1: Overview of Moshi . Moshi is a speech-text foundation model which enables real-time spoken dialogue. The main components of Moshi’s architecture are: a bespoke text language model backbone (Helium, see Section 3.2 ) ; a neural audio codec with residual vector quantization and with semantic knowledge distilled from a self-supervised speech model (Mimi, Section 3.3 ); the streaming, hierarchical generation of semantic and acoustic tokens for both the user and Moshi, along with time-aligned text tokens for Moshi when using Inner Monologue ( Section 3.4 ). Moshi is a multi-stream speech-to-speech Transformer model, which allows for full-duplex spoken dialogue with a user thanks to an innovative architecture summarized in Figure 1 . Moshi is built on top of Helium, a text LLM which we build from scratch ( Section 3.2 ), relying on high-quality text data to provide strong reasoning abilities to the model. We also propose Inner Monologue ( Section 3.4.4 ), a training and inference procedure in which we jointly model text and audio tokens. This allows the model to fully exploit the knowledge imparted from the text modality, while remaining a speech-to-speech system. To enable real-time dialogue, we also design Moshi as a multi-stream architecture from the get-go ( Section 3.4.3 ): The model is able to both speak and listen to the user at the same time, and does not need to explicitly model speaker turns. In addition, to capture the input user audio and output Moshi’s voice with high quality and in an efficient manner, we propose Mimi ( Section 3.3 ), a neural audio codec combining semantic and acoustic information into a single tokenizer by using residual vector quantization and knowledge distillation. To jointly model the audio streams from Moshi and the user, as well as Moshi’s text tokens, we rely on a Depth Transformer compatible with streaming inference ( Sections 3.4.1 , 3.4.2 ). In this section, we further detail each of these components. We then describe the training datasets and the different training phases we used to train Moshi in Section 4 . Finally, in Section 5 , we report thorough evaluation results on Moshi’s abilities, both linguistic and acoustic, as well as ablation experiments on its main components, while Section 6 provides analyses on the safety of our system. 3.2 The Helium Text Language Model 3.2.1 Architecture Helium is an autoregressive language model, based on the Transformer architecture (Vaswani et al., 2017 ) . Following previous work in this area, we make the following changes to the original architecture: First, we use RMS normalization (Zhang and Sennrich, 2019 ) at the input of the attention blocks, the feed-forward blocks and the output linear layer of the model. We use rotation positional embeddings (Su et al., 2024 , RoPE) , a context length of 4,096 tokens and FlashAttention (Dao et al., 2022 ) for efficient training. Finally, we change the architecture of the feed-forward blocks and use Gated Linear Units (Shazeer, 2020 ) , with the SiLU activation as a gating function (Hendrycks and Gimpel, 2016b ) . Our tokenizer is based on the unigram model from SentencePiece (Kudo and Richardson, 2018 ) , and contains 32,000 elements mostly targeting English. We split all numbers into single digits, and use byte-backoff to ensure that our tokenizer does not lose information. We train the model with the AdamW (Loshchilov and Hutter, 2017 ) optimizer, with a fixed learning rate followed by a cosine learning rate decay (Loshchilov and Hutter, 2016 ) . Table 1: Models’ hyper-parameters . Hyper-parameters for the architecture and training of our 7B-parameter Helium language model and of Moshi, our speech-text dialogue model. The training of Moshi goes through 4 phases: Pre-training on unsupervised data (with Temporal Transformer initialized from Helium); Post-training with simulated multi-stream based on diarization; Fine-tuning on the Fisher dataset (Cieri et al., 2004 ) to gain its fully duplex capabilities; Instruction fine-tuning on a custom dataset built from synthetic interaction scripts. During the pre-training phase, we keep training half of the time on full text batches from the same dataset as used for Helium, using a separate optimizer state. Helium training Moshi training Hyper-parameter pre-training pre-training post-training fisher fine Temporal Transformer Model dimension 4096 same MLP dimension 11264 Number of heads 32 Number of layers 32 Context size 4096 3000 steps, e.g. 4 min. Learning rate 3 ⋅ 10 − 4 3\cdot 10^{-4} 3 ⋅ 10 − 5 3\cdot 10^{-5} 3 ⋅ 10 − 6 3\cdot 10^{-6} 2 ⋅ 10 − 6 2\cdot 10^{-6} 2 ⋅ 10 − 6 2\cdot 10^{-6} Depth Transformer Model dimension - 1024 MLP dimension - 4096 Number of heads - 16 Number of layers - 6 Learning rate - 2 ⋅ 10 − 4 2\cdot 10^{-4} 5 ⋅ 10 − 5 5\cdot 10^{-5} 4 ⋅ 10 − 6 4\cdot 10^{-6} 2 ⋅ 10 − 6 2\cdot 10^{-6} Input / Output space Text cardinality 32000 32000 Audio cardinality - 2048 Frame rate - 12.5 Hz Common parameters Batch size (text) 4.2M tok. 1.2M tok. 1.2M tok. - - Batch size (audio) - 16h 8h 40min 2.7h Training steps 500k 1M 100k 10k 30k LR Schedule cosine cosine - - - Acoustic delay - 2 1 1 1 Text delay - ± \pm 0.6 0 0 0 3.2.2 Pre-training data filtering Training data is one of the critical ingredients to train LLMs: we now describe our method to obtain a large and high-quality text dataset. We start from high-quality data sources, such as Wikipedia, Stack Exchange and a large collection of scientific articles. As the quantity of data from these sources is too small to train a LLM, we also rely on web crawled data, specifically from CommonCrawl, to extend our dataset. See more details on data sources in Section 4.1 . Web data requires extensive processing to obtain a high-quality training set: we perform deduplication, language identification and quality filtering. In the following, we describe each operation in more details. Deduplication. We start from the WET files, which contain only the text content of web-pages, which was extracted by the CommonCrawl project. Because this format contains all the text of a page, it includes a lot of boilerplate such as navigation menus. Thus, the first step of our pipeline is to deduplicate each shard (there is 100 shards per crawl) at the line level , to remove this boilerplate. To do so, we compute the FNV-1a 6 6 6 http://www.isthe.com/chongo/tech/comp/fnv hash of each line, and use a bloom filter to remove duplicates. We also train a fastText (Joulin et al., 2016 ) classifier on duplicates vs. non-duplicates, to perform fuzzy deduplication: here we only remove blocks of at least 3 consecutive lines that are classified as duplicates. Language identification. Once deduplication is performed, we apply a language identifier based on fastText to keep English data only. Language identification is performed at the document level, and we only keep documents above a certain threshold (0.85). Quality filtering. The last step is to filter the remaining data, to keep high-quality webpages only. To perform this step, we train a fastText classifier on lines from our high quality data sources and from random CommonCrawl webpages. We obtain a classifier with 9 categories, corresponding to our different high quality sources such as Wikipedia or Wikibooks and to subsets of StackExchange such as STEM or humanities. The motivation is to obtain a finer control over which documents to keep, not only based on similarity to high quality sources, but also based on their domains. This classifier is applied at the line level , and an aggregated score is obtained by computing the average scores of each line, weighted by their length. Again, we keep documents corresponding to scores above a certain threshold. 3.3 Audio Tokenization To discretize waveforms into audio tokens, we introduce Mimi, a neural audio codec (Zeghidour et al., 2022 ; Défossez et al., 2023 ) that operates as an autoencoder with a discrete bottleneck (van den Oord et al., 2017 ) . In the literature, and following the terminology defined by Borsos et al. ( 2022 ) , these tokens are referred to as acoustic tokens, as they model fine audio details and are optimized for high-quality reconstruction. While these acoustic tokens provide appropriate targets for conditioned text-to-audio models (e.g. text-to-speech (Wang et al., 2023 ) or text-to-music (Copet et al., 2023 ) ), unconditioned speech generation requires combining them with semantic tokens extracted from self-supervised speech models (Baevski et al., 2020 ; Hsu et al., 2021 ; Chung et al., 2021 ) . Unlike their acoustic counterpart, semantic tokens do not allow for reconstructing high-quality audio but correlate strongly with linguistic content. This similarity with language allows generating intelligible and consistent speech, even without text conditioning, by using semantic audio tokens as a prefix to predicting acoustic tokens. Yet, this hybrid tokenization approach is not compatible with real-time generation. Semantic tokens are typically not causal and can thus only be computed in an offline manner. Moreover, generating acoustic and semantic tokens with separate encoders represents a non-negligible computational burden. Consequently, and taking inspiration from previous work on SpeechTokenizer (Zhang et al., 2024b ) , Mimi uses distillation to transfer non-causal, high-level semantic information into the tokens produced by a causal model, allowing for streaming encoding and decoding of semantic-acoustic tokens. Figure 2: Architecture and training of Mimi, our neural audio codec, with its split residual vector quantization . During training (blue part, top), we distill non-causal embeddings from WavLM (Chen et al., 2022 ) into a single vector quantizer which produces semantic tokens, and is combined with separate acoustic tokens for reconstruction. 3.3.1 Architecture Our baseline architecture takes inspiration from SoundStream (Zeghidour et al., 2022 ) and Encodec (Défossez et al., 2023 ) and consists of a SeaNet (Tagliasacchi et al., 2020 ) autoencoder and a Residual Vector Quantizer (Zeghidour et al., 2022 ) . The encoder projects a single-channel waveform x ∈ ℝ L x\in\mathbb{R}^{L} to a latent representation enc ​ ( x ) ∈ ℝ S × D \mathrm{enc}(x)\in\mathbb{R}^{S\times D} by cascading residual convolutional blocks that interleave dilated (van den Oord et al., 2016 ) and strided convolutions along with ELU (Clevert et al., 2016 ) non-linearities and Weight Normalization (Salimans and Kingma, 2016 ) . All convolutions are causal, such that this autoencoder can run in a streaming fashion. With 4 convolutional blocks and respective striding factors ( 4 , 5 , 6 , 8 ) (4,5,6,8) , and a final 1D convolution with stride 2, Mimi’s encoder projects a 24kHz waveform to a latent representation of 12.5 12.5 frames per second and dimension D = 512 D=512 . Symmetrically, the decoder adopts a similar structure but with transposed convolutions rather than strided ones, to project the latent representation back to 24kHz audio. We discretize the latent space with a Residual Vector Quantizer (Zeghidour et al., 2022 ) , which iteratively applies vector quantization (VQ) to the residuals of the previous quantizer. With Q Q quantizers, each with a codebook of N A N_{A} centroids, the RVQ discretizes the latent space into { 1 , … , N A } S × Q \{1,\dots,N_{A}\}^{S\times Q} . As a baseline, we train this model with a combination of reconstruction and adversarial losses, following the setup of Encodec (Défossez et al., 2023 ) . We detail below the main changes of Mimi with respect to this default configuration. Transformer-based bottleneck. To improve the ability of Mimi to encode speech into compact representations while reconstructing high-quality audio, we add Transformer modules in the bottleneck, one right before quantization and one after. These Transformers have 8 layers, 8 heads, RoPE position encodings, a finite context of 250 frames (20 seconds), GELU (Hendrycks and Gimpel, 2016a ) activations, a model dimension of 512 and an MLP dimension of 2048. To stabilize training, we use LayerScale (Touvron et al., 2021 ) , with initialization of the diagonal values at 0.01 0.01 . Both Transformers use causal masking, which preserves the compatibility of the whole architecture with streaming inference. Both Transformers prove to be useful with regard to perceived audio quality, while Transformer in the encoder also improves the distillation of semantic information described below (see Table 3 for ablation studies). Causality and streaming. With the mentioned hyper-parameters, Mimi is causal and can be used in a streaming fashion, both for encoding and decoding. Both its initial frame size and overall stride correspond to 80ms, meaning that given a first audio frame of 80ms, Mimi outputs a first latent timestep, which can be decoded to 80ms of output audio. Optimization. Unlike purely convolutional codecs that use Adam (Kingma and Ba, 2015 ) , the introduction of Transformers into the architecture requires additional regularization with weight decay along with using the AdamW (Loshchilov and Hutter, 2019 ) optimizer. More precisely, we apply weight decay only to the parameters of the Transformers, with a weight of 5 ⋅ 10 − 2 5\cdot 10^{-2} . We use a learning rate of 8 ⋅ 10 − 4 8\cdot 10^{-4} , a momentum decay of 0.5 0.5 and a decay of the squared gradient of 0.9 0.9 , and an exponential moving average of weights with a decay of 0.99 0.99 . We train with a batch size of 128 128 on random windows of 12   s 12\text{\,}\mathrm{s} , for 4M steps, while the context of Transformers is limited to 10s (250 frames before the last downsampling layer of the encoder, and symmetrically for the decoder). Quantization rate. We use Q = 8 Q=8 quantizers, each with a codebook size of N A = 2048 N_{A}=2048 . At 12.5Hz, this represents a bitrate of 1.1kbps. While the latent dimension is 512, we project embeddings to 256 dimensions before applying the RVQ, and project back to 512 before the decoder. Consistently with previous work, we use quantizer dropout (Zeghidour et al., 2022 ) to provide the codec with bitrate scalability. We moreover follow the observation of Kumar et al. ( 2023 ) that not applying quantization with a certain probability during training improves audio quality. More precisely, we only apply quantization 50 % 50\% of the time, on a per-sequence basis, during training. Unlike Kumar et al. ( 2023 ) , this means passing unquantized embeddings to the decoder, rather than passing embeddings quantized with all quantizers. Table 3 shows that this significantly improves objective quality metrics, while human evaluations are not conclusive. Across our experiments, we make the somehow counter-intuitive observation that this gain gets more significant as we lower the bitrate. Adversarial-only training. As a baseline, we train Mimi with the same combination of reconstruction and adversarial losses as Défossez et al. ( 2023 ) , namely a multi-scale mel-spectrogram reconstruction loss along with a multi-scale STFT discriminator. The exact parameters can be found in the Audiocraft repository. 7 7 7 https://github.com/facebookresearch/audiocraft/blob/main/config/solver/compression/default.yaml While previous neural codecs rely on such combinations of reconstruction and adversarial losses, we experiment with pure adversarial training, where we only keep the feature loss and discriminator loss. We note that this was previously experimented in the context of bandwidth extension by Tagliasacchi et al. ( 2020 ) and Hauret et al. ( 2023 ) . While removing reconstruction losses majorly degrades objective metrics, we observed during development that the resulting audio sounded much better than expected based on aforementioned metrics. Subjective evaluations reported in Table 4 confirm this observation and demonstrate a remarkable boost in audio quality from training with adversarial losses only. 3.3.2 Learning semantic-acoustic tokens with a split RVQ Similarly to SpeechTokenizer (Zhang et al., 2024b ) , we distill semantic information from a self-supervised model (WavLM 8 8 8 https://huggingface.co/microsoft/wavlm-large (Chen et al., 2022 ) in our case) into the first level of the RVQ. WavLM projects a 16kHz waveform into 1024-dimensional embeddings sampled at 50Hz, while Mimi projects a 24kHz waveform into 512-dimensional at 12.5Hz. During training, we thus produce targets for distillation by downsampling the input waveform to 16kHz before computing WavLM embeddings followed by average pooling with a stride of 4 and a kernel size of 8, to reach 12.5 Hz. Interestingly, we observed that it was critical for performance to perform this average pooling in a non-causal way, which is compatible with streaming inference as these embeddings are only used during training. We apply a linear projection with an output dimension of 1024 to the output of the first RVQ level, parallel to the actual embedding going into the decoder. We then compute a cosine distance between the output of the first quantizer and the transformed WavLM embeddings, to perform distillation. Table 3 shows that this distillation loss conflicts with reconstruction and adversarial losses targeting quality. Indeed, while distillation significantly improves the phonetic discriminability of the first quantizer (as measured by ABX (Schatz et al., 2013 ) ), it also affects audio quality negatively. We hypothesize that this is due to distilling semantic information into the first level of a single RVQ: As higher-order quantizers operate on the residual of the first one, the latter needs to trade audio quality for phonetic discriminability. We address this issue by proposing a split RVQ. Rather than a single RVQ with 8 levels, we distill semantic information into a plain VQ and apply an RVQ with 7 levels in parallel. We sum their outputs, such that while both can be used for reconstruction, we remove the constraint that acoustic information should be conserved in the residual of the semantic quantizer. Figure 2 illustrates this architecture and Table 3 shows that this solution provides a better semantic-acoustic trade-off overall. 3.4 Generative Audio Modeling We now describe how we extend the base Helium model to support the modeling of the audio tokens provided by the Mimi codec. With our goal of achieving realistic spoken dialogue interactions, we further show how to model not just a single stream of audio, but two at the same time, one representing the user, and one the system. Finally, we detail a novel feature, the Inner Monologue , which consists in a joint modeling of the textual and audio modalities on the system side, to improve the quality of interactions. 3.4.1 Hierarchical autoregressive modeling with RQ-Transformer Let U ∈ { 1 , … , N } S U\in\{1,\ldots,N\}^{S} be a discrete random sequence, with cardinality N N and a sequence length S S . For convenience, we also denote U 0 = 0 U_{0}=0 , a deterministic initial token value. Autoregressive modeling consists in estimating the joint distribution ℙ ​ [ U 1 , … , U S ] \mathbb{P}\left[U_{1},\ldots,U_{S}\right] through estimating the conditional distributions ℙ ​ [ U s | U 0 , … ​ U s − 1 ] \mathbb{P}\left[U_{s}|U_{0},\ldots U_{s-1}\right] for all steps 1 ≤ s ≤ S 1\leq s\leq S . Text language models, such as GPT (Radford et al., 2019 ) or Helium, fit this paradigm. Figure 3: Architecture of the RQ-Transformer. The RQ-Transformer breaks down a flattened sequence of length K ⋅ S K\cdot S into S S timesteps for a large Temporal Transformer which produces a context embedding used to condition a smaller Depth Transformer over K K steps. This allows scaling to longer sequences by increasing S S —or to a higher depth by increasing K K — than modeling the flattened sequence with a single model. In this figure, we use K = 4 K=4 for the sake of illustration. When modeling spoken language, relying on the tokenized text yields a much more compact representation than audio tokens: Using the Mimi codec introduced in Section 3.3 , with Q = 8 Q=8 codebooks at a frame rate of 12.5hz, one would require a sequence length of 100 100 steps per second of audio to generate. To model 5 minutes of audio, this would amount to 30,000 timesteps, which represents a significant computational cost and generating 100 tokens per second is incompatible with streaming inference. As a comparison, a sample of English speech can be represented with around 3 to 4 text tokens per second. We are interested in modeling not just a single sequence ( U s ) (U_{s}) , but multiple sub-sequences, e.g. different audio codebooks, along with an optional text stream. We can stack those sub-sequences as V s , k V_{s,k} for 1 ≤ s ≤ S 1\leq s\leq S and 1 ≤ k ≤ K 1\leq k\leq K . Similarly, we define V 0 , k = 0 V_{0,k}=0 , a deterministic initial token value for all sub-sequences. For each 1 ≤ s ≤ S 1\leq s\leq S and 1 ≤ k ≤ K 1\leq k\leq K , V s , k ∈ { 1 , … , N k } V_{s,k}\in\{1,\ldots,N_{k}\} , where N k N_{k} is the cardinality of the k k -th sub-sequence. One can flatten the K K sequences into a single one, increasing the number of predictions by K K . Lee et al. ( 2022 ) propose using a smaller autoregressive model along the dimension K K , combined with a larger model along the time dimension, forming a RQ-Transformer. Later, Yu et al. ( 2024 ) suggested a similar approach for byte-level modeling while Yang et al. ( 2023 ) and Zhu et al. ( 2024 ) applied it to audio token modeling. RQ-Transformer. Formally, the RQ-Transformer consists in two Transformer models, as illustrated in Figure 3 . It consists of a Temporal Transformer, e.g. with the same architecture as the one described for Helium in Section 3.2 , and a smaller Depth Transformer. We denote Tr Temp \mathrm{Tr}_{\mathrm{Temp}} the function represented by the Temporal Transformer, and Tr Depth \mathrm{Tr}_{\mathrm{Depth}} the one for the Depth Transformer. For simplicity, and for all steps s ≤ S s\leq S , we denote V s = ( V s , 1 , … , V s , K ) V_{s}=(V_{s,1},\ldots,V_{s,K}) the joint value of all sub-sequences at step s s . For a given sequence step 1 ≤ s ≤ S 1\leq s\leq S , the Temporal Transformer maps ( V 0 , … , V s − 1 ) (V_{0},\ldots,V_{s-1}) to a temporal context vector z s = Tr Temp ​ ( V 0 , … , V s − 1 ) ∈ ℝ d . z_{s}=\mathrm{Tr}_{\mathrm{Temp}}(V_{0},\ldots,V_{s-1})\in\mathbb{R}^{d}. (1) If we further take a sub-sequence index 1 < k ≤ K 1 1 . \begin{cases}\mathrm{softmax}(l_{s,1})&\approx\mathbb{P}\left[V_{s,1}|V_{0},\ldots,V_{s-1}\right]\\ \mathrm{softmax}(l_{s,k})&\approx\mathbb{P}\left[V_{s,k}|V_{0},\ldots,V_{s-1},V_{s,1},\ldots V_{s,k-1}\right]\quad\text{if $k>1$}.\end{cases} (3) Importantly, the number of steps in the Temporal Transformer is always equal to S S , rather than K ⋅ S K\cdot S , and the number of steps in the Depth Transformer is at most K K . In practice, the Temporal Transformer receives at each step s s as input the sum of K K learnt embedding tables representing the value for the last V s − 1 V_{s-1} . Given 1 < k ≤ K 1 1 V s , q = 0 if s < τ + 1 , q > 1 . \begin{cases}V_{s,1}=A_{s,1}&\\ V_{s,q}=A_{s-\tau,q}&\quad\text{if}\quad s\geq\tau+1,q>1\\ V_{s,q}=0&\quad\text{if}\quad s<\tau+1,q>1.\\ \end{cases} (4) Note that using RQ-Transformers to model audio was successfully used by Yang et al. ( 20