Moshi: a speech-text foundation model for real-time dialogue

Latency of several seconds from the chained components; a text bottleneck that drops non-linguistic information such as emotion and accent; and a turn-based design that cannot handle overlap, interruptions and backchannels, which take up 10–20% of spoken time in real conversation.

A full-duplex speech-to-speech dialogue model: one model that listens and speaks at the same time, with a theoretical latency of 160 ms (about 200 ms in practice), against an average of 230 ms between turns in natural conversation.

Helium, a 7B text LLM (trained on 2.1T English tokens) that supplies the knowledge and reasoning; Mimi, a streaming neural audio codec that turns speech into discrete tokens; and an RQ-Transformer that models Moshi’s and the user’s audio tokens, plus text, jointly.

A causal convolutional encoder with Transformers in the bottleneck downsamples it to 12.5 frames per second (80 ms each), and each frame is quantized by 8 codebooks of 2048 entries, about 1.1 kbps. The frame rate is close to that of text tokens, keeping sequences short.

The first token is distilled from WavLM (a non-causal self-supervised speech model). Distilling into the first level of a single residual quantizer hurt audio quality, so Mimi uses a split RVQ: one plain VQ for the semantic token in parallel with a 7-level RVQ for acoustics, their outputs summed.

Applying quantization only 50% of the time during training, so the decoder also sees unquantized latents (better objective metrics); and training with adversarial and feature-matching losses only, without reconstruction losses, which badly hurt objective metrics but clearly improved quality in human ratings.

With 8 codebooks at 12.5 Hz that is 100 tokens per second, 30,000 steps for 5 minutes, too slow for streaming. English speech is only about 3–4 text tokens per second, so a model that predicts a whole frame per step is much cheaper.

A large Temporal Transformer (initialized from Helium) runs once per 80 ms frame over all previous frames and outputs a context vector. A small Depth Transformer (6 layers, separate weights per level) then predicts that frame’s \(K\) tokens one after another, conditioned on the context vector and the tokens already predicted.

The acoustic tokens are shifted one or two frames after the semantic token, so the large Temporal Transformer, rather than the small Depth Transformer, models how they depend on each other. A delay of 1–2 steps greatly improved generation quality; the final model uses 1 step (160 ms theoretical latency).

It models two audio streams, its own and the user’s, as parallel sub-sequences. At inference it samples only its own stream and feeds in the real user audio. There are no turn boundaries: when Moshi is silent its stream decodes to natural silence, so it can speak, listen, or both at once.

Before the audio tokens of each frame, Moshi predicts time-aligned text tokens of its own speech (word timestamps from Whisper, with PAD and end-of-padding tokens filling the gaps). This text prefix markedly improved the linguistic quality of the generated speech.

By changing the delay between text and audio. Delaying text by 2 s lets the model hear the audio before writing the words: streaming ASR (5.7% WER on LibriSpeech test-clean). Delaying audio by 2 s gives the text a lookahead: streaming TTS (4.7% WER, better than Vall-E’s 5.9%).

Pretraining on 7M hours of Whisper-transcribed audio as a single stream (half the batches text-only, against forgetting); post-training on diarized audio split into two streams; fine-tuning on Fisher telephone conversations (2000 h, one speaker per channel) for real overlap; and instruction tuning on 20k+ hours of synthetic dialogues.

Text instruction sets like Open Hermes did not read well aloud (URLs, bullet points, long lists). Instead Helium wrote realistic spoken dialogue transcripts, and Moshi’s own multi-stream TTS rendered them, with Moshi always in one actor’s voice and the user in randomly sampled voices, plus noise, echo and gain augmentation.

A speaker-verification model compared each of Moshi’s segments in 100 h of generated dialogues with Moshi’s and the other speaker’s first segment: 98.7% were closer to Moshi’s own, with no drift over time, just from training on a single consistent voice.

Audioseal marks were detected well on unchanged audio, but passing the audio through a low-bitrate codec, Mimi itself or RVQGAN, removed them, since such codecs discard anything not needed for reconstruction.