learn.ifs.name

Moshi: a speech-text foundation model for real-time dialogue

Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037)

Subscriber deck — free during early access

2 cards in “3.4.4 Inner Monologue” · whole deck

3.4.4 Inner Monologue

  1. What is Moshi’s “Inner Monologue”?
    Before the audio tokens of each frame, Moshi predicts time-aligned text tokens of its own speech (word timestamps from Whisper, with PAD and end-of-padding tokens filling the gaps). This text prefix markedly improved the linguistic quality of the generated speech.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.4.4
  2. How does Inner Monologue turn the same architecture into streaming ASR or TTS?
    By changing the delay between text and audio. Delaying text by 2 s lets the model hear the audio before writing the words: streaming ASR (5.7% WER on LibriSpeech test-clean). Delaying audio by 2 s gives the text a lookahead: streaming TTS (4.7% WER, better than Vall-E’s 5.9%).
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.4.4

Source

  • Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. In arXiv preprint arXiv:2410.00037. https://arxiv.org/abs/2410.00037