learn.ifs.name

Moshi: a speech-text foundation model for real-time dialogue

Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037)

Subscriber deck — free during early access

2 cards in “1 Introduction” · whole deck

1 Introduction

  1. Which three problems of cascaded voice assistants (ASR → LLM → TTS) does Moshi set out to fix?
    Latency of several seconds from the chained components; a text bottleneck that drops non-linguistic information such as emotion and accent; and a turn-based design that cannot handle overlap, interruptions and backchannels, which take up 10–20% of spoken time in real conversation.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §1
  2. What is Moshi, in one sentence?
    A full-duplex speech-to-speech dialogue model: one model that listens and speaks at the same time, with a theoretical latency of 160 ms (about 200 ms in practice), against an average of 230 ms between turns in natural conversation.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §1

Source

  • Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. In arXiv preprint arXiv:2410.00037. https://arxiv.org/abs/2410.00037