learn.ifs.name

Moshi: a speech-text foundation model for real-time dialogue

Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037)

Subscriber deck — free during early access

1 cards in “3.1 Overview” · whole deck

3.1 Overview

  1. What are the three components of Moshi?
    Helium, a 7B text LLM (trained on 2.1T English tokens) that supplies the knowledge and reasoning; Mimi, a streaming neural audio codec that turns speech into discrete tokens; and an RQ-Transformer that models Moshi’s and the user’s audio tokens, plus text, jointly.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.1

Source

  • Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. In arXiv preprint arXiv:2410.00037. https://arxiv.org/abs/2410.00037