learn.ifs.name

Moshi: a speech-text foundation model for real-time dialogue

Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037)

Subscriber deck — free during early access

2 cards in “3.4.1 Hierarchical autoregressive modeling with RQ-Transformer” · whole deck

3.4.1 Hierarchical autoregressive modeling with RQ-Transformer

  1. Why does Moshi not just flatten its audio tokens into one sequence?
    With 8 codebooks at 12.5 Hz that is 100 tokens per second, 30,000 steps for 5 minutes, too slow for streaming. English speech is only about 3–4 text tokens per second, so a model that predicts a whole frame per step is much cheaper.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.4.1
  2. How does the RQ-Transformer split the work between its two Transformers?
    A large Temporal Transformer (initialized from Helium) runs once per 80 ms frame over all previous frames and outputs a context vector. A small Depth Transformer (6 layers, separate weights per level) then predicts that frame’s \(K\) tokens one after another, conditioned on the context vector and the tokens already predicted.
    Temporal T.: one step per 80 ms →textpadHipadyoupadpadsemantics₁s₂s₃s₄s₅s₆acoustic ×7a₁a₂a₃a₄a₅user sem.u₁u₂u₃u₄u₅u₆user ac. ×7v₁v₂v₃v₄v₅dashed: acoustic delay of one stepframed: Depth T. fills one step, 17 tokens
    Moshi models its own speech, its inner-monologue text and the user’s audio as parallel streams (illustrative tokens)
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.4.1

Source

  • Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. In arXiv preprint arXiv:2410.00037. https://arxiv.org/abs/2410.00037