learn.ifs.name

Moshi: a speech-text foundation model for real-time dialogue

Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037)

Subscriber deck — free during early access

1 cards in “3.3 Audio Tokenization” · whole deck

3.3 Audio Tokenization

  1. How does the Mimi codec represent 24 kHz audio?
    A causal convolutional encoder with Transformers in the bottleneck downsamples it to 12.5 frames per second (80 ms each), and each frame is quantized by 8 codebooks of 2048 entries, about 1.1 kbps. The frame rate is close to that of text tokens, keeping sequences short.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.3

Source

  • Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. In arXiv preprint arXiv:2410.00037. https://arxiv.org/abs/2410.00037