Moshi: a speech-text foundation model for real-time dialogue

3.3.2 Learning semantic-acoustic tokens with a split RVQ

1. How do Mimi’s tokens carry semantic as well as acoustic information?
   The first token is distilled from WavLM (a non-causal self-supervised speech model). Distilling into the first level of a single residual quantizer hurt audio quality, so Mimi uses a split RVQ: one plain VQ for the semantic token in parallel with a 7-level RVQ for acoustics, their outputs summed.
   • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.3.2
2. Name two training choices that improved Mimi’s audio quality.
   Applying quantization only 50% of the time during training, so the decoder also sees unquantized latents (better objective metrics); and training with adversarial and feature-matching losses only, without reconstruction losses, which badly hurt objective metrics but clearly improved quality in human ratings.
   • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.3.2