Moshi: a speech-text foundation model for real-time dialogue

4 Datasets and Training

1. Describe Moshi’s training stages after Helium pretraining.
   Pretraining on 7M hours of Whisper-transcribed audio as a single stream (half the batches text-only, against forgetting); post-training on diarized audio split into two streams; fine-tuning on Fisher telephone conversations (2000 h, one speaker per channel) for real overlap; and instruction tuning on 20k+ hours of synthetic dialogues.
   • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §4
2. How was Moshi’s instruction-tuning data made, and why not use text instruction data directly?
   Text instruction sets like Open Hermes did not read well aloud (URLs, bullet points, long lists). Instead Helium wrote realistic spoken dialogue transcripts, and Moshi’s own multi-stream TTS rendered them, with Moshi always in one actor’s voice and the user in randomly sampled voices, plus noise, echo and gain augmentation.
   • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §4