learn.ifs.name

Moshi: a speech-text foundation model for real-time dialogue

Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037)

Subscriber deck — free during early access

2 cards in “6 Safety” · whole deck

6 Safety

  1. How did Moshi’s authors check that it keeps its own voice rather than mimicking the user’s?
    A speaker-verification model compared each of Moshi’s segments in 100 h of generated dialogues with Moshi’s and the other speaker’s first segment: 98.7% were closer to Moshi’s own, with no drift over time, just from training on a single consistent voice.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §6
  2. Why did signal-based audio watermarking fail for Moshi?
    Audioseal marks were detected well on unchanged audio, but passing the audio through a low-bitrate codec, Mimi itself or RVQGAN, removed them, since such codecs discard anything not needed for reconstruction.
    • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §6

Source

  • Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. In arXiv preprint arXiv:2410.00037. https://arxiv.org/abs/2410.00037