Moshi: a speech-text foundation model for real-time dialogue

3.4.3 Multi-stream modeling

1. How does Moshi handle the user and itself without explicit turns?
   It models two audio streams, its own and the user’s, as parallel sub-sequences. At inference it samples only its own stream and feeds in the real user audio. There are no turn boundaries: when Moshi is silent its stream decodes to natural silence, so it can speak, listen, or both at once.
   • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.4.3