Moshi: a speech-text foundation model for real-time dialogue

3.4.2 Audio modeling

1. What is the acoustic delay and why does Moshi use it?
   The acoustic tokens are shifted one or two frames after the semantic token, so the large Temporal Transformer, rather than the small Depth Transformer, models how they depend on each other. A delay of 1–2 steps greatly improved generation quality; the final model uses 1 step (160 ms theoretical latency).
   • Défossez et al., 2024, Moshi (arXiv preprint arXiv:2410.00037), §3.4.2