Text-to-Speech

17.1 TTS overview

1. How did historical TTS differ from modern TTS in how voices are obtained?
   Historically: record many hours of one talker in a lab and train a system that only speaks in that voice. Modern: train a speaker-independent synthesizer on speech from thousands of talkers, then guide a new voice with a few seconds of that talker’s speech.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.1, p. 384
2. What is zero-shot TTS?
   Generating speech for a text in a voice that may never have been seen in training, given only a short sample (e.g. 3 seconds) of the desired voice.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.1, p. 384
3. What is the core idea of modern LM-based TTS?
   Use an audio tokenizer based on a neural codec to turn speech into discrete audio tokens, train a language model over text and audio tokens, and conditionally generate audio tokens for the text in the prompted voice; then decode the tokens into a waveform.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.1, p. 384
4. What are the two components of systems like VALL-E?
   An audio tokenizer (a codec with an encoder, a quantizer and a decoder) and a two-stage conditional language model that generates audio tokens corresponding to the text.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.1, pp. 384–385