Text-to-Speech

17.3 VALL-E: two-stage LM generation

1. How does VALL-E perform TTS at a high level?
   Tokenize the text (BPE) and a voice sample (codec), then conditionally generate audio tokens for the text in that voice with language models, and decode them into a waveform.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.3, p. 391
2. Why does VALL-E generate audio tokens in two stages?
   Because of the RVQ hierarchy: the first-quantizer tokens are most important. An autoregressive transformer generates the first-quantizer codes for the whole output left to right; then a non-autoregressive transformer, run once per remaining quantizer, fills in each next quantizer’s codes given the previous ones (including codes to the right).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.3, pp. 391–392
3. What does the acoustic code matrix \(C\) look like?
   \(T \times 8\) for EnCodec: one row of 8 codes per frame (one per quantizer), one column per quantizer giving that quantizer’s code sequence over time.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.3, p. 392
4. What is VALL-E trained to maximize?
   The likelihood of the code matrix given the text, \(-\log p(C \mid x)\), as a conditional code language model.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.3, p. 392
5. How does VALL-E do zero-shot synthesis at inference?
   Encode the enrolled speech sample into prompt codes \(C^P\); prepend its transcript to the text to be spoken; then generate the following codes conditioned on the text and the acoustic prompt (AR for the first quantizer, NAR for the rest), and decode with the EnCodec decoder.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.3, pp. 392–393