Text-to-Speech

17.2 Codecs and discrete audio tokens

1. Why are discrete audio tokens useful beyond TTS?
   They let speech systems use language-model technology, which is specialized for sequences of discrete tokens: spoken language models for speech-to-speech translation, diarization, spoken question answering.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 385
2. What is a codec?
   From coder/decoder: originally a device digitizing analog signals; generally a mechanism encoding analog speech into a compressed digital representation for storage or transmission. For TTS, a neural codec converts speech into discrete tokens.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 385
3. Why not use raw digitized samples (e.g. 16-bit values at 16 kHz) as tokens for a language model?
   That would be thousands of tokens per second, far too long for a transformer with quadratic attention; we want tokens for longer chunks, on the order of a few hundred per second.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 385
4. What are the three stages of an audio tokenizer?
   (1) An encoder maps the waveform \(x_1..x_T\) to many fewer embeddings \(z_1..z_\tau\) (downsampling by ~100–1000×); (2) a vector quantizer turns each \(z_t\) into discrete tokens from \(N_c\) codebooks and sums their codewords into \(z_{q,t}\); (3) a decoder reconstructs a lossy waveform \(\hat{x}\) from \(z_{q,t}\).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, pp. 385–386
5. How are audio tokenizers trained?
   End-to-end, with losses that reward tokenizations from which the input waveform can be reconstructed.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 386
6. How does the EnCodec encoder downsample audio?
   With a series of convolutional encoder blocks whose strides are larger than 1, plus residual units, iteratively turning a high-rate waveform into a low-rate sequence of embedding vectors; the decoder mirrors this with upsampling convolutions.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, pp. 386–387
7. What was vector quantization originally for, and why is it used in modern speech systems?
   Originally to compress speech for transmission or storage (send an integer index instead of a vector of floats). Now because it creates discrete tokens, which suit language modeling.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 387
8. How is a basic VQ codebook trained?
   Encode lots of speech into vectors and cluster them into \(k\) clusters (usually with k-means); each cluster’s centroid is its codeword. In end-to-end training the means are updated online, e.g. with exponential moving averages.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 387
9. What is a centroid?
   The geometric center of a set of points in n-dimensional space.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 387
10. Recall the two steps of k-means.
    Start with random centroids, then iterate: assignment (assign each vector to the cluster with the closest centroid by squared Euclidean distance) and re-estimation (recompute each centroid as the mean of its vectors), until convergence.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 387
11. What are a codeword, a codebook and a code?
    A codeword is the vector at a cluster’s centroid; the codebook is the list of cluster ids with their codewords; the code is the cluster id (the discrete token).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 388
12. How does VQ tokenize a new vector at inference time?
    Compare it to every codeword in the codebook and output the id of the closest one.

    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 388
13. How is a VQ token turned back into audio in simple VQ?
    Pass the token’s codeword to the decoder; it only approximates the original vector, especially with a small codebook (e.g. 1024 entries).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 388
14. What is residual vector quantization (RVQ)?
    Hierarchical VQ: quantize \(z_t\) with codebook 1, compute the residual \(z_t - z^{(1)}_{q,t}\) (the part VQ missed, like a rounding error), quantize the residual with codebook 2, and so on, giving several tokens per frame (8 in EnCodec). Reconstruction sums all the chosen codewords.

    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 389
15. Why is RVQ preferred over simple VQ for audio?
    A single codebook (e.g. 1024 codewords) isn’t enough to represent the variety of speech embeddings; successive residual codebooks refine the approximation.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 389
16. In RVQ, which quantizer matters most?
    The first: it captures most of the acoustic information, and later quantizers contribute less and less residual detail.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 389
17. What data are audio tokenizers like EnCodec trained on, and with what target?
    Large speech and audio datasets (e.g. Common Voice, AudioSet); the target output is the input span itself, so the model is a kind of autoencoder.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, pp. 389–390
18. What losses are used to train an audio tokenizer?
    A reconstruction loss (waveform difference, e.g. sum of squares, and/or mel-spectrogram distance), an adversarial (GAN) loss from a discriminator trying to tell real from reconstructed audio, and a VQ loss; the total is a weighted sum.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 390
19. Why does a quantizer cause problems for end-to-end training, and how is this handled?
    Quantization isn’t differentiable. The straight-through estimator copies gradients from the quantizer output \(z_{q,t}\) back to its input \(z_t\), ignoring quantization in the backward pass.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, p. 390
20. How are codewords kept updated during end-to-end codec training?
    Initialize them by k-means on encoder outputs, and add a VQ loss on the distance between the encoder output and the chosen codewords, summed over codebooks: \(L_{VQ} = \sum_t\sum_c \|z^{(c)}_t - z^{(c)}_{q,t}\|\).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.2, pp. 390–391