Text-to-Speech

The first full-sentence speech synthesizer (1769–1790): bellows for lungs, a reed for vocal folds, a rubber mouthpiece and nose aperture, whistles for fricatives and a small bellows for plosive puffs, played with levers by an operator. (Unlike his chess-playing Mechanical Turk, it was not a hoax.)

Speech synthesis: mapping text to an acoustic waveform, the reverse of ASR.

Spoken language models that interact with people, reading text aloud, games, and producing speech for people who have lost their voice to neurological disorders (e.g. ALS).

Historically: record many hours of one talker in a lab and train a system that only speaks in that voice. Modern: train a speaker-independent synthesizer on speech from thousands of talkers, then guide a new voice with a few seconds of that talker’s speech.

Generating speech for a text in a voice that may never have been seen in training, given only a short sample (e.g. 3 seconds) of the desired voice.

Use an audio tokenizer based on a neural codec to turn speech into discrete audio tokens, train a language model over text and audio tokens, and conditionally generate audio tokens for the text in the prompted voice; then decode the tokens into a waveform.

An audio tokenizer (a codec with an encoder, a quantizer and a decoder) and a two-stage conditional language model that generates audio tokens corresponding to the text.

They let speech systems use language-model technology, which is specialized for sequences of discrete tokens: spoken language models for speech-to-speech translation, diarization, spoken question answering.

From coder/decoder: originally a device digitizing analog signals; generally a mechanism encoding analog speech into a compressed digital representation for storage or transmission. For TTS, a neural codec converts speech into discrete tokens.

That would be thousands of tokens per second, far too long for a transformer with quadratic attention; we want tokens for longer chunks, on the order of a few hundred per second.

(1) An encoder maps the waveform \(x_1..x_T\) to many fewer embeddings \(z_1..z_\tau\) (downsampling by ~100–1000×); (2) a vector quantizer turns each \(z_t\) into discrete tokens from \(N_c\) codebooks and sums their codewords into \(z_{q,t}\); (3) a decoder reconstructs a lossy waveform \(\hat{x}\) from \(z_{q,t}\).

End-to-end, with losses that reward tokenizations from which the input waveform can be reconstructed.

With a series of convolutional encoder blocks whose strides are larger than 1, plus residual units, iteratively turning a high-rate waveform into a low-rate sequence of embedding vectors; the decoder mirrors this with upsampling convolutions.

Originally to compress speech for transmission or storage (send an integer index instead of a vector of floats). Now because it creates discrete tokens, which suit language modeling.

Encode lots of speech into vectors and cluster them into \(k\) clusters (usually with k-means); each cluster’s centroid is its codeword. In end-to-end training the means are updated online, e.g. with exponential moving averages.

The geometric center of a set of points in n-dimensional space.

Start with random centroids, then iterate: assignment (assign each vector to the cluster with the closest centroid by squared Euclidean distance) and re-estimation (recompute each centroid as the mean of its vectors), until convergence.

A codeword is the vector at a cluster’s centroid; the codebook is the list of cluster ids with their codewords; the code is the cluster id (the discrete token).

Compare it to every codeword in the codebook and output the id of the closest one.

Pass the token’s codeword to the decoder; it only approximates the original vector, especially with a small codebook (e.g. 1024 entries).

Hierarchical VQ: quantize \(z_t\) with codebook 1, compute the residual \(z_t - z^{(1)}_{q,t}\) (the part VQ missed, like a rounding error), quantize the residual with codebook 2, and so on, giving several tokens per frame (8 in EnCodec). Reconstruction sums all the chosen codewords.

A single codebook (e.g. 1024 codewords) isn’t enough to represent the variety of speech embeddings; successive residual codebooks refine the approximation.

The first: it captures most of the acoustic information, and later quantizers contribute less and less residual detail.

Large speech and audio datasets (e.g. Common Voice, AudioSet); the target output is the input span itself, so the model is a kind of autoencoder.

A reconstruction loss (waveform difference, e.g. sum of squares, and/or mel-spectrogram distance), an adversarial (GAN) loss from a discriminator trying to tell real from reconstructed audio, and a VQ loss; the total is a weighted sum.

Quantization isn’t differentiable. The straight-through estimator copies gradients from the quantizer output \(z_{q,t}\) back to its input \(z_t\), ignoring quantization in the backward pass.

Initialize them by k-means on encoder outputs, and add a VQ loss on the distance between the encoder output and the chosen codewords, summed over codebooks: \(L_{VQ} = \sum_t\sum_c \|z^{(c)}_t - z^{(c)}_{q,t}\|\).

Tokenize the text (BPE) and a voice sample (codec), then conditionally generate audio tokens for the text in that voice with language models, and decode them into a waveform.

Because of the RVQ hierarchy: the first-quantizer tokens are most important. An autoregressive transformer generates the first-quantizer codes for the whole output left to right; then a non-autoregressive transformer, run once per remaining quantizer, fills in each next quantizer’s codes given the previous ones (including codes to the right).

\(T \times 8\) for EnCodec: one row of 8 codes per frame (one per quantizer), one column per quantizer giving that quantizer’s code sequence over time.

The likelihood of the code matrix given the text, \(-\log p(C \mid x)\), as a conditional code language model.

Encode the enrolled speech sample into prompt codes \(C^P\); prepend its transcript to the text to be spoken; then generate the following codes conditioned on the text and the acoustic prompt (AR for the first quantizer, NAR for the rest), and decode with the EnCodec decoder.

Mean opinion score: listeners rate synthesized utterances on how good they are, usually on a 1–5 scale; systems are compared on the same sentences (e.g. with paired t-tests).

Comparative MOS: listeners hear the same sentence from two systems and give a preference, from −3 (much worse than the reference) to +3 (better), over many sentences in random order.

Run the output through ASR and compute word error rate (robustness/intelligibility), and use a speaker verification system to score similarity between the output voice and the enrolled voice.

Determining “who spoke when” in a multi-speaker recording. Typically: voice activity detection to find speech segments, speaker embeddings for them, and clustering; newer end-to-end systems map speech directly to per-frame speaker labels.

Verification is a binary decision (is this speaker X?), e.g. for phone security. Identification is a one-of-N decision, matching a voice against a database of speakers. Both are speaker recognition.

Identifying which language is spoken in a wavefile; important for multilingual models, dataset creation and online systems.

Detecting a short phrase that wakes a voice assistant. It must run fast in small-footprint software on edge devices, to preserve privacy by sending as little speech as possible to the cloud; it typically uses ASR-style features and a whole-word classifier.

Formant synthesis (generating artificial spectrograms, e.g. Haskins Pattern Playback, Klatt’s synthesizer and DECtalk), concatenative synthesis (splicing recorded units), and articulatory synthesis (modeling vocal tract physics).

From splicing magnetic tape for phones (Harris, 1953), to diphone databases with prosodic variants and join costs (implemented decades later), to unit selection synthesis in the 1980s–90s with larger units of varying length and target costs.

Umeda and Teranishi’s system (1968), which included a parser assigning prosodic boundaries, accent and stress; earlier systems took phonemes as input.