learn.ifs.name

Automatic Speech Recognition

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 16

35 cards in “16.3 Encoder-decoder ASR” · whole deck

16.3 Encoder-decoder ASR

  1. What is AED / LAS?
    Attention-based encoder-decoder, also called Listen Attend and Spell after the papers first applying it to speech (Chorowski et al. 2014; Chan et al. 2016). The same encoder-decoder architecture used for MT.
  2. What are the input and output of an ASR encoder-decoder?
    Input: a sequence of t acoustic feature vectors X = x₁…x_t, one per 10 ms frame (often log mel features, or a raw wavefile). Output: letters or tokens (BPE or SentencePiece), Y = (⟨SOS⟩, y₁, …, y_m, ⟨EOS⟩).
  3. Example English character output set for an encoder-decoder?
    {a…z, 0…9, ⟨space⟩, ⟨comma⟩, ⟨period⟩, ⟨apostrophe⟩, ⟨unk⟩}
  4. Order the stages of the encoder-decoder ASR pipeline (Fig. 16.5).
    Waveform → feature computation (e.g. 80-dim log mel per frame) → subsampling → shorter sequence X → encoder → decoder (via H) → output letters.
  5. What is released and not released for OpenAI's Whisper?
    Released: models and inference code. Not released: training code and training data.
  6. Whisper's transcription format tokens?
    Start of Transcript (SOT), a language tag, then an instruction token for whether to transcribe or translate (followed by timestamps/text). Whisper is multitask.
  7. Besides transcription, what other tasks does Whisper do?
    Speech translation and voice activity detection (and language identification).
  8. Why is the encoder-decoder architecture well suited to speech?
    Input and output sequences differ starkly in length: long acoustic feature sequences map to much shorter letter/word sequences.
  9. How much longer is a 10 ms-frame speech signal than its text?
    About 5× (vs letters) to 19× (vs BPE tokens). Words average 5 letters or 1.3 BPE tokens and last about 250 ms (25 frames) in conversation: 25/5 = 5, 25/1.3 ≈ 19.
  10. What is the low frame rate subsampling method?
    At time i, concatenate feature vector fᵢ with fᵢ₋₁ and fᵢ₋₂ into one vector 3× longer, then delete fᵢ₋₁ and fᵢ₋₂. E.g. a 40-dim vector every 10 ms becomes 120-dim every 30 ms; length n = t/3 (Pundak & Sainath, 2016).
  11. What is the most common way to shorten the input sequence?
    Convolutional layers with stride > 1.
  12. Whisper front end: context window and input features?
    30-second audio context window; 128-channel log mel features per frame with 25 ms window and 10 ms stride, normalized to zero mean and range −1 to 1. That gives 3000 input frames.
  13. Whisper's two convolutional layers?
    Each followed by GELU. Conv 1: 128 input channels → model dimensionality, stride 1 (length stays 3000). Conv 2: model dim → model dim, stride 2, halving length to 1500 — one audio token every 20 ms.
  14. What is GELU?
    Gaussian Error Linear Unit: a smoother version of ReLU, used by Whisper and HuBERT.
  15. What is added to Whisper's audio encodings before the transformer encoder?
    Sinusoidal position embeddings. (The decoder uses learned positional encodings.)
  16. HuBERT's front end: how does it differ from Whisper's?
    Its conv layers completely replace spectrum computation: raw 16 kHz audio goes through seven 512-channel conv layers that learn to extract spectral info and shorten the sequence.
  17. HuBERT conv strides and kernel widths, and the total downsampling?
    Strides [5,2,2,2,2,2,2], kernel widths [10,3,3,3,3,2,2]. Total 320× shorter: from 16 kHz (one sample per 0.0625 ms) to a 20 ms frame rate.
  18. What is applied after HuBERT's conv layers before the transformer?
    Positional encodings are added, then GELU and layer norm.
  19. What are the two transformers in an encoder-decoder, and what's special about the decoder?
    An encoder (a basic transformer) mapping X to H^enc = h₁…h_n, and a decoder that adds one new layer: cross-attention.
  20. What is the decoder, conceptually?
    A conditional language model that attends to the encoder representation and generates letters/tokens one by one, conditioning on the audio and previously generated text.
  21. What is cross-attention?
    Multi-head attention where queries come from the previous decoder layer, but keys and values come from the encoder's final output H^enc.
  22. Write the cross-attention Q, K, V computations.
    \(Q = H^{dec[\ell-1]}W^Q;\ K = H^{enc}W^K;\ V = H^{enc}W^V\)
  23. Write the cross-attention formula.
    \(\mathrm{CrossAttention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\)
  24. Which attention is masked in encoder-decoders: encoder or decoder?
    The decoder's self-attention is causal (left-to-right, masked). The encoder's attention can look at the entire source audio, so it is not masked.
  25. How is the output string probability decomposed?
    \(p(y_1,\ldots,y_n) = \prod_{i=1}^{n} p(y_i \mid y_1,\ldots,y_{i-1}, X)\)
  26. What is greedy decoding in an ASR encoder-decoder?
    Pick the most probable character at each step: \(\hat{y}_i = \mathrm{argmax}_{\text{char}\in\text{Alphabet}} P(\text{char}\mid y_1\ldots y_{i-1}, X)\). Whisper/OWSM can also use beam search.
  27. Why add an external language model to an encoder-decoder?
    It already learns an implicit LM, but speech paired with transcripts has far less text than pure text corpora, so a very large LM can usually improve results at least slightly.
  28. What is an n-best list and rescoring?
    The final beam of hypothesized sentences from beam search is an n-best list. Rescoring re-ranks each hypothesis by interpolating an LM score with the encoder-decoder score, weight λ tuned on held-out data.
  29. Why add a length factor in ASR scoring?
    Most models prefer shorter sentences; normalizing by the number of characters |Y|_c counters this.
  30. Listen, Attend and Spell scoring function?
    \(\mathrm{score}(Y|X) = \frac{1}{|Y|_c}\log P(Y|X) + \lambda \log P_{LM}(Y)\)
  31. How are ASR encoder-decoders trained? Give the per-step loss.
    With cross-entropy loss backpropagated end to end: \(L_{CE} = -\log p(y_i \mid y_1,\ldots,y_{i-1}, X)\), summed over i = 1…m for the sentence.
  32. What is teacher forcing, and what variant is possible?
    Forcing the decoder history to be the gold yᵢ rather than the predicted ŷᵢ. Variant: mix, e.g. gold 90% of the time and the decoder's own output with probability 0.1.
  33. Whisper-v2 training data size?
    680,000 hours of speech, mostly English, including 118,000 hours from 96 other languages.
  34. How do web-scraping ASR systems keep data quality high?
    They remove ASR-generated transcripts from training data, e.g. by filtering text that is all uppercase or all lowercase.
  35. OWSM training data?
    180k hours of mainly hand-transcribed public data: LibriSpeech, Multilingual LibriSpeech, Common Voice, FLEURS, Switchboard, AMI, and others.

Source