learn.ifs.name

Automatic Speech Recognition

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 16

25 cards in “16.5 CTC & RNN-T” · whole deck

16.5 CTC & RNN-T

  1. What two properties of speech make encoder-decoders appropriate?
    (1) A very long acoustic input X maps to a much shorter letter sequence Y. (2) It's hard to know which part of X maps to which part of Y.
  2. What is CTC and what's its core intuition?
    Connectionist Temporal Classification (Graves et al., 2006): output one character per input frame (same length as input), then apply a collapsing function that merges repeated letters to get a shorter sequence.
  3. What is an alignment in CTC?
    The sequence of letters corresponding to each input frame, showing where each letter aligns in the acoustic signal.
  4. What two problems does naive collapsing (just merging repeats) have?
    (1) It can't handle double letters: “dinner” collapses to “diner”. (2) It has no symbol for silence, so silence gets transcribed as random letters.
  5. How does CTC fix naive collapsing?
    It adds a special blank symbol ␣ to the alphabet, used where no letter should be transcribed and between letters; repeats aren't merged across a blank.
  6. What is the CTC collapsing function B?
    A mapping B: a → y from alignment to output that first merges consecutive repeated letters, then removes all blanks.
  7. Is B one-to-one? What is B⁻¹(Y)?
    No, it's many-to-one: many alignments produce the same output. B⁻¹(Y) is the set of all alignments that collapse to Y.
  8. What conditional independence assumption does CTC make?
    Given X, the output at time t is independent of outputs at other times: \(P_{CTC}(A|X) = \prod_{t=1}^{T} p(a_t|X)\)
  9. Simple (greedy) CTC inference?
    Choose the most probable character at each step, \(\hat{a}_t = \mathrm{argmax}_{c\in C} p_t(c|X)\), then apply B to get Y.
  10. Why doesn't CTC need a full decoder?
    Because it makes one decision per time step, it's a sequence-labeling task: an encoder produces hᵗ at each timestep, followed by a classifier + softmax over characters.
  11. Why can the most probable alignment give the wrong output?
    Many alignments map to the same output. E.g. best alignment [a b ε] gives “ab”, but [b ε b] and [ε b b] together might make “bb” more probable.
  12. How is P_CTC(Y|X) correctly defined?
    Sum over all alignments collapsing to Y: \(P_{CTC}(Y|X) = \sum_{A\in B^{-1}(Y)} \prod_{t=1}^{T} p(a_t|h_t)\), and \(\hat{Y} = \mathrm{argmax}_Y P_{CTC}(Y|X)\).
  13. How is the sum over alignments approximated at inference?
    A version of Viterbi beam search that keeps high-probability alignments mapping to the same output string in the beam and sums them (see Hannun, 2017).
  14. Why must CTC be combined with a language model?
    Its conditional independence assumption means it does not implicitly learn a language model over the output, unlike attention-based encoder-decoders.
  15. CTC scoring with a language model?
    \(\mathrm{score}_{CTC}(Y|X) = \log P_{CTC}(Y|X) + \lambda_1 \log P_{LM}(Y) + \lambda_2 L(Y)\), with weights trained on a dev set and L(Y) a length factor.
  16. CTC training loss?
    Negative log-likelihood over the dataset: \(L_{CTC} = \sum_{(X,Y)\in D} -\log P_{CTC}(Y|X)\)
  17. How is the CTC sum over alignments computed efficiently in training?
    Dynamic programming that merges alignments: a version of the forward-backward algorithm (as used for HMMs and CRFs).
  18. How are CTC and encoder-decoder losses combined in training?
    \(L = -\lambda \log P_{encdec}(Y|X) - (1-\lambda)\log P_{ctc}(Y|X)\), λ tuned on a dev set.
  19. How are CTC and encoder-decoder combined at inference?
    \(\hat{Y} = \mathrm{argmax}_Y [\lambda \log P_{encdec}(Y|X) + (1-\lambda)\log P_{CTC}(Y|X) + \gamma \log P_{LM}(Y)]\)
  20. CTC vs attention-based encoder-decoder: accuracy and streaming?
    CTC is less accurate (due to its independence assumption) but supports streaming. Attention-based models are more accurate but need the whole input first.
  21. What is streaming ASR and why is it important?
    Recognizing words on-line as the user talks rather than waiting until the end of the sentence; crucial for commands and dictation.
  22. Why can't attention-based models stream easily?
    They must compute the hidden state sequence over the entire input first to provide attention context before decoding. CTC can output letters left to right immediately.
  23. What is the RNN-T?
    RNN-Transducer (Graves 2012; Graves et al. 2013): a streaming model removing CTC's independence assumption. Components: a CTC acoustic encoder plus a separate language model called the predictor that conditions on output history.
  24. How does the RNN-T compute its output at each step?
    The encoder outputs h_t^enc from x₁…x_t; the predictor takes the previous output token (excluding blanks) and outputs h_u^pred; a joint network combines them, then a softmax predicts the next character.
  25. RNN-T probability?
    \(P_{RNN-T}(Y|X) = \sum_{A\in B^{-1}(Y)} \prod_{t=1}^{T} p(a_t \mid h_t, y_{

Source