learn.ifs.name

Automatic Speech Recognition

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 16

19 cards in “16.2 Convolutional networks” · whole deck

16.2 Convolutional networks

  1. What is a CNN (convnet) used for in ASR?
    Extracting features. A convolutional layer takes audio (raw waveform or mel spectra) and outputs a sequence of latent representations; stacked conv layers are the initial layers before transformer layers in Whisper, wav2vec 2.0, and HuBERT.
  2. How does a convolutional layer differ from a feedforward layer?
    A feedforward layer is fully connected (every input to every output). A convolutional layer passes a small kernel (a small network of weights) over the input.
  3. Why are speech CNNs “1d” while vision CNNs are “2d”?
    In vision the kernel slides horizontally and vertically over an image; in speech it slides only along the time dimension.
  4. What is a kernel?
    A small vector of weights w₁…w_k used to extract features by convolving it with the input.
  5. What are the three steps of convolving a kernel with a signal?
    (1) Flip the kernel left-to-right. (2) Pass it frame by frame across the input, computing the dot product with the local input values at each frame. (3) Output the resulting sequence of dot products.
  6. What intuition explains what convolution detects?
    It finds regions of the signal similar to the kernel, since a dot product is high when two vectors are similar.
  7. Convolution with a width-1 kernel?
    \(z_j = x_j w_0\ \ \forall j: 1 \le j \le t\) — the dot product is just a scalar product, and flipping doesn't matter.
  8. What is cross-correlation, and why is it used instead of convolution in ASR?
    Walking a kernel across a signal and taking dot products without flipping it first. Libraries like PyTorch skip the flip; it doesn't matter because the kernel's weights are learned and can be learned in either order. It's still called 1d convolution for historical reasons.
  9. What is padding and why use it?
    Adding p zeros at the start and end of the signal so convolution is defined at the edges; this also lets the output length equal the input length.
  10. Why define kernels with odd length k = 2p + 1?
    So the center element has p elements on each side, which with padding of p keeps output length equal to input length.
  11. General convolution equation for a kernel of width 2p+1?
    \(z_j = \sum_{i=-p}^{p} x_{j+i}\, w_{i+p}\)
  12. What is the receptive field of a kernel?
    Its size k: the number of input frames it is compared against at once. It is designed to be small compared to the signal.
  13. Whisper's conv kernel width and how much speech each kernel sees?
    Width 3 frames. With a frame every 10 ms and 25 ms windows, each kernel covers about 45 ms of speech (10 + 10 + 12.5 + 12.5) — enough for formant transitions, stop closures, or aspiration.
  14. What is the depth of a kernel?
    The number of input channels it has separate vectors for. With 128 log mel channels, the kernel has depth 128 and shape [128, 3].
  15. How is a multi-channel input convolved into one output?
    Convolve the kernel with each input channel and sum: \(z = \sum_{c=1}^{C_i} x^c * w\). Each output integrates information from all channels.
  16. How does a conv layer produce a full embedding (e.g. Whisper's 1280 dims) per frame?
    It learns one separate kernel per model dimension — e.g. 1280 kernels, each with depth = number of input channels (128) and width (3) — so each frame's embedding has 1280 independently computed features.
  17. What is stride?
    How far the kernel moves between steps. Stride 1: x₁, x₂, x₃… Stride 2: x₁, x₃, x₅… A stride of 2 halves the output length.
  18. Why use conv layers with stride > 1 in ASR? (2 reasons)
    (1) Shorter sequences need less memory and compute. (2) They reduce the mismatch between short acoustic frames (10 ms) and letters/words, which span much more signal.
  19. What typically follows a convolutional layer?
    An output nonlinearity such as ReLU.

Source