learn.ifs.name

Automatic Speech Recognition

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 16

21 cards in “16.4 HuBERT & k-means” · whole deck

16.4 HuBERT & k-means

  1. What are self-supervised speech models?
    Models that first bootstrap discrete phonetic units from acoustic input and learn to map waveforms to them, without transcripts, then are fine-tuned for ASR on a smaller labeled set.
  2. Main advantage of self-supervised speech models?
    They can use large amounts of untranscribed audio for most of their training.
  3. What intuition do HuBERT and wav2vec 2.0 share with BERT?
    Masked modeling: mask out part of the input and train the model to guess what was hidden.
  4. Describe the HuBERT forward pass.
    Raw 16 kHz audio → 7 conv layers (to 20 ms frames) → positional encodings, GELU, layer norm → some frames replaced with a MASK embedding → transformer stack → linear projection A → cosine similarity with each class embedding → softmax over classes.
  5. What is HuBERT's MASK token?
    A trained embedding shared by all masked frames, replacing the CNN output for those frames.
  6. How many induced phonetic classes does HuBERT predict?
    100 classes in the first training stage, 500 in the second.
  7. How does HuBERT turn transformer outputs into class probabilities?
    Project with A, compare by cosine to each class embedding, softmax with temperature τ = 0.1: \(p(c|X,t) = \frac{\exp(\mathrm{sim}(Ah_t, e_c)/\tau)}{\sum_{c'}\exp(\mathrm{sim}(Ah_t, e_{c'})/\tau)}\)
  8. How are HuBERT's stage-1 target classes created?
    Extract 39-dim MFCC vectors for the whole training set (960 h of LibriSpeech → 172 million vectors), then cluster them with k-means into 100 clusters.
  9. What is a codebook?
    The output of clustering: k vectors (codewords, templates, or prototypes), one per cluster. Each cluster is an acoustic unit used as a gold training target.
  10. How is each frame's gold target obtained during HuBERT training?
    In parallel with the forward pass, the waveform is converted to MFCCs and each vector is mapped to the most similar centroid in the codebook.
  11. What is HuBERT's training loss?
    Summed over masked tokens only: \(L = \sum_{t\in M} \log p(z_t|X)\) — the (log) probability assigned to the correct units, backpropagated through the model.
  12. Describe HuBERT's second training stage.
    Pass 10% of training data through the stage-1 model, take the intermediate layer 6 output (768-dim), cluster with k-means into 500 units, and train on those targets.
  13. Why does HuBERT use two stages of target induction?
    Initial MFCC clusters bias the model toward phonetic representations; after enough training, the model's own representations are more accurate and fine-grained.
  14. How is HuBERT fine-tuned for ASR?
    Remove the projection and cosine layers; add a randomly initialized linear + softmax layer over 29 classes (26 English letters + a few extras). Freeze the CNN layers; fine-tune the transformer and projection on transcribed audio with CTC loss.
  15. What is k-means?
    A family of algorithms grouping vector data into k clusters; in speech, commonly used to convert real-valued vectors into discrete symbols (e.g. HuBERT targets, acoustic tokens for TTS).
  16. What are k-means' inputs?
    N vectors \(v^{(1)}\ldots v^{(N)} \in \mathbb{R}^d\) and a constant k, usually with N ≫ k.
  17. What is a centroid?
    The geometric center of a set of points in n-dimensional space.
  18. k-means initialization step?
    For each cluster choose a random vector \(\mu_k \in \mathbb{R}^d\) as its codeword, giving a codebook.
  19. k-means assignment step?
    Assign each vector to the cluster with the nearest codeword by squared Euclidean distance: \(\mathrm{cluster}(i) = \mathrm{argmin}_j \lVert v^{(i)} - \mu_j\rVert^2\)
  20. k-means re-estimation step?
    Recompute each codeword as the mean of its cluster's vectors: \(\mu_i = \frac{1}{|S_i|}\sum_{v\in S_i} v\). Iterate assignment and re-estimation until convergence.
  21. Must a k-means centroid be an actual data point?
    No. The mean vector need not be a point from the dataset.

Source