learn.ifs.name

PUMA: language tokens for low-resource multilingual ASR

Oukid et al., 2026, PUMA (Findings of ACL 2026)

Subscriber deck — free during early access

10 cards in “3.1 Architecture” · whole deck

3.1 Architecture

  1. Name PUMA's five components, in order.
    Feature Extractor (FE) → Feature Projection (FP) → Universal Language Projection (ULP) → Encoder → LM head.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3, pp. 372–373, fig. 1
  2. Which PUMA components are frozen and which are trained?
    Frozen: the feature extractor, feature projection and encoder, all taken from a pretrained wav2vec 2.0 model. Trained: the ULP module, the language tokens and the LM head (about 30M parameters).
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §§3.1, 4.3, pp. 373, 375, fig. 1
  3. Equation for the features entering the ULP (Eq. 1)?
    \(h = FP(FE(x)) \in \mathbb{R}^{N \times L \times D_{fp}}\), where N is the batch size, L the number of frames, and \(D_{fp}\) the projection dimension.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3.1, p. 373
  4. What is a PUMA language token?
    A learnable vector \(t_l \in \mathbb{R}^{1 \times D_{fp}}\), one per language l, that tells the shared ULP which language the input is in.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3.1, p. 373
  5. How is the language token combined with the audio features (Eq. 3)?
    It is prepended along the time axis: \(h' = \mathrm{concat}(t_l, h) \in \mathbb{R}^{N \times (L+1) \times D_{fp}}\).
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3.1, p. 373
  6. What is inside the ULP module?
    n = 4 shared Transformer blocks (RMSNorm, multi-head self-attention, feed-forward) applied to the token-plus-features sequence: \(u = \mathrm{ULP}(h'; \theta)\). Through self-attention every frame can attend to the language token.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §§3.1, 4.2, pp. 373, 375, fig. 1
  7. Why is the language token removed after the ULP (Eq. 5), and why does it matter for CTC?
    The token doesn't correspond to any span of audio. Keeping it would add an extra position and break the alignment between output frames and the transcript that the CTC loss relies on.
  8. How does the ULP output reach the encoder (Eq. 6)?
    As a residual: \(e = \mathrm{Encoder}(h + u)\). The ULP learns a language-specific correction added to the original features, and the frozen wav2vec 2.0 encoder processes the sum.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3.1, p. 373
  9. What does the LM head compute (Eq. 7)?
    A per-frame softmax over the character vocabulary: \(z = \mathrm{Softmax}(We + b)\), with \(W \in \mathbb{R}^{D_{enc} \times M}\), where M is the vocabulary size.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3.1, p. 373
  10. Contrast how MMS and PUMA make a shared model language-specific.
    MMS swaps in a separate set of adapter weights \(\phi_l\) per language. PUMA keeps all weights shared and changes only a single input vector, the language token \(t_l\).
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §3.1, p. 373

Sources