learn.ifs.name

PUMA: language tokens for low-resource multilingual ASR

Oukid et al., 2026, PUMA (Findings of ACL 2026)

Subscriber deck — free during early access

4 cards in “Reading it critically” · whole deck

Reading it critically

  1. PUMA's training data: what should make you cautious about the very low Hausa WER (0.029)?
    The Hausa training set cited is a synthetic (XTTS-generated) ASR dataset, and the Yoruba, Igbo and Fon sets are cited as TTS datasets. Synthetic or studio TTS speech is much easier than natural speech. On FLEURS, Hausa WER is 0.324 and MMS does better.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §4.1, Appendix B, pp. 375, 381, dataset citations in the reference list
  2. The ablation text reports Arabic changes as a “relative WER increase of 12.1%” and BLEU drop of 11.3%. What's wrong?
    Those are absolute differences (0.384 → 0.505 and 0.345 → 0.232). The relative changes are about +31.5% WER and −32.8% BLEU. Always recompute relative vs absolute from the table.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §4.5, pp. 377–378, table 5
  3. Compare the MMS rows in Tables 1 (monolingual) and 2 (progressive). What do you notice?
    They are identical, so MMS appears not to have been retrained in the progressive setting; the comparison there is against monolingual MMS.
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §4.3.1–4.3.2, p. 376, tables 1, 2
  4. What details would you need to reproduce PUMA that the paper does not give?
    The exact wav2vec 2.0 / MMS checkpoint used, train/validation/test split sizes, the language order and replay amount in progressive training, and the decoding method (greedy vs beam search, any LM).
    • Oukid et al., 2026, PUMA (Findings of ACL 2026), §§3.1, 3.2, 4.1, pp. 373–375

Sources