learn.ifs.name

Scaling Neural Machine Translation to 200 Languages

Team, 2024, Scaling neural machine translation to 200 languages (Nature, vol. 630)

Subscriber deck — free during early access

3 cards in “Modelling” · whole deck

Modelling

  1. Why did NLLB use a sparsely gated mixture-of-experts model?
    To share capacity for cross-lingual transfer between related languages while limiting interference between unrelated ones: some feed-forward layers in encoder and decoder become MoE layers, routing each token to its top two experts.
  2. How did NLLB counter overfitting on low-resource language pairs?
    Expert Output Masking, which drops the expert outputs for a random fraction of tokens (beating gating dropout), and curriculum learning: pairs that overfit within \(K\) updates are introduced only \(K\) updates before the end of training.
  3. What did NLLB’s expert-routing heat maps show?
    Late decoder layers are language-specific, with related languages (Arabic dialects, Benue–Congo languages, Devanagari-script languages) choosing similar experts; late encoder layers route tokens in a largely language-agnostic way.

Source