Scaling Neural Machine Translation to 200 Languages

Neural MT quality depends on large parallel corpora, which exist for few of the world’s 7,000+ languages, so progress has mostly benefited high-resource languages and widened digital inequities. NLLB-200 is a single model translating among 200 languages, three times as many low- as high-resource.

One with at least 1 million sentences of aligned text (bitext) with another language; a very low-resource language has fewer than 100,000 samples across all its pairings.

An average 44% improvement in translation quality (BLEU-based) over the previous state of the art, while doubling the number of supported languages, showing that wide coverage and quality aren’t mutually exclusive.

FLORES-200, a benchmark covering about 40,000 translation directions; XSTS, a human evaluation protocol focused on meaning that is consistent across languages; and ETOX, a detector of toxicity added by translation, covering every language.

Collecting monolingual web data for 200 languages needs accurate language ID, which is far from solved on web text: domains differ by language, similar languages are confused, and classifiers trained on found data conflate domain with language. NLLB tuned a fastText model on FLORES-200 plus handwritten rules.

It embedded sentences of all languages in a shared multilingual space (LASER3) and treated pairs with high similarity as translations. New languages were added by teacher–student distillation into the existing LASER space, so previously embedded data needn’t be re-encoded.

Over 1.1 billion new sentence pairs for 148 languages; together with back-translation, this improved translation of very low-resource languages into English by 12.5 chrF++.

To share capacity for cross-lingual transfer between related languages while limiting interference between unrelated ones: some feed-forward layers in encoder and decoder become MoE layers, routing each token to its top two experts.

Expert Output Masking, which drops the expert outputs for a random fraction of tokens (beating gating dropout), and curriculum learning: pairs that overfit within \(K\) updates are introduced only \(K\) updates before the end of training.

Late decoder layers are language-specific, with related languages (Arabic dialects, Benue–Congo languages, Devanagari-script languages) choosing similar experts; late encoder layers route tokens in a largely language-agnostic way.

Each language’s set went through alignment, translation, iterative and final quality assurance, and was accepted only when independent raters agreed with at least 90% of the reference translations; evaluation data were filtered from training data by hashing.

A human protocol scoring meaning preservation on a 1–5 scale (3 = acceptable), calibrated across evaluators and languages. Calibrated scores correlated reasonably with spBLEU and chrF++ (Spearman about 0.69–0.71), better than uncalibrated ones.

Largely misaligned training bitext in which one side has toxic words the other lacks. Filtering pairs where one side has at least two more toxic items removed about 30% of data for some languages, improving chrF++ by 5% and reducing added toxicity similarly.

Four months after release it was the third most used machine translation engine among Wikipedia editors, with the lowest share of deleted articles (0.13%) among translation services; the authors stress that technology alone can’t close language gaps without education, access and literacy policies.