Scaling Neural Machine Translation to 200 Languages

Evaluation

1. How was the quality of FLORES-200 assured?
   Each language’s set went through alignment, translation, iterative and final quality assurance, and was accepted only when independent raters agreed with at least 90% of the reference translations; evaluation data were filtered from training data by hashing.
   • Team, 2024, Scaling neural machine translation to 200 languages (Nature, vol. 630), Evaluation, p. 844
2. What is XSTS, and how did automatic metrics correlate with it?
   A human protocol scoring meaning preservation on a 1–5 scale (3 = acceptable), calibrated across evaluators and languages. Calibrated scores correlated reasonably with spBLEU and chrF++ (Spearman about 0.69–0.71), better than uncalibrated ones.
   • Team, 2024, Scaling neural machine translation to 200 languages (Nature, vol. 630), Evaluation, p. 844
3. What causes added toxicity in translation, and how did NLLB reduce it?
   Largely misaligned training bitext in which one side has toxic words the other lacks. Filtering pairs where one side has at least two more toxic items removed about 30% of data for some languages, improving chrF++ by 5% and reducing added toxicity similarly.
   • Team, 2024, Scaling neural machine translation to 200 languages (Nature, vol. 630), Evaluation, pp. 844–845