Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

5 Results

1. Why did mT0x (mT5 fine-tuned on xP3x for 101 languages) do worse than mT0 (46 languages)?
   Not mainly capacity dilution: over half of xP3x comes from a handful of multilingual datasets that add little when oversampled. Downsampling them and adding more diverse data, as in Aya, beat both.
   • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15899
2. How did Aya compare with other massively multilingual models?
   On unseen discriminative tasks the best Aya variant averaged 75.1% vs 72.9% for mT0 (46 languages) despite covering twice the languages; on multilingual MMLU it beat mT0x by 21.1%, mT0 by 18.4% and the per-language Okapi models by 25.1% (relative).
   • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15899
3. How did people and GPT-4 judge Aya’s open-ended generations?
   On held-out Dolly prompts Aya was preferred over mT0 and mT0x in all evaluated languages (87% average win rate against mT0, about 77% for human raters against mT0x in seven languages); GPT-4 favoured Aya more consistently than humans, partly perhaps because Aya’s answers are longer.
   • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15900
4. How did Aya address safety across languages?
   With multilingual safety context distillation, training the model to refuse adversarial prompts in many languages: harmful generations dropped by 78–89% according to human experts. It reduced harm from malicious prompts but had little effect on toxicity in open-ended generation.
   • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15900