learn.ifs.name

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

13 cards

1 Introduction

  1. What gap does the Aya model address?
    Instruction-following models had been built mainly for a few data-rich languages: 73% of popular instruction-tuning datasets are primarily English. Aya aims to follow instructions given directly in many languages, rather than requiring speakers to prompt in English.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15894
  2. Why do Üstün et al. say the dominance of a few languages in NLP data isn’t natural?
    Which languages are favoured reflects historical access to technology and resources rather than which languages are most spoken or written in the world.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15894
  3. What is the Aya model, in numbers?
    A 13B-parameter model instruction-tuned in 101 languages, over half of them lower-resourced, on a mix of 203M examples with only 21.5% English (vs 46 languages, 81M examples and 39% English in xP3), released under Apache 2.0.
    xP346 langs39% EnglishAya101 langs78.5% other languages21.5%51 of Aya’s 101 languages are lower-resourced
    Share of English in the instruction-tuning mix
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15895

2 Data

  1. What is xP3x, and how was it used for Aya?
    An extension of the xP3 prompt collection from 86M examples in 46 languages to 680M in 277 languages; Aya used a pruned subset of 168M examples in mT5’s 101 languages, favouring quality and longer generations.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2, p. 15895
  2. What kinds of instruction data went into the Aya training mix?
    Templated academic datasets (xP3x, Data Provenance collection, Aya collection templates), 199.5K human-written prompt–response pairs from native speakers (the Aya dataset), machine-translated English instruction datasets (via NLLB), and ShareGPT-Custom: ShareGPT prompts with new synthetic completions, translated into 93 languages.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2, p. 15896
  3. Why did ShareGPT-Custom not reuse the original ShareGPT completions?
    Those completions came from user-shared ChatGPT conversations; to comply with the terms of use, new English completions were generated with in-house models before translation.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2, p. 15897

3 Training and Baselines

  1. Why did Aya fine-tune mT5-XXL rather than a newer model like Llama?
    Although mT5 (2019) is weaker than recent models, it was pretrained on 1T tokens covering exactly 101 languages; adapting embeddings during instruction tuning to languages unseen in pretraining is known to be hard, and recent strong models were English-centric.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 15897
  2. How were Aya’s data sources weighted within the 25M-example training budget?
    Three ablations upweighted human-annotated, translated or templated data. Human annotations were only 0.7% of the budget at natural rates, so upweighting them (to 25%) means repeating each example many times; translation-heavy weighting became the final model.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 15897

5 Results

  1. Why did mT0x (mT5 fine-tuned on xP3x for 101 languages) do worse than mT0 (46 languages)?
    Not mainly capacity dilution: over half of xP3x comes from a handful of multilingual datasets that add little when oversampled. Downsampling them and adding more diverse data, as in Aya, beat both.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15899
  2. How did Aya compare with other massively multilingual models?
    On unseen discriminative tasks the best Aya variant averaged 75.1% vs 72.9% for mT0 (46 languages) despite covering twice the languages; on multilingual MMLU it beat mT0x by 21.1%, mT0 by 18.4% and the per-language Okapi models by 25.1% (relative).
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15899
  3. How did people and GPT-4 judge Aya’s open-ended generations?
    On held-out Dolly prompts Aya was preferred over mT0 and mT0x in all evaluated languages (87% average win rate against mT0, about 77% for human raters against mT0x in seven languages); GPT-4 favoured Aya more consistently than humans, partly perhaps because Aya’s answers are longer.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15900
  4. How did Aya address safety across languages?
    With multilingual safety context distillation, training the model to refuse adversarial prompts in many languages: harmful generations dropped by 78–89% according to human experts. It reduced harm from malicious prompts but had little effect on toxicity in open-ended generation.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §5, p. 15900

Limitations

  1. What limitations of Aya do Üstün et al. acknowledge?
    101 languages are a tiny fraction of about 7,000 and ignore dialects; translated and safety data may skew towards Western cultural values; gendered languages and honorifics raise pronoun choices; and data per language remains very uneven, which can disadvantage lower-resourced languages.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), Limitations, p. 15902

Source

  • Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer and Sara Hooker. 2024. Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894–15939. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.845 https://aclanthology.org/2024.acl-long.845/