learn.ifs.name

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

3 cards in “1 Introduction” · whole deck

1 Introduction

  1. What gap does the Aya model address?
    Instruction-following models had been built mainly for a few data-rich languages: 73% of popular instruction-tuning datasets are primarily English. Aya aims to follow instructions given directly in many languages, rather than requiring speakers to prompt in English.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15894
  2. Why do Üstün et al. say the dominance of a few languages in NLP data isn’t natural?
    Which languages are favoured reflects historical access to technology and resources rather than which languages are most spoken or written in the world.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15894
  3. What is the Aya model, in numbers?
    A 13B-parameter model instruction-tuned in 101 languages, over half of them lower-resourced, on a mix of 203M examples with only 21.5% English (vs 46 languages, 81M examples and 39% English in xP3), released under Apache 2.0.
    xP346 langs39% EnglishAya101 langs78.5% other languages21.5%51 of Aya’s 101 languages are lower-resourced
    Share of English in the instruction-tuning mix
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15895

Source

  • Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer and Sara Hooker. 2024. Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894–15939. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.845 https://aclanthology.org/2024.acl-long.845/