learn.ifs.name

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

3 cards in “2 Data” · whole deck

2 Data

  1. What is xP3x, and how was it used for Aya?
    An extension of the xP3 prompt collection from 86M examples in 46 languages to 680M in 277 languages; Aya used a pruned subset of 168M examples in mT5’s 101 languages, favouring quality and longer generations.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2, p. 15895
  2. What kinds of instruction data went into the Aya training mix?
    Templated academic datasets (xP3x, Data Provenance collection, Aya collection templates), 199.5K human-written prompt–response pairs from native speakers (the Aya dataset), machine-translated English instruction datasets (via NLLB), and ShareGPT-Custom: ShareGPT prompts with new synthetic completions, translated into 93 languages.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2, p. 15896
  3. Why did ShareGPT-Custom not reuse the original ShareGPT completions?
    Those completions came from user-shared ChatGPT conversations; to comply with the terms of use, new English completions were generated with in-house models before translation.
    • Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2, p. 15897

Source

  • Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer and Sara Hooker. 2024. Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894–15939. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.845 https://aclanthology.org/2024.acl-long.845/