Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
2 cards in “3 Training and Baselines” · whole deck
3 Training and Baselines
- Why did Aya fine-tune mT5-XXL rather than a newer model like Llama?Although mT5 (2019) is weaker than recent models, it was pretrained on 1T tokens covering exactly 101 languages; adapting embeddings during instruction tuning to languages unseen in pretraining is known to be hard, and recent strong models were English-centric.
- Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 15897
- How were Aya’s data sources weighted within the 25M-example training budget?Three ablations upweighted human-annotated, translated or templated data. Human annotations were only 0.7% of the budget at natural rates, so upweighting them (to 25%) means repeating each example many times; translation-heavy weighting became the final model.
- Üstün et al., 2024, Aya Model (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 15897
Source
- Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer and Sara Hooker. 2024. Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894–15939. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.845 https://aclanthology.org/2024.acl-long.845/