learn.ifs.name

OLMo: Accelerating the Science of Language Models

Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

3 cards in “1 Introduction” · whole deck

1 Introduction

  1. Why did Groeneveld et al. build OLMo?
    The most powerful language models had become closed, with key details (especially training data) undisclosed, which makes it hard to study them scientifically, including their biases and risks. OLMo is a competitive, truly open model to enable that research.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15789
  2. How did earlier “open” model releases differ in openness?
    Mixtral released weights and a brief report; Llama came with adaptation instructions; MPT described its data mix but not the data; Falcon released part of its data; Pythia was the most open, with code, checkpoints and data, but far from state-of-the-art capability.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 15789–15790
  3. What does the OLMo release include?
    Full weights for 1B and 7B models (with variants for different architectures, optimizers and hardware), hundreds of intermediate checkpoints, training code and logs, the full pretraining data (Dolma) with its curation code and analysis tools, evaluation code (Catwalk, Paloma) and adaptation code, all under Apache 2.0.
    weights+ training code+ training data+ checkpoints, logsOLMoeach row adds one kind of artifact(cf. Mixtral: weights; Pythia: weights, code, data)
    What a “truly open” release contains
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15790

Source

  • Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith and Hannaneh Hajishirzi. 2024. OLMo: Accelerating the Science of Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789–15809. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.841 https://aclanthology.org/2024.acl-long.841/