learn.ifs.name

OLMo: Accelerating the Science of Language Models

Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

13 cards

1 Introduction

  1. Why did Groeneveld et al. build OLMo?
    The most powerful language models had become closed, with key details (especially training data) undisclosed, which makes it hard to study them scientifically, including their biases and risks. OLMo is a competitive, truly open model to enable that research.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15789
  2. How did earlier “open” model releases differ in openness?
    Mixtral released weights and a brief report; Llama came with adaptation instructions; MPT described its data mix but not the data; Falcon released part of its data; Pythia was the most open, with code, checkpoints and data, but far from state-of-the-art capability.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 15789–15790
  3. What does the OLMo release include?
    Full weights for 1B and 7B models (with variants for different architectures, optimizers and hardware), hundreds of intermediate checkpoints, training code and logs, the full pretraining data (Dolma) with its curation code and analysis tools, evaluation code (Catwalk, Paloma) and adaptation code, all under Apache 2.0.
    weights+ training code+ training data+ checkpoints, logsOLMoeach row adds one kind of artifact(cf. Mixtral: weights; Pythia: weights, code, data)
    What a “truly open” release contains
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 15790

2.1 OLMo Model and Architecture

  1. What changes did OLMo make to the original Transformer?
    No bias terms (for stability); non-parametric layer norm without learned gain or bias (judged safest and fastest); SwiGLU activations (hidden size about \(\tfrac{8}{3}d\), rounded up to a multiple of 128); rotary position embeddings; and a BPE tokenizer with extra tokens for masking personal information.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.1, p. 15790

2.2 Pretraining Data: Dolma

  1. What is Dolma, and how was it built?
    OLMo’s open pretraining corpus of trillions of tokens from sources common in LM pretraining and publicly accessible (mostly Common Crawl, plus code, Reddit, papers, books and Wikipedia), built by language, quality and content filtering, deduplication, mixing and tokenization; sources are kept separate in the release.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 15791

2.4 Evaluation

  1. What is the difference between OLMo’s online and offline evaluation?
    Online (in-loop) evaluation runs every 1,000 steps (about 4B tokens) during training to guide decisions on architecture, optimizer, schedule and data mix. Offline evaluation compares final and intermediate checkpoints with other models via Catwalk (downstream tasks) and Paloma (perplexity).
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.4, p. 15792
  2. Why does OLMo’s perplexity evaluation require decontamination?
    Paloma covers 585 text domains; OLMo removed any pretraining document containing paragraphs leaked from Paloma’s evaluation data. Without this, models can look better than they are (underestimated perplexity, overestimated out-of-sample fit).
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.4, p. 15792

3.3 Data

  1. How can OLMo’s exact training data order be reproduced?
    Documents are concatenated with an end-of-sequence token, cut into 2,048-token instances and shuffled identically for every run; the released artifacts let anyone reconstruct the data order and the exact composition of each batch.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.3, p. 15793

3.4 Hardware

  1. What did training OLMo on two different clusters show?
    Runs on AMD MI250X GPUs (LUMI) and NVIDIA A100s (MosaicML) reached nearly identical evaluation performance despite small batch-size differences, showing the codebase is portable across hardware.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §3.4, p. 15793

4.1 Downstream evaluation

  1. How did OLMo-7B perform, and what training detail visibly helped?
    It was competitive with comparable open models (Llama 2 7B, Falcon-7B, MPT-7B) on eight zero-shot tasks. Training a further 1,000 steps with the learning rate decayed linearly to zero gave a sharp late jump in accuracy on many tasks.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §4.1, pp. 15794–15795

4.2 Intrinsic language modeling evaluation

  1. What did Paloma show about sample efficiency across text sources?
    Models fit best the kinds of text they were trained on: OLMo-7B was best on C4 (close to its Common-Crawl-heavy data) but less efficient on Wikipedia and scientific papers, which are scarcer than web text; keeping fit to such curated sources will be hard as corpora scale.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §4.2, p. 15795

4.3 Adaptation Evaluation

  1. How much did adaptation help OLMo-7B?
    Instruction tuning on the Tülu mix and then DPO greatly improved chat and safety scores (e.g. ToxiGen, TruthfulQA), with OLMo beating most other chat variants, though still behind Tülu 2 built on Llama 2, for which that data mix was designed.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §4.3, p. 15796

Ethics Statement

  1. How do OLMo’s authors justify releasing everything openly despite risks?
    Openness is essential for scientific understanding and broad participation; others can build on the model instead of retraining (saving compute and emissions); research into harms is accelerated by openness; and comparable models were already permissively licensed, so a stricter license wouldn’t remove the risk.
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), Ethics Statement, p. 15798

Source

  • Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith and Hannaneh Hajishirzi. 2024. OLMo: Accelerating the Science of Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789–15809. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.841 https://aclanthology.org/2024.acl-long.841/