learn.ifs.name

OLMo: Accelerating the Science of Language Models

Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

2 cards in “2.4 Evaluation” · whole deck

2.4 Evaluation

  1. What is the difference between OLMo’s online and offline evaluation?
    Online (in-loop) evaluation runs every 1,000 steps (about 4B tokens) during training to guide decisions on architecture, optimizer, schedule and data mix. Offline evaluation compares final and intermediate checkpoints with other models via Catwalk (downstream tasks) and Paloma (perplexity).
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.4, p. 15792
  2. Why does OLMo’s perplexity evaluation require decontamination?
    Paloma covers 585 text domains; OLMo removed any pretraining document containing paragraphs leaked from Paloma’s evaluation data. Without this, models can look better than they are (underestimated perplexity, overestimated out-of-sample fit).
    • Groeneveld et al., 2024, OLMo (Proceedings of the 62nd Annual Meeting of ACL (Volume 1: Long Papers)), §2.4, p. 15792

Source

  • Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith and Hannaneh Hajishirzi. 2024. OLMo: Accelerating the Science of Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789–15809. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2024.acl-long.841 https://aclanthology.org/2024.acl-long.841/