learn.ifs.name

Lost in the Middle: How Language Models Use Long Contexts

Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)

Subscriber deck — free during early access

3 cards in “1 Introduction” · whole deck

1 Introduction

  1. What question does “Lost in the Middle” ask?
    Whether language models with long context windows actually use information anywhere in their input: if they did, performance shouldn’t depend much on where the relevant information is placed.
  2. What is the paper’s main finding?
    A U-shaped performance curve: models use relevant information best when it is at the very beginning (primacy bias) or end (recency bias) of the input, and much worse when it is in the middle, even for explicitly long-context models.
    firstmiddlelastposition of the relevant documentacc.- - closed book (no documents)primacyrecency
    Lost in the middle (stylized after the paper’s GPT-3.5 results with 20–30 documents)
  3. What does Liu et al.’s analysis imply for evaluating long-context models?
    To claim a model robustly uses long inputs, one must show its performance is barely affected by the position of the relevant information, e.g. a small gap between best- and worst-case positions, not just that long inputs fit.

Source

  • Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/