learn.ifs.name

Lost in the Middle: How Language Models Use Long Contexts

Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)

Subscriber deck — free during early access

14 cards

1 Introduction

  1. What question does “Lost in the Middle” ask?
    Whether language models with long context windows actually use information anywhere in their input: if they did, performance shouldn’t depend much on where the relevant information is placed.
  2. What is the paper’s main finding?
    A U-shaped performance curve: models use relevant information best when it is at the very beginning (primacy bias) or end (recency bias) of the input, and much worse when it is in the middle, even for explicitly long-context models.
    firstmiddlelastposition of the relevant documentacc.- - closed book (no documents)primacyrecency
    Lost in the middle (stylized after the paper’s GPT-3.5 results with 20–30 documents)
  3. What does Liu et al.’s analysis imply for evaluating long-context models?
    To claim a model robustly uses long inputs, one must show its performance is barely affected by the position of the relevant information, e.g. a small gap between best- and worst-case positions, not just that long inputs fit.

2.1 Experimental Setup

  1. How is the multi-document QA task built?
    Each NaturalQuestions-Open question comes with \(k\) Wikipedia passages: exactly one contains the answer, and \(k - 1\) are distractors retrieved by Contriever as most relevant but answer-free. Context length is varied via \(k\) (10, 20, 30), position by reordering the passages.

2.3 Results and Discussion

  1. How bad can “lost in the middle” get for GPT-3.5-Turbo?
    Its multi-document QA accuracy dropped by more than 20 points when the answer document moved to the middle; with 20 or 30 documents its worst case fell below its closed-book accuracy (56.1%) with no documents at all.
  2. Are extended-context models better at using their input than their standard versions?
    No: when an input fits both, e.g. GPT-3.5-Turbo vs its 16K version, their accuracy curves across positions were nearly identical. A longer window doesn’t mean better use of the context.

3 How Well Can Language Models Retrieve From Input Contexts?

  1. What is the synthetic key-value retrieval task, and why use random UUIDs?
    The input is a JSON object of \(k\) key-value pairs (75, 140 or 300), all random 128-bit UUIDs, plus a query key; the model must return its value. Random strings strip away natural-language semantics that could confound the test of pure retrieval.
  2. What did key-value retrieval reveal?
    Claude-1.3 was nearly perfect at all lengths, but other models struggled, again worst when the key was in the middle: even exact-match retrieval of a token string from the context isn’t reliable for all models.

4.1 Effect of Model Architecture

  1. How did encoder–decoder models compare with decoder-only models?
    Flan-UL2 and Flan-T5-XXL were fairly robust to position (e.g. 1.9-point best–worst gap for Flan-UL2) when the input was within their training-time length, but showed the U shape on longer inputs. Their bidirectional encoder may help by contextualizing each document with later ones.

4.2 Effect of Query-Aware Contextualization

  1. What is query-aware contextualization, and did it help?
    Putting the query both before and after the documents or key-value pairs, so a decoder-only model can attend to it while reading the data. It made key-value retrieval near-perfect (GPT-3.5-Turbo 16K: 100% with 300 pairs vs a worst case of 45.6%) but barely changed multi-document QA.

4.3 Effect of Instruction Fine-Tuning

  1. Is instruction tuning responsible for the U-shaped curve?
    No: the base MPT-30B showed the same U shape as MPT-30B-Instruct (instruction tuning only slightly reduced the gap). Among Llama-2 models, the 7B ones were purely recency-biased, while 13B and 70B showed the U shape.

5 Is More Context Always Better? A Case Study With Open-Domain QA

  1. What did the open-domain QA case study show about adding retrieved documents?
    Reader accuracy saturated long before retriever recall did: going from 20 to 50 retrieved documents gained only about 1–1.5 points while greatly increasing context length, latency and cost.
  2. What practical remedies for retrieval-augmented generation do Liu et al. suggest?
    Rerank retrieved documents so relevant ones come near the start of the context, and truncate the ranked list, retrieving fewer documents when appropriate.

6.3 The Serial-Position Effect

  1. What psychological phenomenon does the U-shaped curve resemble, and why is that surprising?
    The serial-position effect: in free recall people remember the first and last items of a list best. It is surprising in Transformers, whose self-attention can in principle retrieve any token equally well.

Source

  • Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/