Lost in the Middle: How Language Models Use Long Contexts
Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)
Subscriber deck — free during early access
2 cards in “3 How Well Can Language Models Retrieve From Input Contexts?” · whole deck
3 How Well Can Language Models Retrieve From Input Contexts?
- What is the synthetic key-value retrieval task, and why use random UUIDs?The input is a JSON object of \(k\) key-value pairs (75, 140 or 300), all random 128-bit UUIDs, plus a query key; the model must return its value. Random strings strip away natural-language semantics that could confound the test of pure retrieval.
- Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12), §3, p. 162
- What did key-value retrieval reveal?Claude-1.3 was nearly perfect at all lengths, but other models struggled, again worst when the key was in the middle: even exact-match retrieval of a token string from the context isn’t reliable for all models.
- Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12), §3, p. 163
Source
- Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/