Lost in the Middle: How Language Models Use Long Contexts
Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)
Subscriber deck — free during early access
2 cards in “2.3 Results and Discussion” · whole deck
2.3 Results and Discussion
- How bad can “lost in the middle” get for GPT-3.5-Turbo?Its multi-document QA accuracy dropped by more than 20 points when the answer document moved to the middle; with 20 or 30 documents its worst case fell below its closed-book accuracy (56.1%) with no documents at all.
- Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12), §2.3, pp. 161–162
- Are extended-context models better at using their input than their standard versions?No: when an input fits both, e.g. GPT-3.5-Turbo vs its 16K version, their accuracy curves across positions were nearly identical. A longer window doesn’t mean better use of the context.
- Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12), §2.3, p. 162
Source
- Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/