learn.ifs.name

Lost in the Middle: How Language Models Use Long Contexts

Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)

Subscriber deck — free during early access

1 cards in “4.2 Effect of Query-Aware Contextualization” · whole deck

4.2 Effect of Query-Aware Contextualization

  1. What is query-aware contextualization, and did it help?
    Putting the query both before and after the documents or key-value pairs, so a decoder-only model can attend to it while reading the data. It made key-value retrieval near-perfect (GPT-3.5-Turbo 16K: 100% with 300 pairs vs a worst case of 45.6%) but barely changed multi-document QA.

Source

  • Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/