learn.ifs.name

Lost in the Middle: How Language Models Use Long Contexts

Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)

Subscriber deck — free during early access

1 cards in “4.1 Effect of Model Architecture” · whole deck

4.1 Effect of Model Architecture

  1. How did encoder–decoder models compare with decoder-only models?
    Flan-UL2 and Flan-T5-XXL were fairly robust to position (e.g. 1.9-point best–worst gap for Flan-UL2) when the input was within their training-time length, but showed the U shape on longer inputs. Their bidirectional encoder may help by contextualizing each document with later ones.

Source

  • Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/