Lost in the Middle: How Language Models Use Long Contexts

Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12)

Subscriber deck — free during early access

1 cards in “4.3 Effect of Instruction Fine-Tuning” · whole deck

4.3 Effect of Instruction Fine-Tuning

Is instruction tuning responsible for the U-shaped curve?
No: the base MPT-30B showed the same U shape as MPT-30B-Instruct (instruction tuning only slightly reduced the gap). Among Llama-2 models, the 7B ones were purely recency-biased, while 13B and 70B showed the U shape.
- Liu et al., 2024, Lost in the Middle (Transactions of ACL, vol. 12), §4.3, p. 165

Source

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. In Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1162/tacl_a_00638 https://aclanthology.org/2024.tacl-1.9/