Efficient Streaming Language Models with Attention Sinks
Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024))
Subscriber deck — free during early access
1 cards in “4.1 Language Modeling on Long Texts Across LLM Families and Scales” · whole deck
4.1 Language Modeling on Long Texts Across LLM Families and Scales
- How far could StreamingLLM go?Llama-2, MPT, Falcon and Pythia models (2.8B–70B) kept stable perplexity over 4 million tokens of PG19 books, matching the sliding-window-with-recomputation oracle, while dense attention failed past the pre-training length and window attention past the cache size.
- Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §4.1
Source
- Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2309.17453