Efficient Streaming Language Models with Attention Sinks
Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024))
Subscriber deck — free during early access
2 cards in “3.1 The Failure of Window Attention and Attention Sinks” · whole deck
3.1 The Failure of Window Attention and Attention Sinks
- Why does window attention collapse when the initial tokens are evicted?Their keys took a large part of the softmax denominator in most layers; removing them shifts the attention distribution far from what the model saw in training, and perplexity spikes.
- Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.1
- How did Xiao et al. show that the initial tokens matter for their position rather than their meaning?Replacing the first four tokens with line-break tokens and keeping them alongside the recent window restored perplexity almost as well as the original tokens: the model attends to them because of where they are.
- Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.1
Source
- Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2309.17453