Efficient Streaming Language Models with Attention Sinks
Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024))
Subscriber deck — free during early access
1 cards in “4.4 Ablation Studies” · whole deck
4.4 Ablation Studies
- Does a larger cache always improve StreamingLLM’s perplexity?No: increasing the cache size didn’t consistently lower perplexity, suggesting the models don’t fully use the context they get. StreamingLLM also doesn’t extend the context window: tokens evicted from the cache are forgotten.
- Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §4.4
Source
- Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2309.17453