Efficient Streaming Language Models with Attention Sinks
Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024))
Subscriber deck — free during early access
1 cards in “4.5 Efficiency Results” · whole deck
4.5 Efficiency Results
- How much faster is StreamingLLM than sliding-window recomputation?Its per-token decoding latency grows linearly with cache size versus quadratically for recomputation, giving up to 22.2× speedup per token with a similar memory footprint.
- Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §4.5
Source
- Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2309.17453