Efficient Streaming Language Models with Attention Sinks

3.2 Rolling KV Cache with Attention Sinks

1. How does StreamingLLM manage the KV cache?
   It keeps the keys and values of a few initial tokens (four suffice) as attention sinks, plus a rolling window of the most recent tokens, evicting everything in between. No fine-tuning is needed.

   • Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.2
2. How does StreamingLLM assign positions to cached tokens?
   By position within the cache, not in the original text: with cached tokens 0–3 and 6–8 while decoding token 9, positions are 0–7. For RoPE, keys are cached before rotation and rotated by their cache position at each step; ALiBi gets a contiguous bias.
   • Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.2
3. Why are four initial tokens needed rather than one?
   Models like Llama-2 weren’t pre-trained with a consistent first token in every sample (the start token is added before text is chunked), so they learned to spread sink attention over several initial tokens; one or two don’t fully restore perplexity.
   • Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.2