Efficient Streaming Language Models with Attention Sinks

3.3 Pre-training LLMs with Attention Sinks

1. What did Xiao et al. propose for future pre-training?
   Prepend a learnable sink token to every training sample. In 160M-parameter models it became a dedicated attention sink: the sink token plus the recent window gave stable streaming perplexity, with no loss in convergence or benchmark accuracy.
   • Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.3
2. How did the “Zero Sink” (softmax-off-by-one) compare with a learnable sink token?
   Letting attention sum to less than one (equivalent to a zero key and value token) helped only somewhat; the model still relied on other initial tokens. A learnable sink token worked much better.
   • Xiao et al., 2024, Efficient Streaming Language Models with Attention Sinks (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.3