Mamba: Linear-Time Sequence Modeling with Selective State Spaces

4.1 Synthetic Tasks

1. How did Mamba do on Induction Heads?
   A 2-layer Mamba trained on sequences of length 256 solved the task perfectly and generalized to length 1,048,576, 4,000× longer than training, while attention models with various position encodings failed beyond about 2× and other SSMs did poorly.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §4.1