Mamba: Linear-Time Sequence Modeling with Selective State Spaces

3.1 Motivation: Selection as a Means of Compression

1. How does Mamba frame the efficiency–effectiveness trade-off of sequence models?
   As how well a model compresses context into its state. Attention doesn’t compress at all (it stores the whole context as a KV cache): effective but slow. Recurrent models have a fixed-size state: fast, but only as good as what that state keeps. Selectivity—choosing what enters the state—is the key.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §3.1
2. Why can’t time-invariant models solve Selective Copying and Induction Heads?
   Plain Copying, with fixed spacing, can be solved by tracking time alone (a convolution kernel of the right length). In Selective Copying the tokens to remember are scattered among noise at random positions, and Induction Heads requires recalling what followed a token earlier; both need content-aware decisions that constant dynamics can’t make.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §3.1