Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Attention can’t model anything outside its finite window, and its cost scales quadratically with window length; efficient variants usually sacrifice the dense information routing that makes attention effective.

They can’t perform content-based reasoning: their dynamics don’t depend on the input, so they can’t decide, token by token, what to remember and what to ignore. This made them weak on discrete, information-dense data like language.

A selection mechanism (SSM parameters become functions of the input), a hardware-aware parallel scan that computes the resulting time-varying recurrence efficiently, and a simple homogeneous architecture without attention or even separate MLP blocks.

A continuous system \(h'(t) = A h(t) + B x(t)\), \(y(t) = C h(t)\) is discretized with a step size \(\Delta\) (e.g. zero-order hold, \(\bar A = \exp(\Delta A)\)), giving the recurrence \(h_t = \bar A h_{t-1} + \bar B x_t\), \(y_t = C h_t\), applied independently to each channel with an \(N\)-dimensional hidden state.

Their parameters \((\Delta, A, B, C)\) were constant over time (linear time invariance), so the recurrence unrolls into a global convolution computable in parallel without materializing the large hidden state; recurrent mode was used for fast autoregressive inference.

As how well a model compresses context into its state. Attention doesn’t compress at all (it stores the whole context as a KV cache): effective but slow. Recurrent models have a fixed-size state: fast, but only as good as what that state keeps. Selectivity—choosing what enters the state—is the key.

Plain Copying, with fixed spacing, can be solved by tracking time alone (a convolution kernel of the right length). In Selective Copying the tokens to remember are scattered among noise at random positions, and Induction Heads requires recalling what followed a token earlier; both need content-aware decisions that constant dynamics can’t make.

It makes \(\Delta\), \(B\) and \(C\) functions of the current input (linear projections of \(x_t\), with softplus for \(\Delta\)), so the model is time-varying: at each step it decides how much of the input to let into the state and how much state to keep.

With kernel fusion (load parameters from slow HBM into fast SRAM, discretize and run the recurrence there, write only outputs back), a work-efficient parallel scan instead of a sequential loop, and recomputation of intermediate states in the backward pass instead of storing them; memory use matches FlashAttention Transformers.

It merges the H3-style SSM block with a gated MLP into one block repeated homogeneously: expand the model dimension (by \(E = 2\)), apply a short convolution, SiLU and the selective SSM on one branch, multiply by a SiLU gate branch, and project back; two blocks match a Transformer layer’s parameters.

With \(N = 1\), \(A = -1\), \(B = 1\) and \(\Delta = \mathrm{softplus}(\mathrm{Linear}(x_t))\), the selective SSM reduces to the gated recurrence \(h_t = (1 - g_t) h_{t-1} + g_t x_t\) with \(g_t = \sigma(\mathrm{Linear}(x_t))\): discretization is the principled basis of gating.

How much to focus on the current input versus keep the state: large \(\Delta\) resets the state and attends to \(x_t\); small \(\Delta\) preserves the state and ignores \(x_t\). This lets the model skip filler tokens, discard irrelevant history and reset at document boundaries.

A 2-layer Mamba trained on sequences of length 256 solved the task perfectly and generalized to length 1,048,576, 4,000× longer than training, while attention models with various position encodings failed beyond about 2× and other SSMs did poorly.

It was the first attention-free model to match a strong modern Transformer recipe (Transformer++) in scaling laws from 125M to 1.3B parameters; Mamba-3B beat Pythia-3B by about 4 points on common-sense benchmarks and matched models twice its size.

Mamba’s pretraining perplexity kept improving as context grew to a million base pairs, while HyenaDNA (a time-invariant model) got worse; selection lets the model ignore noisy history that a long convolution would mix in.

As a recurrent model it needs no KV cache, only a fixed-size state, so each generated token costs constant time and much larger batches fit: about 4–5× the inference throughput of a similar-size Transformer. Its scan was also faster than FlashAttention-2 beyond length 2K.