Mamba: Linear-Time Sequence Modeling with Selective State Spaces

4.5 Speed and Memory Benchmarks

1. Why is Mamba faster at inference than a Transformer?
   As a recurrent model it needs no KV cache, only a fixed-size state, so each generated token costs constant time and much larger batches fit: about 4–5× the inference throughput of a similar-size Transformer. Its scan was also faster than FlashAttention-2 beyond length 2K.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §4.5