Mamba: Linear-Time Sequence Modeling with Selective State Spaces

1 Introduction

1. What two drawbacks of attention motivate Mamba?
   Attention can’t model anything outside its finite window, and its cost scales quadratically with window length; efficient variants usually sacrifice the dense information routing that makes attention effective.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §1
2. What weakness of earlier subquadratic models (linear attention, gated convolutions, structured SSMs) does Mamba identify?
   They can’t perform content-based reasoning: their dynamics don’t depend on the input, so they can’t decide, token by token, what to remember and what to ignore. This made them weak on discrete, information-dense data like language.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §1
3. What are Mamba’s three main ingredients?
   A selection mechanism (SSM parameters become functions of the input), a hardware-aware parallel scan that computes the resulting time-varying recurrence efficiently, and a simple homogeneous architecture without attention or even separate MLP blocks.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §1