Mamba: Linear-Time Sequence Modeling with Selective State Spaces

3.5 Properties of Selection Mechanisms

1. How is Mamba’s selection related to RNN gating?
   With \(N = 1\), \(A = -1\), \(B = 1\) and \(\Delta = \mathrm{softplus}(\mathrm{Linear}(x_t))\), the selective SSM reduces to the gated recurrence \(h_t = (1 - g_t) h_{t-1} + g_t x_t\) with \(g_t = \sigma(\mathrm{Linear}(x_t))\): discretization is the principled basis of gating.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §3.5
2. What does the step size \(\Delta\) control in a selective SSM?
   How much to focus on the current input versus keep the state: large \(\Delta\) resets the state and attends to \(x_t\); small \(\Delta\) preserves the state and ignores \(x_t\). This lets the model skip filler tokens, discard irrelevant history and reset at document boundaries.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §3.5