Mamba: Linear-Time Sequence Modeling with Selective State Spaces

3.4 A Simplified SSM Architecture

1. What does a Mamba block look like?
   It merges the H3-style SSM block with a gated MLP into one block repeated homogeneously: expand the model dimension (by \(E = 2\)), apply a short convolution, SiLU and the selective SSM on one branch, multiply by a SiLU gate branch, and project back; two blocks match a Transformer layer’s parameters.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §3.4