Mamba: Linear-Time Sequence Modeling with Selective State Spaces

4.2 Language Modeling

1. How did Mamba compare with Transformers on language modeling?
   It was the first attention-free model to match a strong modern Transformer recipe (Transformer++) in scaling laws from 125M to 1.3B parameters; Mamba-3B beat Pythia-3B by about 4 points on common-sense benchmarks and matched models twice its size.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §4.2