Mamba: Linear-Time Sequence Modeling with Selective State Spaces

4.3 DNA Modeling

1. What did DNA experiments show about context length?
   Mamba’s pretraining perplexity kept improving as context grew to a million base pairs, while HyenaDNA (a time-invariant model) got worse; selection lets the model ignore noisy history that a long convolution would mix in.
   • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §4.3