learn.ifs.name

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024))

Subscriber deck — free during early access

1 cards in “3.2 Improving SSMs with Selection” · whole deck

3.2 Improving SSMs with Selection

  1. What exactly does Mamba’s selection mechanism change?
    It makes \(\Delta\), \(B\) and \(C\) functions of the current input (linear projections of \(x_t\), with softplus for \(\Delta\)), so the model is time-varying: at each step it decides how much of the input to let into the state and how much state to keep.
    ht−1htytxtĀ(Δₜ)B̄(Δₜ, Bₜ)CₜΔₜ, Bₜ, Cₜ computed from xₜ(fixed for all t in earlier SSMs)
    Selection: the input decides how much enters and stays in the state
    • Gu & Dao, 2024, Mamba (First Conference on Language Modeling (COLM 2024)), §3.2

Source

  • Albert Gu and Tri Dao. 2024. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In First Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2312.00752