learn.ifs.name

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “2.2 Overall Framework” · whole deck

2.2 Overall Framework

  1. What is the overall idea of NSA?
    For each query, replace all previous keys and values by a much smaller set built from them in three branches (compressed, selected, sliding window). Each branch runs its own attention, and the outputs are combined with gates from a small MLP with sigmoid outputs.
    compressed× gselected× gwindow× g← earlier keysquery →output = gated sum of the three branches
    Every block is seen coarsely, a few blocks in full, recent tokens always (illustrative)
    • Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 23080

Source

  • Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang and Wangding Zeng. 2025. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126/