Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “2.3.2 Token Selection” · whole deck
2.3.2 Token Selection
- How does NSA choose which blocks to attend to in full, and why blocks?It reuses the attention scores of the compressed keys as block importance scores (summed over the query heads of a GQA group so they share blocks) and keeps the top-\(n\) blocks (16 blocks of 64 tokens). Contiguous blocks suit GPUs and Tensor Cores, and attention scores tend to be spatially continuous anyway.
- Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3.2, p. 23081
Source
- Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang and Wangding Zeng. 2025. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126/