Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
2 cards in “1 Introduction” · whole deck
1 Introduction
- Why is attention the bottleneck for long contexts, and why hadn’t sparse attention solved it?Attention cost grows with context length; when decoding 64k-token contexts it is estimated at 70–80% of latency. Earlier sparse methods often failed to turn theoretical savings into real speedups, and most were applied only at inference, without training support.
- Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 23078
- What two requirements does NSA set for practical sparse attention?Hardware-aligned speedups in both prefilling and decoding (memory access and GPU scheduling must actually benefit), and training-aware design: sparse operators that can be trained end to end, so pretraining itself becomes cheaper and the model learns to use the sparsity.
- Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 23079
Source
- Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang and Wangding Zeng. 2025. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126/