learn.ifs.name

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “3.3 Performance Comparison” · whole deck

3.3 Performance Comparison

  1. How did NSA’s quality compare with full attention?
    Despite its sparsity it was better on average: 7 of 9 general benchmarks (notably DROP and GSM8K), perfect needle-in-a-haystack retrieval at 64k, the best LongBench average (beating full attention and inference-time sparse methods), and higher AIME accuracy after distillation from DeepSeek-R1.
    • Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §3.3, p. 23084

Source

  • Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang and Wangding Zeng. 2025. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126/