learn.ifs.name

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “2.3.1 Token Compression” · whole deck

2.3.1 Token Compression

  1. What does NSA’s compression branch do?
    It slides over the keys (and values) in blocks of length \(l\) with stride \(d\) (32 and 16 in the experiments) and maps each block, with a learnable MLP and intra-block position encoding, to one compressed key or value. Attending to these gives cheap, coarse access to the whole context.
    • Yuan et al., 2025, Native Sparse Attention (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3.1, p. 23080

Source

  • Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang and Wangding Zeng. 2025. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.1126 https://aclanthology.org/2025.acl-long.1126/