Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “4.1 PagedAttention” · whole deck
4.1 PagedAttention
- How is attention computed over blocks in PagedAttention?With block size \(B\), key block \(K_j\) holds keys of tokens \((j-1)B+1 \ldots jB\), likewise \(V_j\). The kernel fetches each block wherever it is stored, computes the query’s attention scores against \(K_j\) and combines them with \(V_j\); the result is identical to ordinary attention.
- Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §4.1
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180