Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “7.2 Impact of Block Size” · whole deck
7.2 Impact of Block Size
- How should the KV block size be chosen?Too small and the kernel can’t exploit GPU parallelism; too large and internal fragmentation grows and sharing becomes less likely. Block size 16 worked well across workloads and is vLLM’s default.
- Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §7.2
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180