Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “4.2 KV Cache Manager” · whole deck
4.2 KV Cache Manager
- How does vLLM map a request’s KV cache to GPU memory?Each request has logical KV blocks filled left to right; a block table maps them to physical blocks anywhere in GPU memory and records how many slots are filled. A new physical block is allocated only when the last one is full, so waste per request is at most one partly filled block.
A block table maps each sample’s logical KV blocks to scattered physical blocks; shared blocks carry a reference count - Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §4.2
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180