Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “3 Memory Challenges in LLM Serving” · whole deck
3 Memory Challenges in LLM Serving
- How large is the KV cache, for example for OPT-13B?Per token: 2 (key and value) × 5,120 (hidden size) × 40 (layers) × 2 bytes (FP16) = 800 KB; a 2,048-token sequence needs up to 1.6 GB, so even a whole GPU fits only a few tens of requests. GPU compute also grows faster than memory capacity.
- Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §3
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180