Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “2.2 LLM Service & Autoregressive Generation” · whole deck
2.2 LLM Service & Autoregressive Generation
- What are the two phases of LLM generation, and how do they differ computationally?The prompt phase processes all prompt tokens at once (parallel matrix–matrix work, GPU-efficient) and builds their KV cache. The generation phase produces one token per iteration using the cached keys and values (matrix–vector work), which is memory-bound and dominates latency.
- Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §2.2
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180