Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “6.2 Basic Sampling” · whole deck
6.2 Basic Sampling
- How much did vLLM improve serving throughput?At similar latency it sustained 2–4× the request rate of state-of-the-art systems such as Orca and up to 22× that of FasterTransformer; e.g. on ShareGPT with OPT-13B it batched 2.2× more requests than Orca with oracle knowledge of output lengths. Gains were largest for long sequences, large models and complex decoding.
- Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §6.2
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180