Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23))
Subscriber deck — free during early access
1 cards in “7.3 Comparing Recomputation and Swapping” · whole deck
7.3 Comparing Recomputation and Swapping
- When is recomputation better than swapping for recovering evicted KV blocks?Swapping suffers with small blocks, because many small CPU–GPU transfers underuse PCIe bandwidth; recomputation’s cost is independent of block size. Recomputation wins for small blocks, swapping for large ones; for 16–64 they perform similarly.
- Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §7.3
Source
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). doi:10.1145/3600006.3613165 https://arxiv.org/abs/2309.06180