Efficient Memory Management for Large Language Model Serving with PagedAttention

4.4 Application to Other Decoding Scenarios

1. How does vLLM share memory in parallel sampling?
   Samples from the same prompt map their prompt blocks to the same physical blocks with a reference count. When a sample needs to write into a shared block, copy-on-write gives it its own copy, as when an operating system forks a process.
   • Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §4.4
2. Why does beam search benefit even more from PagedAttention?
   Beam candidates share not only the prompt but also long common prefixes, and the sharing pattern changes each step like a process tree; blocks whose reference count drops to zero are freed. Previous systems had to copy large parts of the KV cache between candidates. Memory savings reached 37–55% for beam search vs 6–10% for parallel sampling on Alpaca.
   • Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §4.4
3. How can vLLM speed up requests that share a common prefix such as a system prompt or few-shot examples?
   The provider can keep the prefix’s KV cache in reserved physical blocks, like a shared library across processes; each request maps its first logical blocks to them (last block copy-on-write), so only the request-specific part of the prompt is computed.
   • Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention (Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23)), §4.4