PagedAttention

PagedAttention is an attention algorithm that stores a model’s key-value cache (KV cache) in fixed-size blocks scattered across GPU memory, applying the paging technique that operating systems use for virtual memory.

During autoregressive generation, a request accumulates key and value vectors for every token it has processed, so its cache keeps growing and its final size isn’t known in advance. Serving systems that reserve one contiguous slab per request lose much of that memory to fragmentation and over-reservation. PagedAttention allocates small blocks on demand and keeps a per-request block table that maps logical token positions to physical blocks.

The simulation below runs the same request stream under both schemes, with dashed blocks marking memory that was reserved but never filled.

Interactive diagram — enable JavaScript to view.

The block table also lets requests share blocks, so a common prompt prefix is stored once and copied only when generations diverge. Reclaiming the wasted memory lets a server batch more concurrent requests, which raises throughput at comparable latency. The 2023 vLLM paper introduced the algorithm, and other inference servers such as TensorRT-LLM added paged KV caches soon after.

How to Run Large Language Models Locally With Ollama

Tutorial

How to Use Ollama to Run Large Language Models Locally

Learn how to use Ollama to run large language models locally. Install it, pull models, and start chatting from your terminal without needing API keys.

intermediate ai tools

For additional information on related topics, take a look at the following resources:

By Martin Breuss • Updated Aug. 20, 2026
AI Coding Glossary Share Feedback