key-value cache (KV cache)

A KV cache is a store of the key and value vectors that a transformer has already computed for the tokens in a sequence, kept so that generating each new token doesn’t recompute them.

During autoregressive generation, every new token attends to all the earlier ones. Each attention layer turns a token into three vectors: a query, a key, and a value. Queries are matched against keys to score how much each earlier token matters, and the matching values are blended into the result.

A token’s key and value never change once computed, so the layer keeps them and appends one new pair per step. Only the current token’s query is needed, so queries aren’t cached.

The simulation below generates one token at a time, counting the key and value pairs each step computes.

The cache turns the per-step attention cost from quadratic to linear in sequence length. In exchange, it consumes memory that grows with sequence length, batch size, and model depth. That memory is usually what caps how many sequences a server can batch at once. Serving systems keep it in check with:

• Paged block allocation, as in PagedAttention
• Quantization of the cached vectors to lower precision
• Offloading inactive layers to CPU memory
• Sliding-window attention that evicts distant tokens

Prefix caching extends the idea across requests, reusing the cache for a shared system prompt instead of prefilling it again. Caching applies to inference only, not training.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.

intermediate ai

For additional information on related topics, take a look at the following resources:

• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Tutorial)
• Building Type-Safe LLM Agents With Pydantic AI (Course)
• Connecting LLMs to Your Data With Python MCP Servers (Course)
• Python MarkItDown: Convert Documents Into LLM-Ready Markdown (Tutorial)
• Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Quiz)
• Python MarkItDown: Convert Documents Into LLM-Ready Markdown (Quiz)