continuous batching

Continuous batching is a serving technique that lets an inference server add and drop requests from a running batch after every generation step, rather than holding a fixed group of requests together until the slowest one finishes.

During autoregressive generation, a large language model emits one token per forward pass, so a request needs as many passes as it has tokens to generate. Static batching schedules at the granularity of the request, so a sequence that finishes early still holds its slot while an arriving request waits for the whole batch to drain.

Continuous batching schedules at the granularity of the iteration instead. A finished sequence returns to its client immediately, and a queued request takes the freed slot on the next forward pass.

The timeline below runs one queue of requests through both schedulers, with a row per GPU slot and a column per forward pass.

The technique originates in iteration-level scheduling, introduced by the Orca serving system at OSDI 2022, which reported a 36.9x throughput gain over NVIDIA FasterTransformer on a GPT-3 175B model at matched latency.

Per-iteration scheduling conflicts with batching inside the attention operation, whose tensor shapes vary with how many tokens each request has already processed. Orca resolves that with selective batching, running attention per request while the parameter-bearing layers stay batched.

Inference servers such as vLLM adopt this scheduling by default, and NVIDIA’s TensorRT-LLM calls the same idea in-flight batching.

Admitting more concurrent sequences raises throughput until the KV cache exhausts GPU memory, so continuous batching is usually paired with memory schemes like PagedAttention. Folding a long prompt’s prefill pass, which processes the whole prompt at once, into a batch of decode steps, which each generate one token, stalls those steps, a cost that chunked prefill reduces by spreading the prefill across several iterations.

Related Resources

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.

intermediate ai

For additional information on related topics, take a look at the following resources:

• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• How to Integrate Local LLMs With Ollama and Python (Tutorial)
• Speed Up Your Python Program With Concurrency (Tutorial)
• Python Stacks, Queues, and Priority Queues in Practice (Tutorial)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• How to Get Started With Ollama (Course)
• How to Integrate Local LLMs With Ollama and Python (Quiz)
• Speed Up Python With Concurrency (Course)
• Python Concurrency (Quiz)
• Python Stacks, Queues, and Priority Queues in Practice (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)