throughput

Throughput is the rate at which a system completes useful work, typically measured as units per second averaged over a given time interval.

In AI workloads, serving throughput is reported as output tokens per second (TPS) and requests per second (RPS) for language models, and benchmarks separate system-level TPS across all concurrent requests from per-user TPS. For training and batch inference jobs, it’s images or samples per second.

Throughput can be increased with techniques such as continuous batching, paged KV cache management, quantization, speculative decoding, and tensor or pipeline parallelism. However, larger batches and resource contention can increase latency or memory usage. In practice, practitioners monitor both throughput and latency to balance user experience against hardware utilization and cost.

Related Resources

Course

Profiling Performance in Python

Learn to profile Python programs with built-in and popular third-party tools, and turn performance insights into faster code.

intermediate tools