inference

Inference is the stage in which a trained model takes new inputs and produces outputs, operating under real-world constraints of latency, throughput, and cost.

It encompasses an end-to-end flow:

• Request handling
• Input or pre-processing
• Model execution
• Output decoding or post-processing

Unlike training, which optimizes model parameters, inference applies those fixed parameters to unseen data.

In the case of large language models (LLMs), inference typically means autoregressive generation, proceeding token by token, often starting with a prompt or prefill phase, followed by a decode loop.

Reasoning models stretch this phase further by spending a variable compute budget at inference time, generating internal reasoning tokens before the final answer. That budget, alongside the trained parameters, becomes a lever on output quality.

Production systems further optimize inference through continuous batching, KV cache and prefix caching, quantization, speculative decoding, and hardware or compiler tuning.

Related Resources

Course

Building a Neural Network & Making Predictions With Python AI

In this step-by-step course, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.

intermediate data-science machine-learning