latency

Latency is the time between sending a request and seeing a response. In distributed systems it includes network delay, queuing, and processing time, and is usually reported as percentiles to capture tail behavior and jitter.

Latency is different from throughput, which measures how much work a system does per unit time. It’s affected by batching, concurrency, caching, cold starts, and resource contention.

For AI and LLM applications, important latency aspects include time to first token (TTFT), inter-token latency or time per output token, and end-to-end request latency. Delays from tool calls and retrieval add to it, as does the extra overhead when models or containers must be loaded rather than served from a warm cache.

Related Resources

Course

Speed Up Python With Concurrency

Learn what concurrency means in Python and why you might want to use it. You'll see a simple, non-concurrent approach and then look into why you'd want threading, asyncio, or multiprocessing.

advanced best-practices

For additional information on related topics, take a look at the following resources:

• Build an LLM RAG Chatbot With LangChain (Tutorial)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)