latency

Latency is the time between sending a request and seeing a response. In distributed systems it includes network delay, queuing, and processing time, and is usually reported as percentiles to capture tail behavior and jitter.

Latency is different from throughput, which measures how much work a system does per unit time. It’s affected by batching, concurrency, caching, cold starts, and resource contention.

For AI and LLM applications, important latency aspects include time to first token (TTFT), inter-token latency or time per output token, and end-to-end request latency. Delays from tool calls and retrieval add to it, as does the extra overhead when models or containers must be loaded rather than served from a warm cache.

Related Resources

Tutorial

Build an LLM RAG Chatbot With LangChain

Large language models (LLMs) have taken the world by storm, demonstrating unprecedented capabilities in natural language tasks. In this step-by-step tutorial, you'll leverage LLMs to build your own retrieval-augmented generation (RAG) chatbot using synthetic data with LangChain and Neo4j.

intermediate ai databases data-science

For additional information on related topics, take a look at the following resources:

• Profiling in Python: How to Find Performance Bottlenecks (Tutorial)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Tutorial)
• Speed Up Python With Concurrency (Course)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)
• Profiling Performance in Python (Course)
• Profiling in Python: How to Find Performance Bottlenecks (Quiz)
• Building Type-Safe LLM Agents With Pydantic AI (Course)
• Pydantic AI: Build Type-Safe LLM Agents in Python (Quiz)