LLM observability

LLM observability is the practice of collecting and correlating telemetry about large language model applications across prompts, intermediate steps, tool calls, retrieval operations, and outputs. This data is then used to debug behavior and monitor quality, reliability, safety, and cost.

An LLM observability stack typically combines logs, metrics, and end-to-end traces built from hierarchical spans that link each operation involved in handling a request, from model calls and retrieval steps to the tool calls and reasoning steps an agent takes. It captures behavior details, such as token counts, latencies, errors, costs, and evaluation signals. Tooling increasingly emits these details using the OpenTelemetry GenAI semantic conventions, a vendor-neutral schema of gen_ai.* span, metric, and event definitions that keeps traces portable across backends. It also provides controls and retention policies so that teams can gain deep visibility into LLM behavior while respecting security and governance requirements.