token

A token is a minimal unit of text used by natural language processing (NLP) systems and language models (LLMs), typically produced by a tokenizer that segments text into words, subwords, characters, or bytes.

Tokens are mapped to integer IDs from a fixed vocabulary so that models can process sequences efficiently. Tokens are distinct from words. Practical limits, costs, and context windows for LLMs are measured in tokens.

Related Resources

Course

Building Type-Safe LLM Agents With Pydantic AI

Build type-safe LLM agents in Python with Pydantic AI using structured outputs, function calling, and dependency injection.

intermediate ai