token
A token is the smallest unit of data processed and generated by natural language processing (NLP) systems and language models (LLMs), typically produced by a tokenizer that segments text into words, subwords, characters, or bytes.
Tokens are mapped to integer IDs from a fixed vocabulary so that models can process sequences efficiently. Tokens are distinct from words: for English text, a token averages roughly three to four characters, though the exact ratio depends on the model’s tokenizer, so a long or unusual word can span several tokens.
Practical limits, costs, and context windows for LLMs are measured in tokens. Multimodal models tokenize images, audio, and video, so those inputs are counted and billed in tokens too.
Related Resources
Tutorial
Natural Language Processing With spaCy in Python
In this step-by-step tutorial, you'll learn how to use spaCy. This free and open-source library for natural language processing (NLP) in Python has a lot of built-in capabilities and is becoming increasingly popular for processing and analyzing data in NLP.
For additional information on related topics, take a look at the following resources:
- Build an LLM RAG Chatbot With LangChain (Tutorial)
- Python MarkItDown: Convert Documents Into LLM-Ready Markdown (Tutorial)
- Pydantic AI: Build Type-Safe LLM Agents in Python (Tutorial)
- Natural Language Processing With Python's NLTK Package (Tutorial)
- Hugging Face Transformers: Leverage Open-Source AI in Python (Tutorial)
- Vector Databases and Embeddings With ChromaDB (Course)
- First Steps With LangChain (Course)
- Build an LLM RAG Chatbot With LangChain (Quiz)
- Converting Documents to Markdown With MarkItDown (Course)
- Python MarkItDown: Convert Documents Into LLM-Ready Markdown (Quiz)
- Building Type-Safe LLM Agents With Pydantic AI (Course)
- Pydantic AI: Build Type-Safe LLM Agents in Python (Quiz)
- Natural Language Processing With Python's NLTK Package (Quiz)
- Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.
Ask Mentor AI
By Leodanis Pozo Ramos • Updated Nov. 4, 2026
