token
A token is a minimal unit of text used by natural language processing (NLP) systems and language models (LLMs), typically produced by a tokenizer that segments text into words, subwords, characters, or bytes.
Tokens are mapped to integer IDs from a fixed vocabulary so that models can process sequences efficiently. Tokens are distinct from words. Practical limits, costs, and context windows for LLMs are measured in tokens.
Related Resources
Tutorial
Natural Language Processing With Python's NLTK Package
In this beginner-friendly tutorial, you'll take your first steps with Natural Language Processing (NLP) and Python's Natural Language Toolkit (NLTK). You'll learn how to process unstructured data in order to be able to analyze it and draw conclusions from it.
For additional information on related topics, take a look at the following resources:
By Leodanis Pozo Ramos • Updated July 2, 2026
