tokenization

Tokenization is the process of converting raw text into a sequence of discrete tokens that models can handle, typically by segmenting text into words, subwords, characters, or bytes and mapping them to vocabulary IDs.

Modern natural language processing (NLP) systems favor subword tokenization to balance vocabulary size and coverage, using schemes such as Byte Pair Encoding, WordPiece, and Unigram, often trained directly from raw text with tools like SentencePiece. These methods help handle rare and out-of-vocabulary words, support many languages and scripts, and improve robustness by representing text as variable-length units.

Practical pipelines also define special tokens for boundaries or control signals and include a reversible detokenization step, allowing model outputs to be turned back into readable text. Multimodal models apply the same idea beyond text: images, audio, and video are also converted into tokens, which is why context limits and API costs for non-text inputs are still counted in tokens.

Related Resources

Tutorial

Natural Language Processing With Python's NLTK Package

In this beginner-friendly tutorial, you'll take your first steps with Natural Language Processing (NLP) and Python's Natural Language Toolkit (NLTK). You'll learn how to process unstructured data in order to be able to analyze it and draw conclusions from it.

basics data-science

For additional information on related topics, take a look at the following resources:

• Natural Language Processing With Python's NLTK Package (Quiz)