Skip to content

pretraining

Pretraining is the first and most compute-intensive phase of training a large language model, in which the model learns general patterns of language and code from a vast unlabeled dataset before any task-specific adaptation.

The objective is self-supervised, meaning the data supplies its own labels. Decoder-only transformer models such as GPT learn to predict the next token in a sequence, while encoder models such as BERT learn to recover masked tokens.

The stepper below turns a single line of code into many training examples under each objective, with every label taken from the text itself:

Interactive diagram — enable JavaScript to view.

The corpus usually mixes filtered web crawls, books, and source code. Meta reported pretraining Llama 3 on more than 15 trillion tokens. Data and model size grow together, and DeepMind’s 2022 Chinchilla study found that compute-optimal runs scale parameters and training tokens in equal proportion.

The result is a base model that continues text rather than following instructions. Post-training stages such as fine-tuning and reinforcement learning from human feedback then turn it into an assistant. The two phases together are transfer learning at scale. Gaps or biases in the corpus carry into every later stage.

Hugging Face Transformers: Leverage Open-Source AI in Python

Tutorial

Hugging Face Transformers: Leverage Open-Source AI in Python

Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.

intermediate ai

For additional information on related topics, take a look at the following resources:

Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.


By Martin Breuss • Updated Nov. 3, 2026