pretraining
Pretraining is the first and most compute-intensive phase of training a large language model, in which the model learns general patterns of language and code from a vast unlabeled dataset before any task-specific adaptation.
The objective is self-supervised, meaning the data supplies its own labels. Decoder-only transformer models such as GPT learn to predict the next token in a sequence, while encoder models such as BERT learn to recover masked tokens.
The stepper below turns a single line of code into many training examples under each objective, with every label taken from the text itself:
The corpus usually mixes filtered web crawls, books, and source code. Meta reported pretraining Llama 3 on more than 15 trillion tokens. Data and model size grow together, and DeepMind’s 2022 Chinchilla study found that compute-optimal runs scale parameters and training tokens in equal proportion.
The result is a base model that continues text rather than following instructions. Post-training stages such as fine-tuning and reinforcement learning from human feedback then turn it into an assistant. The two phases together are transfer learning at scale. Gaps or biases in the corpus carry into every later stage.
Related Resources
Tutorial
Hugging Face Transformers: Leverage Open-Source AI in Python
Get hands-on with Hugging Face Transformers in Python. Download, run, and inspect pretrained open-source AI models, and speed up inference on GPUs.
For additional information on related topics, take a look at the following resources:
- Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
- PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
- How to Use Ollama to Run Large Language Models Locally (Tutorial)
- ChromaDB: Embeddings and Vector Databases in Python (Tutorial)
- Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
- Building a Neural Network & Making Predictions With Python AI (Course)
- Python AI: How to Build a Neural Network & Make Predictions (Quiz)
- Python Deep Learning: PyTorch vs Tensorflow (Course)
- How to Use Ollama to Run Large Language Models Locally (Quiz)
- Vector Databases and Embeddings With ChromaDB (Course)
- Embeddings and Vector Databases With ChromaDB (Quiz)
Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.
By Martin Breuss • Updated Nov. 3, 2026