overfitting

Overfitting is a failure mode in which a machine learning model learns the noise and quirks of its training data so closely that it performs poorly on new, unseen data. An overfit model has memorized its examples instead of learning the general pattern behind them.

The usual symptom is a gap between two measurements taken during training. The loss on the training set keeps falling, while the loss on a held-out validation set levels off and then starts to rise.

Models with many parameters relative to the size of their training data are especially prone to overfitting, because they have enough capacity to fit every example exactly.

The slider below gives a polynomial model more parameters. Its training error keeps falling, while its error on held-out validation points bottoms out and then climbs:

Common countermeasures include:

• Collecting more varied training data, so the model has fewer coincidences to latch onto
• Regularization, which penalizes large weights or, in the case of dropout, randomly switches off some of a neural network’s neurons during training
• Early stopping, which halts training once the validation loss stops improving
• Reducing model capacity, for example by capping the depth of a decision tree

In large language model work, overfitting shows up most often during fine-tuning on a small dataset. Running too many epochs, or full passes over the training data, can make a model reproduce its training examples nearly verbatim and handle prompts outside that data worse.

The same failure appears at the evaluation level. A model tuned too closely to a public benchmark, or trained on data that leaked its test questions, can post high scores that don’t hold up on fresh tasks.

Related Resources

Tutorial

Split Your Dataset With scikit-learn's train_test_split()

In this tutorial, you'll learn why splitting your dataset in supervised machine learning is important and how to do it with train_test_split() from scikit-learn.

intermediate data-science machine-learning numpy

For additional information on related topics, take a look at the following resources:

• Linear Regression in Python (Tutorial)
• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• Practical Text Classification With Python and Keras (Tutorial)
• Logistic Regression in Python (Tutorial)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)
• Starting With Linear Regression in Python (Course)
• Linear Regression in Python (Quiz)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)
• Learn Text Classification With Python and Keras (Course)