transfer learning

Transfer learning is a machine learning approach that reuses the knowledge a model acquired on one task to improve its performance on a different but related task. Rather than training from scratch, a neural network starts from weights already fitted to a large source dataset and carries those learned representations into the target task.

A 2010 survey by Pan and Yang formalized the setup as a source domain with its task and a target domain with its task, where either the domains or the tasks differ. The payoff is efficiency, since the target task then needs far less labeled data and compute than it would on its own.

Two adaptation patterns are common. Feature extraction freezes the pretrained layers and trains only a small new output layer, or head, on top of them, treating the frozen network as a fixed source of features. Fine-tuning keeps updating some or all of the original weights instead, which tends to work better when the target dataset is large enough to support it.

The two patterns are the endpoints of a single choice about how deep into the pretrained stack the updates reach. Sliding that boundary shows how much of the network stays frozen at each setting:

Because the later stages hold most of the weights, unfreezing even one of them puts the majority of the network back into training.

Modern large language models are the pattern at its largest scale. One expensive pretraining run over broad text corpora produces a base model that many downstream tasks then adapt cheaply, often with parameter-efficient methods such as low-rank adaptation.

Two failure modes limit the technique. Negative transfer occurs when the source task is unrelated enough that borrowing from it hurts the target task, and catastrophic forgetting occurs when adaptation overwrites general capabilities the model started with.

Related Resources

Tutorial

Python AI: How to Build a Neural Network & Make Predictions

In this step-by-step tutorial, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.

intermediate ai data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Practical Text Classification With Python and Keras (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Learn Text Classification With Python and Keras (Course)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)