backpropagation

Backpropagation is the algorithm that works out how much each weight in a neural network contributed to the error in a prediction. These per-weight contributions, called gradients, are what gradient descent uses to update the model during training.

Each training step runs two passes. The forward pass sends an input through the layers to a prediction and keeps the intermediate values computed along the way. The loss function then scores that prediction against the target.

The backward pass starts at the loss and applies the chain rule of calculus, which says that the derivative of a chain of functions is the product of each step’s derivative, one layer at a time, from the output back toward the input. Each layer multiplies the gradient arriving from above by its own local derivative. That product gives the gradient for the layer’s parameters and becomes the signal handed down to the layer below.

Step through a tiny network below to watch the forward pass store each value, then the backward pass multiply its way from the loss back to the weights:

Because each intermediate gradient is computed once and reused, a backward pass costs only a small constant multiple of a forward pass, even in models with billions of parameters. The price is memory, since the stored intermediate values, known as activations, must stay around until the backward pass consumes them. Gradient checkpointing trades some of that memory for extra compute by recomputing activations instead of storing them.

Backpropagation is a special case of reverse-mode automatic differentiation. Seppo Linnainmaa described reverse-mode differentiation in his 1970 master’s thesis, and Paul Werbos applied it to neural networks in 1982. A 1986 paper in Nature by Rumelhart, Hinton, and Williams showed that it lets hidden layers learn useful internal representations.

Frameworks such as PyTorch automate the whole procedure. They record operations into a computation graph during the forward pass, and a single .backward() call on the loss fills in every parameter’s .grad attribute. The repeated multiplication of local derivatives is also the root of the vanishing gradient problem in deep and recurrent networks.

Related Resources

Tutorial

Python AI: How to Build a Neural Network & Make Predictions

In this step-by-step tutorial, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.

intermediate ai data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Stochastic Gradient Descent Algorithm With Python and NumPy (Tutorial)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Practical Text Classification With Python and Keras (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)
• Learn Text Classification With Python and Keras (Course)