vanishing gradient problem

The vanishing gradient problem is a training failure in deep or recurrent neural networks where the error signal, or gradient, shrinks toward zero as it travels back through the layers, so the earliest layers barely learn. The network still trains, but only its last few layers adjust in any useful way.

The cause lies in training by backpropagation. To find how much each weight should change, backpropagation multiplies one local derivative per layer, starting at the loss. When most of those factors are smaller than one, their product shrinks exponentially with depth. The sigmoid activation function is a common source, since its derivative never exceeds 0.25. Gradient descent takes tiny steps for the early layers, and training stalls.

Recurrent neural networks hit the problem across time rather than depth. Unrolled over a long sequence, an RNN multiplies the same factors once per step, which makes long-range dependencies hard to learn. Sepp Hochreiter analyzed the effect in his 1991 diploma thesis, and Bengio, Simard, and Frasconi described it in a 1994 paper. When the factors are larger than one instead, the same mechanism produces the opposite failure, exploding gradients.

Several techniques keep gradients flowing through deep models:

• Gated architectures: Long short-term memory (LSTM) networks, introduced in 1997, carry information through an additive cell state.
• Non-saturating activations: ReLU and its variants pass a gradient of exactly one for positive inputs.
• Residual connections: Skip paths, popularized by ResNet in 2015, give the gradient a direct route around each layer.
• Normalization and initialization: Batch or layer normalization and careful weight initialization keep each layer’s outputs in a well-scaled range.

In the chart below, each bar shows how much of the output-layer gradient reaches that layer, so changing the depth or switching from sigmoid to ReLU or a residual connection shows the signal collapsing or holding.

The transformer behind modern large language models combines residual connections with layer normalization, which is a large part of why networks with dozens of layers train reliably.

Related Resources

Tutorial

Python AI: How to Build a Neural Network & Make Predictions

In this step-by-step tutorial, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.

intermediate ai data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Stochastic Gradient Descent Algorithm With Python and NumPy (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Practical Text Classification With Python and Keras (Tutorial)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Learn Text Classification With Python and Keras (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)