residual connection
A residual connection, or skip connection, is a shortcut in a neural network that adds the input of a block, or small group of layers, to the block’s output, so the block learns only a correction to its input instead of a whole new transformation.
Written out, the block returns x + F(x), where x is the input and F is the stack of layers that the shortcut skips. A block that changes shape adds a learned projection, a small layer that reshapes x, to the shortcut so the two terms still match. Without that projection, one block looks like this:
During backpropagation, the identity path hands the gradient back unchanged. The error signal therefore reaches the earliest layers of a deep neural network, which counters the vanishing gradient problem.
Kaiming He and colleagues introduced the idea in the 2015 ResNet paper, whose 152-layer image classifiers won the classification task at the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
Every transformer block wraps its self-attention and feed-forward sub-layers in residual connections paired with layer normalization. The original design normalized after each addition, but most modern large language models normalize inside the block instead, which leaves an unbroken additive path from the input embeddings to the output. Interpretability research calls that path the residual stream.
Related Resources
Tutorial
Python AI: How to Build a Neural Network & Make Predictions
In this step-by-step tutorial, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.
For additional information on related topics, take a look at the following resources:
- Hugging Face Transformers: Leverage Open-Source AI in Python (Tutorial)
- PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
- Building a Neural Network & Making Predictions With Python AI (Course)
- Python Deep Learning: PyTorch vs Tensorflow (Course)
- Python AI: How to Build a Neural Network & Make Predictions (Quiz)
- Hugging Face Transformers: Leverage Open-Source AI in Python (Quiz)
Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.
By Martin Breuss • Updated Nov. 4, 2026