Skip to content

residual connection

A residual connection, or skip connection, is a shortcut in a neural network that adds the input of a block, or small group of layers, to the block’s output, so the block learns only a correction to its input instead of a whole new transformation.

Written out, the block returns x + F(x), where x is the input and F is the stack of layers that the shortcut skips. A block that changes shape adds a learned projection, a small layer that reshapes x, to the shortcut so the two terms still match. Without that projection, one block looks like this:

Input x passes through two layers to produce F(x), while a shortcut carries x around them to an Add step that outputs x + F(x).
One Residual Block, x Plus a Learned Correction

During backpropagation, the identity path hands the gradient back unchanged. The error signal therefore reaches the earliest layers of a deep neural network, which counters the vanishing gradient problem.

Kaiming He and colleagues introduced the idea in the 2015 ResNet paper, whose 152-layer image classifiers won the classification task at the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

Every transformer block wraps its self-attention and feed-forward sub-layers in residual connections paired with layer normalization. The original design normalized after each addition, but most modern large language models normalize inside the block instead, which leaves an unbroken additive path from the input embeddings to the output. Interpretability research calls that path the residual stream.

Python AI: How to Build a Neural Network & Make Predictions

Tutorial

Python AI: How to Build a Neural Network & Make Predictions

In this step-by-step tutorial, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.

intermediate ai data-science machine-learning

For additional information on related topics, take a look at the following resources:

Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.


By Martin Breuss • Updated Nov. 4, 2026