Residual Connections and ResNets

Residual connections (also called skip connections) are architectural shortcuts in neural networks that add a layer's input directly to its output, allowing gradients to flow unimpeded through hundreds or thousands of layers during backpropagation. Introduced in ResNets (Residual Networks) by He et al. in 2015, they solved the degradation problem — the counterintuitive observation that adding more layers to a plain network made training worse, not just harder — and enabled networks of unprecedented depth (152+ layers) to be trained reliably. The remarkable fact: a residual connection adds essentially zero parameters yet unlocks depth scaling that was previously impossible, making it one of the highest-leverage architectural decisions in deep learning history.

The core insight

Instead of asking a stack of layers to learn a mapping $H(x)$ directly, residual connections reframe the problem: let the layers learn the residual $F(x) = H(x) - x$ , then reconstruct the target as $H(x) = F(x) + x$ . If the optimal transformation is close to the identity (which it often is in deep networks), learning a small residual perturbation near zero is far easier than learning the full mapping from scratch. The "aha moment" is this: the network now has a free gradient highway — the identity shortcut — so even if the learned layers $F(x)$ are poorly initialized or temporarily unhelpful, the signal still flows backward without vanishing.

Common trapCommon Misconception

Practitioners commonly believe that residual connections prevent vanishing gradients by keeping gradient magnitudes large throughout training — but this is an oversimplification that leads to wrong intuitions. The real mechanism is subtler: residual connections create additive gradient paths, so the gradient of the loss with respect to any early layer is a sum of contributions from all downstream residual paths, not just a single multiplicative chain. This means even if one path's gradient shrinks to near zero, others still carry signal. The practical implication: you can still have vanishing gradients in the $F(x)$ branches themselves — residual connections don't eliminate that problem inside each block, which is why Batch Normalization and careful weight initialization remain essential even in ResNets.

How It Works

The Residual Block

The fundamental building block of a ResNet defines its output as:

\quad \mathbf{y} = F(\mathbf{x},\, \{W_i\}) + \mathbf{x}

Explanation: The output $\mathbf{y}$ is the sum of the learned residual function $F$ (parameterized by weights $\{W_i\}$ ) and the original input $\mathbf{x}$ passed through unchanged via the shortcut. Here $\mathbf{x} \in \mathbb{R}^d$ is the input to the block, $F(\mathbf{x}, \{W_i\})$ is the transformation computed by the stacked layers (typically Conv → BN → ReLU → Conv → BN), and addition is performed element-wise. When dimensions change (e.g., after strided convolutions), a linear projection $W_s \mathbf{x}$ replaces the identity:

\quad \mathbf{y} = F(\mathbf{x},\, \{W_i\}) + W_s \mathbf{x}

Explanation: The projection matrix $W_s$ (a $1\times1$ convolution in practice) matches the spatial and channel dimensions of the shortcut to those of the residual branch so the addition is well-defined.

Gradient Flow Analysis

For a network of $L$ residual blocks where each block satisfies $\mathbf{x}_{l+1} = \mathbf{x}_l + F(\mathbf{x}_l)$ , the feature at any layer $L$ can be written as a direct sum of all preceding residuals:

\quad \mathbf{x}_L = \mathbf{x}_0 + \sum_{l=0}^{L-1} F(\mathbf{x}_l,\, W_l)

Explanation: This telescoping sum shows that the output at depth $L$ is the input plus an accumulated sum of learned residuals — every block contributes additively rather than multiplicatively. Taking the gradient of the loss $\mathcal{L}$ with respect to an early activation $\mathbf{x}_0$ via the chain rule:

\quad \frac{\partial \mathcal{L}}{\partial \mathbf{x}_0} = \frac{\partial \mathcal{L}}{\partial \mathbf{x}_L} \cdot \left(1 + \frac{\partial}{\partial \mathbf{x}_0}\sum_{l=0}^{L-1} F(\mathbf{x}_l, W_l)\right)

Explanation: The gradient always contains the term $\frac{\partial \mathcal{L}}{\partial \mathbf{x}_L}$ multiplied by $1$ — meaning the shortcut path guarantees at least this much gradient reaches $\mathbf{x}_0$ regardless of what the residual branches do.

PyTorch Implementation

A standard BasicBlock (used in ResNet-18/34) looks like: ```python import torch import torch.nn as nn class BasicBlock(nn.Module): expansion = 1 def __init__(self, in_channels, out_channels, stride=1): super().__init__() self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False) self.bn1 = nn.BatchNorm2d(out_channels) self.relu = nn.ReLU(inplace=True) self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False) self.bn2 = nn.BatchNorm2d(out_channels)

# Projection shortcut when dims change self.shortcut = nn.Sequential() if stride != 1 or in_channels != out_channels: self.shortcut = nn.Sequential( nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False), nn.BatchNorm2d(out_channels) ) def forward(self, x): out = self.relu(self.bn1(self.conv1(x))) out = self.bn2(self.conv2(out)) out += self.shortcut(x) # <-- The residual connection out = self.relu(out) return out ``` For deeper variants (ResNet-50/101/152), He et al. use a bottleneck block that first reduces channel dimensions with a $1\times1$ conv, applies a $3\times3$ conv in the reduced space, then expands back — cutting FLOPs by ~3× while preserving representational power: ```python class Bottleneck(nn.Module): expansion = 4 # output channels = in_channels 4 def __init__(self, in_channels, bottleneck_channels, stride=1): super().__init__() out_channels = bottleneck_channels self.expansion self.layers = nn.Sequential( nn.Conv2d(in_channels, bottleneck_channels, 1, bias=False), nn.BatchNorm2d(bottleneck_channels), nn.ReLU(inplace=True), nn.Conv2d(bottleneck_channels, bottleneck_channels, 3, stride=stride, padding=1, bias=False), nn.BatchNorm2d(bottleneck_channels), nn.ReLU(inplace=True), nn.Conv2d(bottleneck_channels, out_channels, 1, bias=False), nn.BatchNorm2d(out_channels), ) self.shortcut = nn.Sequential() if stride != 1 or in_channels != out_channels: self.shortcut = nn.Sequential( nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False), nn.BatchNorm2d(out_channels) ) def forward(self, x): return nn.functional.relu(self.layers(x) + self.shortcut(x)) ```

researcher · Kaiming He | b. 1990 | Inventor of ResNets and He Initialization

Kaiming He completed his PhD at the Chinese University of Hong Kong and joined Microsoft Research Asia, where he authored both the ResNet paper (He et al., CVPR 2016, ~200,000 citations) and the He weight initialization scheme that made training deep ReLU networks tractable. His ResNet-152 won first place in ILSVRC 2015 classification, detection, and localization — a sweep unprecedented in the competition's history. Surprisingly, He has stated in interviews that the core skip-connection idea came from observing that even randomly wired shortcut paths improved optimization, long before the theoretical gradient-flow explanation was formalized.

Figure

Residual Block Architecture and Gradient Flow

Figure

ResNet Family Depth vs. Top-1 Error on ImageNet

Intuition

The Mental Model

Imagine you're hiking up a mountain (optimizing a loss landscape) and you can either take a single steep direct path or a network of switchbacks with convenient rope-handholds at every turn. Plain deep networks are the steep path: if you stumble at any point, you lose all your momentum. Residual connections are the handholds — they don't change where you're trying to go, but they guarantee you always have something stable to grab. A more computational analogy: think of an ensemble of shallow networks sharing parameters. The residual structure effectively lets the network behave like many different-depth models trained simultaneously. At test time, it averages over all these implicit depths — shallow paths dominate early in training when residuals are near zero, and the network gradually specializes deeper paths as capacity is needed. One more way to see it: if the ideal transformation between two layers really is close to the identity (e.g., features barely need changing at that depth), the network just has to push $F(x) \to 0$ . Zeroing out weights is easy — it's the default for near-zero initialization. Without skip connections, learning the identity mapping through two conv layers and batch norms is a non-trivial, fragile optimization problem.

Historical Development

Origins · 1 minhow this was discovered

History

1989: Sepp Hochreiter observes the vanishing gradient problem in his diploma thesis, identifying that deep networks struggle to propagate error signals through many layers — the fundamental problem ResNets would later solve.

1997: Hochreiter and Schmidhuber publish the LSTM, which uses additive updates (cell state

c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t

) rather than multiplicative chains — the first successful architecture to exploit additive gradient paths, a direct conceptual predecessor to skip connections.

2000s: Highway Networks (Srivastava et al., 2015) introduce gated skip connections inspired by LSTM gates, enabling 100-layer networks — but the gating mechanism adds parameters and complexity, and the paper appears concurrently with the ResNet submission.

2015 (Dec): He et al. publish "Deep Residual Learning for Image Recognition" (arXiv:1512.03385), sweeping ILSVRC 2015 with a 152-layer ResNet achieving 3.57% top-5 error — surpassing human-level performance (~5%) on ImageNet for the first time.

2016: He et al. publish "Identity Mappings in Deep Residual Networks," showing that placing BN and ReLU before the weight layers (pre-activation ResNet) produces cleaner gradient paths and slightly better performance — a subtle but important refinement.

2017: Huang et al. introduce DenseNet, taking the skip connection idea to its logical extreme by connecting every layer to every subsequent layer — effectively adding

O(L^2)

shortcut paths rather than

O(L)

2019: Brock et al. develop Big Transfer (BiT) and demonstrate that vanilla ResNets, when trained with careful hyperparameter tuning at scale, match or exceed far more complex architectures on transfer learning benchmarks.

2020: Dosovitskiy et al. publish the Vision Transformer (ViT), which retains residual connections in every transformer block — demonstrating that the residual principle transcends CNNs and is a near-universal component of deep learning architectures.

2021–present: Residual connections appear in virtually every competitive architecture: GPT-3/4, DALL·E, Stable Diffusion's U-Net, AlphaFold 2's Evoformer — confirming them as a foundational primitive rather than a CNN-specific trick.

Why It Matters

Real-World Impact

ResNet-50 (Backbone universality)ResNet-50 became the single most widely deployed computer vision backbone in industry — used as a feature extractor in object detection (Faster R-CNN), semantic segmentation (DeepLab), and face recognition systems, with pretrained weights fine-tuned for thousands of downstream tasks.

GPT-3 / GPT-4 (Transformer residuals)Every transformer layer in GPT-class models uses two residual connections per block — one around the multi-head self-attention sublayer and one around the feed-forward sublayer. Without these, training a 96-layer, 175B-parameter model (GPT-3) would be impossible due to gradient vanishing.

AlphaFold 2 (Protein structure prediction)The Evoformer module in AlphaFold 2 stacks 48 residual transformer blocks to process amino acid pair representations. The residual structure allows AlphaFold to be trained end-to-end on sequence-to-structure tasks, contributing to predictions that matched experimental crystallography within ~1 Ångström.

Stable Diffusion / DALL·E 2 (U-Net with skip connections)The denoising U-Net at the core of latent diffusion models combines residual blocks within each resolution level with long-range skip connections across the encoder-decoder — a doubly hierarchical application of the residual principle enabling high-resolution image synthesis.

YOLO v8 / RT-DETR (Real-time object detection)Modern one-stage detectors use residual bottleneck stacks in their backbone (C2f modules in YOLOv8 are direct descendants of ResNet bottlenecks) to achieve 50+ FPS inference while maintaining high mAP on COCO.

EfficientNet / RegNet (Neural architecture search)Both NAS-derived families retain residual connections as a fixed structural prior — they search over block widths, depths, and expansion ratios but never ablate skip connections, reflecting an empirical consensus that they are non-negotiable for deep networks.