ResNet: Deep Residual Learning

The Degradation Problem

In a vanilla deep network (no shortcuts), stacking more layers should never hurt: at worst, the extra layers could learn the identity function and match the shallower network's performance. In practice, optimizers fail to learn this — training loss increases with depth. This is not overfitting; it is an optimization problem.

The Residual Formulation

Let $\mathbf{x}$ be the input to a residual block and $\mathcal{F}(\mathbf{x}, \{W_i\})$ be the mapping learned by the stacked layers. The block's output is:

Explanation: The block's output is the learned residual correction plus the original input, so the layers only need to model what changes rather than the full transformation. Here $\mathbf{x}$ is the skip-connected input, $\mathcal{F}$ represents two or three convolutional layers with batch normalization and ReLU, and $\mathbf{y}$ is passed through a final ReLU before becoming the next block's input.

Gradient Flow Through Shortcuts

Consider the loss $\mathcal{L}$ and a deep network with $L$ layers. With shortcut connections the activations at any shallow layer $l$ can be expressed in terms of any deeper layer $L$:

Explanation: The deep activation equals the shallow activation plus a sum of residuals, meaning information from any depth is directly accessible at any other depth. The gradient at layer $l$ follows directly:

Explanation: The gradient at any shallow layer receives a direct copy of the deep gradient (the "1" term), preventing it from vanishing even through hundreds of layers. The term $\frac{\partial \mathcal{L}}{\partial \mathbf{x}_L}$ is the gradient from the loss, and the "1" in the parenthesis is the contribution of the identity shortcut — it guarantees the gradient can never fully vanish as long as the loss gradient itself is nonzero.

Dimension Matching with Projection Shortcuts

When the input and output dimensions differ (e.g., when stride=2 halves spatial resolution or channel depth doubles), a projection shortcut is used:

Explanation: The learnable projection matrix $W_s$ (implemented as a 1×1 convolution) reshapes the shortcut signal to match the residual branch's output dimensions before addition. $W_s$ is a $1\times1$ convolution with stride matching the residual branch, with no activation applied to the shortcut path itself.

BasicBlock vs. Bottleneck

For ResNet-18/34, each block uses two $3\times3$ convolutions (BasicBlock). For ResNet-50/101/152, a Bottleneck design reduces computation:

Explanation: The bottleneck block uses 1×1 convolutions to compress channels by 4× before and expand after the expensive 3×3 convolution, cutting FLOPs by roughly 8× compared to two 3×3s at full width. Here $C$ is the full channel width, the factor $C/4$ is the bottleneck dimension, and the three stages are: compress → spatial-mix → expand.

PyTorch Implementation

```python import torch import torch.nn as nn class BasicBlock(nn.Module): expansion = 1 def __init__(self, in_channels: int, out_channels: int, stride: int = 1): super().__init__()

# Main residual branch self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride=stride, padding=1, bias=False) self.bn1 = nn.BatchNorm2d(out_channels) self.relu = nn.ReLU(inplace=True) self.conv2 = nn.Conv2d(out_channels, out_channels, 3, stride=1, padding=1, bias=False) self.bn2 = nn.BatchNorm2d(out_channels)

# Projection shortcut (only when dimensions change) self.shortcut = nn.Sequential() if stride != 1 or in_channels != out_channels: self.shortcut = nn.Sequential( nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False), nn.BatchNorm2d(out_channels) ) def forward(self, x: torch.Tensor) -> torch.Tensor: identity = self.shortcut(x) # ← shortcut path out = self.relu(self.bn1(self.conv1(x))) out = self.bn2(self.conv2(out)) out = self.relu(out + identity) # ← residual addition return out

class Bottleneck(nn.Module): expansion = 4 # output channels = base_width × 4 def __init__(self, in_channels: int, base_width: int, stride: int = 1): super().__init__() out_channels = base_width * self.expansion self.conv1 = nn.Conv2d(in_channels, base_width, 1, bias=False) self.bn1 = nn.BatchNorm2d(base_width) self.conv2 = nn.Conv2d(base_width, base_width, 3, stride=stride, padding=1, bias=False) self.bn2 = nn.BatchNorm2d(base_width) self.conv3 = nn.Conv2d(base_width, out_channels, 1, bias=False) self.bn3 = nn.BatchNorm2d(out_channels) self.relu = nn.ReLU(inplace=True) self.shortcut = nn.Sequential() if stride != 1 or in_channels != out_channels: self.shortcut = nn.Sequential( nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False), nn.BatchNorm2d(out_channels) ) def forward(self, x: torch.Tensor) -> torch.Tensor: identity = self.shortcut(x) out = self.relu(self.bn1(self.conv1(x))) # compress out = self.relu(self.bn2(self.conv2(out))) # spatial mix out = self.bn3(self.conv3(out)) # expand return self.relu(out + identity) ```