Residual Connections and ResNet

A plain deep network stacks layers so that the output of layer $\ell$ feeds directly into layer $\ell+1$ as $x_{\ell+1} = F(x_\ell)$, where $F$ bundles a convolution, normalization, and nonlinearity. As depth grows, two related pathologies appear. First, vanishing/exploding gradients: the chain rule multiplies many Jacobians together, and if their norms are consistently below or above 1, gradients shrink or blow up exponentially with depth. Batch normalization and careful initialization mitigate this but do not eliminate it for very deep stacks. Second, and more surprising, is the degradation problem: even when gradients are numerically healthy, plain networks beyond roughly 20–30 layers become harder to optimize in a way that is not explained by overfitting — the training error itself rises with added depth. This is puzzling because a deeper network can always simulate a shallower one: take the shallow network's layers, then append extra layers that compute the identity function. The composed function loses nothing. So the optimal solution for the deep network is at least as good as the shallow one. The empirical fact that solvers fail to find even this trivial identity solution is the real diagnosis: it is hard for a stack of nonlinear layers to learn an identity mapping, because that requires precisely tuned weights (e.g., driving convolution kernels toward a Dirac-delta-like configuration) rather than the generic small-random-weight regime that gradient descent naturally explores. A residual block reparametrizes the problem. Instead of asking a block to learn the mapping $H(x)$ directly, define

where $F$ is the residual function — typically two or three weight layers — and the raw input $x$ is added back via an identity shortcut (also called a skip connection). If the optimal underlying mapping really is close to the identity, the block only needs to drive $F(x) \to 0$, which is easy: push weights toward zero, a direction gradient descent is very good at finding. The reformulation does not change the hypothesis space — any $H$ expressible by a plain block is still expressible as $F(x) + x$ with $F(x) = H(x) - x$ — but it changes the optimization landscape dramatically, biasing the default (near-zero-weight) behavior toward identity rather than toward an arbitrary, hard-to-discover function.

The basic residual block. For feature maps with matching shape, the canonical two-layer block is:

Two 3×3 convolutions, each followed by batch normalization, with a ReLU after the first conv and a second ReLU applied after the addition. The shortcut adds the raw (or projected) input to the output of the second BN, before the final activation.

Handling shape mismatches. A skip connection requires $F(x)$ and $x$ to have the same shape. Two situations break this: 1. Channel count changes (e.g., 64 → 128 channels between stages). 2. Spatial downsampling (e.g., stride-2 convolution halves height/width). ResNet handles both with a projection shortcut: a 1×1 convolution with the matching stride and output channels, applied to $x$ before addition:

When shapes already match, $W_s$ is simply the identity (no extra parameters) — this is the default, cheaper option, reserved for the first block of each new stage where dimensions change.

The bottleneck block. For deeper variants (ResNet-50/101/152), a three-layer bottleneck design keeps computation tractable: a 1×1 convolution reduces channels (e.g., 256 → 64), a 3×3 convolution operates on the reduced representation, and a final 1×1 convolution restores the channel count (64 → 256). The identity shortcut connects across all three layers. This design has roughly the same time complexity as the two-layer block on wide feature maps but far fewer parameters, because the expensive 3×3 convolution operates on a narrow (bottlenecked) representation.

Stacking into stages. A full ResNet is organized into stages, each stage repeating a residual block a fixed number of times at a constant resolution and channel width; resolution is halved and channels doubled between stages (typically via a strided 1×1 projection in the first block of the new stage), mirroring the pattern from LeNet and the CNN Blueprint of alternating feature extraction with downsampling, but replacing each "layer" with a residual block. ```

# Residual block (basic, 2-layer variant) function ResidualBlock(x, W1, W2, stride, W_shortcut=None):

# Main path h = Conv3x3(x, W1, stride=stride) h = BatchNorm(h) h = ReLU(h) h = Conv3x3(h, W2, stride=1) h = BatchNorm(h)

# Shortcut path if W_shortcut is not None: # shape changes: project shortcut = Conv1x1(x, W_shortcut, stride=stride) shortcut = BatchNorm(shortcut) else: # shapes match: identity shortcut = x out = h + shortcut # element-wise add out = ReLU(out) return out

# Bottleneck block (3-layer variant, used in ResNet-50/101/152) function BottleneckBlock(x, W1, W2, W3, stride, W_shortcut=None): h = Conv1x1(x, W1, stride=1) # channel reduction, e.g. 256 -> 64 h = BatchNorm(h); h = ReLU(h) h = Conv3x3(h, W2, stride=stride) # spatial work on reduced channels h = BatchNorm(h); h = ReLU(h) h = Conv1x1(h, W3, stride=1) # channel expansion, e.g. 64 -> 256 h = BatchNorm(h) if W_shortcut is not None: shortcut = Conv1x1(x, W_shortcut, stride=stride) shortcut = BatchNorm(shortcut) else: shortcut = x out = ReLU(h + shortcut) return out

# Full network: stem + stages of blocks + head function ResNet(image, params): x = Conv7x7(image, stride=2); x = BatchNorm(x); x = ReLU(x) x = MaxPool3x3(x, stride=2) for stage in [1, 2, 3, 4]: num_blocks = params.blocks_per_stage[stage] for i in 1..num_blocks: stride = 2 if (i == 1 and stage > 1) else 1 needs_projection = (i == 1) # first block of a stage x = BottleneckBlock(x, ..., stride=stride, W_shortcut = params.W_s[stage] if needs_projection else None) x = GlobalAveragePool(x) logits = FullyConnected(x, params.W_fc) return Softmax(logits) ```

Time and space complexity. A residual (or bottleneck) block adds only an element-wise addition and, for projection shortcuts, one extra 1×1 convolution to whatever a plain stack of the same layers would already cost. For an input feature map with $C$ channels, spatial size $H \times W$, and a $k\times k$ convolution producing $C'$ output channels, one convolutional layer costs $O(H W C C' k^2)$ multiply-adds. A basic block with two 3×3 layers costs $O(HWC^2 \cdot 9)$ (assuming $C=C'$); a bottleneck block with reduction ratio 4 costs roughly $O(HWC^2/4 \cdot (1 + 9/16 + 1)) \approx O(HWC^2)$ — comparable to a single 3×3 layer at full width, which is precisely why bottlenecks let ResNet-50/101/152 go deep without exploding cost. The identity shortcut itself is $O(HWC)$, negligible next to convolution cost; the projection shortcut adds $O(HWC C')$, the cost of one 1×1 convolution, used only once per stage. Space is dominated by storing activations for backpropagation, $O(HWC)$ per block, unchanged in order from the plain-network case — residual connections are essentially free in asymptotic terms while changing optimization behavior completely.

Why the shortcut fixes gradient flow (the core argument). Consider a chain of $L$ residual blocks:

This follows by induction: $x_{i+1} = x_i + F_i(x_i)$, so unrolling from $x_0$ telescopes into $x_0$ plus a sum of residual contributions — no products, only sums. Differentiate the loss $\mathcal{L}$ with respect to an early activation $x_0$ using the chain rule:

The crucial structural fact is the additive 1: the gradient with respect to $x_0$ is the downstream gradient $\partial \mathcal{L}/\partial x_L$ plus a correction term, never a pure product of many Jacobians each potentially $\ll 1$ or $\gg 1$. In a plain network, $\partial x_L/\partial x_0 = \prod_i \partial x_{i+1}/\partial x_i$, a product of $L$ Jacobians whose norms compound multiplicatively — this is exactly the mechanism behind vanishing/exploding gradients. With the residual formulation, even if every $F_i$'s Jacobian happens to vanish, the gradient signal $\partial \mathcal{L}/\partial x_L$ still reaches every earlier layer undiminished through the shortcut term. This is the same principle later reused explicitly in gated architectures for Recurrent Neural Networks, where a "keep" gate near 1 similarly guarantees an additive identity path through time.

Why deeper is never worse (in principle). Because a residual block reduces to the identity when $F(x) \to 0$ and $x$ has matching shape, a residual network of depth $L+1$ contains, as a special case within its hypothesis space, every function representable by a residual network of depth $L$ (append a block with $F \equiv 0$). Optimization is not guaranteed to find the global optimum, but the existence of an easy-to-reach path to "no worse than the shallower network" removes the specific degradation pathology observed in plain stacks, where discovering an identity block requires a delicate, non-default configuration of weights across an entire nonlinear sub-network rather than simply zeroing out a residual branch.

Tradeoffs, honestly. Residual connections do not repeal the no-free-lunch nature of optimization: extremely deep ResNets (1000+ layers) still show some instability and require care (better initialization, sometimes different normalization). The reformulation also assumes the optimal function really is close to identity per block, which is empirically reasonable for image classification but not universally guaranteed. Skip connections increase peak memory during training somewhat, since both the block input and its output must often be retained for the backward pass. Finally, projection shortcuts add parameters and computation exactly where feature maps change shape, a small but real cost concentrated at stage boundaries.

Consider a tiny basic residual block operating on a single scalar feature (to make the arithmetic fully explicit; real blocks operate on tensors, but the additive mechanics are identical). Suppose $x = 2.0$, and the residual function is a two-layer linear-then-ReLU stack (dropping BN for clarity) with weights $w_1 = 0.5$, $w_2 = -0.3$:

The residual block output is

Compare against a plain (non-residual) block with the same two weights: $y_{\text{plain}} = \text{ReLU}(F(x)) = \text{ReLU}(-0.3) = 0$ — the plain block, with these particular (small, near-initialization-scale) weights, kills the signal entirely, while the residual version preserves most of $x$ because the shortcut dominates. Now trace the backward pass for a loss $\mathcal{L} = \frac{1}{2}(y - y^*)^2$ with target $y^* = 1.5$ (residual case, $y = 1.7$, so $\partial\mathcal{L}/\partial y = 0.2$; ReLU is active since $y>0$ so its local gradient is 1):

Note the gradient reaching $x$ is $0.2 \times (1 + (-0.15)) = 0.17$: the "$1$" from the shortcut guarantees a substantial gradient regardless of how small or even sign-flipped the residual branch's own Jacobian is. If we imagine 50 such blocks stacked and the residual branch's local Jacobian averaging around $-0.15$ each, a plain network's gradient would involve a product $(-0.15)^{50} \approx 0$ — catastrophic vanishing. The residual network's gradient involves a sum of 50 small correction terms around an anchor of $1$, staying $O(1)$ throughout. This numeric miniature is exactly the mechanism formalized in the complexity/correctness section above.

Residual networks were introduced by a team at Microsoft Research Asia. Kaiming He led the work alongside Xiangyu Zhang, Shaoqing Ren, and Jian Sun, building directly on the momentum of deep CNN successes chronicled from AlexNet and Deep CNNs onward, and extending the layer-stacking blueprint established by LeNet and the CNN Blueprint to unprecedented depths.

ResNet architectures (ResNet-50 above all) remain the default backbone for transfer learning in computer vision, embedded in object detectors (Faster R-CNN, Mask R-CNN), semantic segmentation networks, and countless production image-classification pipelines because pretrained ResNet weights are cheap, reliable, and widely available. Beyond vision, the residual-connection pattern itself — add the sublayer's input back to its output — is now load-bearing infrastructure in Transformer architectures that power large language models, where every attention sublayer and every feed-forward sublayer is wrapped in a residual connection (often paired with layer normalization) for exactly the gradient-flow reasons derived above. Autonomous-vehicle perception stacks, medical-imaging diagnostic models, and speech-recognition acoustic models built on deep convolutional or Transformer backbones inherit residual connections as a foundational, largely invisible piece of plumbing. The idea also directly foreshadows the gating mechanisms used to stabilize training across long sequences in Recurrent Neural Networks, where an additive "carry" path plays the same mathematical role across time steps that the shortcut plays across depth.

Ask the tutor: Can you show how stacking many residual blocks compares numerically to a plain network when the residual branch's Jacobian is small at every layer?