Backpropagation

Backpropagation is an algorithm for efficiently computing the gradient of a scalar loss function with respect to every parameter in a neural network by applying the chain rule of calculus in a single reverse-mode pass through the computation graph. It is the foundational engine of virtually all deep learning training — without it, optimizing networks with millions or billions of parameters would be computationally infeasible. A counterintuitive fact: backpropagation does not actually learn anything itself — it only computes gradients; the learning is performed by a separate optimizer that consumes those gradients.

The core insight

Every neural network is just a composed function, and the chain rule tells you how to differentiate composed functions. Backpropagation is simply the chain rule applied systematically and efficiently, reusing intermediate computations so that you compute each partial derivative exactly once rather than redundantly. The "aha moment" is realizing that the forward pass builds a computational graph that stores everything you need, and the backward pass is just a single structured traversal of that graph in reverse — no numerical approximations, no independent perturbations per parameter, just exact symbolic differentiation at the cost of one extra forward-pass worth of computation.

How It Works

The Forward Pass: Building the Computation Graph

A neural network computes a prediction $\hat{y}$ through a sequence of operations. For a single layer with weight matrix $W$ , bias $b$ , activation $\sigma$ , and input $x$ :

\quad z = Wx + b

Explanation: The pre-activation $z$ is a linear combination of the inputs — this is the purely affine part of the layer before any nonlinearity is applied.

\quad a = \sigma(z)

Explanation: The activation function $\sigma$ (e.g. ReLU, sigmoid, tanh) introduces nonlinearity; $a$ is the post-activation output passed to the next layer.

The Loss

For a regression task with target $y$ , the mean-squared loss over a batch of $N$ examples is:

\quad \mathcal{L} = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2

Explanation: $\mathcal{L}$ is the single scalar we want to minimize — backprop's job is to find how every parameter in the network nudges this number.

The Chain Rule: Core of Backprop

Suppose $\mathcal{L}$ depends on $a$ , which depends on $z$ , which depends on $W$ . The chain rule gives:

\quad \frac{\partial \mathcal{L}}{\partial W} = \frac{\partial \mathcal{L}}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial W}

Explanation: The gradient of the loss with respect to $W$ factors into a product of local gradients, each computable from quantities already available at that node of the graph. Here:

$\frac{\partial \mathcal{L}}{\partial a}$ — the error signal arriving from the layer above (already computed by the time we reach this layer in the backward pass)
$\frac{\partial a}{\partial z} = \sigma'(z)$ — the derivative of the activation, computed from $z$ stored during the forward pass
$\frac{\partial z}{\partial W} = x^T$ — the input to this layer, also stored during the forward pass

The General Gradient Update Rule

For a parameter $\theta$ updated with learning rate $\eta$ :

\quad \theta \leftarrow \theta - \eta \cdot \frac{\partial \mathcal{L}}{\partial \theta}

Explanation: Each parameter moves a small step in the direction that most steeply reduces the loss — the gradient $\frac{\partial \mathcal{L}}{\partial \theta}$ is what backpropagation supplies.

PyTorch Implementation

```python import torch import torch.nn as nn

# Simple two-layer network model = nn.Sequential( nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10) ) optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) loss_fn = nn.CrossEntropyLoss()

# --- Training step --- x, y = next(iter(train_loader)) # fetch a batch

# 1. Forward pass: builds the computation graph logits = model(x) loss = loss_fn(logits, y)

# 2. Backward pass: backprop fills .grad for every parameter optimizer.zero_grad() # clear stale gradients from last step loss.backward() # THE backprop call

# 3. Optimizer step: actually update weights (NOT backprop) optimizer.step() print(f"Loss: {loss.item():.4f}") print(f"Grad norm W1: {model[0].weight.grad.norm():.4f}") ```

researcher · Geoffrey Hinton | b. 1947 | Co-inventor of practical backpropagation for neural networks

Geoffrey Hinton, a British-Canadian cognitive scientist and computer scientist at the University of Toronto and Google Brain, co-authored the landmark 1986 Nature paper "Learning representations by back-propagating errors" with David Rumelhart and Ronald Williams, which demonstrated that backpropagation could train multi-layer networks to learn useful internal representations. His persistent advocacy for neural networks during the "AI winters" — decades when the broader community dismissed them — proved decisive in establishing deep learning as the dominant paradigm. Hinton shared the 2024 Nobel Prize in Physics for his contributions to artificial neural networks, and he famously resigned from Google in 2023 partly to speak freely about AI risks.

researcher · Yann LeCun | b. 1960 | Applied backpropagation to convolutional networks at scale

Yann LeCun, a French-American computer scientist at NYU and Meta AI Research, applied backpropagation to train convolutional neural networks (LeNet) for handwritten digit recognition in the late 1980s and early 1990s, producing the first practically deployed deep learning system — AT&T's check-reading machines processed millions of bank checks daily using his backprop-trained networks. His work demonstrated that backpropagation was not merely a theoretical curiosity but an engineering tool capable of real commercial deployment. LeCun, along with Hinton and Yoshua Bengio, received the 2018 Turing Award — often called the "Nobel Prize of computing."

Figure

Backpropagation Forward and Backward Pass

Figure

Computation Graph: Chain Rule Unrolled

Intuition

The Mental Model

Imagine you are a manager in a large company, trying to figure out which employees caused a particular bad outcome. The naive approach would be to interview every single employee independently — but that takes forever. Instead, you hold a "blame meeting" that starts at the top of the org chart (the loss) and cascades downward, each manager passing credit or blame proportionally to their direct reports based on how much each person actually influenced the outcome. The forward pass is like watching the work get done: inputs flow up through the hierarchy, each layer transforms the data, and eventually a final mistake (the loss) is recorded. The backward pass is the blame cascade: the loss says "I was this wrong," the output layer says "here is how much each of my inputs contributed," the hidden layers each receive their share of the blame and pass it further back — all the way to the weights. The key efficiency insight is that the "blame fractions" at each layer — the local gradients like $\sigma'(z)$ and $x^T$ — were already computed and cached during the forward pass. No extra work is needed to compute them during the backward pass; they are just retrieved from memory. This is what makes backprop $O(1)$ backward passes relative to the number of parameters, rather than the $O(N)$ passes naive finite-difference gradient estimation would require. Think of it as a receipt: the forward pass builds a detailed receipt of every operation, and the backward pass reads that receipt from bottom to top, tallying up each item's contribution to the total.

Historical Development

Origins · 1 minhow this was discovered

History

1847

George Boole and later Augustin-Louis Cauchy develop early formulations of the chain rule, the mathematical backbone that backpropagation is built upon.

1960

Henry J. Kelley publishes gradient-descent-based optimal control methods using chain-rule-like calculations, an early precursor in control theory.

1970

Seppo Linnainmaa's Master's thesis introduces reverse-mode automatic differentiation — essentially backpropagation — in its general computational form, though not applied to neural networks.

1974

Paul Werbos, in his Harvard PhD thesis "Beyond Regression," describes backpropagation through time and multi-layer networks, the first explicit application to neural network-like architectures.

1986

Rumelhart, Hinton, and Williams publish "Learning representations by back-propagating errors" in Nature, the landmark paper that popularized backprop and demonstrated it could solve problems thought impossible for perceptrons (e.g. XOR, feature learning).

1989

Yann LeCun applies backpropagation to train convolutional networks (proto-LeNet) for digit recognition; AT&T deploys these networks in check-reading machines — the first large-scale commercial deep learning deployment.

1991

Sepp Hochreiter identifies the vanishing gradient problem in his diploma thesis, showing that backprop in deep or recurrent networks degrades because gradients shrink exponentially with depth.

1997

Hochreiter and Schmidhuber publish the Long Short-Term Memory (LSTM) architecture, the first major architectural solution to vanishing gradients in backprop-trained recurrent networks.

2012

Krizhevsky, Sutskever, and Hinton win ImageNet with AlexNet, trained via backprop on GPUs with ReLU activations — reigniting mainstream interest in deep learning and validating GPU-accelerated backprop at scale.

2017

PyTorch's dynamic computation graph (define-by-run) becomes widely adopted, making backpropagation transparent and debuggable for practitioners via eager execution and autograd.

Why It Matters

Real-World Impact

GPT-4 (OpenAI, 2023)Trained using backpropagation through a transformer with an estimated ~1.8 trillion parameters across mixture-of-experts layers; backprop computes gradients across all layers in a single backward pass via autograd, making this scale tractable on distributed GPU clusters.

AlphaFold 2 (DeepMind, 2021)Backpropagation trains the Evoformer and structure module end-to-end to predict 3D protein structures from amino acid sequences, achieving accuracy competitive with experimental methods — a result impossible without differentiating through complex geometric operations.

Stable Diffusion / DALL·E 3Denoising diffusion models are trained by backpropagating through a U-Net or transformer that predicts noise at each diffusion timestep; the loss is a simple MSE between predicted and actual noise, but backprop threads gradients through hundreds of layers.

Tesla Autopilot / Full Self-DrivingMulti-task convolutional networks for real-time object detection, lane recognition, and depth estimation are trained via backprop on millions of labeled driving frames; gradients flow through shared backbone weights simultaneously from multiple task-specific loss heads.

AlphaGo / MuZero (DeepMind)Policy and value networks are trained by backpropagating gradients from MCTS-derived policy targets and game outcome signals — backprop over deep residual networks (19–40 layers) enabled superhuman Go performance.

Recommendation Systems (YouTube, TikTok)Deep neural networks with embedding layers for users and items are trained via backpropagation on hundreds of billions of interaction events; gradients update embedding vectors to encode user preferences in a continuous space.

BERT / Language Model Fine-tuningThe entire pretrain-then-finetune paradigm depends on backpropagation propagating task-specific gradient signals from a small labeled dataset back through 12–24 transformer layers, updating all 110M–340M parameters from a pretrained initialization.

Backpropagation

How It Works

The Forward Pass: Building the Computation Graph

The Loss

The Chain Rule: Core of Backprop

The General Gradient Update Rule

PyTorch Implementation

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Before this

Where this leads