The Chain Rule and Computational Graphs

The chain rule computes the derivative of a composition of functions by multiplying the derivatives of the constituent functions. In words: to differentiate $f (g (x))$ , multiply the rate of change of $f$ (at $g (x)$ ) by the rate of change of $g$ (at $x$ ). It applies everywhere derivatives appear — single-variable calculus, multivariable analysis, differential geometry, and modern machine learning (where its computational-graph form is called backpropagation). First systematized by Leibniz (1676) via differential notation; the computational-graph perspective matured with reverse-mode automatic differentiation in the 1970s–80s.

The chain rule states that if $y = f (u)$ and $u = g (x)$ are differentiable, then the composite $y = f (g (x))$ is differentiable with derivative $f^{'} (g (x)) g^{'} (x)$ . It matters because almost every function encountered in practice is a composition of simpler primitives, and differentiation must respect that structure. A computational graph is a directed acyclic graph (DAG) representing such a composition node-by-node, turning the chain rule into a mechanical bookkeeping procedure over edges. The surprising fact: when a scalar output depends on many inputs, reverse-mode application of the chain rule computes all partial derivatives in a single backward pass costing roughly the same as one forward evaluation — the algorithmic heart of training neural networks with billions of parameters.

The Core Insight

A composition is a pipeline: $x \to u \to y$ . The chain rule says the overall sensitivity is the product of local sensitivities along the pipeline. On a graph with branches and merges, "product along a path" generalizes to "sum over all paths, each path being a product of edge derivatives." Reverse-mode differentiation is just an efficient order of summation that reuses shared subexpressions.

Formal Statement

Composite Function

Given functions $g : D \subseteq R \to R$ and $f : E \subseteq R \to R$ with $g (D) \subseteq E$ , the composite $f \circ g$ is defined by $(f \circ g) (x) = f (g (x))$ . We call $g$ the inner function and $f$ the outer function. The intermediate value $u = g (x)$ is the node variable connecting them.

Chain Rule — Single Variable

Let $g$ be differentiable at $x_{0}$ and let $f$ be differentiable at $u_{0} = g (x_{0})$ . Then $f \circ g$ is differentiable at $x_{0}$ and

(f \circ g)^{'} (x_{0}) = f^{'} (g (x_{0})) g^{'} (x_{0}) .

⟳Differentiate

f

at the point

u_{0} = g (x_{0})

, then multiply by the inner rate

g^{'} (x_{0})

; see the proof below using the error-form of differentiability.

In Leibniz notation, with $y = f (u)$ , $u = g (x)$ :

\frac{d y}{d x} = \frac{d y}{d u} \cdot \frac{d u}{d x} .

⟳Treat

d y / d u

and

d u / d x

as ratios of differentials; the cancellation of

d u

is suggestive but must be justified by the limit argument in the proof.

Chain Rule — Multivariable

Let $g : R^{n} \to R^{m}$ be differentiable at $x_{0}$ and $f : R^{m} \to R^{k}$ differentiable at $g (x_{0})$ . Then the Jacobian of the composite is the matrix product of the Jacobians:

J_{f \circ g} (x_{0}) = J_{f} (g (x_{0})) J_{g} (x_{0}) .

⟳Apply the linear-approximation definition of total differentiability to both maps; the composite's best linear approximation is the composition of the linear approximations, whose matrix is the product.

Componentwise, for a scalar $z = f (u_{1}, \dots, u_{m})$ where each $u_{j} = g_{j} (x_{1}, \dots, x_{n})$ :

\frac{\partial z}{\partial x _{i}} = j = 1 \sum m \frac{\partial z}{\partial u _{j}} \frac{\partial u _{j}}{\partial x _{i}} .

⟳Expand the matrix product in

[3]

for a single output row and single input column; the sum runs over the shared intermediate variables

u_{j}

— this is the "sum over paths through the

u_{j}

layer."

Computational Graph

A computational graph for an expression is a directed acyclic graph $G = (V, E)$ where each vertex $v$ holds a variable computed by an elementary operation $ϕ_{v}$ of its parent vertices $pa (v)$ : $v = ϕ_{v} ({p : p \in pa (v)})$ . Source vertices are inputs; a designated sink is the output. Each edge $(p \to v)$ carries the local partial derivative $\partial v / \partial p$ , evaluated at the current values. The full derivative of the output with respect to an input is obtained by applying the chain rule across the graph.

Computational graph for $y = \sin(x_1 x_2) + x_1$ with edge-local derivatives; backward pass accumulates $\partial y/\partial x_i$ by summing over paths.

Why It Holds / Derivation

proof[Single-Variable Chain Rule via the Error Form] Step 1 — Differentiability as linear approximation with controlled error. Because $f$ is differentiable at $u_{0}$ , there is a function $ε$ with $lim_{k \to 0} ε (k) = 0 = ε (0)$ such that for all small $k$ :

f (u_{0} + k) - f (u_{0}) = (f^{'} (u_{0}) + ε (k)) k .

⟳Define

ε (k) = \frac{f ( u _{0} + k ) - f ( u _{0} )}{k} - f^{'} (u_{0})

for

k \neq = 0

and

ε (0) = 0

; differentiability forces

ε (k) \to 0

Step 2 — Set the inner increment. Let

h

be a small increment in

x

, and define

k = g (x_{0} + h) - g (x_{0})

. By differentiability of

g

k \to 0

h \to 0

. Step 3 — Substitute. Then

f (g (x_{0} + h)) - f (g (x_{0})) = (f^{'} (u_{0}) + ε (k)) (g (x_{0} + h) - g (x_{0})) .

⟳Replace

u_{0} + k

g (x_{0} + h)

[5]

; this remains valid even when

k = 0

because the right side then vanishes too.

Step 4 — Divide by $h$ and take the limit.

\frac{f ( g ( x _{0} + h )) - f ( g ( x _{0} ))}{h} = (f^{'} (u_{0}) + ε (k)) \frac{g ( x _{0} + h ) - g ( x _{0} )}{h} .

⟳As

h \to 0

ε (k) \to 0

(since

k \to 0

), and the difference quotient of

g

tends to

g^{'} (x_{0})

; multiply the limits.

Taking $h \to 0$ gives $(f \circ g)^{'} (x_{0}) = f^{'} (u_{0}) g^{'} (x_{0}) = f^{'} (g (x_{0})) g^{'} (x_{0})$ . $■$

Why the naïve "cancel $du$" proof fails

The seductive argument $\frac{Δ y}{Δ x} = \frac{Δ y}{Δ u} \cdot \frac{Δ u}{Δ x}$ followed by taking limits breaks when $Δ u = g (x_{0} + h) - g (x_{0}) = 0$ for some sequence of $h \to 0$ (e.g. $g$ locally constant or oscillating). Division by $Δ u$ is then illegal. The error-form proof above avoids any division by $Δ u$ , which is exactly why it is the correct argument.

Gottfried Wilhelm Leibniz | 1646–1716

Leibniz, co-inventor of calculus, gave us the differential notation $d y / d x$ in which the chain rule appears as the almost-tautological cancellation $\frac{d y}{d x} = \frac{d y}{d u} \frac{d u}{d x}$ . His manuscripts from November 1676 already manipulate composite differentials. While the notation makes the rule memorable, it obscures the subtlety (see the warning above) — a gap rigorously closed only with the 19th-century $ε$ – $δ$ definition of the limit by Cauchy and Weierstrass. Leibniz's deeper gift was the operational viewpoint: differentiation as a symbolic process that propagates along the structure of an expression — the conceptual seed of computational graphs three centuries later.

Building Intuition

Think of nested gears. If gear $A$ drives gear $B$ at ratio $3 : 1$ (B turns 3× as fast), and $B$ drives $C$ at ratio $2 : 1$ , then $A$ drives $C$ at $6 : 1$ — ratios multiply. The derivative $g^{'} (x)$ is the local gear ratio from $x$ to $u$ ; $f^{'} (u)$ is the ratio from $u$ to $y$ ; the composite ratio is their product.

Now add branching. If $x$ influences $y$ through two independent gear trains (paths), the total effect is the sum of the two products — small pushes from each route add up. That is exactly $[4]$ : sum over paths, multiply along each. The computational graph is the literal gear diagram, and backpropagation is the disciplined way to total up all the ratios without recomputing shared gears.

Worked Examples

Single-Variable: a nested trig–power function

Given:

y = (sin (3 x))^{4}

. Find

\frac{d y}{d x}

x = π /6

. Step 1 — Identify the layers. Outer:

f (u) = u^{4}

. Middle:

u = sin (w)

. Inner:

w = 3 x

. Three nested functions. Step 2 — Local derivatives.

$\frac{d y}{d u} = 4 u^{3}$
$\frac{d u}{d w} = cos w$
$\frac{d w}{d x} = 3$

Step 3 — Multiply (chain rule for a 3-chain).

\frac{d y}{d x} = 4 (sin 3 x)^{3} \cdot cos (3 x) \cdot 3 = 12 sin^{3} (3 x) cos (3 x) .

⟳Apply

[2]

twice:

\frac{d y}{d x} = \frac{d y}{d u} \frac{d u}{d w} \frac{d w}{d x}

; substitute

w = 3 x

u = sin 3 x

Step 4 — Evaluate at $x = π /6$ . Here

3 x = π /2

, so

sin (3 x) = 1

cos (3 x) = 0

\frac{d y}{d x}_{π /6} = 12 \cdot 1^{3} \cdot 0 = 0.

\frac{d y}{d x}_{x = π /6} = 0 (dimensionless)

The zero makes sense: at $3 x = π /2$ , $sin (3 x)$ is at its maximum, so $sin^{4} (3 x)$ momentarily stops changing.

Multivariable / Graph: reverse-mode pass

Given:

y = sin (x_{1} x_{2}) + x_{1}

(x_{1}, x_{2}) = (2, \frac{π}{4})

. Compute

\partial y / \partial x_{1}

and

\partial y / \partial x_{2}

by a forward pass then a backward pass. Forward pass (compute node values).

$u = x_{1} x_{2} = 2 \cdot \frac{π}{4} = \frac{π}{2} \approx 1.5708$
$v = sin u = sin (\frac{π}{2}) = 1$
$y = v + x_{1} = 1 + 2 = 3$

Backward pass (seed $\overset{y}{ˉ} \equiv \partial y / \partial y = 1$ , propagate adjoints).

\overset{v}{ˉ} = \overset{y}{ˉ} \frac{\partial y}{\partial v} = 1 \cdot 1 = 1, \overset{u}{ˉ} = \overset{v}{ˉ} \frac{\partial v}{\partial u} = 1 \cdot cos u = cos \frac{π}{2} = 0.

⟳Each adjoint

\overset{n}{ˉ} = \partial y / \partial n

; multiply the downstream adjoint by the local edge derivative.

Accumulate to inputs (sum over all incoming paths).

\frac{\partial y}{\partial x _{1}} = via u \overset{u}{ˉ} \frac{\partial u}{\partial x _{1}} + direct \overset{y}{ˉ} \frac{\partial y}{\partial x _{1}} = 0 \cdot x_{2} + 1 \cdot 1 = 1.

⟳

x_{1}

reaches

y

by two paths — through

u

and directly via the

+ x_{1}

term;

[4]

tells us to add them.

\frac{\partial y}{\partial x _{2}} = \overset{u}{ˉ} \frac{\partial u}{\partial x _{2}} = 0 \cdot x_{1} = 0.

⟳

x_{2}

reaches

y

only through

u

; one path, so a single product.

\nabla y = (\frac{\partial y}{\partial x _{1}}, \frac{\partial y}{\partial x _{2}}) = (1, 0) (dimensionless)

Note: a single backward pass produced both partials — the efficiency that powers backpropagation.

Common Misconceptions

Three classic errors

Evaluating the outer derivative at $x$ instead of at $u = g (x)$ . $f^{'}$ must be evaluated at $g (x_{0})$ , not $x_{0}$ . Writing $(sin (x^{2}))^{'} = cos (x) \cdot 2 x$ is wrong; it is $cos (x^{2}) \cdot 2 x$ .
Forgetting the inner derivative entirely. $(sin (x^{2}))^{'} \neq = cos (x^{2})$ . The factor $2 x$ is mandatory.
Treating $d y / d u$ as a genuine fraction. It is notation for a limit. The cancellation works as a mnemonic but is not a proof, and it fails to generalize cleanly to partial derivatives, where $\partial z / \partial x \neq = 1/ (\partial x / \partial z)$ in general.

A composite that is differentiable while neither factor's quotient is "tame"

Let $g (x) = x^{2} sin (1/ x)$ for $x \neq = 0$ , $g (0) = 0$ . Then $g$ is differentiable at $0$ with $g^{'} (0) = 0$ , but $g^{'} (x)$ oscillates wildly near $0$ and $Δ u = g (x_{0} + h) - g (0)$ equals $0$ infinitely often as $h \to 0$ . With $f (u) = u$ (so $f \circ g = g$ ), the naïve "divide by $Δ u$ " proof is invalid on those sequences — yet the chain rule still holds because the error-form proof ( $[5]$ – $[7]$ ) never divides by $Δ u$ . This is the canonical example showing why rigor matters.

Connections

The chain rule is the connective tissue of differential calculus. It powers Implicit Differentiation (differentiate $F (x, y (x)) = 0$ and solve for $y^{'}$ ), Related Rates (link $d V / d t$ to $d r / d t$ through $d V / d r$ ), and The Substitution Rule for Integration (which is the chain rule read backwards). In several variables it underlies the Gradient and Directional Derivative and the Jacobian Matrix. Its graph form is precisely Backpropagation in neural networks, and the abstract version — composition of linear approximations — is the statement of the Total Derivative in Differential Geometry (the functoriality of the tangent map: $T (f \circ g) = T f \circ T g$ ).

Real-World Applications

Deep learning (backpropagation). Training networks like GPT or ResNet computes gradients of a scalar loss over billions of parameters via reverse-mode chain rule on the computational graph — one forward + one backward pass.
Automatic differentiation libraries. PyTorch's autograd, TensorFlow, and JAX build computational graphs and apply the chain rule mechanically; JAX's grad/jvp/vjp expose forward- and reverse-mode directly.
Sensitivity analysis in engineering. Adjoint methods (reverse-mode chain rule) compute how a single performance metric depends on thousands of design variables in aircraft wing and turbine optimization.
Computational fluid dynamics & weather. Adjoint models in data assimilation (e.g. ECMWF's 4D-Var) propagate sensitivities backward through time-stepping graphs to fit initial conditions.
Computer graphics & differentiable rendering. Gradients of rendered images w.r.t. scene parameters flow through the rendering pipeline by the chain rule, enabling inverse rendering and neural radiance fields (NeRF).
Economics & finance. Greeks (option sensitivities like delta, vega) are chain-rule derivatives of a price functional; adjoint algorithmic differentiation accelerates risk computation across portfolios.
Physics — Lagrangian/Hamiltonian mechanics. Coordinate changes transform equations of motion via the multivariable chain rule (Jacobians of the transformation).

Historical Development

1676 — Leibniz manipulates composite differentials in unpublished manuscripts, foreshadowing $\frac{d y}{d x} = \frac{d y}{d u} \frac{d u}{d x}$ .
1684 — Leibniz publishes his first calculus paper (Nova Methodus), establishing differential notation that encodes the chain rule.
1690s — The Bernoullis and L'Hôpital apply chain-rule reasoning to curves and nested expressions across Europe.
1820s — Cauchy's Cours d'analyse (1821) introduces limit-based rigor; the chain rule gets its first careful, if still informal, treatment.
1860s–1870s — Weierstrass's $ε$ – $δ$ framework enables the modern error-form proof, closing the "divide by $Δ u$ " gap.
1870s — The multivariable / Jacobian chain rule is formalized in the work of Jacobi's successors on transformations and partial differentiation.
1964 — Wengert publishes a method for the automatic evaluation of derivatives — an early forward-mode automatic differentiation.
1970 — Seppo Linnainmaa describes the reverse-mode accumulation of derivatives (the algorithmic chain rule on a graph) in his master's thesis.
1986 — Rumelhart, Hinton & Williams popularize backpropagation for neural networks, igniting modern deep learning.

Subtleties & Limits

Boundaries of the rule

The chain rule requires differentiability; at points where the inner or outer function is merely continuous (e.g. $∣ x ∣$ at $0$ ), the composite may fail to be differentiable or require subgradients — leading to the calculus of Clarke subdifferentials and questions about which "nonsmooth chain rules" hold for compositions like $ReLU$ networks, where backpropagation is used in practice despite kink points. A related modern theme is numerical stability: repeated multiplication of edge derivatives causes vanishing or exploding gradients in deep graphs; understanding when the product of many Jacobians stays well-conditioned (via residual connections, normalization, orthogonal initialization) remains an active research area. Finally, in infinite-dimensional and stochastic settings, generalizations (Fréchet-derivative chain rule, Itô's formula as a "stochastic chain rule" with an extra second-order term) show that the simple product law is only the smooth, deterministic special case of a richer family of composition rules.