Forward Propagation

A feedforward neural network with $L$ layers is defined by weight matrices $\mathbf{W}^{[1]}, \dots, \mathbf{W}^{[L]}$, bias vectors $\mathbf{b}^{[1]}, \dots, \mathbf{b}^{[L]}$, and activation functions $g^{[1]}, \dots, g^{[L]}$. Forward propagation is the procedure that, given an input $\mathbf{x}$, computes the network's output $\hat{\mathbf{y}}$ by applying these layers in sequence. Concretely, define $\mathbf{a}^{[0]} = \mathbf{x}$ as the input, and for each layer $l = 1, \dots, L$ compute two quantities: the pre-activation (or logit)

and the activation

The final activation $\mathbf{a}^{[L]}$ is the network's output $\hat{\mathbf{y}}$. The word "forward" refers to the direction of information flow: data moves strictly from the input layer toward the output layer, with no loops or backward edges, which is why this class of models is called a feedforward network. The key structural fact is that $\mathbf{W}^{[l]}$ has shape $n_l \times n_{l-1}$, where $n_{l-1}$ is the width (number of units) of the previous layer and $n_l$ is the width of the current layer. This shape convention guarantees that $\mathbf{W}^{[l]} \mathbf{a}^{[l-1]}$ is well-defined and produces a vector of length $n_l$, which is exactly the width expected by layer $l$. This is the "shape invariant" that forward propagation preserves at every step, and it is the single most useful fact for debugging real networks: any dimension mismatch in a forward pass is a violation of this invariant, almost always caused by a wrongly transposed matrix or a misconfigured layer width. In practice, we never run one input vector through the network at a time — we run a batch of $m$ examples simultaneously. Stacking the examples as columns of a matrix $\mathbf{A}^{[0]} = \mathbf{X} \in \mathbb{R}^{n_0 \times m}$, the exact same equations apply with vectors replaced by matrices:

where the addition of $\mathbf{b}^{[l]} \in \mathbb{R}^{n_l}$ to a matrix of shape $n_l \times m$ is broadcast — the bias vector is added to every column. This batched form is what actually runs on GPUs: a handful of large matrix multiplications rather than millions of small vector operations, which is essential for computational efficiency (discussed in detail below).

Forward propagation proceeds as a simple loop over layers, each iteration performing one matrix multiply, one broadcasted bias addition, and one elementwise nonlinearity. The only subtlety is bookkeeping: every intermediate $\mathbf{Z}^{[l]}$ and $\mathbf{A}^{[l]}$ must typically be cached, because although forward propagation alone does not need them after use, the backpropagation algorithm that computes gradients will need them.

Step-by-step mechanism: 1. Load the input batch $\mathbf{X}$ into $\mathbf{A}^{[0]}$. 2. For each layer $l$ from $1$ to $L$: a. Compute the pre-activation $\mathbf{Z}^{[l]} = \mathbf{W}^{[l]} \mathbf{A}^{[l-1]} + \mathbf{b}^{[l]}$. b. Apply the nonlinearity elementwise: $\mathbf{A}^{[l]} = g^{[l]}(\mathbf{Z}^{[l]})$. c. Cache $\mathbf{Z}^{[l]}$ and $\mathbf{A}^{[l]}$ for later use (e.g. by backpropagation). 3. Return $\mathbf{A}^{[L]}$ as the prediction $\hat{\mathbf{Y}}$. The choice of $g^{[l]}$ varies by layer. Hidden layers commonly use ReLU, $g(z) = \max(0, z)$, or tanh; the output layer's activation is chosen to match the task — sigmoid for binary classification, softmax for multiclass classification, or the identity function for regression. This choice does not change the mechanics of forward propagation, only the specific function applied in step 2b. ``` function FORWARD_PROPAGATE(X, weights[1..L], biases[1..L], activations[1..L]):

# X has shape (n0, m): n0 input features, m examples in the batch A[0] = X cache = {} # will store Z[l], A[l] for every layer for l = 1 to L: W = weights[l] # shape (n_l, n_{l-1}) b = biases[l] # shape (n_l, 1), broadcast over m columns g = activations[l] Z[l] = MATMUL(W, A[l-1]) + b # shape (n_l, m) A[l] = ELEMENTWISE(g, Z[l]) # shape (n_l, m), same shape as Z[l] cache[l] = (Z[l], A[l-1]) # store what backprop will need

# (A[l] itself is available as A[l-1] of the next iteration,

# and also cached explicitly if this is the last layer) Yhat = A[L] return Yhat, cache ``` Two invariants make this loop correct by construction. First, the shape invariant: at the start of iteration $l$, $\mathbf{A}^{[l-1]}$ has shape $n_{l-1} \times m$ by the induction hypothesis (true at $l=1$ since $\mathbf{A}^{[0]} = \mathbf{X}$ has shape $n_0 \times m$ by definition), and since $\mathbf{W}^{[l]}$ has shape $n_l \times n_{l-1}$, the product has shape $n_l \times m$, matching what the next iteration expects. Second, the semantic invariant: $\mathbf{A}^{[l]}$ always represents "the representation of the input batch as seen after $l$ layers of processing," so that composing $L$ layers computes exactly the function the architecture defines, namely $f(\mathbf{x}) = g^{[L]}(\mathbf{W}^{[L]} g^{[L-1]}(\cdots g^{[1]}(\mathbf{W}^{[1]}\mathbf{x} + \mathbf{b}^{[1]}) \cdots) + \mathbf{b}^{[L]})$.

Time complexity. The dominant cost at layer $l$ is the matrix multiplication $\mathbf{W}^{[l]} \mathbf{A}^{[l-1]}$, where $\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}$ and $\mathbf{A}^{[l-1]} \in \mathbb{R}^{n_{l-1} \times m}$. A standard (non-strassen) matrix multiply of an $n_l \times n_{l-1}$ matrix by an $n_{l-1} \times m$ matrix costs $O(n_l \cdot n_{l-1} \cdot m)$ multiply-add operations, since each of the $n_l \times m$ output entries is an inner product of length $n_{l-1}$. The bias addition and elementwise activation cost only $O(n_l \cdot m)$, which is dominated by the matrix multiply whenever $n_{l-1} > 1$. Summing over all $L$ layers, the total time for one forward pass over a batch of $m$ examples is

This is tight (not just an upper bound) for standard dense matrix multiplication, since every entry of every weight matrix must be read and multiplied by some activation at least once — there is no way to compute a general matrix-vector product faster than examining every matrix entry, in the absence of special structure (sparsity, low rank, etc.). If all hidden layers have roughly the same width $n$, this simplifies to $O(L n^2 m)$.

Space complexity. Two components matter. First, storing the parameters themselves costs $O\!\left(\sum_l n_{l-1} n_l\right)$, independent of the batch size — this is fixed at training/deployment time. Second, storing the activations for a forward pass costs $O\!\left(m \sum_l n_l\right)$, since each layer's output $\mathbf{A}^{[l]}$ has $n_l \times m$ entries. If the goal is inference only, one can discard $\mathbf{A}^{[l-1]}$ as soon as $\mathbf{A}^{[l]}$ is computed, needing only $O(m \max_l n_l)$ memory at any instant (two buffers). However, if a backward pass will follow (training), every $\mathbf{Z}^{[l]}$ and $\mathbf{A}^{[l]}$ must be retained, since the chain rule needs them — this is precisely why training consumes far more memory than inference, and it is the reason techniques like gradient checkpointing exist (trading recomputation for memory).

Correctness. The forward pass computes exactly the function specified by the network architecture, and this can be shown by induction on the layer index $l$. The base case is trivial: $\mathbf{A}^{[0]} = \mathbf{X}$ is the input by definition, matching the architecture's specification with zero layers applied. For the inductive step, assume $\mathbf{A}^{[l-1]}$ correctly holds $f_{1:l-1}(\mathbf{X})$, the composition of the first $l-1$ layers applied to the input. Then by the algorithm's definition,

which is exactly the definition of applying $l$ layers. By induction this holds for all $l$ up to $L$, so $\mathbf{A}^{[L]}$ is the network's full function applied to $\mathbf{X}$, as required. Correctness of the shapes follows by the same induction using the shape invariant argued in the previous section.

Tradeoffs. Batching increases throughput (better hardware utilization, amortized memory-access overhead) but increases peak memory use linearly in $m$; very large batches can exceed device memory, forcing gradient accumulation or smaller batches at the cost of noisier gradient estimates during training. Caching every activation is essential for training but wasteful for inference-only deployment, where "forward-only" implementations deliberately drop this cache to save memory. Numerically, computing $\mathbf{Z}^{[l]}$ and applying $g^{[l]}$ separately (rather than some fused approximation) is exact but two full passes over the data; production systems often fuse the matmul, bias-add, and activation into a single kernel purely for speed, without changing the mathematical result.

Consider a tiny network with $n_0 = 2$ input features, one hidden layer of width $n_1 = 2$ with ReLU activation, and a scalar output layer $n_2 = 1$ with sigmoid activation (binary classification). Take a batch of $m = 2$ examples:

Here column $j$ of $\mathbf{X}$ is example $j$: example 1 is $(1, 2)$, example 2 is $(-1, 0)$.

Layer 1 pre-activation:

Layer 1 activation (ReLU, elementwise):

Layer 2 pre-activation:

Layer 2 activation (sigmoid):

The table below traces the shapes at every step, confirming the shape invariant argued earlier:

Example 1 (features $(1,2)$) is predicted "positive class" with probability $\approx 0.269$, and example 2 (features $(-1,0)$) with probability $\approx 0.119$; both are cached along with $\mathbf{Z}^{[1]}, \mathbf{A}^{[1]}, \mathbf{Z}^{[2]}$ for the backward pass that would compute gradients against a loss such as binary cross-entropy — the topic of Loss Functions for Deep Networks.

The forward pass as we use it today is the computational realization of ideas that trace back to the earliest formal models of neurons and to the linear-algebraic view of computation that became standard with the rise of vectorized numerical computing.

Every neural network deployed in production — image classifiers, speech recognizers, recommendation systems, large language models generating the next token — executes a forward pass to produce its output; training additionally requires a forward pass before every backward pass to compute gradients. Deep learning frameworks such as PyTorch and TensorFlow implement forward propagation as a graph of tensor operations, which is precisely the notion of a computational graph that later supports automatic differentiation. Hardware accelerators (GPUs, TPUs) are designed around the exact operation forward propagation demands most: dense matrix multiplication at scale, which is why matrix-multiply throughput (measured in FLOPs) is the headline specification of modern AI chips. Inference-serving systems (the software that answers a live user's request) optimize forward propagation specifically — through quantization, kernel fusion, and batching strategies — since inference-time forward passes, run at massive scale, dominate the operational cost of deployed AI products. Understanding forward propagation precisely is also the direct prerequisite for understanding Loss Functions for Deep Networks, since a loss is always computed as a function of the forward pass's output, and for understanding backpropagation, which is defined as a reverse traversal of the same computational graph.

:::quiz[If layer $l$ has $n_l$ units and layer $l-1$ has $n_{l-1}$ units, what is the shape of $\mathbf{W}^{[l]}$?]

• $n_{l-1} \times n_l$
• $n_l \times n_{l-1}$ ✓
• $n_l \times n_l$
• $m \times n_l$

:::quiz[Why is a bias vector $\mathbf{b}^{[l]} \in \mathbb{R}^{n_l}$ added correctly to $\mathbf{Z}^{[l]} \in \mathbb{R}^{n_l \times m}$?]

• Because matrix addition requires identical shapes, so no adjustment is needed
• Because the bias is broadcast across the $m$ columns, adding the same vector to every example ✓
• Because the bias is transposed before addition
• Because $m$ must always equal $n_l$

:::quiz[Why must $\mathbf{Z}^{[l]}$ and $\mathbf{A}^{[l]}$ typically be cached during a forward pass used for training?]

• They are needed to compute the loss directly, and nothing else uses them
• The backward pass (gradient computation) reuses these exact values via the chain rule ✓
• Caching improves numerical precision of the forward computation
• Only $\mathbf{A}^{[L]}$, the final output, is ever needed later

Ask the tutor: How does the cached forward-pass data get reused, step by step, when backpropagation computes gradients?