Weight Initialization Strategies

Weight initialization is the assignment of numerical values to a network's trainable parameters at the moment before the first forward pass, before any gradient descent step has occurred. Formally, for a layer computing

initialization specifies the distribution from which each entry of $W^{(l)} \in \mathbb{R}^{n_{out} \times n_{in}}$ and $b^{(l)} \in \mathbb{R}^{n_{out}}$ is drawn, before any data-dependent update is applied. Two quantities describe the shape of a layer: fan-in ($n_{in}$), the number of connections feeding into each output unit, and fan-out ($n_{out}$), the number of connections a given input unit feeds forward into. These two numbers, not just the total parameter count, govern how variance propagates. The central quantity of interest is the variance of the pre-activation $z^{(l)}_j$ across the population of neurons and across training examples, assuming (for the purposes of this analysis) that weights, inputs, and biases are drawn independently with zero mean. Under this assumption, for a single output unit $j$ with fan-in $n_{in}$,

using independence and zero-mean assumptions to drop cross terms $\mathbb{E}[W_{ji}a_i W_{jk}a_k] = 0$ for $i \neq k$, and using $\mathrm{Var}(W_{ji}a_i) = \mathrm{Var}(W_{ji})\mathrm{Var}(a_i)$ for zero-mean independent factors. This single identity is the entire engine behind every initialization scheme discussed in this article and the next: the variance of the output of a layer is the fan-in multiplied by the per-weight variance multiplied by the input variance. If $n_{in} \cdot \mathrm{Var}(W)$ is not close to $1$, variance compounds geometrically across $L$ stacked layers, since applying the identity recursively gives $\mathrm{Var}(z^{(L)}) \approx \left(n_{in}\mathrm{Var}(W)\right)^{L} \cdot \mathrm{Var}(a^{(0)})$ up to the effect of the nonlinearity. Two disastrous special cases dominate the historical failure record. Zero initialization sets $\mathrm{Var}(W) = 0$ deterministically — every weight equal to a constant, most commonly zero — which does not merely shrink variance, it eliminates the mechanism by which different neurons in the same layer come to compute different functions (a phenomenon called symmetry breaking, discussed below). Naive random initialization avoids that particular trap by drawing weights from, say, $\mathcal{N}(0, \sigma^2)$ with a fixed $\sigma$ independent of layer width, but this leaves $n_{in}\sigma^2$ essentially arbitrary — for wide layers it is large, for narrow layers small — so the geometric compounding described above is nearly guaranteed to occur unless $\sigma$ happens to be tuned exactly for that layer's fan-in, which naive schemes do not do.

The mechanism has two parts worth separating cleanly: why zero initialization fails structurally (symmetry), and why naive random initialization fails quantitatively (variance drift), even though it avoids the symmetry trap.

Symmetry breaking. Consider a fully connected layer with two hidden units, both initialized to identical weight rows $W_{1,:} = W_{2,:}$ (as happens when everything is set to zero, or to any other single shared constant, or when random weights happen to be tied). On the forward pass, both units compute the same pre-activation $z_1 = z_2$, hence the same activation $a_1 = a_2$. On the backward pass, the gradient with respect to $W_{1,:}$ and $W_{2,:}$ are computed from identical downstream quantities and identical upstream errors (since both units contribute identically to every downstream computation), so $\partial \mathcal{L}/\partial W_{1,:} = \partial \mathcal{L}/\partial W_{2,:}$. The gradient update therefore preserves the tie: $W_{1,:}$ and $W_{2,:}$ remain equal after any number of gradient steps. This is an invariant of the training dynamics, not a transient effect corrected once weights start moving — no amount of subsequent training breaks it. All that varies is the shared bias term, which does receive nonzero gradient even under all-zero weight initialization (because the bias gradient depends on the upstream error signal, not on the tied weights), but this only lets the whole layer shift together, not diversify into distinct feature detectors. A layer of $n_{out}$ tied neurons in this state behaves, for all forward and backward computation, exactly like a layer with a single neuron — an enormous, silent reduction in model capacity that produces no error message and no obviously anomalous loss curve at first glance (loss often decreases slightly, from the shared bias adapting, masking the deeper failure).

Variance drift. Assume the tie is broken (weights are i.i.d. random, not constant), so this failure mode is avoided. The remaining question is scale. Using the identity derived above, $\mathrm{Var}(z^{(l)}) \approx n_{in}^{(l)} \mathrm{Var}(W^{(l)}) \cdot \mathrm{Var}(a^{(l-1)})$, and treating the nonlinearity as approximately variance-preserving near the origin (a reasonable local approximation for $\tanh$ or linear regimes, though not for ReLU, which needs its own correction factor — foreshadowed but not derived here), the accumulated variance after $L$ layers of identical fan-in $n$ and identical per-weight variance $\sigma^2$ is

If $n\sigma^2 > 1$, this quantity grows exponentially in $L$ — exploding activations, which saturate nonlinearities like sigmoid/tanh into their flat regions (where derivatives are near zero) or produce numerical overflow in deep linear stacks. If $n\sigma^2 < 1$, the quantity shrinks exponentially — vanishing activations, indistinguishable numerically from zero after enough layers, carrying no discriminative signal forward. Only $n\sigma^2 = 1$ keeps the forward signal's scale roughly constant regardless of depth. The identical argument, run in reverse through the chain rule for backpropagation, shows that the gradient variance compounds by the same multiplicative factor per layer (with fan-out replacing fan-in, since gradients flow backward along the same edges used forward but summed over the outputs a unit feeds into), directly producing the Vanishing and Exploding Gradient Problem as the natural consequence of the same mechanism applied to the backward pass. ```

# Diagnostic algorithm: trace activation variance through a naive-initialized MLP function trace_variance(layer_sizes, sigma, num_layers):

# layer_sizes[l] = fan-in of layer l (n_in)

# sigma = fixed std-dev used for every layer (naive scheme) var_a = 1.0 # assume standardized input, Var(a0) = 1 for l in range(1, num_layers + 1): n_in = layer_sizes[l] var_z = n_in (sigma <strong> 2) var_a # forward variance identity var_a = var_z # approx: nonlinearity ~ variance-preserving locally record(l, var_z) return history # inspect for geometric blow-up or decay function diagnose(history): ratios = [history[l] / history[l-1] for l in range(2, len(history))] if all(r > 1.05 for r in ratios): return "EXPLODING: variance compounding by factor > 1 each layer" if all(r < 0.95 for r in ratios): return "VANISHING: variance compounding by factor < 1 each layer" return "STABLE: variance approx constant across depth" ``` This pseudocode is not a training algorithm — it is a diagnostic** any practitioner can run mentally or on paper before training: multiply fan-in by weight variance, compare to $1$, and predict the qualitative failure mode without running a single epoch.

Time and space cost of initialization itself. Initializing a layer with $n_{in}$ inputs and $n_{out}$ outputs requires drawing $n_{in} \times n_{out}$ random numbers for the weight matrix and $n_{out}$ for the bias, so the cost is $\Theta(n_{in} \cdot n_{out})$ time and space per layer — identical to the cost of merely storing the weight matrix, so initialization adds no asymptotic overhead beyond allocating the parameters that training needs anyway. Across a network with $L$ layers of width $n$, total initialization cost is $\Theta(L n^2)$, matching the parameter count exactly; this is optimal, since any initialization scheme must write at least one value per parameter, giving a matching $\Omega(Ln^2)$ lower bound.

Correctness argument for the failure modes (why they are provably bad, not just empirically bad). Zero initialization. The symmetry argument above is an exact invariant, provable by induction on the number of gradient steps $t$. Base case ($t=0$): all rows of $W^{(l)}$ within a layer are identical by construction. Inductive step: assume rows $i$ and $k$ of $W^{(l)}$ are identical after step $t$. Because they are identical, units $i$ and $k$ produce identical activations on every training example (forward pass is a deterministic function of identical weights and identical inputs). Because their activations are identical, every downstream unit's gradient with respect to unit $i$'s output equals its gradient with respect to unit $k$'s output (they are literally the same function evaluated on the same value). Therefore the gradient update $\Delta W_{i,:} = -\eta \, \partial \mathcal{L}/\partial W_{i,:}$ equals $\Delta W_{k,:}$, preserving equality of rows $i$ and $k$ at step $t+1$. By induction the rows remain equal for all $t$, so no amount of training ever differentiates units $i$ and $k$. This proves zero (or any constant) initialization caps the effective width of every hidden layer at $1$ regardless of the nominal width $n_{out}$, a structural capacity loss no learning rate or optimizer can repair. Naive random initialization. The variance-drift argument is not a strict inequality-based proof of failure in the way the symmetry argument is — for some combinations of $\sigma$ and $n$, naive initialization is fine, precisely when $n\sigma^2 \approx 1$. The correctness claim here is more subtle: it is that the failure is generic, i.e., for a fixed $\sigma$ chosen without reference to $n_{in}$, the quantity $n_{in}\sigma^2$ is an arbitrary positive real number, and the map $L \mapsto (n_{in}\sigma^2)^L$ is exponential in $L$ whenever this base differs from $1$ — so unless the practitioner has tuned $\sigma$ specifically for that layer's fan-in, drift is the expected outcome, not an edge case. This reasoning is exactly why fan-in-aware schemes (Xavier, He) exist: they solve the equation $n_{in}\sigma^2 = 1$ directly for $\sigma$, making the stable regime the default rather than a coincidence.

Honest tradeoffs. The variance analysis above rests on simplifying assumptions — that pre-activations are approximately zero-mean and that the nonlinearity is approximately variance-preserving near the origin. Real networks violate both assumptions once training begins and weights move away from their initial values; initialization only controls the first few forward and backward passes, not the entire trajectory. Nonetheless, those first few passes are precisely what determines whether learning signal exists at all when training starts, which is why the analysis, despite being approximate, has proven durable and practically predictive.

Consider a plain feedforward network with four hidden layers, each of width $n = 256$, no bias for simplicity, and a linear activation (so the variance-preserving approximation is exact, isolating the initialization effect from nonlinearity effects). Input $a^{(0)}$ is standardized so $\mathrm{Var}(a^{(0)}) = 1$. We compare three initializations for the weight standard deviation $\sigma$, all applied to the same architecture:

Each row applies the recursive identity $\mathrm{Var}(z^{(l)}) = n\sigma^2 \cdot \mathrm{Var}(z^{(l-1)})$ starting from $\mathrm{Var}(z^{(0)}) = 1$. Row 1 shows variance multiplying by $2.56$ each layer, reaching nearly $43\times$ the input scale after only four layers — with a sigmoid or tanh activation in the loop, this pushes almost every unit into the saturated tail of the nonlinearity, where local derivative $\approx 0$, so the backward pass would see gradients crushed to near zero despite the forward activations being large (a subtlety: exploding forward activations under a saturating nonlinearity often cause vanishing gradients, not exploding ones — a common point of confusion addressed in the pitfalls table below). Row 2 shows the opposite: variance shrinks by roughly $10\times$ every layer, so by layer 4 the signal is four orders of magnitude smaller than the input, numerically indistinguishable from zero in low-precision arithmetic, carrying no information for later layers or for the loss to discriminate between different inputs. Row 3, using $\sigma = 1/\sqrt{n_{in}}$ so that $n\sigma^2$ is exactly $1$, holds variance perfectly constant across all four layers regardless of depth — this is exactly the scaling that Xavier initialization generalizes (with an averaged fan-in/fan-out term) and that He initialization adapts for ReLU's asymmetric variance-halving effect. The same computation run backward — replacing fan-in with fan-out in the recursive identity, since a gradient at layer $l$ is a sum over the units in layer $l+1$ that received input from it — produces an identical qualitative table for gradient variance, confirming that the forward failure and the Vanishing and Exploding Gradient Problem are two faces of the same variance-drift phenomenon, not two separate problems.

Weight initialization strategy grew out of the broader struggle to make deep networks trainable at all, a struggle intertwined with the study of vanishing and exploding gradients in deep and recurrent architectures.

Every deep learning framework — PyTorch, TensorFlow, JAX — ships default initializers for linear, convolutional, and recurrent layers that are direct implementations of fan-in/fan-out variance analysis, because framework authors cannot assume users will hand-derive the correct $\sigma$ for every layer shape they construct. Production systems training large Convolutional Neural Networks or transformer stacks depend on this analysis at initialization time, since a poorly scaled first forward pass can produce NaN losses within the first mini-batch, long before any data-dependent debugging is possible. Recurrent architectures, where the same weight matrix is applied repeatedly across time steps, are especially sensitive, since the effective "depth" for variance-drift purposes is the sequence length, not the number of physical layers — this is a major reason gated architectures like LSTMs and careful recurrent weight initialization co-evolved with the general theory presented here. Even modern architectures that rely heavily on normalization layers (batch norm, layer norm) to reduce sensitivity to scale still specify initialization schemes derived from this variance analysis, because normalization mitigates but does not eliminate the benefit of starting from a well-scaled point, particularly at the very first forward pass before any normalization statistics have been estimated.

Ask the tutor: Can you derive, step by step, how the He initialization constant accounts for ReLU zeroing half its inputs, starting from the variance identity in this article?