A hidden layer takes as input the activations of the previous layer

PHYSICS · Multi-part · Fermi University Physics · Computational Physics · High-performance and data-driven computational physics · High-performance and data-driven computational physics

Scientific Computing for Physicists, Ch Neural networks on physical data: backpropagation, convolution and symmetry

Problem

A hidden layer takes as input the activations hi=ReLU(zi(l))h_i = \mathrm{ReLU}(z_i^{(l)}) of the previous layer, where z(l)z^{(l)} is symmetric about zero, and produces zj=∑i=1ninWijhi+bj,z_j = \sum_{i=1}^{n_{\rm in}} W_{ij} h_i + b_j, where the weights WijW_{ij} are independent, identically distributed, symmetric about zero with variance Var(W)\mathrm{Var}(W), and independent of every hih_i; the biases bjb_j are constants. ((a)) Show that Var(zj)=nin Var(W) E[hi2]\mathrm{Var}(z_j) = n_{\rm in}\,\mathrm{Var}(W)\,\mathrm{E}[h_i^2]. ((b)) Using E[ReLU(z)2]=12 Var(z)\mathrm{E}\big[\mathrm{ReLU}(z)^2\big] = \tfrac12\,\mathrm{Var}(z) for zz symmetric about zero, find the value of Var(W)\mathrm{Var}(W) that makes Var(zj)=Var(z(l))\mathrm{Var}(z_j) = \mathrm{Var}(z^{(l)}), i.e. that preserves the variance from one layer to the next. ((c)) The layer takes the 900900 ReLU outputs of a previous hidden layer of width 900900 (nin=900n_{\rm in}=900) and has nout=300n_{\rm out}=300 units. Give the standard deviation of the weights that (b) prescribes. ((d)) The backward pass sends hˉi=∑j=1noutWijzˉj\bar h_i=\sum_{j=1}^{n_{\rm out}}W_{ij}\bar z_j to the layer below, with zˉj=hˉjout ReLU′(zj)\bar z_j=\bar h^{\rm out}_j\,\mathrm{ReLU}'(z_j). Assume the incoming adjoints hˉjout\bar h^{\rm out}_j have mean zero and are independent of the weights and of whether zj>0z_j>0, and that each zjz_j is positive with probability 12\tfrac12. Find the weight variance that keeps Var(hˉi)=Var(hˉjout)\mathrm{Var}(\bar h_i)=\mathrm{Var}(\bar h^{\rm out}_j), give the standard deviation it prescribes for the layer in (c), and say when the two prescriptions agree.

Browse the problem library