A fully connected network with ten hidden layers of logistic-sigmoid units
PHYSICS · Multi-part · Fermi University Physics · Mathematical Methods for Physics · Computational Physics · Calculus of change and accumulation
Scientific Computing for Physicists, Ch Neural networks on physical data: backpropagation, convolution and symmetry
Problem
A fully connected network with ten hidden layers of logistic-sigmoid units, initialised with small weights, is trained one step on Ising configurations. The measured Euclidean norms of the weight-gradient at each layer (layer 1 nearest the input, layer 10 nearest the output) are: Measured gradient norms ∥Wˉ(l)∥ for layers 1 to 10 are, in order: 9.1×10−8, 4.4×10−7, 2.1×10−6, 1.0×10−5, 4.8×10−5, 2.3×10−4, 1.1×10−3, 5.2×10−3, 2.5×10−2, 1.2×10−1.
((a)) Compute the ratio ∥Wˉ(1)∥/∥Wˉ(10)∥.
((b)) The logistic sigmoid's derivative never exceeds 1/4. Count the sigmoid-derivative factors the adjoint picks up between the weight gradient of layer 10 and that of layer 1, write down the bound they set on the ratio in (a), and compare the two. What else in the backward pass could account for the difference?
((c)) The first hidden layer has nin=256 inputs. If the network were rebuilt with ReLU units and the variance-preserving initial weight variance derived for ReLU layers, what value of Var(Wij) should be used for that layer?