The reverse-mode example is extended by inserting one hidden ReLU unit between the…

PHYSICS · Multi-part · Fermi University Physics · Computational Physics · High-performance and data-driven computational physics · High-performance and data-driven computational physics

Scientific Computing for Physicists, Ch Neural networks on physical data: backpropagation, convolution and symmetry

Problem

The reverse-mode example L=(wx+b−y)2\mathcal{L} = (wx+b-y)^2 is extended by inserting one hidden ReLU unit between the input and the output. The computation graph is z1=w1x+b1,h=ReLU(z1),z_1 = w_1 x + b_1, \qquad h = \mathrm{ReLU}(z_1), y^=w2h+b2,L=(y^−y)2,\hat{y} = w_2 h + b_2, \qquad \mathcal{L} = (\hat{y}-y)^2, where xx is the input, yy the target, w1,b1,w2,b2w_1, b_1, w_2, b_2 the four parameters, and ReLU(z)=max⁡(0,z)\mathrm{ReLU}(z)=\max(0,z). ((a)) Starting from Lˉ=1\bar{\mathcal{L}}=1, derive in order the adjoints y^ˉ\bar{\hat y}, wˉ2\bar w_2, bˉ2\bar b_2, hˉ\bar h, zˉ1\bar z_1, wˉ1\bar w_1, bˉ1\bar b_1, using uˉ=∑v:u→vvˉ ∂v∂u\bar u = \sum_{v: u \to v} \bar v\,\dfrac{\partial v}{\partial u}. ((b)) Evaluate every adjoint found in (a) numerically for x=1.5x=1.5, y=1.0y=1.0, w1=2.0w_1=2.0, b1=−1.0b_1=-1.0, w2=0.5w_2=0.5, b2=0.2b_2=0.2. ((c)) Check wˉ1\bar w_1 by differentiating L\mathcal{L} directly with respect to w1w_1 at the same point (note z1>0z_1>0 there, so the ReLU is on its linear branch).

Browse the problem library