The temperature of a rod of length is evolved by the explicit (forward-time

PHYSICS · Multi-part · Fermi University Physics · Computational Physics · High-performance and data-driven computational physics · High-performance and data-driven computational physics

Scientific Computing for Physicists, Ch Many processors: speed-up, decomposition and the cost of communication

Problem

The temperature of a rod of length

L=1\,\mathrm{m}

is evolved by the explicit (forward-time, centred-space) diffusion scheme from the first course on

N_g=2\times10^6

grid points. The rod (not periodic) is decomposed into

P

equal contiguous blocks of

N_g/P

points, one per process placed one per core; before every step, each process exchanges one ghost (halo) point with each neighbour it has (an interior process has two neighbours, the two end processes have one each). Ping-pong measurements on the cluster give message latency

\alpha_{\rm lat}=1.0\,\mu\mathrm{s}

and bandwidth

\beta_{\rm net}=8\,\mathrm{GB\,s^{-1}}

, related to the message time by

t_{\rm msg}(m)=\alpha_{\rm lat}+m/\beta_{\rm net}

; each halo message carries one 8-byte float64 value. Updating one grid point (reading its neighbours, writing the new value) takes

t_{\rm cell}=5.0\,\mathrm{ns}

per process, independent of

P

. The best serial (undecomposed) code takes

T_1=10.0\,\mathrm{ms}

per step. ((a)) Describe, in words or pseudocode, the exchange-then-update loop each process runs every step, and state an ordering of the sends and receives that cannot deadlock, naming what can go wrong with the naive ordering "every process sends to both neighbours before it receives from either". ((b)) Compute the time for one halo message,

t_{\rm msg}

, and the total communication time an interior process adds to a step by exchanging with both of its neighbours. ((c)) Derive the number of grid points per process,

n^{*}

, at which this communication time equals the computation time

t_{\rm cell}\,n^{*}

of updating that share of the rod, and the process count

P^{*}=N_g/n^{*}

at which the whole rod first reaches this crossover. ((d)) Measured strong-scaling step times are

T_P=0.64\,\mathrm{ms}

P=16

and

T_P=0.19\,\mathrm{ms}

P=64

. Compute the strong-scaling efficiency

E_s(P)=T_1/(PT_P)

at both process counts, and say, using the crossover from part (c), whether the fall in efficiency by

P=64

is explained by the communication model or points to something else.

Browse the problem library