A chemical concentration diffuses around a thin circular hoop (periodic boundary…

PHYSICS · Multi-part · Fermi University Physics · Computational Physics · High-performance and data-driven computational physics · High-performance and data-driven computational physics

Scientific Computing for Physicists, Ch Many processors: speed-up, decomposition and the cost of communication

Problem

A chemical concentration diffuses around a thin circular hoop (periodic boundary conditions), evolved by the explicit (forward-time, centred-space) diffusion scheme from the first course. The ring of grid points is decomposed into

P

equal contiguous blocks, one per process, arranged in a ring topology: process

k

exchanges one halo point with process

k-1

and one with process

k+1

(indices modulo

P

), so every process has exactly two neighbours and there is no end process. Processes are placed

16

per compute node. Ping-pong measurements give

\alpha_{\rm lat}=0.8\,\mu\mathrm{s}

and

\beta_{\rm net}=10\,\mathrm{GB\,s^{-1}}

, with

t_{\rm msg}(m)=\alpha_{\rm lat}+m/\beta_{\rm net}

; each halo message carries one 8-byte float64 value. A weak-scaling run keeps

n=5\times10^4

grid points per process fixed as

P

grows; the best serial per-step time for that fixed per-process problem size is

T_1=0.30\,\mathrm{ms}

. Measured step times at fixed

n

are

T_P=0.38\,\mathrm{ms}

P=4

T_P=0.81\,\mathrm{ms}

P=16

, and

T_P=0.83\,\mathrm{ms}

P=64

. ((a)) Suppose the library holds no message in a buffer, so that a blocking send cannot finish until the matching receive has been posted. In the first half of each exchange, every process

k

sends its edge value to process

k+1

and only then receives from process

k-1

. Explain why this ordering always deadlocks the ring, while a non-periodic line of

P

processes using the same ordering does not. State one fix that removes the deadlock risk in both topologies. ((b)) Compute the time for one halo message,

t_{\rm msg}

, and the total communication time an interior process adds to a step by exchanging with both of its ring neighbours. ((c)) Using the weak-scaling data given, compute the weak-scaling efficiency

E_w(P)=T_1/T_P

P=4

P=16

and

P=64

. ((d)) Using the pattern of

E_w(P)

from part (c) and the fact that processes share one node's memory bandwidth

16

at a time, state at which of the three process counts the run first shows the node's bandwidth ceiling, and explain why

E_w

then stays roughly flat rather than continuing to fall as

P

grows further at fixed

n

Browse the problem library