Pipeline Parallelism

Pipeline parallelism is a model-parallel training and inference strategy that partitions a neural network's layers across multiple accelerators and processes micro-batches in a staged, assembly-line fashion to maximize device utilization. It is essential for training models whose parameter counts exceed single-device memory capacity—the dominant constraint for frontier LLMs such as GPT-4, PaLM-2, and Megatron—and enables near-linear scaling of throughput with device count when tuned correctly. A counterintuitive fact: naïve pipeline parallelism can achieve lower hardware utilization than a single GPU due to pipeline bubbles, yet with careful micro-batch scheduling the bubble fraction approaches zero asymptotically as the number of micro-batches grows.

The core insight

The fundamental insight is that a deep neural network is a sequential computation graph that can be cut into contiguous layer-slabs, each assigned to a different device, so that while device $k$ processes the forward pass of micro-batch $m$ , device $k-1$ can simultaneously process micro-batch $m+1$ —turning serial latency into parallel throughput. The "aha moment" is recognizing that temporal overlap between stages is only possible because micro-batches are independent of one another, so the only synchronization required is the activation tensor passed between adjacent stages. The cost of this independence is a pipeline bubble—idle time at startup and drain—whose relative overhead shrinks as the number of micro-batches per batch increases. Minimizing bubble fraction and activation memory simultaneously is the central engineering tension in all modern pipeline-parallel systems.

Common trapCommon Misconception

Practitioners frequently conflate pipeline parallelism with data parallelism or assume they are interchangeable: "both run the same layer on multiple devices, so they must be equivalent." This is wrong in a precise technical sense. Data parallelism replicates the entire model on each device and shards the data; every device performs a full forward-backward pass and gradients are all-reduced across replicas. Pipeline parallelism instead shards the model along the depth dimension across devices, and each device holds only a subset of layers; activations (not gradients of weights) are communicated between adjacent stages. The correct mental model is a factory assembly line versus a set of identical factories running in parallel: assembly lines require coordination between stations (activation hand-offs) and suffer start/stop overhead (bubble), but use far less total memory per station.

How It Works

Formal Setup

Let a neural network $f = f_S \circ f_{S-1} \circ \cdots \circ f_1$ be partitioned into $S$ stages, where stage $s$ consists of consecutive layers $\{l_{a_s}, \ldots, l_{b_s}\}$ assigned to device $d_s$ . A global mini-batch $\mathcal{B}$ of size $B$ is split into $M$ micro-batches $\{\mu_1, \ldots, \mu_M\}$ each of size $B/M$ .

Throughput model. Define the per-stage compute time as $t_s$ (assumed uniform $t_s = t$ for analysis). The ideal pipeline throughput for $M$ micro-batches across $S$ stages is:

\quad T_{\text{ideal}} = M \cdot t

Explanation: In a perfectly filled pipeline, total useful compute scales linearly with the number of micro-batches, independent of stage count.

Bubble time. Pipeline startup requires filling $S-1$ stages; drain requires another $S-1$ steps. Total wall-clock time is:

\quad T_{\text{wall}} = (M + S - 1)\cdot t

Explanation: The actual wall time equals the ideal time plus the $(S-1)$ bubble steps that are unavoidable at the beginning and end of every batch.

Bubble fraction. The fraction of time all devices are idle (the pipeline bubble ratio) is:

\quad \phi = \frac{T_{\text{wall}} - T_{\text{ideal}}}{T_{\text{wall}}} = \frac{S - 1}{M + S - 1}

Explanation: As $M \to \infty$ the bubble fraction vanishes, but large $M$ inflates peak activation memory, creating the fundamental micro-batch count trade-off. Here $S$ is the number of pipeline stages, $M$ the number of micro-batches, and $t$ the per-micro-batch per-stage latency.

1F1B Schedule

The one-forward-one-backward (1F1B) schedule (Narayanan et al., 2019) interleaves forward and backward passes to cap the number of in-flight activations at $S$ rather than $M$ . In 1F1B, once the pipeline is full, each device alternates executing one forward pass and one backward pass per clock cycle. Activation memory per device is bounded:

\quad \mathcal{M}_{\text{1F1B}} = S \cdot \frac{B}{M} \cdot A_s

Explanation: Only $S$ micro-batches have outstanding (unreleased) activations at any moment, so memory scales with stage count rather than total micro-batch count. Here $A_s$ is the activation tensor size for one micro-batch at stage $s$ .

Interleaved Stages (Virtual Stages)

Narayanan et al. (2021) introduced interleaved pipeline parallelism, assigning each device $V$ non-contiguous virtual stage chunks. This reduces bubble fraction by factor $V$ :

\quad \phi_{\text{interleaved}} = \frac{1}{V}\cdot\frac{S-1}{M+S-1}

Explanation: By interleaving $V$ virtual stages per device, each device stays active more of the time, shrinking the relative bubble overhead at the cost of $V\times$ more point-to-point communications per micro-batch. The trade-off: inter-device communication volume scales as $\mathcal{O}(V \cdot S \cdot B/M \cdot d_{\text{model}})$ where $d_{\text{model}}$ is the hidden dimension, so $V$ cannot be made arbitrarily large without saturating interconnect bandwidth.

researcher · Deepak Narayanan | b. 1993 | GPipe and Megatron-LM Pipeline Scheduling

Deepak Narayanan, a PhD student and later researcher at Stanford and Microsoft Research, was the primary architect of the Megatron-LM pipeline parallelism infrastructure, introducing both the 1F1B schedule and interleaved virtual-stage scheduling that became the de facto standard for training hundred-billion parameter models. His 2019 PipeDream paper and 2021 Megatron-LM v2 paper together resolved the memory-bubble tension that had made earlier systems like GPipe impractical for very deep models. Remarkably, the interleaved schedule paper was developed concurrently with the Megatron-Turing NLG 530B training run, meaning the scheduling theory was being finalized while the largest model of its time was being trained on it.

Figure

Pipeline Parallelism: 1F1B Schedule

Gradient Staleness and Weight Update Semantics

A subtle correctness issue: in the 1F1B schedule, the backward pass for micro-batch $\mu_m$ is computed with the same weight snapshot used for its forward pass (weight version at step $t$ ), but in async variants (PipeDream, 2019) later micro-batches may begin their forward pass with a more recent weight snapshot. The weight staleness for micro-batch $m$ in the async case is:

\quad \delta_m = \theta_{t + \lfloor m/S \rfloor} \quad \text{vs.} \quad \theta_t \text{ (synchronous)}

Explanation: Each micro-batch in the asynchronous regime uses weights that are $\lfloor m/S \rfloor$ optimizer steps newer than the batch-start weights, introducing a weight-version mismatch between forward and backward passes that can bias gradient estimates. This motivates the weight stashing technique (PipeDream-Flush, Narayanan et al., 2021): devices cache the weight version used during each micro-batch's forward pass and use the identical version during backprop, eliminating gradient staleness at the cost of $S$ weight copies per device.

researcher · Yanping Huang | b. 1985 | GPipe: Gradient Checkpointing for Pipeline Parallelism

Yanping Huang, a research scientist at Google Brain, led the GPipe project (Huang et al., NeurIPS 2019) which was the first large-scale demonstration that pipeline parallelism combined with re-materialization (gradient checkpointing) could train otherwise-infeasible models—specifically AmoebaNet-D with 557M parameters on image classification. GPipe's key contribution was trading compute for memory by discarding intermediate activations during the forward pass and recomputing them during backprop, bounding activation memory at $\mathcal{O}(S \sqrt{L/S})$ rather than $\mathcal{O}(L)$ for $L$ layers. Interestingly, GPipe's name was chosen before the project scope was defined—the team named it after Pipe and only later decided it should stand for "Giant Pipeline."

Figure

3D Parallelism: Pipeline + Tensor + Data