Loading lesson…
GPUs & Hardware / Fitting & Scaling
Splitting a model and its work across many GPUs.
Reviewed by Yuvaraj
When a model, or even a single training batch, will not fit inside one GPU's VRAM, or when it simply trains too slowly on one device, the work is spread across many accelerators. But splitting is not free. The moment computation lives on more than one device, those devices must communicate, and on modern hardware it is that communication, not the arithmetic, that usually becomes the bottleneck. Floating-point throughput has grown far faster than interconnect bandwidth, so the real question in every parallelism scheme is: what must the GPUs exchange, and how often? The three classic axes, data, tensor, and pipeline parallelism, differ precisely in what they split and therefore what they are forced to send over the wire.
The simplest axis. Replicate the entire model on every GPU, then split the batch across them. Each device runs a full forward and backward pass on its own shard of the data and produces gradients for the complete parameter set. Because every replica saw different examples, those gradients differ, so before the optimizer step all devices average their gradients via an all-reduce. After the all-reduce every replica holds identical averaged gradients, applies the same optimizer update, and stays in lock-step.
Data parallelism is the right first move when the model already fits but you want to chew through more data per second. Its limit is memory: it buys throughput, not capacity. Sharded variants, ZeRO and FSDP, lift that limit by partitioning the optimizer state, then the gradients, then the parameters themselves across the data-parallel group, gathering each shard on demand. You trade extra communication for the ability to train a model far larger than any single device could hold.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Here we split inside a layer. A single large weight matrix is partitioned across GPUs so that one matrix multiply is computed collaboratively, each device multiplies its slice of the weights against the input and produces a partial result. Those partials are then combined with an all-reduce (or all-gather) within the layer to reconstruct the correct output before the next layer runs.
Reach for tensor parallelism when an individual layer is too large to hold on one GPU, or when you need to cut both per-GPU memory and per-token latency by dividing the arithmetic itself.
This axis splits the model by depth. Different stages, contiguous groups of layers, are placed on different GPUs, and activations are handed from one stage to the next like work moving down an assembly line: GPU 0 runs layers 1–8, passes its output to GPU 1 for layers 9–16, and so on.
Pipeline parallelism shines for very deep models spread across multiple nodes, where the inter-node interconnect is slower and its light, boundary-only communication is exactly what you want.
For a curious beginner
How it is actually used
The underlying mechanism
| Axis | What it splits | What it communicates | Main limit / cost |
|---|---|---|---|
| Data parallelism | The batch (data), model replicated in full | Gradients, once per step via all-reduce (∝ parameter count) | Whole model must fit on one GPU (unless sharded via ZeRO/FSDP) |
| Tensor parallelism | The weight matrices inside a layer | Activations, several times per layer (all-reduce) | Very bandwidth-hungry; keep within one fast-interconnect node |
| Pipeline parallelism | The layer stack, by depth (stages) | Activations, only at stage boundaries | Pipeline bubble; needs many microbatches to stay full |
Large-scale training rarely picks one axis, it composes all three. Tensor parallelism runs inside a node across GPUs joined by fast links; pipeline parallelism spans across nodes where the interconnect is slower and only boundary activations travel; and data parallelism sits on top of that arrangement to replicate the whole tensor-plus-pipeline group and scale raw throughput. This layered arrangement, matching each axis's communication pattern to the bandwidth available at that level of the hardware hierarchy, is how frontier models are trained across thousands of GPUs.
Common mistakes