Multi-GPU & model parallelism.

One GPU is not enough for big models, so you split work across many — and the moment you do, a new bottleneck appears: communication. There are four ways to split (data, tensor, pipeline, sequence parallelism), each with a different communication pattern, and which combination works depends entirely on your interconnect (fast NVLink inside a node, slower InfiniBand between nodes). Megatron-LM is the reference implementation. The recurring theme — and the lesson from the multi-GPU Python work of Oden & Nölp — is that synchronization and memory-access overheads, not raw compute, decide how well you scale.

Everything so far in this series has been about making one GPU fast. But a frontier model's weights do not fit on one GPU — GLM-5.2 and DeepSeek V4 are hundreds of billions to over a trillion parameters — and even when they fit, you want to train or serve them faster than one device allows. So you reach for many GPUs. The catch: as soon as work spans multiple devices, they must talk to each other, and that communication is slow compared to on-chip compute. The entire discipline of multi-GPU programming is managing that communication so it does not erase the benefit of the extra hardware.

The new bottleneck: the interconnect

Recall the memory hierarchy from article 1.1. Multi-GPU adds two more, much slower, levels at the bottom:

Link	Scope	Approx. bandwidth	Relative speed
HBM (on-device)	within a GPU	2–8 TB/s	baseline
NVLink / NVSwitch	GPUs in one node	~400–900 GB/s	~5–10× slower than HBM
PCIe 5.0	GPU ↔ CPU / GPU ↔ GPU (no NVLink)	~64 GB/s	~50–100× slower
InfiniBand / Ethernet	between nodes	~25–100 GB/s per link	~100× slower

This table is the master constraint of distributed training. Communication that stays inside a node over NVLink is an order of magnitude cheaper than communication that crosses between nodes over InfiniBand. Every parallelism decision below is really a decision about which data crosses which link. The art is putting the chattiest communication on the fastest link.

The four ways to split work

1. Data parallelism — replicate the model, split the batch

The simplest and most common. Every GPU holds a full copy of the model and processes a different slice of the input batch. After each backward pass, the GPUs must average their gradients so all replicas stay identical — an all-reduce across all devices. Data parallelism scales throughput beautifully as long as the model fits on one GPU and the gradient all-reduce does not dominate. Its limit is memory: it does nothing to help a model too big for a single device.

The modern refinement is sharded data parallelism (ZeRO / FSDP), which splits the optimizer states, gradients, and even parameters across the data-parallel group instead of replicating them — trading extra communication for a large memory saving, which lets data parallelism handle much bigger models than naive replication.

2. Tensor parallelism — split each layer across GPUs

When a single layer is too big or you want to cut per-layer latency, tensor parallelism splits the math of one layer across GPUs. A big matrix multiply Y = XW is partitioned by splitting W into column blocks; each GPU computes part of the output, and the partial results are combined with an all-reduce or all-gather. This is the core idea of Megatron-LM (below). Because it requires an all-reduce within every layer, tensor parallelism is extremely communication-heavy and is therefore almost always confined inside a single node, where NVLink can absorb the traffic.

3. Pipeline parallelism — split the layers across GPUs

Put layers 1–10 on GPU 0, layers 11–20 on GPU 1, and so on — the model is a pipeline of stages. A batch flows through stage by stage. The problem is the pipeline bubble: while GPU 0 works on the first layers, GPU 1 sits idle waiting for GPU 0's output, and vice versa. The fix is micro-batching: split the batch into many micro-batches and keep them flowing so all stages stay busy, the way a CPU instruction pipeline overlaps stages. Pipeline parallelism communicates only the activations at stage boundaries — far less than tensor parallelism — so it tolerates the slower inter-node links and is used to span nodes.

4. Sequence / context parallelism — split the sequence

For very long contexts, the activations and attention computation can be split along the sequence dimension across GPUs. This is increasingly important as context windows reach hundreds of thousands of tokens (the KV-cache pressure from the DeepSeek series), and it adds its own communication pattern for the attention step.

Megatron-LM: the reference for tensor parallelism

NVIDIA's Megatron-LM is the canonical implementation of tensor (and combined) parallelism for transformers, and its design choices are worth knowing because nearly every large-model training stack borrows them. Its central trick is splitting the transformer's two big sublayers — the MLP and the multi-head attention — across GPUs with the minimum possible synchronization:

• MLP block. The first linear layer's weight is split by columns (so each GPU produces a slice of the hidden activation with no communication), the nonlinearity is applied locally, and the second linear is split by rows so that a single all-reduce at the end recombines the result. Two matrix multiplies, one communication.
• Attention block. Different attention heads go to different GPUs (heads are independent), and again a single all-reduce recombines the output projection. The natural head-parallelism of multi-head attention maps cleanly onto tensor parallelism.

The result is just two all-reduces per transformer layer (one for attention, one for MLP) in the forward pass. Megatron also introduced sequence parallelism to split the parts of the layer (layernorm, dropout) that tensor parallelism leaves replicated, shaving more memory. Combined with pipeline and data parallelism, Megatron's "3D parallelism" is how models with hundreds of billions of parameters are trained on thousands of GPUs.

Collective communication: NCCL and the operations that matter

All four parallelism styles are built from a small set of collective operations, implemented for NVIDIA GPUs by NCCL (the NVIDIA Collective Communications Library). Knowing the collectives is knowing the vocabulary of distributed GPU work:

Collective	What it does	Used for
all-reduce	Every GPU ends with the sum (or mean) of all GPUs' inputs	Gradient averaging (DP); tensor-parallel layer outputs
all-gather	Every GPU ends with the concatenation of all inputs	Reassembling sharded params (FSDP); tensor parallel
reduce-scatter	Sum across GPUs, each keeps one shard of the result	Sharded gradient reduction (ZeRO/FSDP)
broadcast	One GPU's data copied to all	Distributing weights at startup
point-to-point send/recv	One GPU to one GPU	Pipeline stage boundaries

NCCL is topology-aware: it knows whether two GPUs are connected by NVLink, PCIe, or InfiniBand and chooses ring or tree algorithms accordingly. Most of the time you never call NCCL directly — PyTorch DDP/FSDP, DeepSpeed, and Megatron issue the collectives for you — but understanding which collective each parallelism style triggers is how you reason about where the time goes.

The real lesson: synchronization & access overhead dominate

It is tempting to think distributed training is about adding more compute. In practice, as the work of Oden & Nölp on efficient multi-GPU programming in Python shows, the things that actually decide your scaling efficiency are synchronization and data-access overhead — not how many FLOP/s you added. Their findings generalize well beyond Python:

• Synchronization is the silent tax. Every barrier and every collective forces GPUs to wait for the slowest participant. A single straggler — one slow GPU, one congested network link — stalls the entire group. Reducing the number and frequency of synchronization points often matters more than speeding up any individual GPU.
• Implicit data movement is expensive and easy to trigger accidentally. In high-level multi-GPU Python, an innocuous-looking operation can silently copy a large array across PCIe or between devices. Reducing these implicit transfers — by keeping data resident on the device that needs it and being explicit about placement — was where Oden & Nölp found the largest speedups.
• Overlap communication with computation. The best distributed code never waits idly for a collective to finish. It launches the communication asynchronously and keeps computing other work meanwhile, hiding the transfer behind compute — the exact same latency-hiding principle as warp scheduling in article 1.1, now at cluster scale. (This is precisely the idea behind the KOG article's "Delayed Tensor Parallelism," which hides the all-reduce behind several layers of compute.)

Scaling efficiency: the number that matters

The metric for all of this is scaling efficiency: if one GPU does work at rate R, do N GPUs reach N×R? Perfect (linear) scaling is the goal; reality falls short because of the communication and synchronization above.

• Strong scaling — fixed problem size, more GPUs. Hardest, because per-GPU work shrinks while communication stays, so the communication fraction grows and efficiency drops. Amdahl's law in hardware form.
• Weak scaling — grow the problem with the GPU count (e.g. bigger batch or model). Easier to keep efficient, because each GPU stays well-fed. Large-model training is mostly a weak-scaling story.

A well-tuned 3D-parallel training run on thousands of GPUs can hold scaling efficiency in the 50–70% range — and getting there is almost entirely about the communication management in this article, not about faster kernels. Past a point, the kernels are fine; the network is the frontier.

Practical guidance

FAQ

If I have 8 GPUs in one server, do I need any of this, or does PyTorch just handle it?

For a model that fits on one GPU, DistributedDataParallel handles 8-GPU data parallelism with a few lines and you get near-linear scaling over NVLink. You only reach for tensor/pipeline parallelism when the model itself does not fit, or when you need lower per-step latency than data parallelism alone provides.

What's the difference between DDP and FSDP?

DDP replicates the full model on every GPU and all-reduces gradients — simple, but every GPU needs to hold the whole model plus optimizer states. FSDP (Fully Sharded Data Parallel) shards parameters, gradients, and optimizer states across the group, gathering each layer's params just-in-time for its forward/backward. FSDP uses more communication but far less memory, letting data parallelism scale to models that would never fit under DDP.

Why is tensor parallelism limited to ~8 GPUs in practice?

Because its per-layer all-reduce is so communication-intensive that it only stays efficient within a single NVLink/NVSwitch domain — which is typically 8 GPUs in a node. Beyond that you would be doing per-layer collectives over InfiniBand, and scaling falls off a cliff. So tensor parallelism handles the "within a node" axis; pipeline and data parallelism handle "across nodes."

Does inference need model parallelism too?

Yes — a model whose weights exceed one GPU's memory must be sharded for inference as well, usually with tensor parallelism inside a node (as in vLLM/TensorRT-LLM). The communication pressure is lower than training (no gradient all-reduce), but the same NVLink-domain rule applies. The KOG article is an extreme case study of inference-time tensor parallelism.

Is the pipeline bubble ever fully eliminated?

Never fully, but micro-batching and clever schedules (interleaved/1F1B) shrink it to a small fraction. The bubble is fundamental: at the start and end of each batch, some stages have no work. More micro-batches amortize the fixed bubble over more useful work, which is why pipeline parallelism wants many micro-batches in flight.

Takeaways

• Past one GPU, communication is the bottleneck, and the interconnect hierarchy (HBM ≫ NVLink ≫ PCIe ≫ InfiniBand) governs every decision.
• Four ways to split: data (replicate, split batch), tensor (split each layer), pipeline (split depth), sequence (split context).
• Megatron-LM does tensor parallelism with just two all-reduces per layer — and must stay inside an NVLink domain.
• Collectives (all-reduce, all-gather, reduce-scatter) via NCCL are the vocabulary; each parallelism style triggers specific ones.
• Synchronization and implicit data movement — not raw compute — decide scaling efficiency (Oden & Nölp).
• Overlap communication with compute — the same latency-hiding idea as warp scheduling, at cluster scale.
• Reach for the smallest parallelism that fits; map each axis to the link that suits its communication cost.

The final article, 3.2, zooms out one more level — from a training job to the whole datacenter: how GPU clusters are scheduled, virtualized, and shared safely between many tenants, and why a $40M cluster can sit at 50% utilization.

References & further reading

• Shoeybi et al. — Megatron-LM: Training Multi-Billion Parameter Models Using Model Parallelism — the tensor-parallel reference.
• Narayanan et al. — Efficient Large-Scale Training on GPU Clusters Using Megatron-LM — 3D parallelism and scaling to thousands of GPUs.
• Rajbhandari et al. — ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — sharded data parallelism.
• Oden & Nölp — Efficient Multi-GPU Programming in Python: Reducing Synchronization and Access Overheads — the synchronization/access lesson.
• NVIDIA NCCL documentation — the collective operations and topology awareness.