Six articles, one repeating idea: keep the expensive resource busy by hiding the latency of the slower level below it. Start from the silicon — SIMT, warps, the memory hierarchy. Learn the rules the hardware rewards — coalescing, shared memory, occupancy. Get a repeatable optimization method — the Roofline model and the modern toolchain (Nsight, CUTLASS, TensorRT, Triton, Helion). Then scale out — multi-GPU model parallelism with Megatron-LM, and the whole datacenter (scheduling, virtualization, multi-tenancy). Plain language, deep technical detail, grounded in the canonical books, NVIDIA/AMD whitepapers, and the research literature.
The GPU from the silicon up: why it's a throughput machine, not a fast CPU, and the rules that machine rewards — warps, the SM, the memory hierarchy, coalescing, shared memory, and occupancy.
A GPU hides memory latency by keeping thousands of threads in flight. Threads run in lock-step warps of 32; divergence is a per-warp cost; the SM switches warps for free. The memory hierarchy, and how it maps onto Ampere, Hopper, RDNA & CDNA.
The three rules that decide kernel speed: coalesce a warp's 32 reads into one transaction, stage reused data in shared memory (and dodge bank conflicts with the [N][N+1] trick), and keep enough warps resident to hide the rest.
Stop guessing. Diagnose the bottleneck with the Roofline model, match it to a technique family, and use the modern toolchain to measure what's slow and write fast kernels without raw PTX.
Arithmetic intensity and the Roofline tell you whether you're memory- or compute-bound — which decides which technique family (the Hijma taxonomy) can help. Fusion, the Occupancy Calculator, mixed precision, and a worked optimization session.
Nsight Systems & Compute to see what's slow; CUTLASS, TensorRT, Triton & Helion to write fast kernels without raw PTX. The abstraction ladder and how to pick the right rung.
Past one GPU, communication is the bottleneck. Map parallelism to the interconnect, train trillion-parameter models with Megatron-LM, and keep a fleet of the most expensive compute on earth from sitting half-idle.
Data, tensor, pipeline & sequence parallelism; Megatron-LM's two-all-reduces-per-layer design; the NCCL collectives; the NVLink-vs-InfiniBand rule; and why synchronization and access overhead — not raw compute — decide scaling.
Why a $40M cluster sits half-idle, and the two fixes: scheduling (the Wei Gao taxonomy) and sharing (MIG, MPS, time-slicing, kernel interception) — under the hard constraints of isolation and predictable performance.