A GPU cluster is a fleet of the most expensive compute on earth, and the central problem is utilization: huge clusters routinely sit at 40–60% because jobs queue for whole GPUs they barely use. Two levers fix it. Scheduling decides which job runs where and when (the Wei Gao taxonomy maps the design space). Sharing lets multiple workloads use one GPU at once — via partitioning (MIG), process multiplexing (MPS), time-slicing, or kernel-level interception — with the hard constraints being isolation (one tenant must not hurt another) and performance predictability. This is where GPU programming meets systems and economics.
The series has climbed from a single warp to a multi-node training job. The last level is the whole datacenter: thousands of GPUs shared by many teams and many jobs. At this scale the question is no longer "is my kernel fast" but "is this $40-million cluster actually being used." The answer is depressingly often "not much" — and recovering that wasted capacity is one of the highest-value problems in the field, because every idle GPU-hour is money burned. This article covers the two mechanisms that fight idleness: scheduling and sharing.
Why utilization is the problem
Deep-learning clusters are notoriously underutilized, and the reasons are structural, not careless:
- Whole-GPU allocation. The default unit of allocation is an entire GPU. A job that needs 8 GB of an 80 GB GPU, or that keeps the device busy 20% of the time, still holds the whole device — the other 90% of memory and 80% of compute sit idle but unavailable to anyone else.
- Gang scheduling. A distributed job needing 64 GPUs cannot start until all 64 are free simultaneously. So GPUs sit reserved-but-idle waiting for the gang to assemble — the "fragmentation" problem.
- Bursty, interactive workloads. Inference and notebook/interactive jobs have spiky demand; provisioning for the peak leaves the trough empty.
- Heterogeneous, evolving demand. A mix of huge training jobs, small fine-tunes, and latency-sensitive inference competes for the same pool, and any static partition is wrong most of the time.
Wei Gao et al.'s survey of deep-learning workload scheduling in GPU datacenters frames the whole field around this: the scheduler's job is to maximize cluster utilization and throughput while honoring fairness and the very different needs of training (throughput, long-running, gang) versus inference (latency, bursty, SLO-bound).
The scheduling taxonomy
Following the Wei Gao taxonomy, datacenter GPU schedulers vary along a few key axes. Understanding the axes is understanding why no single scheduler wins:
| Axis | Choices | Trade-off |
|---|---|---|
| Objective | utilization · fairness · job-completion-time · SLO/deadline | Maximizing one usually costs another. |
| Preemption | non-preemptive vs preemptive (checkpoint & requeue) | Preemption raises utilization but adds checkpoint overhead. |
| Job knowledge | black-box vs profiled (knows the job's scaling curve) | Profiling enables smarter packing but needs instrumentation. |
| Placement | consolidated (locality-aware) vs spread | Locality cuts communication (Phase 3.1!); spreading aids fault tolerance. |
| Elasticity | fixed vs elastic (grow/shrink GPU count mid-run) | Elastic fills gaps but needs the job to support reconfiguration. |
Two ideas from the literature recur because they directly attack the utilization problem:
- Topology / locality-aware placement. A scheduler that knows the interconnect (article 3.1) places a tensor-parallel group's GPUs within one NVLink domain and a job's GPUs close on the network fabric. Ignoring topology means putting chatty workers across slow links — the scheduler can hand you a scaling disaster before your code runs a single kernel.
- Preemption and elasticity for filling gaps. Systems like Gandiva, Tiresias, Themis, and Pollux (representative of the research line Gao surveys) use job profiling, preemption, and elastic resource allocation to pack the cluster tighter — checkpointing a low-priority training job to free GPUs for an urgent one, or growing a job to use idle capacity and shrinking it when demand returns.
Sharing one GPU: the four mechanisms
Scheduling decides which whole GPUs go to which job. But the deepest waste is within a single GPU — the job that uses 10% of it. GPU sharing (a.k.a. multi-tenancy or GPU virtualization) lets several workloads use one physical GPU at once. There are four mechanisms, from hardware-rigid to software-flexible, and they trade isolation against flexibility:
Fig 1 — The sharing spectrum. Hardware partitioning gives the strongest isolation but the least flexibility; software interception gives the finest-grained packing but must build isolation itself.
1. MIG (Multi-Instance GPU) — hardware partitioning
On Ampere and later data-center GPUs, MIG physically partitions one GPU into up to seven isolated instances, each with its own dedicated slice of SMs, L2 cache, and memory. The isolation is hardware-enforced: one instance cannot touch another's memory or steal its compute, and a crash in one does not affect the others. This makes MIG ideal for multi-tenant clouds and for guaranteeing predictable performance. The cost is rigidity — the partitions are fixed sizes set at configuration time, so you cannot dynamically give a busy tenant more than its slice, and unused capacity in one slice cannot be lent to another.
2. MPS (Multi-Process Service) — spatial multiplexing
MPS lets multiple processes submit work to one GPU concurrently, sharing a single GPU context so their kernels can run side by side on different SMs rather than time-slicing. This raises utilization when individual processes each under-fill the GPU (common for inference). The catch: isolation is weak. Processes share memory space, so one can exhaust memory or, in older versions, a fault in one could affect others. MPS is great for trusted co-located workloads (e.g. several inference replicas you own); it is not a security boundary between untrusted tenants.
3. Time-slicing — temporal multiplexing
The simplest software sharing: the GPU round-robins between workloads, each getting the whole device for a time quantum. Easy and universally supported (it is how a single display GPU serves many apps), but it provides no memory isolation (workloads must fit together or evict each other) and adds context-switch overhead. It suits low-intensity, bursty, latency-tolerant workloads where each only needs the GPU occasionally.
4. Kernel-space interception — software-defined sharing
The most flexible and the most active research area. A software shim sits between the application and the GPU driver, intercepting CUDA/kernel launches and API calls. By controlling which kernels are submitted and when, it can enforce fine-grained limits on compute and memory per tenant, throttle one workload to protect another's latency, and oversubscribe memory transparently — all without hardware partitioning and without changing the application. The Efficient Performance-Aware GPU Sharing literature builds exactly here: intercepting at the kernel/driver boundary to deliver both compatibility (apps run unmodified) and isolation (one tenant's bursts cannot starve another), with much finer control than MIG's fixed slices. The challenge is doing it with low overhead and strong enough isolation to be trustworthy.
The two hard constraints: isolation and predictability
Every sharing mechanism is judged on two things, and the Survey of Multi-Tenant Deep Learning Inference on GPU returns to both repeatedly:
- Isolation. One tenant must not be able to read another's data (security), exhaust shared memory and crash a neighbor (fault isolation), or hog compute and wreck a neighbor's latency (performance isolation). MIG gives all three in hardware; software methods must build them, and performance isolation is the hardest to guarantee.
- Performance predictability (QoS). A latency-sensitive inference service co-located with a throughput-hungry training job must still meet its SLO. This is the central tension in multi-tenant inference: the interference between co-located workloads — contention for SMs, cache, and memory bandwidth — makes latency unpredictable. Performance-aware sharing systems monitor this interference and throttle the aggressor to protect the latency-critical tenant.
How this maps onto Kubernetes
In practice most of this is operated through Kubernetes, which schedules GPUs via the NVIDIA device plugin. Plain Kubernetes treats a GPU as an indivisible resource (nvidia.com/gpu: 1) — the whole-GPU-allocation problem in container form. The ecosystem layers the sharing mechanisms on top:
- MIG via the device plugin — advertise each MIG slice as a schedulable resource so pods request a fraction of a GPU with hardware isolation.
- Time-slicing and MPS configured through the plugin for soft sharing of a single GPU among several pods.
- Dynamic Resource Allocation (DRA) — the newer Kubernetes API for expressing richer GPU requirements (specific topology, sharing mode) than a flat integer count.
- Gang/topology-aware schedulers (Volcano, Kueue, and similar) to handle the all-or-nothing placement of distributed jobs with locality awareness.
This is the production realization of everything above: the cluster scheduler (Wei Gao's domain) and the sharing mechanism (MIG/MPS/intercept) made operable for real multi-tenant fleets.
Choosing a sharing strategy
| You have… | Use… | Because |
|---|---|---|
| Untrusted tenants, strict SLAs | MIG | Hardware isolation; predictable performance. |
| Your own inference replicas, want density | MPS | Concurrent execution; you trust the co-tenants. |
| Bursty dev/notebook workloads | Time-slicing | Simple; each needs the GPU only occasionally. |
| Mixed latency + throughput, need QoS | Kernel interception | Fine-grained, performance-aware throttling. |
| Big distributed training jobs | Whole GPUs + gang/topology scheduler | They saturate the GPU anyway; locality matters. |
Where the series has taken you
Step back and see the whole arc. The same principle — keep the expensive resource busy by hiding the latency of the cheap-but-slow level below it — appears at every scale:
- Phase 1: hide HBM latency behind warps; coalesce and tile to feed the ALUs.
- Phase 2: diagnose the bottleneck with the Roofline, raise intensity by fusing, and use the tools to measure and write fast kernels.
- Phase 3.1: hide interconnect latency behind compute; map parallelism to the link.
- Phase 3.2: hide whole-GPU idleness behind other tenants' work; schedule and share to keep the fleet busy.
It is latency hiding and resource packing all the way down — from 32 threads in a warp to 32,000 GPUs in a datacenter. Learn the pattern once and it transfers across every level.
FAQ
Can I use MIG and MPS together?
Yes — a common pattern is to MIG-partition a GPU for coarse isolation between tenants, then run MPS within a MIG instance to pack several of one tenant's processes into their slice. They operate at different granularities and compose.
Why not just buy more GPUs instead of sharing?
Because GPUs are scarce and extremely expensive, and the waste is enormous: a cluster at 50% utilization is effectively paying double. Sharing recovers capacity you already own. At hyperscale, a few points of utilization improvement is worth millions, which is why this is such an active research and product area.
Is GPU sharing safe for untrusted tenants?
Only with hardware isolation (MIG) can you make strong security claims between untrusted tenants. MPS and most software sharing share memory space and are meant for co-tenants you trust. Kernel-interception systems aim to add isolation in software, but proving it is robust enough for hostile tenants is hard — for true multi-tenant clouds with untrusted users, MIG (or whole-GPU allocation) remains the safe default.
What makes inference scheduling different from training scheduling?
Training is throughput-oriented, long-running, gang-scheduled, and tolerant of preemption (checkpoint and resume). Inference is latency-oriented, bursty, SLO-bound, and intolerant of jitter. A scheduler optimal for one is wrong for the other, which is why mixed clusters are hard and why inference often gets its own pool with QoS-aware sharing.
Does sharing help training jobs at all?
Rarely — a well-tuned training job already saturates its GPUs (that was the whole point of Phases 1–2), so there is little spare capacity to share. Sharing pays off most for under-utilizing workloads: small models, inference at low load, interactive notebooks, and data-preprocessing steps. Match the mechanism to the workload's actual GPU appetite.
Takeaways
- The datacenter problem is utilization: whole-GPU allocation, gang scheduling, and bursty demand leave clusters at 40–60%.
- The Wei Gao taxonomy maps schedulers by objective, preemption, job knowledge, placement, and elasticity — no single design wins.
- Topology-aware scheduling and topology-aware parallelism (Phase 3.1) are two halves of one optimization.
- Four sharing mechanisms: MIG (hardware partition, best isolation), MPS (concurrent, weak isolation), time-slicing (temporal, no memory isolation), kernel interception (software, most flexible).
- The two hard constraints are isolation and performance predictability; interference between co-located workloads is the multi-tenancy tax.
- In production this runs on Kubernetes (device plugin, MIG, DRA, gang schedulers).
- The whole series is one idea at six scales: hide latency, pack the resource, keep the expensive thing busy.
That closes the GPU Programming & Optimization series — from a single warp of 32 threads to a datacenter of tens of thousands of GPUs, held together by one repeating principle. Thanks for reading the whole thing.
References & further reading
- Gao et al. — Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision — the scheduling survey behind this article.
- A Survey of Multi-Tenant Deep-Learning Inference on GPU — isolation, interference, and QoS in sharing.
- Efficient Performance-Aware GPU Sharing via Kernel-Space Interception — compatibility + isolation at the kernel boundary.
- NVIDIA MIG User Guide — hardware GPU partitioning.
- NVIDIA Multi-Process Service (MPS) — concurrent process execution on one GPU.
- NVIDIA Kubernetes Device Plugin — MIG, time-slicing, and DRA in Kubernetes.