GPU Hunter — Global GPU Availability with MultiKueue — KubeCon Mumbai 2026 Day 2 Deep Dive

Day 2 opens with the GPU problem nobody can buy their way out of: scarcity is regional. Kishore Jagannath and Ram J A (Google) showed how to treat a fleet of Kubernetes clusters spread across regions as one global GPU pool using Kueue (job queueing + quotas in one cluster) and MultiKueue (a manager-worker meta-scheduler that dispatches jobs to whichever cluster actually has accelerators). The talk built up the Kueue object model (ResourceFlavor, ClusterQueue, LocalQueue, quotas), extended it across clusters with MultiKueue, wired it to physical capacity through ProvisioningRequest and the Cluster Autoscaler, and then — crucially — walked the real production failure: a SchedulingGated deadlock where the manager pins a job to an exhausted region while idle GPUs sit unused in another. The honest "here's the bug" moment is the most valuable part.

This is the first of the Day 2 deck-backed deep dives. It connects straight to Day 1's serving and GPU talks — DD13 (serving LLMs) and DD12 (zero-GPU autopilot) — but where those were about using accelerators efficiently, this is about finding them in the first place.

The problem — GPU scarcity is a regional silo

The framing was vivid (the deck literally drew cavemen hunting GPUs through a jungle): everyone needs compute, and the good accelerators keep hiding. But the technical claim underneath is precise. GPU capacity is locked to local regional availability. A cluster or VM in us-central1 cannot tap idle accelerators sitting one region over in asia-southeast1. When demand spikes locally, you hit the dreaded "Insufficient Regional Capacity" and your training or inference job fails or stalls — even though, globally, your organisation has plenty of idle GPUs.

The operational damage compounds:

Disrupted jobs — training and inference stall on regional capacity errors.
Poor customer experience — high latency and service timeouts erode trust.
Innovation block — teams wait weeks for hardware expansion instead of iterating.

Why the obvious fixes don't work. Two traditional provisioning models both fail GenAI's volatility. Upfront provisioning reserves capacity for peak demand — high OpEx, low utilisation, wasted spend in idle periods. Queued provisioning makes workloads wait for regional availability — time-to-market delays and stalled innovation. The core failure is the same: static models can't adapt to uneven, spiky traffic, so regional spikes cause local failures while global capacity sits idle.

It's worth understanding why GPU scarcity is regional rather than global, because it's the physical fact the whole architecture works around. A cloud "region" is a specific cluster of datacenters in a geography, and GPUs are physical cards racked in those buildings. When NVIDIA's latest accelerators are supply-constrained (as they perpetually are), a cloud provider can't instantly stock every region equally — some regions get the new H100/B200 capacity first, others wait. On top of that, your own org has per-region quotas. So "we have 200 idle H100s" can be globally true while us-central1 specifically is dry. You cannot teleport a GPU between regions; the only thing you can move cheaply is the job. That asymmetry — hardware is fixed in place, jobs are portable — is the entire premise of MultiKueue: bring the work to the silicon, since you can't bring the silicon to the work.

Foundation — what Kueue actually is

Before you can chase GPUs across regions, you need a sane way to manage them in one cluster. That's Kueue: a Kubernetes-native job queueing system that manages quotas and fair access to high-demand hardware for batch and training workloads. Instead of every pod fighting the scheduler directly, jobs enter a queue and Kueue admits them only when quota and capacity allow. It's built for exactly the four jobs the deck listed — multi-team sharing, large-scale training, batch inference, and squeezing utilisation out of scarce datacenter GPUs.

Why batch GPU jobs need a queue at all — the default scheduler can't do it. Kubernetes' built-in scheduler places pods one at a time, greedily. That's fine for a web service, but a distributed training job needs all its workers running at once (it's gang-scheduled — 8 GPUs or nothing useful happens). With the default scheduler, a 8-GPU job can grab 5 GPUs, then block waiting for 3 more that never come — holding 5 idle GPUs hostage while a second job does the same. That's resource deadlock and partial-allocation waste. Kueue fixes it with all-or-nothing admission: a job is admitted only when its full quota is available, so GPUs aren't fragmented across half-started jobs. Add quotas and fair-sharing on top and you have orderly multi-team access to scarce hardware instead of a scheduler free-for-all.

The object model

Kueue's hierarchy is small but precise, and getting it straight is the whole game:

Object	Scope	Role
ResourceFlavor	cluster	Describes the physical characteristics of a resource — GPU model (H100, B200), spot vs on-demand, a node label selector.
ClusterQueue	cluster	A cluster-scoped pool of ResourceFlavors with quotas and fair-sharing rules. The global policy lives here.
LocalQueue	namespace	The user-facing entry point. A team submits jobs to a LocalQueue in their namespace; it maps to a ClusterQueue.
Resource Quota	cluster	Enforces consumption limits across teams and flavors so nobody hogs the pool.

Fig 1 — Kueue flow: jobs enter via namespace-scoped LocalQueues, pass through a cluster-scoped ClusterQueue (global rules + quota), and consume specific ResourceFlavors (CPU / H100 / B200).

Multi-team quota in practice

The deck made it concrete with a dev/prod split:

ClusterQueue: Development — a shared resource pool on the n1-standard-8 flavor (N1 GPU, optimised for cheap prototyping). Team Research (research-lq) and Team Engineering (eng-lq) both point at it and share a single cluster-level quota: CPU 60, GPU 8, Mem 126Gi. The shared cap stops anyone hogging dev resources.
ClusterQueue: Production — dedicated quotas on the a4-highgpu-8g flavor (A4-series GPU for high-throughput inference). Strictly enforced limits for live traffic, priority-based preemption disabled, guaranteed GPU availability: CPU 64, GPU 8, Mem 256Gi, single entry via prod-lq.

MultiKueue — beyond local limits

Kueue is single-cluster. The leap is MultiKueue: a meta-orchestrator that dispatches jobs from a central manager cluster to one or more worker clusters based on where capacity actually exists. The deck's four headline capabilities:

Capability	What it does
Global accelerator chasing	Dynamically dispatches batch AI/ML jobs to worker clusters across regions the moment GPU/TPU capacity frees up anywhere.
Unified global quotas	Enforces hierarchical org quotas across the whole fleet, so no single team monopolises a multi-region environment.
Topology abstraction	Researchers submit to one centralised manager cluster; MultiKueue hides the cluster topology and manual targeting.
Global spot harvesting	Opportunistically routes flexible batch workloads to whichever worker currently has the cheapest preemptible capacity.

The manager-worker topology

Fig 2 — Manager-worker topology: the manager hosts the MultiKueue controller, watches LocalQueues and creates remote jobs; workers are standard Kueue clusters that execute and report status back.

The split of responsibilities:

Manager — hosts the MultiKueue controller. Manages global limits, watches LocalQueues, and creates "remote" jobs on workers. Users submit to one LocalQueue and the job runs wherever capacity is.
Worker — a standard Kueue cluster. It receives the dispatched job, admits it within its own local ClusterQueues, executes it, and reports status back to the manager.

The mental model. MultiKueue is to Kueue what a load balancer is to a single server: the manager doesn't run your GPU job, it places it. Centralised quota lives on the manager; execution lives on the workers; the user sees one queue. That transparency — "submit to one LocalQueue, run anywhere" — is the entire value proposition.

Bridging logical to physical — ProvisioningRequest

Quotas are logical. GPUs are physical. The piece that connects them is the ProvisioningRequest API, which coordinates the Cluster Autoscaler to guarantee node provisioning before a job is admitted. Kueue picks the provisioning class per worker, and the choice matters enormously:

Provisioning class	Behaviour
best-effort-atomic-scaleup	Fulfils batch demand by scaling a hardware group atomically — all nodes or none. Good for gang-scheduled training that needs every node at once.
check-capacity-autoscaling	Checks real-time physical resource and cloud capacity quotas before committing.
queue-dws-provisioning	Coordinates pipelines natively with GKE's Dynamic Workload Scheduling (DWS) API.

The deck's architecture used GKE worker clusters with GPU nodepools, scale-from-zero enabled, the Cluster Autoscaler for dynamic provisioning, and best-effort-atomic-scaleup as the admission controller for workers. That last choice is exactly what triggers the failure mode below — atomicity is a double-edged sword.

Scale-from-zero is what makes federation economical — and what makes it fragile. The worker clusters run GPU nodepools that scale to zero when idle: you pay for no GPU nodes until a job needs them, then the Cluster Autoscaler provisions hardware on demand. That's the cost win — no idle reservation per region. But it widens a dangerous gap: between "Kueue's quota says this job fits" (logical) and "a GPU node actually booted and joined the cluster" (physical), there's now a provisioning step that can fail — the region may simply have no spare hardware to give. ProvisioningRequest is the handshake across that gap, and the provisioning class decides what happens when the hardware isn't there: check-capacity-autoscaling verifies real capacity before committing (fail fast), while best-effort-atomic-scaleup optimistically accepts and tries to scale all-or-nothing (which, as the next section shows, can accept-then-hang).

The real bug — the SchedulingGated deadlock

This is the part worth the price of admission. In a clean global federation you'd expect the manager to route around an exhausted region. In practice, the team hit a regional dispatch failure: the manager commits a job to a region that has no capacity, while valid capacity sits idle in a secondary worker — and nothing fails over.

Fig 3 — The SchedulingGated issue: the manager commits to an exhausted region on a stale view; the pod enters SchedulingGated and stalls with no automated failover, while idle capacity in Worker B is bypassed.

Walking the failure step by step:

The manager observes the global workload and, on a stale view of capacity, selects us-central1.
The job is pinned to Worker A.
Worker A's best-effort-atomic-scaleup controller accepts the provisioning request — status ACCEPTED — but there's no capacity.
The pod enters SchedulingGated and stays there. Because the provisioning request was accepted, nothing triggers a re-dispatch.
Meanwhile Worker B in asia-southeast1 has idle capacity — but it's bypassed, because the manager already pinned the job to region A.

The trap, named. SchedulingGated is a pod state that means "admission webhook/gate hasn't cleared this pod for scheduling yet." Combined with an atomic provisioning class that reports ACCEPTED before nodes actually exist, you get a silent deadlock: the manager believes the job is placed and making progress, so it never reconsiders — and the job waits forever on hardware that will never arrive in that region, while the right hardware idles next door.

What to take away

The fix space the talk pointed at — and the practical lessons for anyone building global GPU federation on Kueue:

Stale capacity views are the enemy. A global scheduler is only as good as its freshness. If the manager pins on a stale snapshot, it will pin wrong.
Atomic provisioning needs a timeout / fallback. "ACCEPTED but no capacity" must be a re-dispatch trigger, not a terminal state. Without that, SchedulingGated becomes a black hole.
Pinning is the anti-pattern. The whole point of MultiKueue is "run anywhere"; hard-pinning to a region on first decision defeats it. You want the job to remain eligible for other workers until it's genuinely admitted.
Scale-from-zero amplifies the gap between "logical quota says yes" and "a node physically exists." The ProvisioningRequest class is where that gap is either handled or hidden.

The demo wired it all together on GKE: a manager cluster (mkueue-manager-us-central1) and two worker clusters (mkueue-worker-us-central1, mkueue-worker-asia-southeast1) in one fleet, with the GKE console showing scaling issues, unschedulable pods, and webhook-verification warnings — i.e. exactly the SchedulingGated symptoms in the wild before the dispatch logic is hardened.

Fixing the deadlock — what actually works

The talk left the fix as "harden the dispatch logic," but it's worth spelling out concretely, because each lever fixes a different part of the failure chain. The deadlock has three independent contributing causes — a stale capacity view, an optimistic provisioning class, and region pinning — and you can attack any of them.

Lever	What it changes	Trade-off
Swap to `check-capacity-autoscaling`	Verifies real cloud capacity before admitting, so the job is never pinned to an empty region. Fail-fast instead of accept-then-hang.	Slightly slower admission (a capacity check round-trip); can still thrash if every region is genuinely full.
ProvisioningRequest timeout + retry	Treat a long-pending ACCEPTED request as a failure and release the job back to the manager for re-dispatch.	Need to tune the timeout — too short and you abandon a node that was about to boot; too long and the stall persists.
Don't hard-pin — keep workloads eligible	Leave the job admissible on multiple workers until one genuinely admits it, so a stuck region doesn't strand the job.	More cross-cluster chatter; quota accounting on the manager must avoid double-counting in-flight dispatches.
Shorten the manager's capacity refresh	Fresher snapshots mean fewer wrong pins in the first place.	More API load on workers; never eliminates the race, only narrows it.

The pragmatic combination. In production you don't pick one — you layer them. Use check-capacity-autoscaling so most bad pins never happen, add a ProvisioningRequest timeout so the ones that slip through self-heal instead of hanging, and keep jobs eligible on more than one worker so a single stuck region can't strand work. Fresh capacity views are the cheap baseline under all of it. Defence in depth: each layer catches what the previous one missed.

Detecting it before users complain

The symptom is quiet — no crash, no error event, just a pod that never schedules. Watch for these signals:

Pods stuck in SchedulingGated for longer than your provisioning SLO (alert on age, not just count).
ProvisioningRequest objects in Accepted state with no backing nodes appearing — the ACCEPTED-but-empty tell.
A worker cluster reporting idle GPU capacity while the manager's queue depth is non-zero — the "idle next door" signature.
Webhook-verification warnings in the GKE console, which the demo showed accompanying the stuck state.

FAQ

What's the difference between Kueue and MultiKueue?

Kueue manages job queueing and resource quotas within a single Kubernetes cluster (ResourceFlavor / ClusterQueue / LocalQueue). MultiKueue sits above it as a meta-orchestrator: a manager cluster dispatches jobs to one or more worker clusters (each a standard Kueue cluster) based on where capacity is available. Users submit to one LocalQueue; the job runs wherever there's room.

Why not just over-provision GPUs in every region?

Because that's the "upfront provisioning" anti-pattern: high OpEx and low utilisation, with capacity idle most of the time. GenAI traffic is spiky and uneven across regions, so static per-region reservation either wastes money or still fails on local spikes. Federation lets one global pool absorb the spikes.

What is the SchedulingGated deadlock, in one line?

The manager pins a job to a region whose atomic provisioning request returns ACCEPTED despite having no capacity, so the pod sits in SchedulingGated forever with no automated failover — while idle GPUs in another worker cluster go unused.

What is a ProvisioningRequest class and why does it matter here?

It's how Kueue asks the Cluster Autoscaler to guarantee nodes before admitting a job. best-effort-atomic-scaleup scales a node group all-or-nothing; check-capacity-autoscaling verifies real capacity first; queue-dws-provisioning uses GKE Dynamic Workload Scheduling. The atomic class is what produced the ACCEPTED-but-empty state behind the deadlock.

Does MultiKueue move running pods between clusters?

No. MultiKueue places a job before it runs — it dispatches the workload to a worker cluster, which then executes it locally. It does not live-migrate a running pod across clusters. That's why a wrong placement is costly: once a job is pinned and provisioning is accepted, there's no built-in mechanism to pick it up and run it elsewhere unless you keep it eligible on multiple workers in the first place.

Why does the manager use a "stale" view at all — can't it just check live?

A global scheduler can't synchronously query every worker's real-time capacity for every decision without huge latency and API load, so it works off periodically refreshed snapshots. There's always a window between snapshot and decision where reality has moved on — a spot node got reclaimed, another job grabbed the last GPUs. Shrinking the refresh interval narrows the window but never closes it; that's why fail-fast admission (capacity-check) and re-dispatch on timeout matter more than chasing perfect freshness.

How does spot/preemptible capacity fit in?

Spot harvesting is part of the cost win — workers can run GPU nodepools on preemptible instances, and the federation chases whichever region has cheap spare capacity. The catch: spot nodes can vanish mid-job, and a reclaimed node looks a lot like "no capacity," so the same dispatch-hardening (timeout, re-dispatch, multi-worker eligibility) is what keeps spot from turning into stuck jobs.

Is this Kueue-specific, or a general federation problem?

The pattern is general. Any system that (1) makes placement decisions on a cached view of capacity, (2) commits optimistically before the resource physically exists, and (3) pins the work to that decision will hit some version of the accept-then-hang deadlock. Kueue/MultiKueue is just where this talk hit it; the lessons — fresh views, fail-fast admission, re-dispatch, don't hard-pin — transfer to any global scheduler.

Takeaways

GPU scarcity is regional, not absolute. The win is treating many clusters as one global pool, not buying more cards.
Kueue gives you the object model — ResourceFlavor, ClusterQueue, LocalQueue, quotas — for fair, queued GPU access in one cluster.
MultiKueue federates it — manager-worker topology, global accelerator chasing, unified quotas, spot harvesting, submit-to-one-run-anywhere.
ProvisioningRequest is the logical-to-physical bridge, and the provisioning class you pick decides your failure modes.
The SchedulingGated deadlock is the cautionary tale: stale views + atomic provisioning + region pinning = a job stuck on capacity that will never arrive while idle GPUs sit next door.

Next in Day 2 — A gRPC Transport for the Model Context Protocol, giving MCP a production-grade transport.

References

KubeCon Mumbai 2026 — Day 2 index · the rest of Day 2
Kueue — concepts · ResourceFlavor, ClusterQueue, LocalQueue
Kueue — MultiKueue · manager-worker dispatch
Kueue — ProvisioningRequest · provisioning classes & the autoscaler bridge
Day 1 DD13 — Serving LLMs on Kubernetes · the consuming side of GPUs

GPU hunter — chasing global GPU capacity with Kueue and MultiKueue.