Inference in progress — please monitor responsibly.

Gaurav Sharma (NVIDIA) shared observability learnings from running AI inference at population scale — NVIDIA Cloud Functions (NVCF) powers 500+ government websites and processes 15M+ inferences daily, 6B+ total, at sub-second latency (per a March 2026 PIB/MeitY note). The thesis: inference observability is not ordinary web monitoring. You have a three-plane platform (control / invocation / compute), each with its own failure modes, and the right structure is an SLO at the top that fans down into single-pane dashboards and progressively deeper drill-downs. Two critical journeys carry the whole thing: worker readiness (is the GPU node actually healthy?) and the inference request itself (success rate + time-to-first-token). DCGM, XID/ECC errors, and burn-rate alerting are the connective tissue.

This is the third Day 2 deep dive, and it sits right next to GPU Hunter (finding GPUs) and Day 1's serving LLMs (DD13) (running them). This one answers: once they're serving, how do you see whether they're healthy?

The scale framing matters because it changes what "monitoring" even means. 15M+ inferences a day is roughly 170 per second sustained, with peaks far higher, spread across many models, GPU types, and regions — and "sub-second response" is a published promise to the public, not an internal nicety. At that volume you cannot watch individual requests, and you cannot afford a human to ask "which layer broke?" on every blip. The monitoring has to be self-routing: an alert must encode where to look, and the dashboards must let you descend from "the whole platform is unhealthy" to "this one node's GPU is throwing ECC errors" in a few clicks. Everything below — the three planes, the SLO-on-top hierarchy, the two journeys — is in service of that property. It's also, notably, all assembled from open-source CNCF-adjacent pieces, which is why the talk doubles as a parts list anyone can copy.

The inference request lifecycle

Start with the path a single prompt travels — because every monitoring decision hangs off it:

Walk that path slowly, because each hop is a different kind of thing and fails for a different reason. The user prompt arrives at a global endpoint — possibly from a government portal, a chatbot, or a batch job. The inference platform (control plane) authenticates it, decides which model and which region/cluster should serve it, and applies rate limits and quotas. The model endpoint (data plane) is the addressable front door of a specific deployed model. Model inference is the actual GPU work — the forward pass that turns tokens into tokens. The response stream is where it gets interesting: an LLM doesn't return one blob, it emits tokens incrementally over a held-open connection (SSE or gRPC streaming), and the user watches text appear word by word.

That streaming shape is the single fact that breaks classic web monitoring. In a normal HTTP service, a request has one duration: start to response. You measure p50/p95/p99 of that number and you're mostly done. An LLM request has at least three distinct durations that matter: how long until the first token (the user sees "thinking" until then), how fast tokens stream after that, and how long the whole completion takes. A request can have a great total time but a terrible first-token time and the user will still feel it as slow. A monitoring stack that only records "request took 4.2s" has thrown away the number the user actually cares about.

Why inference orchestration is hard

The deck laid the challenges next to the requirements they impose — a clean way to see why a plain web stack doesn't cut it:

Challenge	Orchestration requirement
Multiple consumers	Global endpoint + auth
Multiple models	Model abstraction
Multiple GPU types	GPU-aware placement
Multiple clusters	Multi-cluster deployment
Multiple traffic patterns	Routing + autoscaling

Each row hides a real operational problem. Multiple consumers means you can't trust a single client's behaviour — one team's batch job can starve another team's interactive traffic unless quotas and a global auth/endpoint layer sit in front. Multiple models means consumers shouldn't hard-code which GPU or which container serves "the summarizer" — they ask for a logical model name and the platform maps it, so you can swap a model version without breaking callers. Multiple GPU types is the brutal one: an A100, an H100, an L40S, and a GH200 have wildly different memory, throughput, and cost, and a 70B model simply won't fit on the small ones — placement has to be GPU-aware or pods will fail to schedule or OOM at load. Multiple clusters (often across regions, for sovereignty or latency) means a request from one geography should land near it, and a cluster outage should fail over rather than black-hole. Multiple traffic patterns — bursty interactive vs steady batch vs spiky launch-day — means one static replica count is always wrong; you need routing plus autoscaling that understands GPU warm-up time.

The three-plane platform

NVCF is structured as three planes, and this structure is the key to monitoring it — because each plane fails differently:

Plane	Responsibilities	Failure modes
Control plane	NVCF APIs (global endpoint + auth), function definitions (model abstraction), function deployment (GPU-aware placement), secrets, rate limiting, autoscaling	Routing, scaling, orchestration failures
Invocation plane	Routing inference requests to function workloads	Latency, errors, timeouts, throughput
Compute plane	NVCF Cluster Agent (GPU cluster discovery), worker orchestration on GPU clusters to execute inference	GPU, node, network, storage failures

A quick grounding — SLI, SLO, error budget

The whole monitoring design rests on three SRE terms, so it's worth pinning them precisely before the dashboards. An SLI (service level indicator) is a measurement — a ratio of good events to valid events, e.g. "fraction of inference requests that returned a first token under 2 seconds." An SLO (service level objective) is a target on that SLI — "99.9% of requests over 30 days." The error budget is the inverse — the 0.1% you're allowed to fail. The budget is the budget for change: spend it on risky deploys when it's healthy, freeze and stabilise when it's nearly gone.

Translated to metrics, the success-rate SLI is a pair of counters and the TTFT SLI is a histogram. In PromQL the shapes look like this:

# Success-rate SLI over 5m (invocation plane)
sum(rate(invocation_requests_counter{result="success"}[5m]))
  /
sum(rate(invocation_requests_counter[5m]))

# TTFT SLI: fraction of requests with first token under 2s
sum(rate(ttft_latency_histogram_bucket{le="2.0"}[5m]))
  /
sum(rate(ttft_latency_histogram_count[5m]))

Note the TTFT query keys off a histogram bucket, not an average. Averaging latency is a trap — a bimodal distribution (most fast, a few catastrophic) has a fine average and a terrible p99, and it's the p99 user who complains. Histograms let you ask "what fraction beat the target" directly, which is exactly the shape an SLO wants.

Monitoring structure — SLO at the top, drill-downs below

The monitoring philosophy is a hierarchy: alert via SLO, correlate via drill-downs. A single top-level Inference SLO (e.g. 99.9%) is what pages you. From there you descend through single-pane-of-glass (SPOG) dashboards into progressively finer scopes:

The two critical journeys

Underneath the dashboards, everything reduces to two critical journeys — and each is taken through the same SRE loop: instrument metrics → define SLIs → set targets & budgets → alert on burn rate.

1. Worker readiness — is the GPU worker actually able to serve before traffic arrives?
2. Inference request — is each request succeeding, and fast enough?

Journey 1 — worker readiness

"Ready" for a GPU worker is a layered claim, and the deck broke it into four columns:

Kubernetes	GPU software stack	GPU hardware	Custom monitoring
Node Ready	GPU Operator	GPU availability	Compute-plane agent checks
Pod health	Driver	XID errors	Model-download checks
Resource pressure	Container Toolkit	ECC errors	Custom network monitoring
Kubelet	Device Plugin	Temperature	GPU provisioning workflows

It's worth being concrete about what DCGM actually gives you, because "watch the GPU" is too vague to act on. The dcgm-exporter publishes a field for nearly every health and utilisation signal, scraped by Prometheus like any other exporter. The ones that earn a place on a readiness dashboard:

DCGM field	What it tells you	Why you alert on it
DCGM_FI_DEV_XID_ERRORS	Last XID error code on the GPU	Non-zero fatal XIDs (e.g. 79 "GPU fell off the bus", 48 "double-bit ECC") mean evict the node now
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL	Uncorrectable (double-bit) ECC errors	Memory is corrupting silently — inferences may be wrong, not just failed
DCGM_FI_DEV_GPU_TEMP	Core temperature	Sustained thermal throttling tanks throughput before anything "fails"
DCGM_FI_DEV_GPU_UTIL / DCGM_FI_DEV_FB_USED	Compute utilisation / VRAM in use	Capacity planning + spotting a wedged worker holding VRAM but doing no work
DCGM_FI_DEV_NVLINK_*	NVLink bandwidth/errors	Multi-GPU models (tensor-parallel) degrade hard if NVLink flaps

Journey 2 — inference request health

For the request itself, the deck pinned two SLOs to two points in the path:

• Success Rate SLO at the invocation plane — successful invocations / total invocations, from invocation_requests_counter and invocation_errors_counter.
• TTFT Latency SLO at the inference server / worker pod — first token within a target, from a ttft_latency_histogram.

Burn-rate alerting — the SRE glue

The loop ends on alert on burn rate, and this is the discipline that keeps a 99.9% SLO from being either useless or noisy. Rather than paging on every error, you alert when the error budget is being consumed too fast. A fast burn (budget gone in an hour) is an immediate page; a slow burn (budget gone over days) is a ticket. This is standard Google-SRE-workbook practice, applied to inference SLIs — and it's what makes the top-level SLO actionable instead of decorative.

The mechanics: "burn rate" is how fast you're spending budget relative to "even" consumption. Burn rate 1 means you'll exactly exhaust the 30-day budget in 30 days. Burn rate 14.4 means you'd exhaust it in ~2 days — clearly an incident. The workbook's standard trick is multi-window, multi-burn-rate alerts: page on a fast burn confirmed over a short and a slightly longer window (e.g. 14.4× over 1h AND 5m) to catch acute outages, and open a ticket on a slow burn (e.g. 3× over 6h) to catch the quiet degradation that would otherwise eat the month. The two-window requirement is what kills false pages from a 90-second blip.

# Fast-burn page: 14.4x over 1h confirmed by 5m window
(
  slo:error_rate:ratio_1h > (14.4 * 0.001)
  and
  slo:error_rate:ratio_5m > (14.4 * 0.001)
)

A worked troubleshooting playbook

Tie it together with the path an on-call actually walks when the inference SLO burns. This is the value of the structure — it turns a vague "AI is slow" into a decision tree:

Symptom on the SPOG	Likely plane	First checks
Success rate drops, TTFT normal	Invocation / control	Routing errors, auth/rate-limit rejections, a region with no healthy workers to route to
TTFT spikes, success rate normal	Compute	GPU saturation, queue depth at the server, a model cold-start storm, thermal throttling on a cluster
Both degrade in one region only	Compute (regional)	Per-GPU-cluster drill: XID/ECC spikes, node NotReady, NVLink errors, a bad driver rollout
Slow steady decline over days	Compute (creeping)	Memory fragmentation, a leaking worker holding VRAM, gradual thermal drift, a noisy-neighbour model
Errors right after a deploy	Control	Bad function definition, wrong GPU placement constraint, image/weights pull failing on new workers

Where the metrics actually come from

The first step of every critical journey is "metrics instrumentation," and it's worth saying where those numbers originate, because you rarely write them by hand. Three layers each emit their own telemetry:

• The inference server emits the token-level metrics natively. Modern serving stacks — Triton, vLLM, TGI, and NVIDIA Dynamo — expose Prometheus endpoints with TTFT, TPOT, queue depth, batch size, KV-cache utilisation, and per-request token counts out of the box. You scrape them; you don't compute them. This is why the TTFT histogram lives "at the inference server / worker pod" in Fig 3 — that's the only place that actually knows when the first token left.
• The invocation/control plane emits the request-level counters (invocation_requests_counter, invocation_errors_counter) because it's the layer that sees every request and its final verdict, including ones that never reached a worker (auth rejected, rate-limited, no capacity).
• The node/GPU layer emits hardware health via dcgm-exporter and standard node/kubelet metrics — the readiness signals.

The other half of instrumentation is traces. Metrics tell you the success-rate dropped; a distributed trace (OpenTelemetry spanning control → invocation → worker → model) tells you which hop ate the time for a specific slow request. Metrics are for alerting and SLOs; traces are for the post-page investigation when the drill-down narrows you to a plane but not a root cause. The two are complementary, not competing.

The NVIDIA OSS toolbox

The deck closed by naming the open-source pieces this is built on — useful as a shopping list if you're assembling your own inference observability:

• NVIDIA Cloud Functions — the inference platform itself.
• GPU Operator — installs and manages driver, toolkit, device plugin, DCGM.
• DCGM-Exporter — exposes GPU metrics (utilisation, XID/ECC, thermal, power) to Prometheus.
• Dynamo — NVIDIA's inference serving framework.
• KAI Scheduler — GPU-aware scheduling.
• NVSentinel — GPU health/fault monitoring.

FAQ

How is inference observability different from normal web monitoring?

You're watching a three-plane platform (control / invocation / compute) where the compute plane is GPUs that fail in hardware-specific ways (XID/ECC errors, thermal, NVLink). And the latency that matters is time-to-first-token on a streaming response, not a simple request duration. A standard RED/USE dashboard misses the GPU health layer entirely.

What's the difference between worker readiness and inference request health?

Worker readiness is "is this GPU node fit to serve before traffic arrives" — pods ready, driver/device-plugin up, clean DCGM signals, a test workload passing. Inference request health is "is each live request succeeding and fast" — success-rate SLO at the invocation plane, TTFT SLO at the server. The first prevents routing to a sick node; the second catches problems in flight.

What are XID and ECC errors?

XID errors are NVIDIA GPU/driver error codes surfaced via the kernel and DCGM — they flag hardware or driver faults (some recoverable, some fatal). ECC errors are GPU memory error-correction events; uncorrectable ones indicate failing memory. Both are leading indicators that a node will start producing bad or failed inferences.

Why alert on burn rate instead of raw error count?

Because a 99.9% SLO allows some errors. Burn-rate alerting pages you when the error budget is being spent too fast (fast burn = urgent page, slow burn = ticket), which keeps alerts meaningful and avoids both noise and missed slow degradations.

Why measure TTFT with a histogram instead of an average latency?

Averages hide bimodal pain. If 95% of requests get a first token in 300ms and 5% take 8 seconds, the average looks acceptable while one in twenty users has a terrible experience. A histogram lets you write the SLI as "fraction of requests under the target," which is what an SLO needs, and lets you watch p95/p99 directly — the tail is where users churn.

Can't I just reuse my existing RED/USE dashboards?

Partly. RED (rate, errors, duration) maps reasonably to the invocation plane, and USE (utilisation, saturation, errors) maps to compute. But neither captures GPU-specific health (XID/ECC/NVLink/thermal via DCGM) or the streaming latency split (TTFT vs TPOT vs E2E). You keep the RED/USE habits but extend them with a GPU health layer and token-level latency, or you'll be blind to the failures that are unique to inference.

What's the role of NVSentinel vs DCGM-Exporter?

DCGM-Exporter exposes GPU metrics for Prometheus to scrape — it's the sensor. NVSentinel is about acting on GPU health/faults — detecting a bad GPU and driving remediation (cordon/drain) rather than just graphing it. One observes, the other closes the loop. Together with the GPU Operator (which installs driver/toolkit/device-plugin/DCGM) they form the health spine under the readiness journey.

Takeaways

• Inference observability is platform observability. Three planes — control, invocation, compute — each with distinct failure modes; separate them so an alert tells you where to look.
• One SLO at the top, drill-downs below. Alert via a single inference SLO, correlate via SPOG dashboards down to per-region, per-cluster, per-node.
• Two critical journeys. Worker readiness (healthy GPU node: ready pods + clean DCGM signals + validation checks) and the inference request (success rate + TTFT).
• Watch the silicon, not just the pod. XID/ECC errors, thermal, NVLink via DCGM — a "Ready" node can still be a bad GPU.
• Burn-rate alerting is what makes the SLO actionable; the NVIDIA OSS stack (GPU Operator, DCGM-Exporter, Dynamo, KAI Scheduler, NVSentinel) is the toolbox.

Next in Day 2 — Zero Trust for Autonomous Agents, on isolating AI workloads on Kubernetes.

References

• KubeCon Mumbai 2026 — Day 2 index · the rest of Day 2
• DCGM-Exporter · GPU metrics to Prometheus
• NVIDIA GPU Operator · driver, toolkit, device plugin, DCGM
• Google SRE — alerting on SLOs · burn-rate alerting
• Day 1 DD13 — Serving LLMs on Kubernetes · the serving side