Service Networking Within Air-Gapped Environments — KubeCon Mumbai 2026 Day 2 Deep Dive

Anirban Nandi (Google) walked how service networking is done inside a true air-gapped cloud — Google Distributed Cloud (GDC) Air-Gapped, a hardware+software product that delivers cloud capabilities with zero connectivity to the public internet. The shape of the problem: deliver observability, security, and traffic management for 45+ infrastructure services, each ~10 microservices, across multiple bare-metal clusters and networks, with no SaaS to lean on. The answer is Istio as the service-networking substrate — a primary cluster running Istiod, remote user/service clusters joined over east-west gateways, and the full Istio API surface (Gateway, VirtualService, DestinationRule, PeerAuthentication, AuthorizationPolicy, Telemetry, EnvoyFilter) doing edge proxying, mTLS, L7 routing, and telemetry — all wired into an air-gapped OSS stack (Cilium, MetalLB, CoreDNS, Prometheus/Cortex, Loki, Cert Manager). The honest ending is a frank list of the operational scars.

This opens the Day 2 networking block. It's the air-gapped, multi-cluster cousin of Day 1's lean observability with Istio (DD14) — same mesh, harder constraints.

What "air-gapped" actually means here

An air-gapped environment — sometimes called a "private cloud" — has no connectivity to the public cloud, by design. GDC Air-Gapped is a HW+SW solution that delivers modern cloud capabilities into exactly that setting: Virtual Machines, IAM, Key Management, Database, Storage, and Networking, all Kubernetes-orchestrated with OSS for the infrastructure services, and an Organization/Project tenancy model with distinct personas for infrastructure, platform, and application operators.

It's worth being precise about why anyone signs up for this pain, because the constraints below only make sense against the motivation. Air-gapped deployments exist where connectivity is a liability, not a convenience: defence and intelligence systems handling classified data, central banks and payment switches under sovereignty rules, critical infrastructure (power grids, telco cores) that must keep running even if the internet is cut, and regulated industries in jurisdictions that legally forbid data leaving the country or the building. The "gap" is a deliberate security control — if there's no path to the public internet, an entire class of remote attacks, data-exfiltration channels, and supply-chain phone-homes simply cannot happen. The trade is that everything the platform needs must already be inside the gap.

The first thing the air gap breaks is the thing nobody thinks about: pulling stuff. Every habit of cloud-native operations quietly assumes internet access — docker pull from Docker Hub, helm repo add, apt-get update, fetching a Let's Encrypt cert, an operator calling a vendor SaaS, even a sidecar checking for updates. In an air gap, all of that fails. Container images must be mirrored into an internal registry; Helm charts and OS packages mirrored into internal repos; certificates issued by an internal CA (there is no public ACME); time synced from an internal NTP source. Bootstrapping the environment is a deliberate "carry it across the gap" exercise — physical media or a tightly controlled one-way sync. This is the hidden tax behind every architecture decision in this talk: there is no fallback to "just download it."

Fig 1 — GDC Air-Gapped: a primary Organizational Infra Cluster (Istiod, management API) plus remote user and service clusters, all on bare metal, no public-cloud link.

The cluster split is itself a security and tenancy decision, not just an org chart. The Organizational Infra Cluster (primary) holds the management API server and Istiod — the brains — and is operated by infrastructure personas. Service Clusters run shared infra services (the platform's own building blocks). User Clusters run customer/tenant workloads alongside the infra services they need. Each lives on its own network, and the separation means a compromise or a noisy tenant in a user cluster is contained — it can't reach the management plane or another tenant's cluster except through the mesh's authorized, mTLS-protected paths. The "Organization and Project" tenancy with distinct infra/platform/application personas mirrors the public cloud's IAM model, rebuilt inside the gap so the same least-privilege boundaries apply.

North-south vs east-west — the two traffic axes. Everything in this talk splits along two directions. North-south (N-S) is traffic crossing the platform boundary: a request from the Customer or Operator network coming in to an infra service, handled by ingress gateways. East-west (E-W) is service-to-service traffic inside the platform — often across cluster and network boundaries — handled by east-west gateways. The reason this distinction is load-bearing: N-S is where you enforce the perimeter (TLS termination, authN, rate limiting, DoS protection) and E-W is where you enforce zero-trust between services (mТLS, authorization, identity). Mixing them up is how you accidentally trust internal traffic just because it's internal — exactly the implicit-trust failure the zero-trust talk warns against.

The requirement — at painful scale

Every service in the platform needs the same three things, and there's no managed service to outsource them to:

Observability	Security	Traffic management
QPS, latency, throughput metrics; audit logs + tracing across multiple hops	Encryption in transit; authorized access only; DoS prevention (rate limiting)	L7 attribute-based routing; custom load balancing; N-S and E-W reachability; traffic mutation (header transform, gRPC/HTTP transcoding)

The scale that forces a mesh. The speaker's framing: imagine doing all of the above for 45+ services, each averaging 10 microservices. That's ~450 microservices needing consistent mTLS, routing, rate limiting, and telemetry — by hand, in an air gap, that's impossible. A service mesh is what makes it tractable, because the behaviour lives in the data plane, not in 450 codebases.

Why Istio

Istio is a service mesh with Envoy as the data plane, configured by the control plane via xDS. Quick orientation for the air-gapped use:

Two modes — sidecar (a proxy per pod) and ambient (a shared per-node proxy).
Configured by declarative APIs — VirtualService, DestinationRule, Gateway, and more.
Supports multi-cluster, multi-network, multi-primary topologies — essential here.
Offers traffic shifting, mirroring, circuit-breaking, automatic mTLS, and telemetry out of the box.

What xDS actually is. xDS is the family of discovery APIs Envoy speaks to its control plane — LDS (listeners), RDS (routes), CDS (clusters), EDS (endpoints), SDS (secrets/certs). Istiod watches the Kubernetes API and your Istio CRDs, compiles them into Envoy's native config, and streams it to every proxy over a gRPC connection. The power is that you write intent (VirtualService: "send 10% to v2") and Istiod translates it into the low-level Envoy config on hundreds of proxies. The cost — which becomes the central scaling problem in an air gap — is that every config change can fan out to every proxy, and that fan-out is bounded by control-plane CPU and network, both finite on bare metal.

Sidecar vs ambient — and why it matters more here

Istio's two data-plane modes are a real architectural fork. Sidecar mode injects an Envoy proxy container into every pod — maximum per-workload control, but it doubles pod count and adds a proxy's memory/CPU to every workload. At ~450 microservices on fixed bare-metal compute, that overhead is not academic; it's a meaningful slice of a cluster you can't elastically grow. Ambient mode moves L4 handling to a per-node component (ztunnel) and makes L7 proxying an opt-in per-namespace waypoint, cutting the per-pod tax. The deck notes both modes exist; the picture shown is sidecar. The honest read: in a resource-constrained air gap, the sidecar tax is exactly the kind of thing that pushes teams toward ambient over time — fewer proxies to schedule, patch, and feed.

Istio in GDC Air-Gapped

The topology: Istiod runs in the primary (Organizational Infra Cluster, Network 1). User Cluster (Network 2) and Service Cluster (Network 3) are remote clusters joined to it. Three kinds of gateway carry traffic: Ingress Gateways (north-south, from the Customer/Operator networks in), East-West Gateways (cross-cluster, cross-network service-to-service), and an Egress Gateway (controlled exit). Infra service pods carry Envoy sidecars; Istiod pushes config to all of them over xDS across the network boundaries.

The east-west gateway deserves a closer look because it's the piece that makes multi-network mesh work, and it's specific to this kind of topology. In a single flat network, sidecars reach each other directly by pod IP. But here the clusters sit on different networks (1, 2, 3) that can't route to each other's pod IPs. The east-west gateway solves this: a sidecar in Network 2 that wants to reach a service in Network 1 sends the traffic to that network's east-west gateway, which terminates the mesh mТLS, looks at the SNI, and forwards to the right backend — all while preserving the workload identity so authorization still works end-to-end. It's effectively a mesh-aware bridge between network islands, and it's why "multi-network, multi-primary" is more than a bullet point: the gateways are load-bearing infrastructure, which is why they get HPA and auto-scaling attention in the lifecycle section.

Gateways — the workhorses

Gateways do most of the heavy lifting, and each capability maps to a specific Istio API:

Function	What it does	API
Edge & transit proxy	N-S (customer → infra services), E-W (between infra services), expose API servers securely	Gateway
Authentication	TLS termination / TLS origination	Gateway / DestinationRule
Authorization	JWT, CIDR-based, external authorizers	RequestAuthentication, AuthorizationPolicy
Load balancing	L7 routing, custom LB	VirtualService / DestinationRule
Observability	QPS, throughput, latency	Telemetry

Application security

Three Istio mechanisms cover the security requirement entirely:

Mutual TLS — automatic mTLS for service-to-service comms (PeerAuthentication).
Certificate management — a plug-in CA for flexible identity control (IstioOperator). This matters in an air gap: you bring your own CA rather than trusting a public root.
Policy control — granular allow/deny for traffic governance (AuthorizationPolicy).

Application telemetry

With no SaaS observability backend, everything is local and OSS — and the same Telemetry API does triple duty:

OTel audit logs — standardized auditing/logging for requests traversing the mesh (Telemetry API).
Prometheus metrics — rich Istio/Envoy stats for SLIs (Telemetry API).
Selective metrics — flexible include/exclude of metrics at the pod level (pod annotations).

The cardinality trap, again. "Selective metrics" isn't a nicety — it's survival. The mesh emits richly-labelled metrics for ~450 microservices; keep them all and Prometheus storage explodes. This is the exact discipline Day 1's lean observability talk (DD14) and the Kafka observability talk (DD03) warned about — here it's enforced via pod annotations and (per the operational-challenges slide) selective aggregation.

Traffic management & data-plane extensibility

The traffic toolkit, all via standard Istio APIs:

Capability	API
L7 routing (headers, paths, methods)	VirtualService
Traffic shifting (weighted, gradual migration)	VirtualService
Circuit breaking (outlier detection, fail-fast)	DestinationRule
Fault injection (delays, aborts for resiliency testing)	VirtualService

And when a need has no first-class Istio API, the EnvoyFilter API exposes raw Envoy features: Lua (header/payload transforms), Wasm (sandboxed high-performance scripts), transcoding (native HTTP/gRPC), rate limiting (global & local), compression (gzip/Brotli), external auth (delegate decisions) — and many more filters. This is the escape hatch that makes "the platform can do anything Envoy can" true.

Lifecycle & the OSS stack

Everything is installed and reconciled via Helm: a controller reconciles the control plane (Istiod deployment, east-west gateway), plus plug-in CA, ingress/egress gateways with HPA, the Istio APIs for global config/observability, and platform-dependency APIs (IPAM, SLO). The primary cluster manages remote clusters.

Because there's no cloud to call, the mesh integrates with a fully self-hosted OSS toolbox:

Concern	OSS tools
L4 networking	Cilium
External IP allocation / advertisement	MetalLB
DNS forwarding across clusters	CoreDNS
Mesh audit + operational logs	Loki, Fluent Bit, OTel Collector
Metrics & dashboards	Prometheus + Cortex, Grafana, AlertManager
Plug-in CA	Cert Manager

The operational scars

The most useful slide was the honest list of what's hard — the things you only learn by running this:

Area	The challenge
Advanced features	Some needs have no first-class API; direct xDS modification conflicts with Istiod-generated xDS, so you can't just hand-edit.
Debuggability	Selective metric aggregation to prevent cardinality explosion; selective audit/proxy-log aggregation to prevent storage overload.
Co-existence with 3P workloads	Resource conflicts with third-party Istio deployments; must prevent intra-mesh communication between first-party and third-party workloads.
Scalability	Large config propagation contained by config scoping + delta xDS; ultimately constrained by the underlying bare-metal compute.
Lifecycle	Ordered upgrades across primary-remote multi-cluster; special handling for unhealthy user clusters; minimal data-plane downtime during upgrades; auto-scaling gateways.

The two scaling levers worth stealing. When Istiod has to push config to ~450 sidecars across networks, naive full-state pushes melt the control plane. The fixes named here are the standard-but-essential ones: config scoping (use Sidecar resources / namespace isolation so each proxy only learns about the services it actually talks to) and delta xDS (push only what changed, not the whole config snapshot). On bare metal with no elastic compute, these aren't optimisations — they're what keeps the mesh from falling over.

Why upgrades are the hardest part

The lifecycle row on the scars slide is easy to skim, but it hides the genuinely hard operational problem. Upgrading a mesh that spans a primary plus multiple remote clusters, on bare metal, with no maintenance window where you can "just take it down," is a choreography problem:

Order matters. The control plane (Istiod) generally upgrades before the data plane (proxies), and the primary before the remotes — but a remote cluster pinned to an old proxy version must still talk to a newer control plane, so version-skew compatibility windows have to be respected, not assumed.
Data-plane downtime must be near zero. Restarting every sidecar to pick up a new Envoy means rolling them carefully; do it too fast and you drop in-flight requests across 450 services. Gateways (the east-west/ingress paths) are especially sensitive because they're shared chokepoints.
Unhealthy clusters can't block the fleet. If one user cluster is wedged, the upgrade of everything else can't stall waiting for it — hence "special handling for unhealthy user clusters." You need an upgrade controller that can proceed around a sick member and reconcile it later.
No internet means no canary against prod-like SaaS. You validate the new version entirely inside the gap, against mirrored artifacts, which makes pre-flight testing and rollback plans more important, not less.

Why this specific OSS stack

The tool list isn't arbitrary — each piece replaces a managed cloud service that simply doesn't exist behind the gap. It's a useful template for anyone assembling a self-hosted platform:

Cilium handles L4 networking and network policy in the kernel via eBPF — the same dataplane the fintech zero-trust talk builds on. It's the L4 floor under the L7 mesh.
MetalLB gives you type: LoadBalancer services with no cloud load balancer — it allocates and advertises external IPs (via ARP or BGP) on bare metal, so the ingress gateways are reachable.
CoreDNS does cross-cluster DNS forwarding so a name in one cluster resolves in another — essential when there's no cloud DNS to lean on.
Loki + Fluent Bit + OTel Collector capture mesh audit and operational logs locally; Prometheus + Cortex store Istio/Envoy metrics at scale (Cortex for long-term, horizontally-scalable storage); Grafana visualises; AlertManager pages.
Cert Manager drives the plug-in CA — the in-gap replacement for public certificate authorities, issuing and rotating the certs that back mesh mTLS.

The pattern: the mesh is the integration point. Notice how every concern — networking, IP allocation, DNS, logs, metrics, certs — plugs into the service mesh rather than into a cloud. In a connected environment you'd wire half of these to managed services; in an air gap the mesh becomes the universal seam that ties self-hosted OSS together. That's the deeper lesson of the talk: a service mesh isn't just traffic management, it's the substrate that lets a pile of OSS behave like a coherent cloud.

FAQ

Why use a full service mesh in an air-gapped cloud instead of plain ingress?

Because you need consistent mTLS, L7 routing, authorization, rate limiting, and telemetry for ~450 microservices, with no SaaS to outsource any of it to. A mesh puts that behaviour in the Envoy data plane (configured declaratively) instead of in 450 codebases, and Istio's multi-cluster/multi-network support matches the primary-remote topology.

How does Istio span multiple isolated networks here?

Istiod runs in the primary (Organizational Infra) cluster; remote user/service clusters join it. East-West Gateways carry cross-cluster, cross-network service traffic; Ingress Gateways handle north-south from the customer/operator networks; an Egress Gateway controls exit. Istiod pushes xDS config across the boundaries to every sidecar.

What do you do when there's no first-class Istio API for a feature?

Use the EnvoyFilter API to reach raw Envoy capabilities — Lua, Wasm, transcoding, rate limiting, compression, external auth, and more. The caveat (from the challenges slide): direct xDS modification can conflict with Istiod-generated xDS, so EnvoyFilter has to be applied carefully.

How is certificate trust handled with no internet?

With a plug-in CA (configured via IstioOperator, backed by Cert Manager) rather than any public root. You bring your own CA into the air gap, and Istio's automatic mTLS (PeerAuthentication) issues workload certs from it.

What keeps the control plane from melting at ~450 services?

Config scoping (Sidecar resources so each proxy only learns relevant services) and delta xDS (push only changes, not full snapshots). On bare metal these are necessities, not tuning — the platform is ultimately constrained by the physical compute available.

How do certificates work with no public CA?

You run your own. Istio's plug-in CA lets you feed it a root/intermediate CA you control (managed via Cert Manager), and Istiod issues short-lived workload certs from it via SDS. There's no Let's Encrypt or public ACME inside the gap, so the entire trust chain — root, intermediates, workload identities — is generated and rotated internally. This is actually cleaner for a high-security environment: you own the whole PKI rather than trusting an external root.

Why not just use plain Kubernetes NetworkPolicy and skip the mesh?

NetworkPolicy gives you L3/L4 allow/deny, which Cilium provides here as the floor. But the requirements list is mostly L7 and cross-cutting: attribute-based routing, mTLS with workload identity, rate limiting, retries, gRPC/HTTP transcoding, per-request telemetry, traffic shifting. None of that is expressible in NetworkPolicy. Doing it per-service across ~450 microservices by hand is the impossibility the mesh exists to remove — the behaviour lives in the data plane, configured declaratively, instead of in 450 codebases.

Is ambient mode better than sidecars for an air gap?

It's a strong fit for the resource constraint. Sidecars add a proxy (and its memory/CPU) to every pod, which on fixed bare-metal compute is a real cost at scale. Ambient moves L4 to a per-node ztunnel and makes L7 opt-in per namespace, cutting the per-pod overhead and the number of proxies to patch. The trade-off is a newer, somewhat less battle-tested model and a different mental model for L7 (waypoints). The deck shows sidecars, but the resource math is exactly what nudges air-gapped operators to evaluate ambient.

Takeaways

Air-gapped means no SaaS crutches. Observability, security, and traffic management for ~450 microservices, all self-hosted.
Istio is the substrate — primary Istiod + remote clusters over east-west gateways, the full API surface (Gateway, VirtualService, DestinationRule, PeerAuthentication, AuthorizationPolicy, Telemetry) doing the work.
EnvoyFilter is the escape hatch — Lua, Wasm, transcoding, rate limiting when no first-class API exists.
Self-hosted OSS everywhere — Cilium, MetalLB, CoreDNS, Prometheus/Cortex, Loki, Cert Manager — because there's no cloud to call.
The hard parts are operational — cardinality and log volume, 1P/3P co-existence, config scoping + delta xDS for scale, and careful ordered multi-cluster upgrades.

Next in Day 2 — When the Edge Can't Afford a Third Node, the dedicated two-node-storage follow-up to Day 1's Rook talk.

References

KubeCon Mumbai 2026 — Day 2 index · the rest of Day 2
Istio — multi-cluster install · primary-remote topologies
Istio EnvoyFilter API · the raw-Envoy escape hatch
Google Distributed Cloud · the air-gapped product context
Day 1 DD14 — Lean observability with Istio · the same mesh, telemetry-first

Service networking in an air-gapped cloud, with Istio.