Rajat Khanna (Senior Tech Lead, CommerceIQ) told one of the sharpest "we left the paved road on purpose" stories of the day. CommerceIQ runs a browser farm — ~8K concurrent browser pods, 10M+ pages scraped daily, 100% of workers on spot — where each request needs one specific, free browser, held until the job is done. That's a routing requirement a Kubernetes Service simply cannot express: a Service load-balances across interchangeable ready pods, but a browser mid-job is exclusive and a caller may need to wait for a free one. The fix isn't a mesh or an eBPF rewrite — it's a small custom proxy that reads the same EndpointSlice API kube-proxy reads and adds two verbs the Service lacks: hold and lease. The talk is a masterclass in "bypass surgically" — one hot path goes custom; everything else stays stock.
This is the second networking deep dive on Day 2, alongside air-gapped Istio and the fintech Cilium talk. Where those add a mesh, this one shows the opposite move: drop below Services to raw Kubernetes primitives.
What CommerceIQ runs
The scale sets up the problem:
| Metric | Value |
|---|---|
| Pages scraped / day | 8–16M (10M typical) |
| Retailers, daily | 1,000+ |
| Concurrent browser pods | ~8K (varies with queue) |
| Workers on spot/preemptible | 100% (proxy + control plane on a stable pool) |
The defining fact: a pod serves one session at a time. It's reused across many jobs, but recycled only when the HPA scales down or spot reclaims its node. One browser, one job, right now.
Why the platform exists
Platform v1 coupled browsers to business logic: workers pulled jobs off pub/sub queues, with no service routing anywhere near the hot path. Platform v2 isolated browser automation as a service: request in → rendered page out, clean API, business logic kept out of the browser fleet.
The requirement a Service can't express
Stated plainly: a request doesn't want a load-balancer; it wants one specific, free browser. Two properties:
| Property | Meaning |
|---|---|
| Exclusive | One job per browser at a time. A browser mid-job is not a shared backend — no second caller, ever. |
| Held | The incoming connection waits until a browser is free. None free? Start one and keep holding. |
How kube-proxy + Services actually work
To see the mismatch, recall the normal model: kube-proxy load-balances each connection across all ready pods — any ready replica will do, because the pods are stateless and interchangeable.
Fig 1 — the normal Service model: kube-proxy spreads connections across any ready, interchangeable pod. Perfect for stateless backends — wrong for an exclusive lease.
Two things a Service fundamentally cannot do here:
- It can't hold. A Service answers now or never — there is no "wait until a browser is free." CommerceIQ's caller often must wait, sometimes for one that doesn't exist yet.
- It can't lease. It spreads connections across every ready browser. They need one free browser, given to one request, untouchable until the job is done.
EndpointSlice — the public primitive
The EndpointSlice API is how Kubernetes tracks which pods back a Service and whether each is ready — it's what kube-proxy watches to build its routing tables. The insight: that API is public. Anything can watch it. CommerceIQ's proxy watches EndpointSlices to maintain an in-memory free-pool of browsers, then leases from it.
iptables rules, increasingly via IPVS or eBPF in modern CNIs. When a packet hits a ClusterIP, the kernel picks a backend pod, classically with random/round-robin selection per connection. That's the crux: the choice is made in the kernel datapath with no application state, no notion of "this backend is mid-job," and no ability to block. It's a stateless connection spreader by design — which is exactly right for stateless replicas and exactly wrong for an exclusive lease. You can't teach iptables "wait until a browser frees up"; that's an application concern, so it has to live in an application-level proxy.It's worth being precise about EndpointSlice itself, because it's the load-bearing primitive. It replaced the older Endpoints object (which crammed every backend of a Service into one resource that didn't scale past a few thousand endpoints) with sliced, paginated sets of ~100 endpoints each — far kinder to the API server and to watchers at 8K pods. Each endpoint carries conditions: ready, serving, and terminating. CommerceIQ's trick hinges on redefining what ready means for their pods (via the readiness probe): a browser reports ready only when it's free for a job, so the EndpointSlice ready condition becomes a live "free/busy" flag the proxy can watch. They're not adding a new API — they're overloading a condition that already exists and is already maintained by the kubelet.
The browser-farm machine
The full system, with the proxy as the only long-lived component near the hot path:
Fig 2 — the browser-farm: a caller hits the proxy (in-memory free-pool); free → lease + dial the pod IP directly; none free → park in the hold-queue; the proxy publishes queue depth to a metric server → HPA → scale the spot fleet.
Following one request through it (the deck's numbered flow):
- A request arrives needing its own browser → hits the proxy.
- Free → lease + dial the pod IP directly. None free → the caller parks in the hold-queue.
- The proxy publishes queue depth; the metric server scrapes it.
- Metrics flow out via the External Metrics API to the HPA.
- The HPA scales the fleet → a new browser pod starts.
- The pod passes readiness → its EndpointSlice flips → it joins the free-pool (via the EndpointSlice watch).
- A parked request wakes, leases it, and dials it directly.
The sharp edge — "ready" is not "free"
They got to redefine readiness: a browser is "ready" when it's free for a job, so readiness flips on every job and the proxy only ever targets a genuinely free browser. That's clean. The one rough edge comes from the watch, not from readiness.
And mid-session death is handled honestly: if a pod dies mid-job, that one job is lost and retried fresh upstream — blast radius of one. They explicitly don't fake a resume. Knowing what not to recover is as important as the recovery.
A bonus: MinIO twice — cache + profiles
A neat aside with real economics. 10M pages/day re-fetch the same JS bundles, fonts, and images from the same retail sites, over and over, across the priced public internet. So an S3-compatible MinIO cache sits in-cluster next to the workers:
- Repeat asset → served on the cluster network, never crossing the internet again.
- First/uncached fetch → priced egress to the retailer.
- Locality is the whole point — a managed bucket would re-introduce the egress you're trying to kill.
- The bigger the fleet, the higher the repeat rate — the cache earns more at scale, not less.
- A second MinIO stores Chrome profiles — warm cookies + fingerprints that outlive ephemeral pods, so recycled pods don't look ephemeral to the sites they visit.
Why not the obvious alternatives?
The natural question — "couldn't you do this with something off the shelf?" — is worth walking, because the answer is what justifies the custom proxy.
| Alternative | Why it doesn't fit |
|---|---|
| Headless Service + client-side LB | Gives you the pod IPs (good!) but still no exclusive lease or hold — every client would independently pick, and two could pick the same free browser. You'd end up building the lease layer anyway. |
Session affinity (sessionAffinity: ClientIP) | Pins a client to a pod by source IP — but that's stickiness, not exclusivity. It doesn't reserve a pod, doesn't wait for a free one, and breaks under NAT/shared egress. |
| A job queue (back to v1) | Workers pulling jobs is exactly what v2 moved away from — it re-couples business logic into the browser fleet and loses the clean request-in/page-out API. The queue still exists here, but as a hold queue inside the proxy, not a coupling. |
| Service mesh (Istio/Linkerd) | Meshes add L7 routing, retries, mTLS — none of which express "lease one exclusive backend and block until free." You'd pay the sidecar tax and still write the lease logic. |
| A scheduler/CRD + operator | Modeling each lease as a custom resource works but adds API-server write load per job (millions/day), reconcile latency on the hot path, and a lot of machinery. In-memory lease in the proxy is faster and simpler for a per-request decision. |
Service is a convenience over EndpointSlice, and when the convenience is the wrong shape, you can drop one level without rebuilding the floor.What they'd tell you
The closing five points are the transferable lesson, and they're good enough to quote whole:
- Match the primitive to the problem. A Service load-balances; they needed an exclusive, availability-aware lease.
- The primitives are public. EndpointSlice is the same API kube-proxy reads — you can read it too.
- Bypass surgically. One hot path on the EndpointSlice API; everything else stays on stock kube-proxy.
- "Ready" is not ready. Design for stale endpoints and cold starts from day one.
- No fancy tooling — just deep Kubernetes. No mesh, no eBPF rewrite, no new CRDs.
FAQ
Did they actually remove kube-proxy from the cluster?
No — the title is provocative. They bypassed it for one hot path (browser dispatch), where a Service is the wrong shape. Everything else in the cluster still uses stock kube-proxy and Services. "Bypass surgically" is the precise claim.
Why can't a Kubernetes Service do this?
A Service load-balances connections across interchangeable ready pods. It can't hold a request until a specific resource is free, and it can't lease one pod exclusively to one caller until a job finishes. Browser dispatch needs both — exclusive and held — so the Service abstraction doesn't fit.
What is the EndpointSlice API and why use it directly?
EndpointSlice is the Kubernetes API that tracks which pods back a Service and their readiness — it's what kube-proxy watches. Because it's a public API, the custom proxy watches it too, building an in-memory free-pool of browsers, then adds hold + lease semantics on top. Same data, different routing logic.
How do they scale, if not on CPU?
On hold-queue depth — the number of parked callers waiting for a free browser. The proxy publishes that, a metric server scrapes it, and it flows via the External Metrics API to the HPA, which scales the (spot) browser fleet. CPU can't tell you whether a browser is available; the queue can.
What happens when a dial lands on a busy or dead pod?
Because the EndpointSlice watch lags reality, a stale pod can be in the pool. A failed dial sets that pod aside for a couple of seconds and re-queues the caller to the next free browser — invisibly, before the session starts. If a pod dies mid-session, that single job is lost and retried fresh upstream (blast radius of one); they don't fake a resume.
Why not just use a headless Service and let clients pick?
A headless Service gives you the pod IPs, which is half the answer — but with no coordination, two clients can pick the same "free" browser, and there's still no waiting for a free one. You'd have to build the exclusive-lease and hold-queue logic on top anyway, so they built exactly that as a small central proxy rather than scattering it across every client.
Why is the EndpointSlice watch "always a half-step behind"?
Kubernetes is eventually consistent: when a pod goes busy or is reclaimed, the kubelet updates its readiness, the control plane updates the EndpointSlice, and only then does the watcher receive the event — milliseconds-to-seconds later. So any cached free-pool is a slightly stale snapshot. That's not a bug to fix but a property to design around: assume a dial can hit a stale pod, and recover by re-queuing on a failed dial.
Does redefining readiness as "free for a job" have side effects?
It means the EndpointSlice ready condition flips on every job, which is high-churn but exactly the live free/busy signal they want. The trade-off is that anything else relying on a normal Service in front of these pods would see them constantly leaving and rejoining — which is why they bypass the Service for this hop entirely and consume EndpointSlice directly. Overloading readiness only works because they control both the probe and the consumer.
Is this a pattern other teams should copy?
Only when the requirement genuinely doesn't fit a Service — an exclusive, availability-aware, hold-until-free lease. For ordinary stateless backends, a Service is correct and you should not touch EndpointSlice. The transferable lesson isn't "write a custom proxy," it's "know that Kubernetes' routing primitives are public and you can drop one level surgically when, and only when, the abstraction is the wrong shape."
Takeaways
- Some routing needs are exclusive + held — one specific free resource, leased until done — which a Service can't express.
- EndpointSlice is a public API. Watch the same data kube-proxy reads and add the semantics you need (hold, lease) in a small proxy.
- Bypass surgically — one hot path goes custom; everything else stays stock. No mesh, no eBPF, no CRDs.
- Scale on the real signal — hold-queue depth via External Metrics → HPA, not CPU.
- "Ready" is your definition, and the watch lags reality — design for stale endpoints; accept a blast radius of one for mid-job death instead of faking resumes.
Next in Day 2 — Using Buildpacks To Reduce Container Bloat and Attack Surface.
References
- KubeCon Mumbai 2026 — Day 2 index · the rest of Day 2
- Kubernetes EndpointSlices · the API kube-proxy reads
- Demo — kubecon-endpointslice-demo · watch → free-pool → lease → dial
- HPA & External Metrics · scale on queue depth
- Day 1 DD13 — Serving LLMs on Kubernetes · another custom-scaling story