Sai Bharadwaj Avvari (Salesforce) made the security case for Cloud Native Buildpacks. The problem: 87% of container vulnerabilities hide in packages never loaded at runtime, so platform teams drown in CVE noise triaging dormant utilities — which are also a real attack surface (Living off the Land). The two usual "fixes" both fail: hyper-bloat (one >10GB global base image with every agent baked in) or fragmentation (hundreds of unverified per-team Dockerfiles). Cloud Native Buildpacks (CNB) cut a third path: turn source into OCI images without Dockerfiles, in clean layers, and then rebase — swap the OS layer by SHA without rebuilding the app. At fleet scale that turns one CVE from O(N) rebuilds (hours-to-days) into an O(1) stack update (minutes) — "the patching waterfall."
This is the supply-chain anchor of Day 2's platform thread. It's the build-time counterpart to Day 1's verifiable SBOMs (DD08) and dovetails with the TAG DevEx push toward signed, OCI-packaged artifacts.
The vulnerability crisis
The headline stat reframes the whole CVE-triage problem:
It helps to understand why images bloat in the first place, because the bloat isn't laziness — it's the path of least resistance the tooling pushes you onto. A typical Dockerfile starts FROM ubuntu or FROM node, and those base images ship a full distro: a package manager, a shell, coreutils, networking tools, often a compiler toolchain. Then every apt-get install pulls a transitive dependency tree — install one library and you get its ten dependencies, each with their own. The app might call exactly none of those at runtime, but they're all in the image, all scanned, all counted against you. A scanner like Trivy or Grype walks the package database and reports every CVE in every installed package, with no knowledge of whether the code is ever executed. That's the 87%: real CVEs, in real packages, in code paths the app never touches.
curl or wget to pull a payload, bash to run it, openssl to exfiltrate over TLS, a compiler to build a rootkit, kubectl if it leaked in. An image with none of those is dramatically harder to operate inside — the attacker has to bring their own tools, which is noisier and often blocked by a read-only filesystem. "Minimal image" isn't a tidiness preference; it's defense in depth. The Day-1 user-namespaces talk and this one are two angles on the same goal: shrink what an intruder can do.The middle-tier trap
Most enterprises pick one of two losing strategies for their base images:
| Option A — Hyper-Bloat | Option B — Fragmentation |
|---|---|
| Bake every shared agent, dependency, and certificate into one global enterprise base image. Result: massive container size (>10GB), huge security noise, a severe attack footprint. | Give developers custom base images, but they add their own libraries inside raw, unverified Dockerfiles. Result: hundreds of unique, unpatched configurations in active production — unauditable, each its own attack surface. |
Both are dead ends: one image too fat to secure, or too many images to track. Buildpacks are the escape from this dichotomy.
What Cloud Native Buildpacks are
A buildpack is a standard for converting source code into OCI images without Dockerfiles. You point it at your source; it detects the language, pulls the right runtime, and produces a layered application image — no hand-written Dockerfile, no per-team drift.
Fig 1 — buildpacks turn source into a layered OCI image with no Dockerfile. CNCF-graduated, supported by Heroku, Google Cloud, Paketo, Tanzu, GitLab, DigitalOcean, Azure, Spring Boot, Tekton, and more.
The idea isn't new — buildpacks were born at Heroku (Joe Kutner, who opened this talk, is a long-time Heroku/buildpacks figure) and were the engine behind git push heroku main "just working." Cloud Native Buildpacks is the CNCF standardisation of that idea on top of OCI, so the same model works on any registry and any Kubernetes. What makes it more than "magic Dockerfile generator" is the precise vocabulary, which is worth learning because the rebasing trick depends on it:
| Term | What it is |
|---|---|
| Buildpack | A unit that detects whether it applies (is there a pom.xml? a package.json?) and contributes layers (a JDK, node_modules, the compiled app). |
| Stack (now "run/build images") | The pair of base images: a fat build-image (compilers, tools) and a slim run-image (only what runtime needs). Rebase swaps the run-image. |
| Builder | A packaged, versioned bundle of buildpacks + a lifecycle + a stack — the thing a platform team curates and hands to developers. |
| Lifecycle | The orchestrator that runs the phases in order and produces the final OCI image with reproducible layers. |
| Platform | What invokes the lifecycle: the pack CLI locally, or kpack/Tekton/Spinnaker in CI. |
Under the hood the lifecycle runs distinct phases — roughly detect (which buildpacks apply), analyze (read metadata from any previous build), restore (pull cached layers), build (each buildpack contributes its layers), and export (assemble the OCI image, reusing unchanged layers by digest). The export phase is the quiet hero: because each layer is content-addressed and the lifecycle knows exactly which buildpack produced which layer, it can reuse layers across builds and — crucially — swap one layer out later without disturbing the others. That property is what makes rebase possible at all.
Reduce, Reuse, Rebase
The buildpack mantra. The image is cleanly layered — OS → runtime (e.g. jdk) → app — and each layer has its own content hash. The magic verb is rebase: swap the bottom OS layer for a patched one while keeping the app and runtime layers byte-for-byte identical.
Fig 2 — rebase is a SHA swap: the OS layer changes from a vulnerable hash to a patched one; the app (4a090ac) and jdk (738fafa) layers are untouched. No recompile.
The patching waterfall
The architecture splits responsibilities into tiers, and patches "waterfall" down without rebuilds:
| Tier | Owner / role |
|---|---|
| Base OS tier | Platform SecOps manages minimal run stacks. |
| Middleware builder | Language runtimes & certificates cached once. |
| App compilation | Business logic compiled independently of the OS. |
| Waterfall flow | Security patches flow down to every image without rebuilds. |
The two tiers in config
CNB decouples the environment where you compile from the environment where you run — that separation is the whole point.
# Tier 1 — the base stack (stack.toml)
[stack]
id = "io.buildpacks.stacks.jammy"
build-image = "corp/build-jammy" # fat: compilers, tools
run-image = "corp/run-jammy-tiny" # tiny: only runtime needs
# Tier 2 — custom builder (builder.toml)
# platform teams compile & cache domain-specific deps once
[[buildpacks]]
uri = "docker://gcr.io/paketo-buildpacks/java"
[[order]]
group = [{ id = "corp/ca-certs" }, { id = "corp/java-runtime" }]
build-image and ships only the slim run-image (just the runtime). That alone eliminates most of the "Living off the Land" binaries — the app runs without curl, gcc, or bash in the image at all.Rebase: the SHA256 swap, at scale
# single image pack rebase my-app:latest # the entire fleet (the Waterfall) kpack-controller --monitor corp/run-base:latest
For one image you run pack rebase; for the whole fleet, a controller (kpack) monitors the base run-image and rebases every dependent image automatically when it changes. What used to take days now takes minutes.
A worked example — from source to rebased fleet
Concretely, here's the loop a platform team lives in. A developer never writes a Dockerfile; they just build:
# Developer: source -> image, no Dockerfile pack build payments-api \ --builder corp/builder:jammy \ --publish registry.corp/payments-api:1.4.2 # The lifecycle ran: detect -> analyze -> restore -> build -> export # Result image layers (bottom to top): # run-image (OS) sha256:8e3e... <- owned by SecOps # ca-certs sha256:11aa... <- owned by platform # jre sha256:738f... <- owned by platform # application sha256:4a09... <- owned by the dev
Weeks later, a critical glibc CVE lands in the OS layer. Nobody touches application code. SecOps publishes a patched run-image, and the controller fans it out:
# SecOps: patch the base once docker push corp/run-jammy-tiny:2026-06-19 # glibc fixed # kpack notices the base digest changed and rebases every # dependent image automatically — app/jre/ca-certs layers # are reused by digest; only the bottom OS layer is swapped. # payments-api:1.4.2 -> new digest, same app layer 4a09... # orders-api:3.1.0 -> new digest, same app layer ... # ... (N services, each a metadata-only SHA swap)
The key thing to notice: the application layer digest 4a09… is identical before and after. The running app is bit-for-bit the same code; only the foundation moved. That's why you don't re-run the app's test suite for an OS patch — nothing about the app changed. Contrast the Dockerfile world, where bumping the base means a full rebuild that could change anything, so you're obligated to re-test everything.
Why it wins — four advantages
1 · O(1) patch propagation vs O(N) rebuilds
The central economic argument. Traditionally, one CVE means N rebuilds — every service runs its CI pipeline (build, test, push image), taking hours to days. With the waterfall, one stack update triggers a rebase per service that takes minutes.
Fig 3 — "fix once, protect the fleet": an O(1) stack update replaces O(N) build queues. Same CVE, minutes instead of days.
2 · True separation of ownership
Developers own application code and user features; security/platform engineers own the base OS and compliance (glibc, certs). Updates flow automatically without developer intervention. Contrast with fragmented Dockerfiles where "everyone owns everything, so nothing is secured."
3 · Elimination of bloat / attack surface
A hyper-bloated corporate image carries Java, Python, Node, Ruby, GCC/build tools, curl, bash, git, vim, openssl, monitoring and logging agents — more packages = more CVEs = more risk. A purpose-built runtime image carries only the app, the JRE/runtime, and required libraries: only what you need = minimal attack surface.
4 · Fleet-wide consistency & auditability
Traditional chaos: service A on Ubuntu 22.04, B on Debian 11, C on Alpine 3.18, D on Ubuntu 20.04, N on "unknown" — different base OSes, hard to audit, easy to miss vulnerabilities. The waterfall: every service on one standardized buildpack stack, yielding consistent SBOMs, provenance, and compliance visibility across the fleet.
An adoption playbook
The talk's architecture is the destination; getting there is a migration. A sane order of operations for a platform team that wants the waterfall without a big-bang rewrite:
- Pick or build a builder. Start with an off-the-shelf builder (Paketo, Google Cloud, Heroku) for your dominant language. Don't build a custom builder on day one — prove the model first.
- Define the run-image as a security artifact. Have SecOps own a minimal run-image with a patch cadence. This is the layer the whole waterfall pivots on, so it gets the governance a base image deserves.
- Onboard one noisy service. Convert a single high-CVE-churn service from Dockerfile to
pack build. Compare the scan results — the drop in CVE count from removing build tools and dormant packages is usually the moment leadership buys in. - Wire kpack into CI. Replace the per-service "docker build" step with a buildpack build, and stand up the controller that watches the run-image. Now you have the fan-out machinery.
- Flip the patch process. The next time a base CVE lands, patch the run-image and let the controller rebase, instead of opening N rebuild tickets. Measure the exposure-window reduction — that's the number that justifies the program.
- Expand the ownership split. Once the mechanics work, formalise it: devs own app layers, platform owns runtime/certs, security owns the OS. The org change is what turns a tool into a durable practice.
Where buildpacks sit vs the alternatives
Buildpacks aren't the only way to escape the bloat trap, and being honest about the landscape makes the choice defensible. The main contenders for "build a container image without a hand-rolled Dockerfile" or "build a minimal image":
| Approach | How it works | Trade-off vs CNB |
|---|---|---|
| Cloud Native Buildpacks | Detect language, contribute layers, export reproducible OCI image; rebase to patch. | The only one with first-class fleet rebasing; polyglot; needs a curated builder. |
| Distroless (base image) | A hand-written Dockerfile FROM a minimal Google distroless base (no shell/pkg-mgr). | Tiny & LotL-resistant, but you still write/maintain the Dockerfile and rebuild per CVE — no O(1) rebase. |
| ko (Go) | Builds Go apps straight to a minimal image, no Dockerfile. | Excellent but Go-only; CNB is polyglot. |
| Jib (Java) | Maven/Gradle plugin builds layered Java images, no Docker daemon. | Great for JVM; CNB covers many languages with one model. |
| Hand-written Dockerfile | Full control, multi-stage builds for slimness. | Maximum flexibility, maximum drift — the fragmentation problem this talk is about. |
The honest framing: a disciplined multi-stage Dockerfile FROM distroless gets you a small, LotL-resistant image too. What it does not get you is the O(1) patching waterfall — when the base has a CVE, you still rebuild and re-test every service. CNB's distinctive value is less "smaller images" (others do that) and more "patch the base once and propagate by digest swap to the whole fleet, without re-running anyone's build." If your fleet is small, distroless may be simpler; if you operate hundreds of services and patch constantly, the rebase economics are the whole argument.
Limitations & when to think twice
No tool is free. The realistic caveats a platform team hits:
- Builder governance is now your job. The flip side of "one standardized stack" is that someone must own, version, test, and patch the builder and run-image. You've traded N Dockerfiles for one critical shared artifact — better, but not zero.
- Less escape-hatch control. Highly custom native dependencies, exotic system libraries, or unusual build steps can fight the buildpack model. Most stacks support custom buildpacks, but it's more work than a one-line
RUN apt-get install. - Image size isn't automatically tiny. A Paketo full builder run-image is small but not distroless-tiny; if you need the absolute minimum, you tune the run-image or combine approaches.
- ABI discipline on rebase. As the caveat above notes, the run-image must stay ABI-compatible. A major glibc bump may require a real rebuild, not a rebase — rebase is for the steady stream of patch-level CVEs, which is most of them.
- Org change, not just tooling. The ownership split (security owns base, devs own app) only pays off if the org actually adopts it. Drop buildpacks into a team that still wants per-service base-image control and you get the costs without the benefits.
FAQ
What's the difference between a buildpack rebuild and a rebase?
A rebuild reruns the whole build (compile app, assemble all layers) — slow, and N of them for N services. A rebase swaps only the OS (base) layer by SHA while keeping the app and runtime layers byte-identical — fast, and it can be applied fleet-wide from one stack update. Rebase is what turns O(N) patching into O(1).
Why is 87% of vulnerabilities "never loaded" such a big deal?
Because it means most CVE triage is wasted effort on code the app never executes — and those dormant binaries are also the attack surface for Living-off-the-Land techniques. Removing them (slim run-image, no build tools at runtime) cuts both the noise and the real risk at once.
Do I have to give up Dockerfiles entirely?
Buildpacks replace hand-written Dockerfiles for application images — you point them at source and get a layered OCI image. The point is to remove the per-team, unverified Dockerfile sprawl (Option B) and the one giant base image (Option A) in favour of a governed buildpack stack.
What is kpack's role?
kpack is the Kubernetes controller that operationalises the waterfall: it monitors the base run-image and automatically rebases every dependent image when the base changes. That's how "fix once" propagates to the whole fleet in minutes without per-service CI runs.
How do I try it?
The Pack CLI: brew tap buildpack/tap, brew install pack, then pack build myimage. Community at buildpacks.io and #buildpacks on CNCF Slack.
If I rebase the OS, don't I still need to re-test the whole app?
For a patch-level, ABI-compatible OS change, no — the application and runtime layers are byte-identical (same digests), so the app you tested is literally the app that ships. You test the run-image once (the platform team does this), not once per dependent service. A major, ABI-breaking base change is the exception: that's a real rebuild with real re-testing, but those are rare compared to the constant stream of patch CVEs that rebase handles.
How does this compare to just using distroless base images?
Distroless gets you a small, shell-less, LotL-resistant image — but you still hand-write the Dockerfile and rebuild every service when the base has a CVE. Buildpacks add the O(1) patching waterfall (patch base once, rebase the fleet by digest swap) and remove the per-team Dockerfile entirely. For a handful of services distroless may be simpler; for a large, constantly-patched fleet the rebase economics win. They're not mutually exclusive — a buildpack run-image can itself be very minimal.
Does the buildpack image include an SBOM?
Yes — CNB attaches build metadata including an SBOM describing exactly which buildpacks and dependencies produced each layer. Combined with reproducible builds, that gives you provenance you can verify, which ties directly into the supply-chain story (signed artifacts, SLSA provenance) the rest of the platform thread is about.
Takeaways
- Most container CVEs are noise — 87% sit in packages never loaded — but they're still attack surface (LotL).
- Hyper-bloat and Dockerfile fragmentation both lose. CNB is the third path: source → layered OCI image, no Dockerfiles.
- Rebase is the superpower — swap the OS layer by SHA, app/runtime untouched; build-image ≠ run-image strips build tools from runtime.
- O(1) beats O(N) — one stack update rebases the whole fleet in minutes (kpack), vs N rebuilds over hours-to-days.
- Clean ownership + auditability — security owns the base, devs own the app, and one standardized stack gives fleet-wide SBOMs and provenance.
Next in Day 2 — Validating RK3588 for KubeEdge, simulating ARM64 edge fleets without the hardware.
References
- KubeCon Mumbai 2026 — Day 2 index · the rest of Day 2
- Cloud Native Buildpacks · the spec, Pack CLI, builders
- kpack · fleet rebase controller
- Paketo Buildpacks · language-family buildpacks
- Day 1 DD08 — Verifiable, private SBOMs · the supply-chain sibling