← BPF Performance Tools

BOOK NOTES · BPF PERFORMANCE TOOLS · CHAPTER 17

BPF Performance Tools Chapter 17 — Other BPF Tools.

bpf-performance-toolschapter-17ecosystemkubernetesobservabilitycilium

// the one-minute version

BCC and bpftrace are the command-line workhorses, but the BPF ecosystem is much wider. Metrics: ebpf_exporter turns BPF histograms into Prometheus/Grafana dashboards. Kubernetes: kubectl-trace, Inspektor Gadget, and Pixie bring BPF tracing to clusters with auto-correlation to pods. Networking/security: Cilium/Hubble and Tetragon build production platforms on BPF. Continuous profiling: Parca/Pyroscope profile fleets always-on. The skill from this book — knowing what BPF measures — transfers directly; these tools just package and scale it.

The command-line tools you've learned are the foundation, but BPF has grown into a whole ecosystem of platforms that package its power for dashboards, clusters, and always-on production use. You don't need to abandon bpftrace — these build on the same kernel events you now understand. This chapter surveys the wider landscape so you know what exists beyond the terminal, and how the analysis skills from this book scale up to fleet-wide observability.

01 Beyond the command line

BCC and bpftrace are interactive, per-host tools — perfect for ad-hoc analysis on a box. But production observability also wants continuous metrics, dashboards, cluster-wide visibility, and always-on collection. The ecosystem fills those gaps by wrapping BPF in agents, exporters, and platforms. Crucially, they all stand on the same ground you've learned: the same probes, the same in-kernel aggregation, the same "measure latency as a histogram" patterns. Understanding what BPF can measure — the real skill of this book — is exactly what lets you use, evaluate, and trust these higher-level tools.

key ideaThe ecosystem tools don't replace your knowledge — they scale it. Every dashboard, cluster gadget, and continuous profiler is BPF underneath, doing the same in-kernel aggregation you've been writing by hand. So the understanding of probes, latency histograms, and overhead transfers directly: you know what these platforms are really measuring, where their numbers come from, and what they can't see. Command-line BPF is how you learn; the platforms are how you operate at scale.

02 Metrics and dashboards: ebpf_exporter

To turn BPF into monitoring, you need its histograms in your metrics system. ebpf_exporter (from Cloudflare) runs BPF programs and exposes their results as Prometheus metrics, which Grafana then graphs. So a BPF latency histogram (like biolatency's) becomes a continuous, alertable dashboard panel — heatmaps of disk latency, run-queue latency over time, custom in-kernel metrics no off-the-shelf exporter provides. This bridges ad-hoc tracing and continuous monitoring: you prototype a metric with bpftrace, then promote it to a permanent dashboard via the exporter. It's the natural path from "I traced this once" to "we watch this always."

03 Kubernetes tracing: kubectl-trace, Inspektor Gadget, Pixie

Clusters need BPF that's cluster-aware. kubectl-trace runs bpftrace programs on cluster nodes via kubectl, so you can trace a node from your normal workflow. Inspektor Gadget packages many BPF tools as "gadgets" that automatically correlate events to pods and namespaces — solving the container-attribution problem (Chapter 15) for you. Pixie goes further: auto-instrumenting a cluster with BPF to capture requests, latencies, and traces with no code changes, presented in a UI. These bring the per-tool power you've learned to Kubernetes with the identity mapping handled — exactly the friction point of raw container tracing, automated.

The BPF ecosystem, all on the same foundationmetricsebpf_exporter → GrafanakubernetesPixie · Inspektor Gadgetnet / securityCilium/Hubble · Tetragoncontinuous profilingParca · Pyroscopecli foundationBCC · bpftraceBPF: probes · maps · in-kernel aggregation

Fig 1 — Dashboards, cluster gadgets, network platforms, and profilers all sit on the same BPF foundation you've learned.

04 Networking and security platforms: Cilium, Hubble, Tetragon

Some of the most mature BPF software is in networking and security. Cilium uses BPF for high-performance container networking, load balancing, and network policy enforcement — replacing iptables with BPF programs in the data path. Hubble (its observability layer) gives flow-level visibility into cluster networking. Tetragon applies BPF to runtime security observability and enforcement (the security ideas from Chapter 11, productized). These show BPF's range beyond performance tracing — into the data plane and the security plane — and they're widely deployed, so understanding their BPF basis helps you reason about their behavior and overhead.

05 Continuous profiling

A category that BPF made practical: always-on, fleet-wide profiling. Tools like Parca and Pyroscope use BPF to sample stacks across every host continuously, storing the profiles so you can get a flame graph for any service at any past moment — and compare "before vs after a deploy" to catch regressions. Because BPF sampling is low-overhead (Chapter 6), running it always-on across a fleet is affordable, which was impossible with heavier profilers. This turns profiling from a reactive, one-host activity into a standing capability — performance data is just always there when you need it.

06 Other front-ends and libraries

A few more worth knowing. ply is a lightweight bpftrace-like tracer with minimal dependencies, handy on embedded or constrained systems. libbpf (with CO-RE, Chapter 2) is the modern C library for building portable BPF tools that compile once and run across kernels — increasingly the foundation under newer tools, replacing BCC's runtime compilation. bpftool (seen in Chapter 2) inspects and manages BPF objects directly. And language bindings (Go's cilium/ebpf, Rust's aya) let developers embed BPF in their own software. The ecosystem spans from quick tracers to production libraries, all on the same kernel capability.

the catchIt's tempting to think these polished platforms mean you no longer need to understand BPF — that a Grafana panel or a Pixie UI replaces knowing what's underneath. The opposite is true: when a dashboard shows a surprising number, a heatmap looks wrong, or a tool reports "nothing," you need to know what BPF was actually measuring to interpret or debug it — which probe, which aggregation, what it can't see (the hypervisor wall, the cache, the off-CPU time it isn't capturing). The platforms are only as trustworthy as your ability to reason about their BPF foundation. The command-line skills aren't obsoleted by the GUIs; they're what make the GUIs usable.

07 Choosing where to operate

A rough guide. For ad-hoc, deep, one-host analysis, drop to BCC/bpftrace — nothing beats their flexibility and immediacy. For continuous metrics and alerting, promote your key BPF measurements to ebpf_exporter + Grafana. For Kubernetes, reach for Inspektor Gadget or Pixie to get pod-correlated tracing without the attribution legwork. For always-on profiling, run Parca/Pyroscope. For networking/security as infrastructure, that's Cilium/Tetragon territory. The command-line tools remain your scalpel; the platforms are the standing instrumentation around it.

08 A practical adoption path

(1) Master the command-line tools first (this book) — they teach what BPF measures. (2) When a one-off trace becomes a recurring need, promote it to a dashboard via ebpf_exporter. (3) On Kubernetes, adopt a cluster gadget (Inspektor Gadget/Pixie) for pod-aware tracing. (4) Add continuous profiling (Parca/Pyroscope) so flame graphs are always available. (5) Let networking/security platforms (Cilium/Tetragon) handle the data and security planes. Throughout, your understanding of probes and aggregation lets you trust and debug each layer. Start at the terminal; grow outward.

common catches & gotchas

  • Thinking GUIs replace understanding — When a panel surprises you, you need to know what BPF measured to debug it. The platforms rest on the skills in this book.
  • Ignoring overhead at scale — A trace that's cheap on one host can add up across a fleet. Continuous/always-on tools still cost; understand their sampling and aggregation.
  • Reinventing what exists — Before scripting cluster-wide bpftrace, check if Inspektor Gadget/Pixie already do it with pod correlation built in.
  • libbpf vs BCC confusion — Modern CO-RE/libbpf tools avoid BCC's runtime compilation; on minimal hosts prefer them. Know which your tooling uses.
  • Platform blind spots — These tools inherit BPF's limits (the hypervisor wall, cache invisibility, off-CPU gaps). A dashboard can't show what BPF can't see.
  • Version/kernel requirements — Ecosystem tools often need recent kernels and BTF; check requirements before relying on them in production.

09 Questions engineers actually ask

How do I turn a bpftrace metric into a Grafana dashboard?

Use ebpf_exporter (Cloudflare): it runs BPF programs and exposes their histograms/counters as Prometheus metrics, which Grafana graphs and alerts on. You prototype the measurement with bpftrace, then promote it to a permanent, alertable dashboard panel — turning a one-off trace into continuous monitoring.

What's the best way to use BPF on Kubernetes?

Cluster-aware tools: kubectl-trace to run bpftrace on nodes, Inspektor Gadget for BPF tools with automatic pod/namespace correlation, or Pixie for auto-instrumented request/latency tracing with a UI. They solve the container-attribution problem (mapping events to pods) that raw tracing makes tedious.

Is continuous profiling really affordable across a whole fleet?

Yes — that's what BPF enabled. Low-overhead BPF stack sampling (a fixed sample rate, in-kernel aggregation) makes always-on profiling cheap enough to run everywhere. Parca and Pyroscope store the profiles so you can get a flame graph for any service at any past time, including before/after a deploy.

Do these platforms mean I don't need to learn bpftrace?

No — they're built on the same BPF foundation, and you need that understanding to interpret their output, debug surprising numbers, and know their blind spots. The command-line tools teach what BPF measures; the platforms package and scale it. The skills compound, not replace.

What are Cilium and Tetragon?

Production BPF platforms beyond performance tracing: Cilium uses BPF for container networking, load balancing, and policy (with Hubble for flow observability); Tetragon applies BPF to runtime security observability and enforcement. They show BPF's reach into the data plane and security plane, and they're widely deployed.

10 Key takeaways

  • BCC/bpftrace are the command-line foundation; the ecosystem packages BPF for dashboards, clusters, and always-on use.
  • ebpf_exporter turns BPF histograms into Prometheus/Grafana dashboards.
  • Kubernetes: kubectl-trace, Inspektor Gadget, and Pixie add pod-aware tracing.
  • Cilium/Hubble and Tetragon build networking and security platforms on BPF.
  • Continuous profiling (Parca/Pyroscope) makes fleet-wide, always-on flame graphs affordable.
  • libbpf/CO-RE, ply, bpftool, and language bindings round out the toolkit.
  • The platforms scale your knowledge, not replace it — and inherit BPF's blind spots.
// chapter cheatsheetBPF ecosystem

metrics & dashboards

ebpf_exporter (Cloudflare)BPF histograms → Prometheus → Grafana.

kubernetes

kubectl-traceRun bpftrace on cluster nodes.
Inspektor GadgetBPF tools with pod/namespace correlation.
PixieAuto-instrumented requests/latency, UI.

networking & security

Cilium / HubbleBPF networking, policy, flow observability.
TetragonRuntime security observability + enforcement.

continuous profiling

Parca · PyroscopeAlways-on fleet-wide flame graphs.

libraries & light tracers

libbpf + CO-RE · cilium/ebpf · ayaBuild portable BPF tools (Go/Rust/C).
ply · bpftoolLight tracer / inspect BPF objects.

11 Wrapping up

BPF has grown from command-line tracers into a full ecosystem — dashboards, cluster gadgets, network and security platforms, and continuous profilers — all resting on the foundation you've learned. The platforms scale your skills outward; they don't replace them. One chapter remains, and it's the most practical: the traps, tricks, and common problems that bite everyone using BPF tools. Next: Tips, Tricks & Common Problems.

← prev: Chapter 16next: Chapter 18 →
© cvam — written in plaintext, served warm