// the one-minute version
BPF (eBPF) runs small, verified programs inside the kernel attached to probes (tracepoints, kprobes, uprobes, USDT). The breakthrough: it can filter and aggregate in the kernel — building a latency histogram in place and returning only the summary — so you get deep, custom visibility at low, bounded overhead, safe for production. Two front-ends: BCC (a library of polished tools like biolatency, execsnoop) and bpftrace (an awk-like language for quick custom one-liners). Learn a dozen BCC tools and basic bpftrace and you can answer almost any "what exactly is happening" question.
BPF is the reason the last decade of Linux observability looks nothing like the one before it. Questions that used to require a custom kernel module, a reboot, and a prayer — "show me the latency distribution of every disk I/O, by process, right now, on production, without restarting anything" — are now a one-liner. This chapter is about that power: how BPF works, why it's safe, and the two tools (BCC and bpftrace) you'll actually use to wield it.
01 What BPF is
BPF (originally the Berkeley Packet Filter, now the general "eBPF") lets you load small programs into the kernel that run when an event fires — a function is called, a tracepoint hits, a packet arrives. The programs are written in a restricted instruction set, run in a kernel virtual machine, and attach to the same probe points from Chapter 4. Crucially they can read kernel and user state, do arithmetic, and store results in maps (in-kernel key-value tables) — which means they can compute summaries on the fly instead of shipping raw events to user space.
02 The verifier: why it's safe
Running arbitrary code in the kernel sounds terrifying — a bug could crash the whole machine. BPF's answer is the verifier: before a program loads, the kernel statically proves it's safe — it terminates (no unbounded loops), only touches memory it's allowed to, and can't crash the kernel. Only verified programs run. This is what makes BPF production-safe in a way kernel modules never were: you can't accidentally panic the box. The trade-off is that the verifier rejects programs it can't prove safe, which is why BPF code has constraints (bounded loops, limited stack) that ordinary C doesn't.
03 Why BPF changed observability: in-kernel aggregation
The killer feature, returning to a theme from Chapter 4. Old tracing dumped every event to user space, so high-frequency tracing was too expensive for production. BPF aggregates in the kernel using maps: it can increment a counter, sum a value, or bucket a latency into a histogram at the probe, and only emit the compact result. Tracing a million events per second becomes cheap because you never move a million events — you move one histogram. This is what turns "deep tracing" from a dangerous luxury into a routine production tool.
Fig 1 — BPF moves the aggregation into the kernel, so you ship a histogram instead of a flood of events. That's the overhead breakthrough.
04 BCC: the tool collection
BCC (BPF Compiler Collection) is a framework and, more importantly to most users, a big library of ready-made, polished BPF tools. You don't write code — you run biolatency for a disk-latency histogram, execsnoop to watch new processes, opensnoop for file opens, tcplife for connection lifetimes, ext4slower for slow file ops, offcputime for blocking analysis, profile for CPU flame graphs. These are the tools referenced throughout this book's resource chapters. Learning maybe a dozen of them covers an enormous fraction of real production questions, with no programming required.
05 bpftrace: the language
When no canned tool fits, bpftrace gives you an awk-like language to write custom tracers in one line. The shape is probe { action }: you specify where to attach and what to do. For example, bpftrace -e 'tracepoint:syscalls:sys_enter_open { @[comm] = count(); }' counts open calls by process name. Key pieces: probes (tracepoint:, kprobe:, uprobe:, profile:), actions (print, assign), maps (the @name variables that aggregate), builtins (comm, pid, nsecs, arg0), and aggregating functions (count(), hist(), sum()). It's the fastest way to ask a question nobody wrote a tool for.
06 One-liners that earn their keep
A few patterns show the power. Latency histogram of a syscall: attach to its enter/exit tracepoints, store the start time in a map keyed by tid, compute the delta on exit, and hist() it. Count events by stack: @[kstack] = count() to see which kernel paths dominate. Profile on-CPU: profile:hz:99 { @[ustack] = count(); } for a flame-graph-able sample. Trace a specific function's arguments with a kprobe. Each is a few lines, runs immediately, aggregates in-kernel, and tears down cleanly when you Ctrl-C. The book's appendices are full of these; a handful become reflexes.
07 Attach points and requirements
BPF can attach to the full menu of sources: tracepoints (stable, preferred), kprobes/kretprobes (any kernel function, dynamic), uprobes (user functions), USDT (app-defined static probes), perf events (timed profiling, PMCs), and software events. Prefer stable tracepoints/USDT where they exist; use k/uprobes when you must see something unexposed (with the upgrade-fragility caveat from Chapter 4). Requirements: a reasonably modern kernel (the newer, the more capable), root or the right capabilities, and for some tools kernel headers/BTF for type info. On current distros it mostly just works.
08 A BPF workflow
(1) Is there a BCC tool for it? Usually yes — biolatency, tcplife, offcputime, execsnoop, etc. Run it; done. (2) If not, write a bpftrace one-liner: pick the probe, aggregate into a map, print on exit. (3) Prefer tracepoints over kprobes for stability. (4) Aggregate in-kernel (histograms, counts) rather than printing per-event, for both readability and overhead. (5) Target the narrowest probe at the lowest frequency that answers the question, and sanity-check overhead on busy boxes. BPF is the deepest, most flexible tool — reach for it when counters and perf can't see what you need.
common catches & gotchas
- "Safe" ≠ "free" — The verifier prevents crashes, not overhead. A BPF program on a million-events-per-second probe can still cost. Target narrow, aggregate in-kernel.
- kprobes break on upgrades — Dynamic probes target internal function names that change. Prefer tracepoints/USDT; re-verify after kernel updates.
- Old kernels, missing features — BPF capability grows with kernel version; some tools need BTF/CO-RE or recent kernels. Check requirements before relying on a tool.
- Printing per-event — Dumping every event defeats BPF's advantage and floods output. Use maps and
hist()/count()to summarize. - Forgetting it needs root — Most BPF tracing needs root or specific capabilities; plan access on locked-down hosts (or fall back to Ftrace).
- Reinventing BCC tools — Before writing bpftrace, check if a BCC tool already does it well. Don't rebuild
biolatency.
09 Questions engineers actually ask
What's the difference between BCC and bpftrace?
BCC is a collection of polished, ready-to-run BPF tools (and a library for building them) — use it when a tool already exists for your question. bpftrace is a concise language for writing custom one-line tracers on the spot — use it when nothing canned fits. Most days you run BCC tools and occasionally drop to a bpftrace one-liner.
Is BPF really safe to run in production?
Yes for crash-safety — the verifier proves each program can't hang or corrupt the kernel before it loads. But "safe" isn't "free": a program on a very hot probe can add overhead. Target narrow, low-frequency probes, aggregate in-kernel, and measure the tool's cost on busy systems.
Why is BPF such a big deal versus older tracing?
In-kernel aggregation. Old tracing shipped every event to user space, making high-frequency tracing too costly for production. BPF computes summaries (counts, histograms) in the kernel and emits only the result, so deep tracing becomes cheap and routine — plus the verifier makes it safe.
Do I need to learn to program BPF?
Not to get enormous value. Learning a dozen BCC tools (biolatency, execsnoop, tcplife, offcputime, ext4slower...) covers most needs with zero coding. bpftrace one-liners are worth learning next for custom questions, but they're awk-simple, not kernel C.
When should I use Ftrace instead of BPF?
When BPF isn't available — older kernels, no root, or hardened hosts where you can't install BCC/bpftrace. Ftrace is built in and needs nothing. For programmable aggregation and the rich tool library, BPF wins; for "it's already here and I can't add software," Ftrace wins.
10 Key takeaways
- BPF/eBPF runs small, verified programs in the kernel on probes — deep, custom visibility.
- The verifier proves programs can't crash the kernel, making BPF production-safe (but not free).
- In-kernel aggregation (maps, histograms) is the breakthrough — ship a summary, not an event flood.
- BCC gives ready-made tools (
biolatency,execsnoop,tcplife...); bpftrace gives an awk-like language for one-liners. - Attach to tracepoints (stable) preferentially, kprobes/uprobes when needed (upgrade-fragile).
- Target the narrowest, lowest-frequency probe and aggregate in-kernel to keep overhead low.
- BPF is the deepest tool — reach for it when counters and perf can't answer the question.
BCC tools (no coding)
bpftrace one-liners
building blocks
11 Wrapping up
BPF is the capstone of the toolbox: verified in-kernel programs that aggregate as they observe, giving you any custom view at production-safe (if not free) overhead. Run BCC tools first, drop to bpftrace one-liners when needed, prefer tracepoints, and always summarize in-kernel. With every method and tool now in hand, one thing remains — watching them combine on a single real problem. Next: the Case Study.