BPF Performance Tools Chapter 1 — Introduction Explained: What BPF Is, BCC, bpftrace

// the one-minute version

BPF (eBPF) lets you run small, verified programs inside the Linux kernel, attached to events — function calls, tracepoints, packets. The breakthrough is programmable, in-kernel observability: you can measure almost anything (latency, counts, stacks) and aggregate in the kernel, so deep tracing is cheap and production-safe. Two front-ends do the work: BCC (a library of polished tools) for power, and bpftrace (an awk-like language) for quick custom one-liners. This book is a tour of both, resource by resource.

For decades, asking the Linux kernel a question it wasn't built to answer meant patching it, rebooting, and hoping. BPF ended that. It turns the running kernel into something you can safely program on the fly — ask "show me the latency distribution of every disk I/O by process, right now, on production" and get an answer in one line, no restart. This chapter is the orientation: what BPF is, why it matters, and the two tools you'll spend the rest of the book learning.

01 What BPF is

BPF began as the Berkeley Packet Filter — a tiny in-kernel virtual machine for filtering packets. Modern eBPF generalized it into a safe way to run small programs anywhere in the kernel, triggered by events. When the event fires (a function runs, a tracepoint hits, a timer ticks), your BPF program runs, can read kernel and user data, do arithmetic, and store results in in-kernel maps. People still write "BPF" to mean this modern, general capability — and that's how this book uses it.

think of it likeBefore BPF, observing the kernel was like a sealed engine — to inspect anything you had to open it up (patch + reboot), risking a wreck. BPF is a set of safe diagnostic ports you can plug a custom probe into while the engine runs, with a guarantee the probe can't damage the engine. Same machine, suddenly transparent and live.

02 Why BPF changed observability

Three properties combine into something genuinely new. Programmable: instead of a fixed counter, you write the exact logic you need — measure this, filtered by that, bucketed how you like. In-kernel: the program runs at the event, with full access to kernel context. Aggregating: it can summarize as it goes (a histogram, a count) and emit only the result. Older tracing dumped every event to user space, too expensive for production; BPF moves the math into the kernel, so tracing a million events a second is cheap because you ship one summary, not a million events.

key ideaThe shift is from "collect raw events, analyze later (expensive)" to "compute the answer in the kernel, return just the answer (cheap)." That single change is why BPF tracing is safe to run on busy production systems — and why it replaced a generation of heavier tools. Power plus low, bounded overhead is the whole story.

03 The verifier: safety first

Running custom code in the kernel sounds dangerous — a bug could crash the whole machine. BPF's answer is the verifier: before any program loads, the kernel statically proves it is safe — it terminates (no unbounded loops), reads only memory it's allowed to, and cannot crash the kernel. Programs that can't be proven safe are rejected. This is why BPF is allowed in production where loadable kernel modules never were: you genuinely cannot panic the box with a verified program. The cost is constraints (bounded loops, limited stack) that ordinary code doesn't have.

04 Dynamic and static tracing

BPF attaches to two kinds of instrumentation. Dynamic tracing (kprobes for kernel functions, uprobes for user functions) can hook almost any function at runtime — incredibly flexible, but tied to internal names that can change between versions. Static tracing (tracepoints in the kernel, USDT probes in applications) are stable, pre-placed hooks with a documented interface that survive upgrades. The rule of thumb: prefer stable tracepoints/USDT when one exists for your question; reach for dynamic kprobes/uprobes when you must see something nobody exposed — and re-check after kernel upgrades.

Fig 1 — An event fires, a verified BPF program runs, aggregates into a map, and emits only the summary. Deep visibility, low overhead.

05 BCC: the tool collection

BCC (BPF Compiler Collection) is a framework and, for most users, a big library of ready-made BPF tools. You don't write code — you run them: execsnoop (new processes), opensnoop (file opens), biolatency (disk-latency histogram), tcplife (connection lifetimes), ext4slower (slow file ops), profile (CPU flame graphs), runqlat (run-queue latency). These are production-grade, well-documented, and cover an enormous range of questions. Much of this book is a guided tour of which BCC tool answers which problem — learning a few dozen makes you dangerous in the best way.

06 bpftrace: the language

When no canned tool fits, bpftrace gives you a high-level, awk-like language to write a custom tracer in one line. The shape is probe { action }: say where to attach and what to do. For example bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }' counts file opens by process. You get probes, filters, per-event actions, and in-kernel maps (the @ variables) with aggregating functions like count() and hist(). bpftrace is the fastest way to ask a question nobody wrote a tool for — and the book teaches it as a first-class skill.

Reach for BCC when…

A polished tool already exists for your question (disk latency, new processes, TCP lifetimes). Run it; done. Great for repeatable, shareable analysis.

Reach for bpftrace when…

You need a quick, custom view nothing canned provides. Write a one-liner, get the answer, tear it down. Great for ad-hoc exploration.

07 What BPF does not replace

BPF is the deepest tool, not the first. The everyday counters — vmstat, iostat, top, mpstat — are still where you start, because they're free and give the broad picture. BPF earns its place when those counters tell you something is wrong but not what, and you need to trace specific events or build a custom latency histogram. Think of it as the precision instrument you bring out after the broad survey, not a replacement for the survey itself. The book's analysis chapters all follow this broad-to-deep order.

the catchBPF's safety guarantee is about not crashing the kernel — it is not a promise of zero overhead. A poorly targeted program on an extremely hot probe (a kprobe on a function called millions of times per second, doing heavy work per event) can still add real, measurable cost, even though it can't crash anything. The verifier proves your program is safe, not cheap. Always target the narrowest, lowest-frequency probe that answers the question, aggregate in-kernel, and sanity-check the tool's own overhead on busy systems.

08 How the book is organized

The structure mirrors how you'll actually use BPF. First the foundations: technology background (the verifier, maps, probe types), performance methodologies, then deep dives on the two tools — BCC and bpftrace. Then the resource chapters — CPUs, memory, file systems, disk, networking — each a catalog of BPF tools and one-liners for that subsystem. Finally specialized topics: security, languages, applications, the kernel, containers, hypervisors, plus tips and traps. You can read it cover to cover or jump to the resource you're debugging today.

common catches & gotchas

"Safe" ≠ "free" — The verifier prevents crashes, not overhead. Target narrow probes and aggregate in-kernel.
Starting with BPF — It's the deep tool, not the first. Begin with counters (vmstat/iostat/top); reach for BPF when you need the what.
Dynamic probes break on upgrades — kprobes/uprobes target internal names that move. Prefer tracepoints/USDT; re-verify after kernel changes.
Needs a modern kernel + root — BPF capability grows with kernel version, and most tracing needs root/capabilities. Check before relying on it.
Reinventing BCC tools — Before writing bpftrace, check if a BCC tool already does it well. Don't rebuild biolatency.
Printing per-event — Dumping every event defeats BPF's advantage. Use maps and hist()/count() to summarize.

09 Questions engineers actually ask

What's the difference between BPF, eBPF, and BCC/bpftrace?

BPF/eBPF is the kernel technology — verified programs that run on events. BCC and bpftrace are front-ends that make it usable: BCC is a library of ready-made tools, bpftrace is a language for custom one-liners. You use BCC/bpftrace; they use BPF under the hood.

Is BPF safe to run on production?

Yes for crash-safety — the verifier proves each program can't hang or corrupt the kernel before loading. But "safe" isn't "free": a program on a very hot probe can add overhead. Target narrow probes, aggregate in-kernel, and measure the tool's cost on busy systems.

Should I learn BCC or bpftrace first?

Start by running BCC tools — huge value with zero coding (execsnoop, biolatency, tcplife, etc.). Then learn bpftrace one-liners for custom questions; the language is awk-simple, not kernel C. Most workflows use both.

Do I still need vmstat and iostat?

Absolutely. Those counters are free and give the broad first picture. BPF is the precision follow-up when counters say "something's wrong" but not what. Use the cheap broad tools first, then BPF to drill into the specific events.

What kernel version do I need?

Newer is better — BPF features expanded rapidly across 4.x and 5.x kernels, and some tools need recent kernels or BTF/CO-RE support. Modern distributions generally work well; check a specific tool's requirements if it fails to load.

10 Key takeaways

BPF/eBPF runs verified programs in the kernel on events — programmable, in-kernel, aggregating observability.
The verifier proves programs can't crash the kernel, making BPF production-safe (but not free).
In-kernel aggregation is the breakthrough — ship a summary, not a flood of events.
BCC is a library of ready tools; bpftrace is an awk-like language for one-liners.
Prefer static tracing (tracepoints/USDT) over dynamic (kprobes/uprobes) when possible.
BPF is the deep tool — start with counters, drill in with BPF for the what.

// chapter cheatsheetgetting started

check your setup

uname -rKernel version — newer = more BPF features.

bpftrace --version · which execsnoopConfirm bpftrace + BCC tools installed.

apt install bpfcc-tools bpftraceInstall on Debian/Ubuntu (or dnf on RHEL).

first BCC tools to try

execsnoopWatch every new process as it execs.

opensnoopTrace files being opened, live.

biolatencyDisk I/O latency as a histogram.

tcplifeTCP connection lifetimes + bytes.

first bpftrace one-liners

bpftrace -l 'tracepoint:*'List available probes.

-e 'tracepoint:syscalls:sys_enter_openat { @[comm]=count(); }'Count file opens by process.

-e 'profile:hz:99 { @[comm]=count(); }'Sample what's on-CPU at 99 Hz.

mindset

counters → BPFBroad survey first, BPF for the precise what.

safe ≠ freeTarget narrow probes; aggregate in-kernel.

11 Wrapping up

BPF turns the kernel from a black box into a live, safely programmable instrument — and BCC and bpftrace are how you wield it. Keep the model: verified programs, on events, aggregating in-kernel, with counters first and BPF for depth. Before the tools, though, you need the foundation they stand on — the verifier, maps, and probe types in detail. Next: Technology Background.

BPF Performance Tools Chapter 1 — Introduction.