// the one-minute version
A BPF program is compiled to bytecode, checked by the verifier (proven safe — terminates, no bad memory access), then JIT-compiled to native code and attached to an event. It stores results in maps (in-kernel key-value tables) and reads them from user space. Events come from kprobes/uprobes (dynamic, any function), tracepoints/USDT (static, stable), and perf events/PMCs (timed sampling, hardware counters). It can capture stack traces for flame graphs. BTF/CO-RE makes tools portable across kernels.
To use BPF tools well — and to debug them when they misbehave — you need a working picture of what happens under the hood. Not the kernel-developer level, but enough to know why a program is rejected, why a stack trace is broken, or why a tool that worked yesterday fails after a kernel upgrade. This chapter is that foundation: the lifecycle of a BPF program, the maps it uses, the probe types it attaches to, and the modern pieces (BTF/CO-RE) that make tools portable.
01 From classic BPF to eBPF
BPF started in 1992 as the Berkeley Packet Filter — a tiny in-kernel VM that ran filters for tcpdump so the kernel could drop unwanted packets without copying them to user space. In 2014, extended BPF (eBPF) massively generalized it: more registers, larger programs, maps for storing state, and the ability to attach to events all over the kernel, not just packets. That generalization is what turned a packet filter into a universal, safe, in-kernel programming environment for tracing, networking, and security. Today "BPF" means eBPF.
02 How a BPF program runs
The lifecycle has clear stages. You write the program (in restricted C for BCC, or bpftrace's language); it's compiled to BPF bytecode. The kernel runs the verifier to prove it's safe. If it passes, the kernel JIT-compiles the bytecode to native machine code (so it runs at near-native speed), then attaches it to the chosen event. When the event fires, the program runs, reads context, and writes results into maps. User space periodically reads the maps to print results. Understanding these stages tells you exactly where a failure happened — compile, verify, attach, or read.
Fig 1 — Source → bytecode → verifier → JIT → attach. Knowing the stages tells you where any failure occurred.
03 The verifier, in depth
The verifier is the gatekeeper that makes BPF safe. It walks every possible path through the program and proves several things: the program terminates (historically no loops at all; modern kernels allow bounded loops), it only accesses memory it's permitted to (no wild pointers), it doesn't leak kernel addresses to unprivileged users, and it stays within size and stack limits. If any path can't be proven safe, the whole program is rejected with an error. This is strict by design — the price of running code in the kernel without the ability to crash it.
04 BPF maps
Programs are event handlers that run and exit — so to keep state across events (and to get data out to user space), they use maps: in-kernel key-value data structures. A hash map keyed by PID can accumulate per-process counts; an array can hold a histogram's buckets; special map types support per-CPU storage (to avoid contention), stack-trace storage, and more. Maps are the backbone of in-kernel aggregation — the histogram you build in biolatency lives in a map, updated at each event, and read out periodically. No maps, no summaries.
05 Probe types: dynamic vs static
BPF attaches to events through several probe types, split into dynamic and static.
kprobes / kretprobes
Dynamic. Instrument the entry/return of (almost) any kernel function by name, at runtime. Hugely flexible, but tied to internal names that change between kernel versions.
uprobes / uretprobes
Dynamic. The same for user-space functions in any binary or library. Lets you trace inside applications without modifying them.
tracepoints
Static. Stable, pre-placed kernel hooks with a documented interface (e.g. sched:sched_switch). Survive upgrades — prefer them.
USDT
Static. Application-defined probes (User Statically-Defined Tracing) baked into programs like databases and language runtimes. Stable app-level events.
tcp_sendmsg may silently stop matching — or match something subtly different — after a kernel or library upgrade. Always prefer a stable tracepoint or USDT probe when one exists for your question, and re-verify dynamic-probe tools after any upgrade. Much of BPF tooling's version fragility comes from kprobes on moving targets.06 perf events and PMCs
BPF also rides on the perf_events subsystem (Chapter from Systems Performance). This gives two more capabilities: timed sampling — run a BPF program at a fixed frequency (e.g. 99 Hz) to sample what's on-CPU, the basis of BPF flame graphs — and PMCs (Performance Monitoring Counters), the CPU's hardware counters for cycles, instructions, cache misses, and the like. PMCs let BPF tools reason about why cycles are productive (IPC) or wasted (cache misses), not just where time goes. Together with the probe types, this rounds out the event sources BPF can observe.
07 Stack traces and flame graphs
A huge part of BPF's value is capturing stack traces at an event — the chain of function calls that led there. Store stacks in a stack-trace map keyed by their contents, count how often each appears, and you have the data for a flame graph. But stacks depend on being walkable: kernel stacks usually are; user-space stacks need frame pointers (or DWARF/LBR) and symbols (debug info) to resolve addresses to function names. When stacks are broken or full of hex addresses, the cause is almost always missing frame pointers or symbols — a recurring theme in later chapters.
[unknown] — are the most common BPF frustration, and the usual culprit is missing frame pointers. Many distributions compile binaries (and libc) with -fomit-frame-pointer for a tiny speed gain, which destroys default stack walking. Fixes: rebuild the hot code with frame pointers, use DWARF or LBR-based unwinding where supported, and install debug info (or debuginfod) so addresses resolve to names. If a BPF flame graph looks suspiciously flat or unnamed, suspect frame pointers before doubting the tool.08 BTF and CO-RE: portable tools
Early BPF tools needed kernel headers present at runtime to know struct layouts — fragile and slow. BTF (BPF Type Format) embeds type information in the kernel itself, and CO-RE ("Compile Once – Run Everywhere") uses it so a BPF tool compiled on one machine runs on many different kernels without recompiling. This is why modern libbpf-based tools are small, fast to start, and portable across distributions, where older BCC tools compiled on each target. You don't need to write CO-RE code to benefit — but knowing it exists explains why newer tools "just work" and need a BTF-enabled kernel.
common catches & gotchas
- Verifier rejections — Most "won't load" errors are the verifier proving safety. Read its message; it's usually an unbounded loop, an unchecked pointer, or a stack/size limit.
- kprobe/uprobe drift — Dynamic probes break on upgrades because they target internal names. Prefer tracepoints/USDT; re-verify after kernel/library changes.
- Broken stacks — Missing frame pointers or symbols give
[unknown]flame graphs. Rebuild with frame pointers, use DWARF/LBR, install debuginfo. - Old kernels, no BTF — CO-RE/libbpf tools need BTF; very old kernels lack it. Check kernel version and BTF availability if a modern tool won't run.
- Forgetting maps are the channel — Data leaves BPF only via maps (or perf buffers). If a tool shows nothing, the program may be running but not populating/reading a map.
- Per-CPU vs global maps — High-frequency counters use per-CPU maps to avoid contention; reading them requires summing across CPUs. A naive read can mislead.
09 Questions engineers actually ask
Why was my BPF program rejected by the verifier?
Because the verifier couldn't prove it safe — common causes are an unbounded loop, accessing memory without a bounds check, exceeding the stack/instruction limit, or potentially leaking a kernel pointer. The verifier's error message points at the offending instruction. It's rejecting "unprovable," not necessarily "wrong."
What's the difference between a kprobe and a tracepoint?
A kprobe dynamically instruments any kernel function by name at runtime — flexible but tied to internals that change across versions. A tracepoint is a stable, pre-placed hook with a documented interface that survives upgrades. Prefer tracepoints when one exists; use kprobes to see what nobody exposed.
What are BPF maps for?
BPF programs run per-event and exit, so maps are the in-kernel key-value structures that hold state across events and pass data to user space. Histograms, per-PID counts, and stack-trace tables all live in maps — they're the foundation of in-kernel aggregation.
Why are my flame-graph stacks full of [unknown]?
Almost always missing frame pointers (binaries built with -fomit-frame-pointer) or missing symbols/debug info. Rebuild hot code with frame pointers, use DWARF or LBR unwinding where supported, and install debuginfo (or debuginfod) so addresses resolve to function names.
What is BTF/CO-RE and why should I care?
BTF embeds kernel type info in the kernel; CO-RE ("Compile Once – Run Everywhere") uses it so a BPF tool compiled once runs across many kernels without recompiling. It's why modern libbpf tools are portable and fast to start, where older BCC tools compiled on each machine. You benefit automatically on BTF-enabled kernels.
10 Key takeaways
- A BPF program goes source → bytecode → verifier → JIT → attach; knowing the stages locates any failure.
- The verifier proves safety (terminates, valid memory), not correctness — most load errors are it doing its job.
- Maps are in-kernel key-value tables that hold state and pass results out — the basis of aggregation.
- Probes: kprobes/uprobes (dynamic, flexible, fragile) and tracepoints/USDT (static, stable — prefer them).
- perf events add timed sampling and PMCs; BPF captures stack traces for flame graphs.
- Broken stacks usually mean missing frame pointers/symbols; BTF/CO-RE makes modern tools portable.
inspect the BPF subsystem
list available probes
stacks & symbols
check BTF (for CO-RE tools)
11 Wrapping up
You now have the machinery: the lifecycle through the verifier and JIT, maps as the state and output channel, the four probe families, perf events and PMCs, stack traces and their frame-pointer dependency, and BTF/CO-RE for portability. This is enough to understand — and troubleshoot — every tool that follows. Before the tools, one more foundation: the method for using them effectively. Next: Performance Analysis.