// the one-minute version
The traps everyone hits: event flooding (tracing a hot event without aggregating — fix with maps/hist()), dropped events (buffers overflow under load — losing exactly what you wanted), missing stacks/symbols (frame pointers and debuginfo, per language), overhead (cost scales with event rate — estimate before production), event ordering (don't assume across CPUs), and kprobe drift (internal names change). Plus practical reflexes: 49/99 Hz sampling, filter early, prefer tracepoints, and always sanity-check that "no output" means "no events" and not "broken probe."
Every BPF tool works perfectly in the demo and then surprises you in the field — a flooded terminal, an empty result that should have data, a flame graph full of hex. These aren't deep mysteries; they're a known set of traps with known fixes. This closing chapter collects them, so when a tool misbehaves you recognize the pattern instead of doubting BPF. Master these and you'll spend your time on the actual problem, not on fighting the tools.
01 Event flooding
The most common beginner mistake: attaching a per-event action (a printf, or per-event output) to a high-frequency event — a syscall, a packet, a scheduler switch on a busy box — and getting a firehose that floods the terminal, adds real overhead, and tells you nothing. The fix is the central BPF discipline: aggregate in the kernel. Replace per-event printing with a map and count() or hist(), so you emit one compact summary instead of millions of lines. If you truly need per-event output, filter hard first (a specific PID, a latency threshold). Flooding is almost always a sign you should be summarizing.
hist()/count() and a map by default; reserve per-event printf for low-frequency events or tightly filtered captures. Flooding isn't a BPF limitation, it's a signal to change approach.02 Dropped events
The flip side, and more dangerous because it's silent. When events arrive faster than user space can consume them, the kernel's perf/ring buffer overflows and drops them — and the dropped ones may be exactly the rare slow events you were hunting. Tools usually report a dropped-event count; always check it, because a result computed from a buffer that dropped 40% of events is misleading. Fixes: aggregate in-kernel (so you're not shipping every event), increase the buffer size, filter to reduce volume, or sample. Silent data loss that skews your conclusion is worse than an obvious flood — watch for the drop counter.
03 Missing stacks and symbols
The recurring frustration (Chapters 2, 6, 12): flame graphs and stack traces full of [unknown] or raw hex. The cause is one of two things, and it differs by language. Frame pointers: binaries built with -fomit-frame-pointer can't be stack-walked — rebuild with frame pointers, or use DWARF/LBR unwinding. Symbols: without debug info, addresses don't resolve to names — install debuginfo or debuginfod. For JIT (Java/Node) you also need a symbol map; for interpreted code, native stacks show the interpreter, so you need USDT/runtime tools. Diagnose by asking: walkable (frame pointers)? nameable (symbols)? right layer (model)?
Fig 1 — The four traps that account for most BPF tool surprises, each with its standard fix.
04 Estimating overhead
"Safe" (won't crash) isn't "free." A tool's overhead scales with event rate × per-event work: counting a rare event is negligible; tracing every malloc on a hot allocator, or every packet on a 1M-pps box, can be significant. Before running on production, estimate: how often does this event fire, and how much does my program do per event? Prefer aggregation (cheap per event), filter at the source, and for sampling tools keep the rate sane (49/99 Hz). When unsure, test on a non-critical host first and measure the tool's own impact. The verifier guarantees safety; you are responsible for overhead.
05 "No output" — events or broken probe?
A subtle trap: a tool runs and prints nothing. Two very different meanings. Either the event genuinely didn't fire (good — that's a real answer), or the probe is broken and silently matched nothing (a kprobe on a renamed/inlined function, a typo, a USDT probe absent from this build). Don't assume the first. Sanity-check the probe: does bpftrace -l list it? Does a known-active event produce output? For kprobes, did the kernel upgrade rename or inline the target? Treat empty output as a question, not a conclusion — "nothing happened" and "I traced nothing" look identical and mean opposite things.
06 Event ordering and timestamps
A correctness subtlety. Events from different CPUs may arrive in the user-space buffer out of order — don't assume the order you read them is the order they happened. For timing, use the kernel timestamp (nsecs) captured at the event, not the time you process it in user space. Per-CPU buffers and aggregation help, but if your logic depends on cross-CPU ordering (e.g. matching a request start on one CPU to its end on another), key by an identifier (tid, a request ID) rather than relying on arrival order. Most aggregation sidesteps this, but per-event correlation needs care.
07 Practical reflexes
A grab-bag of habits that save time. 49 or 99 Hz for sampling (offset from round numbers to avoid aliasing with periodic kernel activity). Filter early — narrow to a PID or threshold before aggregating, to cut overhead and noise. Prefer tracepoints/USDT over kprobes/uprobes for durability across versions. Re-verify after kernel upgrades — kprobe-based tools especially. Use -p PID to scope to one process. Read the man/examples — every tool's output differs. Aggregate by default, print per-event only when filtered. None are deep, but together they're the difference between fighting the tools and using them.
08 A troubleshooting checklist
When a BPF tool misbehaves, run through this. Flooding? → aggregate (map + hist/count), filter. Empty output? → verify the probe attached (bpftrace -l, known-active event), check for kprobe drift. Hex/unknown stacks? → frame pointers + debuginfo; for JIT add a symbol map; for interpreted use runtime tools. Suspiciously clean result? → check the dropped-event counter; you may have lost the outliers. Slowing the target? → estimate event rate, filter, sample. Worked yesterday, not today? → kernel upgrade moved a kprobe target; switch to a tracepoint. Recognize the pattern, apply the fix, get back to the actual problem.
common catches & gotchas
- Printing instead of aggregating — Per-event output on hot events floods and adds overhead. Use maps +
hist()/count(). - Ignoring dropped events — Silent buffer overflow removes the rare events you wanted, skewing results. Always check the drop counter.
- Empty = "nothing happened" — It may be a broken/renamed probe matching nothing. Verify the probe attached before concluding.
- "Safe" = "free" — Overhead scales with event rate × per-event work. Estimate before production; aggregate and filter.
- Trusting cross-CPU order — Events arrive out of order; use kernel timestamps and key by identifier for correlation.
- kprobe drift after upgrades — Internal function names move; tools silently break. Prefer tracepoints; re-verify after kernel changes.
09 Questions engineers actually ask
My tool floods the terminal — how do I fix it?
You're printing per-event on a high-frequency event. Aggregate in the kernel instead: use a map with count() or hist() so you emit one summary, not millions of lines. If you genuinely need per-event output, filter hard first (a specific PID or a latency threshold) to cut the volume.
The result looks clean but I don't trust it — why might it be wrong?
Likely dropped events: under load the buffer overflowed and silently lost events — possibly the rare slow ones you were chasing — making the histogram look healthy. Check the tool's dropped-event counter. A "clean" result from a lossy buffer is misleading; aggregate in-kernel or enlarge the buffer to avoid drops.
My tool prints nothing — is that good or bad?
Ambiguous, so verify. Either the event genuinely didn't fire (a real answer) or the probe is broken and matched nothing (renamed/inlined kprobe, typo, absent USDT). Check bpftrace -l lists the probe and that a known-active event produces output before concluding "nothing happened."
Why are my stacks full of hex addresses?
Missing frame pointers (can't walk the stack) or missing symbols (can't name addresses). Rebuild with frame pointers or use DWARF/LBR, and install debuginfo/debuginfod. For JIT (Java/Node) add a symbol map; for interpreted languages, native stacks show the interpreter — use USDT/runtime tools instead.
My tool worked last week and now returns nothing — what changed?
Probably a kernel upgrade. kprobe-based tools target internal function names that get renamed, inlined, split, or removed across versions, so they silently match nothing. Switch to a stable tracepoint if one covers your question, and always re-verify kprobe-based tools after a kernel update.
10 Key takeaways
- Flooding means print-where-you-should-aggregate — use maps +
hist()/count(), filter early. - Dropped events are silent and dangerous — always check the drop counter; they may be the outliers you wanted.
- Missing stacks/symbols: frame pointers + debuginfo, with JIT symbol maps and interpreted-runtime tools per language.
- Overhead scales with event rate × per-event work — estimate before production; "safe" isn't "free."
- Empty output is ambiguous — verify the probe attached before concluding "nothing happened."
- Don't trust cross-CPU ordering; use kernel timestamps and key by identifier.
- Reflexes: 49/99 Hz, prefer tracepoints, re-verify after upgrades, aggregate by default.
symptom → fix
bpftrace -l; known-active event; kprobe drift.reflexes
11 Wrapping up the book
That's the whole companion: from "what is BPF" through the verifier and probes, the two tools (BCC and bpftrace), the resource-by-resource toolkits, the specialized domains, the ecosystem, and finally the traps that bite everyone. The throughline never changed — aggregate in the kernel, prefer stable probes, fix your stacks, mind the overhead, and verify what you measured. Learn what BPF can see and you can answer almost any question about a running Linux system. Back to the chapter index — or start again at Chapter 1.