// the one-minute version
BPF is powerful but useless without a method to point it. Lead with the same methodologies as the rest of performance work: a 60-second checklist of counters for a fast read, the USE method (Utilization, Saturation, Errors per resource), workload characterization (who/why/what/how), and drill-down from symptom to cause. BPF's role is the deep step — when counters say "something's wrong here" but not what, BPF traces the specific events or builds the custom latency histogram that names the cause. Method first, BPF second.
It's tempting, with a tool as capable as BPF, to start tracing immediately and hope a pattern jumps out. It rarely does — you just drown in events. The engineers who solve problems fast use BPF as the precision instrument at the end of a method, not the beginning. This chapter is that method: the checklists and approaches that decide where to point BPF, so its depth lands on the actual bottleneck instead of a random subsystem.
01 Start with a goal
Before any tool, define what "done" looks like — usually a latency target ("p99 under 200 ms"), a throughput goal, or a resource budget. Without a number you can't tell whether a change helped or when to stop, and you can't tell a real problem from normal behavior. The goal also picks the path: a latency target sends you hunting tail outliers and off-CPU waits; a throughput target sends you after per-request cost and contention. BPF gives the same answer faster either way — but only if you know which question you're asking.
02 The 60-second checklist
The fast first read, using plain counters, before BPF: uptime (load trend), dmesg | tail (errors, OOM), vmstat 1 (run queue, swap), mpstat -P ALL 1 (per-CPU balance), pidstat 1 (per-process CPU), iostat -xz 1 (disk latency/queue), free -m (memory), sar -n DEV/TCP 1 (network). In one minute you've spotted the obviously stressed resource — or learned that all counters look clean, which itself points toward off-CPU waiting. This checklist is the cheap survey that tells BPF where to dig.
03 The USE method
The flagship methodology for a complete sweep. For every resource (CPUs, memory, disks, network interfaces, controllers), check three things: Utilization (how busy), Saturation (how much work is queued — where latency is born), and Errors (failed operations). Walk the list, and the first resource showing saturation or errors is your lead. It's exhaustive by design — you can't skip the resource you forgot to suspect. BPF then drills into whatever USE flags, with tools that expose the saturation (queue lengths, wait times) counters only summarize.
04 Workload characterization
Before optimizing the machine, understand the demand. Characterization answers: who is causing load (which process, client, IP), why (which code path or request type), what the load is (read/write mix, sizes, rates), and how it changes over time. BPF shines here — tools like execsnoop, opensnoop, and custom bpftrace counts attribute load to exact processes, files, and calls. Often the biggest win is reshaping the workload (a chatty client, a redundant query) rather than tuning the hardware. The cheapest operation is the one you stop doing.
05 Drill-down analysis
Start at a high-level symptom and peel down, narrowing at each step until you reach the root cause. "Requests slow" → which endpoint? → it waits on the DB → the DB waits on disk reads → those reads are slow because of a saturated device → because of one table scan. BPF is the ideal drill-down instrument because at each layer you can trace exactly what's happening — file-system latency, then block I/O latency, then the specific slow operation with its stack. Each BPF tool answers one layer's question and points to the next, deeper one.
Fig 1 — Cheap counters narrow the search; BPF drills into the flagged layer; verify against the original metric.
06 Counters vs BPF: event vs summary
Understand what BPF adds over the counters you started with. Counters give pre-defined aggregates — total context switches, average disk latency. BPF gives custom, event-level visibility — the full latency distribution (not just the average), broken down by process or file, with the stack that caused each slow event. Where a counter says "average disk latency is 5 ms," BPF's biolatency shows the histogram revealing a bimodal tail at 50 ms hitting 1% of I/O. That tail — invisible to the average — is often the actual problem.
hist(), biolatency, etc.) reveal the distribution and the outliers. When a metric looks fine but users don't, the average is lying — get the histogram.07 Off-CPU analysis with BPF
A theme that recurs through the book: most user-facing latency is waiting, not computing. On-CPU profiling (where the CPU is spent) misses time threads spend blocked on locks, disk, network, or sleeps. BPF makes off-CPU analysis practical — tools like offcputime record where threads block, with stacks, so you see the waits that dashboards and CPU flame graphs can't. When the 60-second checklist comes back clean but the app is slow, off-CPU analysis is usually the next move, and BPF is the tool that makes it cheap.
08 A BPF analysis workflow
Putting the method together: (1) Set a goal (latency + target). (2) Run the 60-second checklist and a USE sweep to find the stressed resource — or learn nothing's saturated. (3) Characterize the workload with BPF (who/why/what). (4) Drill down with the resource's BPF tools; if counters are clean, do off-CPU analysis. (5) Use histograms, not averages, to see the tail. (6) Confirm the cause, fix it, and verify against the original metric under real load. BPF accelerates steps 3-5; the method makes those steps land in the right place.
common catches & gotchas
- Tracing before triaging — Pointing BPF without the 60-second/USE survey wastes its depth on a random subsystem. Narrow first.
- Averages over histograms — The mean hides the tail you're chasing. Use BPF's in-kernel histograms to see the distribution and outliers.
- On-CPU only — A clean CPU profile doesn't clear the app; off-CPU waiting is where most latency hides. Use
offcputime. - Skipping the goal — Without a latency target you can't tell a problem from normal or know when you're done. Quantify first.
- Tuning before characterizing — Optimizing the machine while the workload is wasteful. Characterize who/why/what before changing hardware.
- Forgetting to verify — "Feels faster" isn't done. Re-measure the same metric under real, peak load.
09 Questions engineers actually ask
Where do I point BPF first?
Don't lead with BPF. Run the 60-second checklist and a USE sweep with plain counters to find the stressed resource (or learn nothing's saturated). Then point BPF at that resource to drill down. BPF is the precision follow-up, not the survey.
If counters already exist, why use BPF?
Counters give pre-defined aggregates and averages; BPF gives custom, event-level visibility — full latency distributions, per-process/per-file breakdowns, and the stack behind each slow event. The tail an average hides is often the real problem, and BPF's histograms reveal it.
What's the USE method again?
For every resource, check Utilization (how busy), Saturation (how much work is queued — where latency comes from), and Errors. Walk all resources; the first with saturation or errors is your lead. It's exhaustive, so you don't miss the resource you forgot to suspect.
My checklist is all clean but the app is slow — now what?
That points to off-CPU waiting — locks, blocking I/O, slow downstream calls — which saturates no resource and so passes the counter survey. Run BPF off-CPU analysis (offcputime) to see where threads block, with stacks. That's where the missing latency usually is.
How do I know when I'm done?
When the metric you defined as the goal crosses its target and holds under real, peak load — not when something subjectively feels faster. Bracket the whole investigation with the same objective number.
10 Key takeaways
- BPF needs a method to point it — start with a goal (latency target), not a trace.
- The 60-second checklist and USE method narrow the search cheaply before BPF.
- Workload characterization (who/why/what/how) — BPF excels here and often the fix is the workload, not the hardware.
- Drill down from symptom to cause; BPF traces each layer and points to the next.
- BPF's advantage over counters is event-level histograms that reveal the tail averages hide.
- When counters are clean, do off-CPU analysis — most latency is waiting, not computing.
- Always verify the fix against the original metric under real load.
60-second checklist (counters first)
then BPF, by question
the order
11 Wrapping up
BPF is a microscope, and a microscope needs you to know where to look first. Lead with a goal, survey with counters, sweep with USE, characterize the workload, then drill down with BPF's event-level histograms and off-CPU analysis — and verify objectively. With the method set, it's time to learn the two tools in depth, starting with the polished library you'll run most often. Next: BCC.