// the one-minute version
For CPUs, BPF answers questions counters can't. Run-queue latency — how long threads wait for a CPU — is measured by runqlat (a histogram) and runqlen (queue length); it's the saturation metric that %CPU hides. On-CPU profiling with profile samples stacks for a flame graph to find what burns CPU. Off-CPU analysis with offcputime shows where threads block. Plus execsnoop (short-lived processes), cpudist (on-CPU time), syscount, and scheduler tracing. Use run-queue latency for "am I CPU-saturated," flame graphs for "what's hot," off-CPU for "why am I waiting."
Traditional CPU tools tell you the processors are 80% busy. They can't tell you that threads are waiting 5 ms in the run queue before they even start, or that 60% of your CPU is in one regex, or that the app spends most of its life blocked off-CPU. BPF can. This chapter is the CPU toolkit: the handful of BCC tools and bpftrace one-liners that turn "CPU is busy" into a precise diagnosis — saturation, hot code, or hidden waiting.
01 What to measure on a CPU
Three questions, three kinds of tool. Saturation: are threads waiting for a CPU? — run-queue latency. On-CPU: when running, what code is executing? — sampled profiling and flame graphs. Off-CPU: when not running, why are threads blocked? — off-CPU analysis. Utilization (the classic %CPU) tells you the processors are busy but not whether work is queued, where cycles go, or why threads wait. BPF supplies all three missing answers, and knowing which question you're asking picks the tool instantly.
02 Run-queue latency: runqlat and runqlen
The most underused CPU metric. When more threads are runnable than there are CPUs, the extras wait in the run queue — and that wait is pure scheduler-induced latency added to your request before your code runs. runqlat traces scheduler events and prints a histogram of that wait time; a tail in the milliseconds means CPU saturation is hurting you. runqlen samples the run-queue length per CPU. Together they answer "is the CPU saturated, and how badly?" far better than a utilization percentage, which can't distinguish a busy-but-fine box from a queued-and-suffering one.
runqlat is how you tell those apart — and it's the BPF metric that turns a vague "high CPU" into a concrete latency number you can target.03 On-CPU profiling: profile and flame graphs
To find what burns CPU, sample. profile takes timed samples of the running stacks across all CPUs (e.g. 49 or 99 Hz) and counts them in-kernel — far cheaper than tracing every function. Feed its output to a flame graph and the widest towers are where the CPU goes, with the leaf being the hot function. This is the BPF-native way to produce CPU flame graphs: low overhead, whole-system or per-process (-p PID), and it aggregates in the kernel so it scales to busy machines. "CPU is high, why?" → profile → flame graph → the answer in one glance.
Fig 1 — Pick the tool by the question: run-queue latency for saturation, profile for hot code, offcputime for blocking.
04 Off-CPU analysis: offcputime
The other half of the picture. On-CPU profiling only sees running code; for most request-serving apps the wall-clock latency is dominated by waiting — blocked on locks, disk, network, or sleeps. offcputime records how long threads spend off-CPU and the stacks that led there, so you see exactly where they block. A clean CPU flame graph plus a fat off-CPU profile is the classic "it's not compute, it's waiting" signature. You can render off-CPU stacks as their own flame graph (width = time blocked) — the complement to the on-CPU one, and often the more revealing of the two.
05 Short-lived processes: execsnoop
A blind spot of sampling tools: processes that live milliseconds — a shell spawning thousands of short commands, a misconfigured loop fork-bombing tiny helpers — burn CPU but vanish between samples, so top and even profile miss them. execsnoop traces every exec() as it happens, catching these ephemeral processes by name and arguments. It's the go-to when CPU is high but no persistent process accounts for it — frequently revealing a runaway script or a build step spawning a storm of subprocesses that periodic tools never caught.
top refreshes every second or two and a process that lives 20 ms is gone before it's seen; even a 99 Hz profiler may never sample it. So a system can be visibly burning CPU while every process-listing tool shows nothing accounting for it — a genuinely baffling situation until you reach for execsnoop, which catches each exec() at the moment it happens. "High CPU, no culprit process" almost always means short-lived processes; trace execs, don't keep refreshing top.06 Distributions and syscalls: cpudist, syscount
Two more useful angles. cpudist shows a histogram of how long threads run on-CPU between being scheduled off — short bursts (lots of tiny on-CPU times) suggest heavy context switching or I/O-bound behavior; long bursts suggest CPU-bound work. It characterizes the shape of CPU use, not just the amount. syscount counts system calls by type and by process — a high %sys CPU figure becomes "which syscalls, how many," instantly pointing at, say, a flood of tiny read()s fixable by buffering (the application lesson from Systems Performance, now traced precisely).
07 Scheduler tracing and one-liners
For deeper questions, bpftrace traces scheduler tracepoints directly. tracepoint:sched:sched_switch fires on every context switch; sched:sched_wakeup on wakeups. With the per-thread timestamp pattern you can build custom run-queue-latency or wakeup-latency tools, or count switches by process. Hardware and software interrupts have their own tools — hardirqs and softirqs show time spent servicing interrupts, which on busy network boxes can itself be a CPU bottleneck hiding in system time. These let you go past the canned tools when a CPU question is truly bespoke.
08 A CPU analysis workflow
(1) Confirm CPU is the issue with counters (mpstat, run queue in vmstat). (2) Measure saturation with runqlat — are threads waiting, and how long? (3) If utilization is genuinely high, profile → flame graph to find the hot code. (4) If the app is slow but on-CPU looks innocent, run offcputime for the blocking. (5) If CPU is high with no obvious process, execsnoop for short-lived ones. (6) High %sys? syscount to find the syscall storm. (7) Deeper still, trace scheduler tracepoints with bpftrace. Each tool answers one question and points to the next.
common catches & gotchas
- Utilization ≠ saturation —
%CPUcan't tell busy-but-fine from queued-and-suffering. Userunqlatfor the run-queue latency that actually hurts. - Missing short-lived processes — Sampling tools and
topmiss millisecond processes. Useexecsnoopwhen CPU is high with no visible culprit. - On-CPU tunnel vision — A clean profile doesn't clear the app; the time may be off-CPU. Run
offcputimetoo. - Broken flame-graph stacks — Missing frame pointers/symbols give
[unknown]. Rebuild with frame pointers or use DWARF (Chapter 2). - Ignoring interrupt time — On busy network boxes,
hardirqs/softirqscan reveal CPU lost to interrupt servicing, hidden in system time. - Profiling too fast — Very high sample rates add overhead for little gain. 49-99 Hz is the sane default.
09 Questions engineers actually ask
How do I know if my CPUs are saturated?
Run runqlat — it shows a histogram of how long threads wait in the run queue before getting a CPU. A tail in the milliseconds means saturation is adding latency. Utilization percentage alone can't tell saturation from healthy busyness; run-queue latency can.
CPU is high but no process shows it — what's happening?
Almost certainly short-lived processes that live milliseconds and vanish between top refreshes and profiler samples. Run execsnoop to catch every exec() as it happens — it usually reveals a runaway script or a build spawning a storm of subprocesses.
My CPU flame graph looks busy but the app is still slow — why?
The on-CPU profile only shows running code; the latency may be off-CPU — threads blocked on locks, I/O, or downstream calls. Run offcputime to see where threads block, and render it as an off-CPU flame graph. That's where waiting-dominated latency hides.
My system CPU (%sys) is high — how do I find the cause?
Run syscount to count syscalls by type and process. High system time usually means a flood of syscalls — often tiny repeated read()/write() calls fixable by buffering. syscount turns "%sys is high" into "this process makes 2M reads/sec."
What sample rate should profile use?
49 or 99 Hz is standard — low overhead and offset from round numbers to avoid aliasing with periodic kernel timers. Higher rates add cost and data with diminishing returns. The default is almost always right.
10 Key takeaways
- Three CPU questions: saturation (run-queue latency), on-CPU (what's hot), off-CPU (why waiting).
runqlat/runqlenmeasure run-queue latency — the saturation metric%CPUhides.profilesamples stacks for a flame graph — the fast path from "high CPU" to "this function."offcputimereveals blocking; a clean on-CPU profile plus fat off-CPU = "it's waiting, not computing."execsnoopcatches short-lived processes that sampling tools miss.cpudistcharacterizes on-CPU burst length;syscountturns high%sysinto a named syscall storm.- Go deeper with scheduler tracepoints and
hardirqs/softirqsfor interrupt time.
saturation (run-queue latency)
on-CPU (what's hot)
off-CPU (why waiting)
processes & syscalls
interrupts & bpftrace
11 Wrapping up
For CPUs, BPF turns "busy" into a precise diagnosis: runqlat for saturation, profile for hot code, offcputime for hidden waiting, and execsnoop for the processes that slip between samples. Pick the tool by the question. The same pattern — purpose-built tools per question — now repeats for the next resource. Next: Memory.