// the one-minute version
perf is the standard Linux profiler — one tool with many subcommands. perf stat reads hardware counters (PMCs) to show cycles, instructions, IPC, and cache misses — why CPU cycles are (un)productive. perf record samples stacks (e.g. -F 99 -ag) and perf report / flame graphs show where CPU is spent. It also taps tracepoints and dynamic probes for event tracing. Master stat (counters) and record→flame graph (sampling) and you've covered most CPU analysis.
Of all the tools in this book, perf is the one to learn first and deepest. It ships with the Linux kernel, needs no extra agents, and reaches everything from high-level CPU sampling down to individual hardware counters. The catch is that it's really a toolbox of subcommands with terse flags, and people bounce off the syntax. This chapter cuts it down to the handful of invocations that solve 90% of real problems — and shows how they connect to the concepts from earlier chapters.
01 What perf is
perf (a.k.a. perf_events) is a profiling and tracing framework built into the kernel, driven by the perf command. It can count events (how many cache misses happened), sample them (grab the stack every N events or N times per second), and trace them (record each occurrence). It draws on the kernel sources from Chapter 4 — hardware PMCs, software events, tracepoints, and dynamic kprobes/uprobes. One tool, three modes, every source. That breadth is why it's the default first reach for CPU work.
02 perf stat: counting and IPC
The fastest way to characterize a workload. perf stat <command> (or -p PID) runs it and prints hardware-counter totals: cycles, instructions, and the all-important IPC (instructions per cycle), plus branch and cache misses. IPC tells you whether the CPU is doing real work (high) or stalling on memory (low) — the compute-bound vs memory-bound distinction from Chapter 6. perf stat -d adds cache detail. Because counting has near-zero overhead, you can run it on production. It's the natural follow-up to "CPU is at 100%" — is that 100% productive or wasted?
perf stat answers a question no software tool can: are these CPU cycles productive? Two processes both at 100% CPU with IPC 2.5 vs 0.4 need opposite fixes — one is genuinely compute-bound (optimize the algorithm), the other is starved by cache misses (fix data locality). Without PMCs you'd treat them the same and fail on one. perf stat is how you tell them apart in seconds.03 perf record: sampling stacks
To find where CPU goes, sample. perf record -F 99 -a -g -- sleep 30 captures stack traces at 99 Hz across all CPUs (-a) with call graphs (-g) for 30 seconds, writing to perf.data. The 99 Hz (not 100) avoids lockstep with periodic kernel activity. Use -p PID to target one process. Sampling overhead is low and roughly fixed regardless of how busy the system is — the key property that makes profiling production-safe (Chapter 4). The result is a statistical map of CPU time you then explore with report or turn into a flame graph.
04 perf report and flame graphs
Two ways to read perf.data. perf report gives an interactive text tree of where samples landed — quick but tree-shaped. The far better view for most people is the flame graph: pipe perf script through Brendan Gregg's stackcollapse-perf.pl and flamegraph.pl to get an SVG where box width is CPU time and the leaf of each wide tower is the hot function (Chapter 5). One glance replaces minutes of tree-walking. This record → script → flame graph pipeline is the single most useful perf recipe; commit it to muscle memory.
Fig 1 — The canonical recipe: perf record → perf script → stackcollapse → flamegraph. Memorize it.
05 perf top, list, and events
Supporting subcommands. perf top is a live, top-like view of the hottest functions right now — great for a quick "what's eating CPU this second." perf list enumerates everything perf can observe: hardware PMCs, software events, and the kernel's tracepoints (e.g. sched:sched_switch, block:block_rq_issue). You select events with -e — for example perf stat -e cache-misses,cycles or perf record -e block:block_rq_issue. Knowing perf list exists turns perf from "CPU profiler" into "observe any kernel event" — the gateway to its tracing side.
06 perf trace and tracepoints
Beyond CPU sampling, perf can trace events. perf trace is a lower-overhead strace-like view of syscalls. More powerfully, you can record any tracepoint — perf record -e sched:sched_switch -a to study scheduling, or block-layer tracepoints for disk I/O. Because tracepoints are stable (Chapter 4), these recipes survive kernel upgrades. perf's tracing is less programmable than BPF (next chapter) — it records events for later analysis rather than aggregating in-kernel — but for "show me every X" it's immediate and always available.
07 Overhead, frequency, and gotchas of use
Practical realities. Counting (stat) is essentially free. Sampling at 99 Hz is cheap; cranking frequency to thousands of Hz raises overhead and data volume for little extra insight — 99 is the sane default. Tracing high-frequency events (every sched_switch on a busy box) can generate huge perf.data files and real overhead, so filter and bound it. Two frequent snags: you need frame pointers or DWARF (--call-graph dwarf) for good stacks (many distros compile with frame pointers omitted, giving broken call graphs), and symbols — without debug info you get raw addresses instead of function names.
-fomit-frame-pointer for a tiny speed gain, which destroys perf's default stack walking — your flame graph becomes a forest of stubs. Fixes: use --call-graph dwarf (heavier but works without frame pointers), --call-graph lbr on supported Intel CPUs, or rebuild the hot code with frame pointers. If a flame graph looks suspiciously flat or full of [unknown], suspect frame pointers before anything else.08 Putting it together: a perf workflow
(1) perf stat -p PID — get IPC and cache behavior; is the CPU productive or stalled? (2) perf record -F 99 -p PID -g -- sleep 30 then build a flame graph — find the hot functions. (3) If memory-bound (low IPC), look at the hot stacks for cache-unfriendly access; if compute-bound, optimize the widest tower. (4) For non-CPU questions, perf list the relevant tracepoints and perf record -e them — scheduling, block I/O, syscalls. (5) Validate stacks (frame pointers/symbols) before trusting the picture. Counters to characterize, sampling to locate, tracing to detail.
common catches & gotchas
- Missing frame pointers — Broken stacks and flat flame graphs. Use
--call-graph dwarforlbr, or rebuild with frame pointers. - No symbols — Raw addresses instead of names. Install debug info / debuginfod, or build with symbols.
- Over-high sampling frequency — Thousands of Hz adds overhead and data for little gain. 99 Hz is the sane default.
- Tracing floods — Recording a high-frequency tracepoint (sched_switch) unfiltered makes huge files and real overhead. Filter and time-box.
- Reading stat without context — IPC is only meaningful relative to the workload; pair it with a flame graph to know what code has that IPC.
- Forgetting
-avs-p—-ais whole-system;-ptargets one process. Mixing them up profiles the wrong thing.
09 Questions engineers actually ask
Where do I start with perf?
Two commands cover most needs: perf stat -p PID for IPC and cache behavior (is CPU productive?), and perf record -F 99 -p PID -g -- sleep 30 → flame graph for where CPU is spent. Learn those cold; add tracepoints and perf trace as you go.
Why is my flame graph broken or full of [unknown]?
Almost always missing frame pointers (binaries built with -fomit-frame-pointer) or missing symbols. Use --call-graph dwarf (or lbr on Intel) for stack walking without frame pointers, and install debug info so addresses resolve to function names.
What's the difference between perf stat and perf record?
perf stat counts events (cycles, instructions, IPC, cache misses) for a whole run — it tells you why cycles are productive or not. perf record samples stacks over time — it tells you where the CPU is spent. Use stat to characterize, record to locate.
What sampling frequency should I use?
99 Hz is the standard default — low overhead, avoids lockstep with periodic kernel timers (which a round 100 Hz could alias with), and gives plenty of samples over tens of seconds. Higher frequencies add cost and data with diminishing returns.
Can I run perf safely in production?
Counting (perf stat) is essentially free and fine. Sampling at 99 Hz is low-overhead and commonly run in production. Be careful with high-frequency event tracing, which can add real overhead and large files — filter and time-box it.
10 Key takeaways
perfis the built-in Linux profiler — one tool that counts, samples, and traces across all kernel sources.perf statreads PMCs for cycles, instructions, IPC, and cache misses — why cycles are (un)productive.perf record -F 99 -agsamples stacks cheaply;perf report/ flame graphs show where CPU goes.- The
record→script→ flame graph pipeline is the must-know recipe. perf list+-etaps tracepoints and events for scheduling, disk, syscalls;perf topis a live view.- Mind frame pointers and symbols — their absence breaks stacks and produces flat or
[unknown]graphs. - Counting is free, 99 Hz sampling is cheap, high-frequency tracing needs filtering.
counting & IPC (why)
sampling → flame graph (where)
stacks & symbols
events & tracing
11 Wrapping up
Learn perf stat for the why and perf record→flame graph for the where, keep frame pointers in mind, and you can analyze almost any CPU problem on any Linux box without installing a thing. perf records events for offline analysis; the next two tracers let you program the kernel to answer questions perf can only approximate. First, the tracer that's also already built in. Next: Ftrace.