// the one-minute version
Observability tools come in three kinds: fixed counters (always-on numbers, cheap), profiling (sampling to see where time concentrates), and tracing (recording individual events, detailed but costlier). They draw on two instrumentation styles: static (predefined tracepoints/USDT probes baked in) and dynamic (kprobes/uprobes that instrument any function on the fly). The big frameworks — ftrace, perf, and BPF — sit on those probes. Pick the cheapest tool that answers your question, and always know its overhead.
Linux has hundreds of performance tools, and that abundance is paralyzing until you see the structure underneath. Almost every tool is one of three types reading from one of a handful of kernel sources. Once you can classify a tool on sight — "that's a counter, that's a tracer" — you instantly know its cost, its detail level, and whether it's safe to run on a production box at peak. This chapter is the map of the toolbox.
01 The three tool types
Fixed counters
Numbers the kernel/app maintain continuously — packets, faults, switches. Tools like vmstat, iostat, top just read and print them. Negligible overhead; safe anywhere; your default first look.
Profiling
Take samples at a fixed rate (e.g. 99 Hz) — "what's running right now?" — and aggregate. Builds a statistical picture of where CPU goes. Low overhead, scales to busy systems. Produces flame graphs.
Tracing
Record every event of a type — each syscall, each I/O, each function call. Maximum detail, answers "exactly what happened," but cost rises with event frequency. Use surgically.
02 Static vs dynamic instrumentation
Both profiling and tracing need instrumentation points — places to hook. There are two ways to get them.
Static instrumentation means stable hooks the developers placed in advance: kernel tracepoints (e.g. sched:sched_switch) and user-space USDT probes. They're documented and have a stable API, so tools built on them keep working across versions. Dynamic instrumentation — kprobes (any kernel function) and uprobes (any user function) — lets you hook almost anything at runtime, even code nobody planned to expose. Infinitely flexible, but tied to internal names that can change between releases.
tcp_sendmsg may silently stop matching (or match something different) after a kernel upgrade. Prefer stable tracepoints/USDT when one exists for your question; reach for kprobes/uprobes when nothing else can see what you need — and re-verify after upgrades.03 The tracing frameworks
Three big front-ends turn those probes into usable tools.
ftrace
Built into the kernel, no extra packages. Great for function tracing and tracepoints with very low overhead. The Swiss-army knife already on every box (Chapter 14).
perf
The standard profiler and event recorder: CPU profiling, hardware counters (PMCs), tracepoints, flame graphs. The first reach for "where's the CPU going?" (Chapter 13).
BPF (BCC / bpftrace)
Programmable, in-kernel, low-overhead tracing. Aggregate in the kernel and emit summaries (histograms) instead of raw events — production-safe power tools (Chapter 15).
04 The observability sources
Under the tools sit a handful of kernel interfaces. Knowing them lets you reach past a coarse tool to a richer one.
| Source | What it gives | Typical consumers |
|---|---|---|
/proc | Per-process & system counters | top, ps, vmstat, free |
/sys | Device & kernel object details | tooling, scripts |
| Tracepoints | Stable kernel events | perf, ftrace, BPF |
| kprobes / uprobes | Dynamic kernel/user hooks | perf, BPF |
| PMCs | CPU hardware counters (cache misses, cycles) | perf stat |
| USDT | App-defined static probes | BPF, perf |
| netlink | Network & socket state | ss, ip |
Fig 1 — The stack of observability: a tool is a front-end over a framework over a kernel source. Trace any number to its source.
05 Counters in practice: /proc and friends
The everyday tools are thin readers of /proc. vmstat summarizes /proc/stat and /proc/meminfo; top walks /proc/[pid]; free parses /proc/meminfo; iostat reads /proc/diskstats. Because these are just counter reads, you can run them at high frequency in production with no fear. They give rates and totals but no per-event detail — perfect for the broad first pass, useless for "which specific call was slow."
06 Profiling in practice
Profiling answers "where is the CPU spent" by sampling the running stack many times per second and counting how often each function appears. Aggregate those samples and the hot paths rise to the top — visualized as a flame graph, where width is time-on-CPU. The magic is cost control: sampling at 99 Hz adds tiny overhead regardless of how busy the app is, because you're taking a fixed number of snapshots, not tracing every instruction. This is why profiling, not tracing, is the right tool for "my CPU is high, why?"
07 Tracing in practice — and its cost
Tracing records events. The risk is volume: trace every tcp_sendmsg on a box doing 500k packets/s and you can generate gigabytes and add measurable load. Two defenses. First, filter at the source so only events you care about are recorded. Second — BPF's superpower — aggregate in the kernel: compute a latency histogram in-place and emit only the summary, never the raw stream. That turns "unsafe in production" into "safe and routine," which is the whole reason BPF tooling took over.
strace on a busy process can slow it by an order of magnitude because each traced syscall stops the process and context-switches to the tracer. The lesson isn't "don't trace," it's "know each tool's overhead before you point it at production." Counters: free. Profiling: cheap and flat. Per-event tracing: cost scales with event rate — bound it with filters and in-kernel aggregation.08 Monitoring vs ad-hoc analysis
Two modes of use. Ad-hoc analysis is you on a box right now, running tools interactively to chase a live problem — most of this book. Monitoring records metrics continuously over time (sar locally; Prometheus/Grafana at fleet scale) so you can see trends, correlate a regression with a deploy, alert on thresholds, and plan capacity. The two complement each other: monitoring tells you when and roughly where; ad-hoc tools tell you exactly why. Good USE-method metrics make excellent monitoring targets.
common catches & gotchas
- Tracing when profiling would do — "Where's my CPU?" is a profiling question. Tracing every function to answer it is slow and overkill.
- Ignoring overhead —
straceand unfiltered tracers can cripple a busy process. Always know a tool's cost class before production use. - Trusting dynamic probes across upgrades — kprobe/uprobe targets are internal names that move. Prefer tracepoints/USDT; re-verify after kernel changes.
- Confusing rate and total — Counter tools show totals or rates depending on flags. Misreading a cumulative counter as a per-second rate gives nonsense.
- One-shot snapshots — A single
iostatincludes since-boot averages on its first line. Always take an interval (e.g.iostat -xz 1) and ignore the first sample. - No monitoring, only firefighting — Without recorded history you can't tell "always been like this" from "started at 2pm." Capture baselines continuously.
09 Questions engineers actually ask
There are hundreds of tools — how do I choose?
Classify by type and question. Broad "is something wrong?" → counters (vmstat/iostat/top). "Where's the CPU?" → profiling (perf, flame graphs). "What exactly happened to this event?" → tracing (bpftrace, ftrace). Then pick the cheapest tool in that class that answers it.
What's the difference between a tracepoint and a kprobe?
A tracepoint is a stable, pre-placed hook with a documented API — safe across kernel versions. A kprobe dynamically instruments any kernel function by name at runtime — far more flexible but tied to internals that can change. Use tracepoints when one exists; kprobes when you need to see something nobody exposed.
Is it safe to run these in production?
Counters and profiling: yes, routinely. Per-event tracing: only with care — filter at the source and prefer BPF's in-kernel aggregation so you emit summaries, not raw event floods. Never point an unfiltered strace at a hot production process.
What is BPF and why is everyone excited?
BPF runs small, verified programs inside the kernel attached to probes. It can filter and aggregate events in the kernel and return just the results — giving deep, custom visibility at low, bounded overhead. That combination of power and safety is why it replaced many older tracing approaches.
What are PMCs?
Performance Monitoring Counters — special CPU hardware registers that count low-level events like cache misses, branch mispredictions, and cycles. Read via perf stat, they explain CPU behavior that software counters can't see, like why IPC (instructions per cycle) is low.
10 Key takeaways
- Tools are counters (cheap, broad), profiling (where time goes, cheap+flat), or tracing (every event, costly).
- Instrumentation is static (stable tracepoints/USDT) or dynamic (flexible kprobes/uprobes that can break on upgrade).
- The frameworks ftrace, perf, and BPF ride on those probes; classic tools just read
/proc. - Profiling samples to find hot paths (flame graphs); tracing records events and must be filtered/aggregated to stay safe.
- Every tool has an observer effect — know its overhead before production use.
- Monitoring (sar, Prometheus) records over time; ad-hoc tools explain the moment. Use both.
counters (always safe, first look)
profiling (where's the CPU)
tracing (exactly what happened)
list available probes
overhead rule of thumb
11 Wrapping up
The toolbox is no longer a jumble: three tool types, two instrumentation styles, three frameworks, a handful of sources — and a cost ordering to keep you safe. From here the book applies this toolbox resource by resource. We start where most application latency is born and analyzed: the application itself — thread pools, locks, and on- versus off-CPU analysis.