// the one-minute version
The final chapter is a worked investigation: a service is slow, the dashboard looks green, and we find the cause without guessing. The path is the whole book in miniature — get a latency number and target, run the 60-second USE triage, follow the trail with drill-down, notice the time is off-CPU (not on-CPU), trace the blocking to a surprising root cause, apply a fix, and verify with the same metric. The lesson isn't the specific bug — it's that a disciplined method beats intuition every time.
Everything so far has been pieces — methods, resources, tools. This chapter assembles them on one realistic problem, the way a real investigation actually unfolds: messy, with false leads and a surprise at the end. Follow the process, not just the answer. The specific bug here is a stand-in; the sequence of moves is the transferable skill, and it's the same sequence whether the culprit turns out to be a disk, a lock, a kernel setting, or a noisy neighbor.
01 The report: "the API is slow"
It starts vague, as it always does: users report the checkout API "feels slow," intermittently. The dashboard shows CPU around 45%, memory fine, disk quiet, no errors. Everything green — the classic trap from Chapter 1. The first move is to refuse to act on a feeling: get a number and a target. We pull request latency and find p50 is a healthy 40 ms but p99 is 1.8 seconds, against a target of 200 ms. Now we have a real, bounded problem: a tail-latency issue, not a broad slowdown. That distinction shapes everything next.
02 60-second triage with USE
Before theorizing, the fast broad sweep (Chapters 1-2). uptime — load normal for the core count. vmstat 1 — run queue low, no swapping. mpstat -P ALL 1 — no single hot core; CPU genuinely ~45%. iostat -xz 1 — disk await low, aqu-sz near zero; the disk is idle, not a suspect. sar -n DEV/EDEV — network throughput modest, no drops or retransmits. dmesg — clean, no OOM, no errors. The USE sweep comes back clean on every resource. That's not a dead end — it's a strong signal: the bottleneck isn't raw resource saturation. The time is being spent waiting, not working.
03 The on-CPU profile looks innocent — and that's the clue
Natural next step: profile the CPU (Chapters 5-6, 13). perf record -F 99 -p PID -g for 30 seconds, build a flame graph — and it's unremarkable. Some JSON work, some business logic, nothing dominating, and crucially the app spends most wall-clock time not on CPU at all. This is the pivotal moment: a clean on-CPU profile on a slow service means the latency is off-CPU — the threads are blocked, waiting on something, not burning cycles. Chapter 5 warned about exactly this tunnel vision. We switch lenses from "where is the CPU?" to "where do threads wait?"
Fig 1 — Each step either finds the bottleneck or rules out a layer. The clean on-CPU profile pointed straight at off-CPU waiting.
04 Off-CPU analysis finds the wait
Now BPF earns its place (Chapter 15). offcputime -p PID 30 records where threads block, with stacks. The output is decisive: the bulk of off-CPU time, on the slow requests, sits in two places — a lock wait (futex) inside a shared client object, and beneath some requests, a blocking name resolution (DNS) on a downstream call. The lock is serializing requests that contend for one shared connection object; occasionally a DNS lookup stalls for over a second when the resolver cache misses and a slow upstream resolver is hit. The "green dashboard" never showed either, because neither is a saturated resource — they're waits.
05 Confirming and fixing
We confirm each (Chapter 2's "don't trust a guess"). For the lock: a quick bpftrace on the futex shows wait time scaling with concurrency — more parallel requests, longer waits — consistent with one shared object serializing access. The fix: give each worker its own connection (a pool) instead of sharing one, removing the contention. For DNS: traces show the stalls correlate with cache-miss lookups; the fix is a local resolver cache and a sane timeout so a slow upstream can't hang a request for 1.8 s. Two targeted changes, each tied to evidence, neither a random tweak.
06 Verifying with the same metric
The discipline that closes the loop: verify with the same number you started with (Chapters 2 & 12). After deploying, p99 drops from 1.8 s to 140 ms — under the 200 ms target — while p50 is unchanged (the fast path was never the problem). We watch it across a full traffic cycle, not one lucky minute, to be sure it holds under peak concurrency where the lock contention was worst. The investigation is done because the defining metric crossed its target and stayed there — not because something "felt faster." That objective close-out is what separates a fix from a hope.
07 Why intuition would have failed
Worth dwelling on the false paths intuition offered. The CPU expert would have profiled on-CPU, seen nothing, and concluded "not a code problem." The infra engineer, seeing 45% CPU, might have scaled out — more nodes — which would have done nothing for a per-request lock and DNS stall, just multiplied the cost. Someone might have "tuned" the database that wasn't involved. Every one of these is the street-light or random-change anti-method from Chapter 2. Only the disciplined sequence — number, USE, drill-down, off-CPU, confirm, verify — walked straight to causes that no hunch would have suggested.
08 The reusable playbook
Strip the specifics and you have a procedure for any performance incident. (1) Quantify: latency metric + percentile + target. (2) Triage: 60-second USE sweep across resources. (3) Branch on the result: a saturated resource → drill into it; everything clean → suspect off-CPU waiting. (4) Profile on-CPU; if innocent, do off-CPU analysis. (5) Drill down the blocking stacks to a concrete cause. (6) Confirm the cause with a targeted trace. (7) Fix with an evidence-tied change. (8) Verify against the original metric, under real load. This is the entire book compressed into eight repeatable steps.
common catches & gotchas
- Stopping at the green dashboard — Clean resource metrics don't clear the system; off-CPU waits (locks, DNS, downstream stalls) are invisible to utilization monitoring.
- On-CPU tunnel vision — An innocent CPU flame graph on a slow service is itself the clue: the time is off-CPU. Switch lenses, don't conclude "not code."
- Acting on a hunch — Scaling out, tuning the database, or restarting before quantifying just burns time and can hide the real cause. Quantify first.
- Averages over percentiles — A tail problem (p99) is invisible in the mean. Always look at the distribution that matches the complaint.
- Unverified fixes — "Feels faster" isn't done. Confirm the defining metric crossed its target and holds under peak load.
- One lucky measurement — Verify across a full traffic cycle; contention and stalls are worst at peak concurrency, not at idle.
09 Questions engineers actually ask
My dashboards are all green but users say it's slow. Where do I look?
Suspect off-CPU waiting — locks, blocking I/O, slow downstream calls, DNS stalls — which consume no CPU and saturate no resource, so utilization dashboards can't see them. Quantify the latency (with percentiles), confirm the on-CPU profile is innocent, then run off-CPU/blocking analysis (BPF offcputime) to find the wait.
Why not just add more servers when it's slow?
Because many causes — a per-request lock, a slow downstream call, a DNS stall — aren't relieved by more nodes; you just multiply the same per-request cost and the bill. Quantify and diagnose first; scale only when you've confirmed you're genuinely resource-bound doing necessary work.
How do I know when I'm actually done?
When the metric you started with crosses its target and stays there under real, peak load — not when something subjectively "feels faster." Bracket the whole investigation with the same objective number, and watch it across a full traffic cycle before declaring victory.
What's the single most useful habit from this?
Quantify before you act. Converting "it's slow" into "p99 is 1.8s, target 200ms" prevents guessing, focuses the search (tail vs broad), and gives you an objective finish line. Almost every failed investigation skipped this step and chased a hunch instead.
Does this process change for different bugs?
No — that's the point. Whether the cause is a disk, a lock, a kernel setting, or a noisy neighbor, the sequence (quantify → USE triage → branch → profile → off-CPU → drill down → confirm → verify) is the same. Only which branch you take and which tool you reach for changes.
10 Key takeaways
- Turn "slow" into a number + percentile + target before doing anything else.
- A clean USE triage isn't a dead end — it points away from resource saturation toward waiting.
- An innocent on-CPU profile on a slow service means the time is off-CPU; switch lenses.
- Off-CPU analysis reveals locks, blocking I/O, and downstream/DNS stalls that dashboards can't see.
- Confirm each cause with a targeted trace; fix with evidence-tied changes, not hunches.
- Verify against the original metric, under peak load — objective close-out, not "feels faster."
- The eight-step playbook is the whole book: it works for any bottleneck because method beats intuition.
the 8 steps
tools by step (all from this book)
the one rule
11 Wrapping up the book
That's the whole companion: from "what is systems performance" to a complete investigation that uses nearly every idea along the way. The throughline never changed — latency is the metric, method beats intuition, measure broad-to-narrow, watch the waits not just the work, and verify objectively. Internalize the eight-step playbook and you can walk into any slow system, on any stack, and find the truth without guessing. Back to the chapter index — or start again at Chapter 1.