Systems Performance Chapter 2 — Methodologies Explained: USE Method, Drill-Down, Latency Analysis

// the one-minute version

A methodology is a repeatable procedure for finding performance problems, so you stop guessing. Skip the anti-methods (random tweaking, blaming whatever you happen to know, watching until something looks off). Lead with the USE method: for every resource, check Utilization, Saturation, and Errors. Then characterize the workload (who is asking for what, how much), drill down from symptom to cause, and use latency analysis to break a slow operation into its parts. Methods turn panic into a checklist.

Two engineers face the same slow server. One opens random config files and changes things, hoping. The other runs a fixed checklist, finds a saturated disk queue in ninety seconds, and traces it to one runaway query. The difference isn't talent — it's method. This chapter is the most valuable in the book: a set of procedures you can run under pressure that lead you to the answer instead of around it.

01 Why you need a method at all

Performance problems are intimidating because the search space is huge — any of a dozen layers, any of dozens of resources, could be the cause. Without a procedure, people fall back on instinct, and instinct is biased toward whatever they know best. The database expert blames the database; the network engineer blames the network. A methodology replaces "what do I suspect?" with "what does the system actually show?" — and it makes your work repeatable, teachable, and honest.

think of it likeA pilot's pre-flight checklist. Pilots are experts, but under stress experts forget steps and skip checks. The checklist guarantees coverage regardless of mood, fatigue, or hunch. A performance methodology is the same: it's not there because you're dumb, it's there because every human is biased and forgetful exactly when the stakes are highest.

02 The anti-methods (what not to do)

Gregg names the bad habits so you can catch yourself doing them.

Street-light anti-method

Looking only where the light is — using whatever tool you already know, not the one the problem needs. Like the drunk searching for keys under the lamp because that's where it's bright.

Random change anti-method

Tweak a setting, see if it helps, repeat. Sometimes it "works," but you don't know why, can't reproduce it, and may have caused two new problems.

Blame-someone-else

Decide it's another team's component without evidence, hand it off, and move on. Wastes everyone's time and is often wrong.

Ad hoc checklist

Run the same few commands you always run, regardless of the symptom. Better than nothing, but blind to anything outside your habitual list.

watch outThe random-change anti-method is seductive because it occasionally appears to work. But a change that "fixes" things by coincidence — or by masking the symptom — leaves the real cause in place to bite you again, usually at 3 a.m. If you can't explain why a change helped, you haven't actually found the problem.

03 The USE method

The flagship. For every resource in the system, check three things:

Utilization

The percent of time the resource was busy (or the fraction of capacity used). A disk 90% busy, memory 80% full.

Saturation

The degree of queued, unserviced work. Run-queue length, I/O wait queue, swap activity. Saturation means demand exceeds supply — and that is where latency comes from.

Errors

The count of error events — dropped packets, failed allocations, I/O errors. Often checked first because errors are quick to read and frequently the real story.

You build a list of resources (CPUs, memory, disks, network interfaces, buses, controllers) and walk it, asking U, S, E of each. The first resource with high saturation or errors is your lead. It's powerful precisely because it's exhaustive: you can't skip the resource you forgot to suspect.

Fig 1 — Walk every resource through Errors → Saturation → Utilization. The first that's saturated or erroring is your lead.

04 Workload characterization

Before optimizing, understand what you're being asked to do. Workload characterization answers four questions: who is causing the load (which process, client, IP), why (which code path, which endpoint), what the load is (read vs write, request sizes, rates), and how it changes over time. This shifts attention from the machine to the demand placed on it — and often the fix is to the workload, not the hardware. The fastest query is the one you never run.

key ideaYou can tune a system endlessly, but if the workload itself is wasteful — duplicate requests, an N+1 query, a client retrying in a tight loop — no amount of resource tuning fixes it. Characterizing the workload frequently reveals that the cheapest, biggest win is to remove or reshape the work, not to make the machine faster at doing the wrong thing.

05 Drill-down analysis

Start at the top (a high-level symptom: "requests are slow") and peel down layer by layer, at each step narrowing to the component responsible, until you reach the root cause. Request slow → which endpoint? → that endpoint waits on the DB → the DB waits on disk reads → those reads are slow because of a saturated disk → because of one table scan → because of a missing index. Each step is a question whose answer points to the next, deeper question.

think of it likeFollowing water back upstream to find the leak. You don't randomly dig; you start at the puddle and trace the dampness, room by room, pipe by pipe, until you reach the burst joint. Drill-down is structured upstream-tracing for latency.

06 Latency analysis

Take the total latency of an operation and split it into the time spent in each component, then attack the biggest slice. A 100 ms request might be 5 ms app, 10 ms network, 80 ms database, 5 ms render — so the database is 80% of the answer and everything else is a rounding error. This is drill-down expressed in the currency of time, and it keeps you honest about where the effort should go. Optimizing the 5 ms render to zero saves nothing a user would notice.

the catchLatency breakdowns must be measured, not assumed. Engineers routinely optimize the part they understand (the app code) while 80% of the time hides in a layer they rarely look at (a DNS lookup, a lock wait, a cold cache). Until you've actually attributed the milliseconds, you don't know where the time is — and "obvious" answers are wrong often enough to keep you humble.

07 Other methods worth knowing

Gregg catalogs many. A few you'll reach for: The 60-second checklist — a fixed tour of ten tools (from Chapter 1) for a fast first read. Off-CPU analysis — instead of asking where the CPU is spent, ask where threads block and wait (locks, I/O, sleeps), which often dominates real-world latency. Static performance tuning — check the configuration and limits before load even arrives (wrong thread-pool size, a 1 Gbps cap, default buffer sizes). Baselines — record normal so you can recognize abnormal; "is this slow?" needs a "compared to what?".

08 Monitoring and modeling, briefly

Two long-horizon ideas. Monitoring records metrics over time (USE metrics make great monitoring targets) so you can spot trends, correlate a slowdown with a deploy, and do capacity planning before you hit a wall. Modeling predicts behavior under load you haven't yet seen — queueing theory says latency stays flat as utilization rises, then explodes near 100%, which is why a resource at 80% can be fine and at 95% can be on fire. You don't need the math, but you need the shape of that curve in your head.

common catches & gotchas

Skipping errors — Errors are the fastest USE check and frequently the actual cause. Don't jump to utilization graphs before reading dmesg and error counters.
USE without a resource list — The method only works if your list is complete. The bottleneck is often the bus, controller, or interconnect nobody listed.
Tuning before characterizing — Optimizing the machine while the workload is the problem. Always ask what work is being demanded before making the machine faster at it.
Guessing the latency breakdown — Assuming where time goes instead of measuring. The dominant slice is regularly in a layer you weren't watching.
On-CPU tunnel vision — Profiling only on-CPU time and missing that threads spend most of their life blocked off-CPU on locks and I/O.
No baseline — "Is 50 ms slow?" is unanswerable without a recorded normal. Capture baselines while things are healthy.

09 Questions engineers actually ask

Which method do I use first?

USE for a fast, complete sweep of resources, plus the 60-second checklist for a first read. Once USE flags a saturated or erroring resource, switch to drill-down and latency analysis to chase the cause into that resource.

What exactly is saturation, versus utilization?

Utilization is how busy a resource is (a percentage). Saturation is how much work is waiting because the resource is busy (a queue length). You can be 100% utilized with zero saturation (fully used, nothing waiting) — that's healthy. Saturation is the painful one.

Isn't a checklist too rigid for weird problems?

The checklist gets you 80% of the way fast and rules out the common causes. For genuinely novel problems you still need drill-down and tracing — but you'll reach them having already eliminated the obvious, instead of chasing a hunch from minute one.

What is off-CPU analysis and why care?

It measures time threads spend not running — blocked on locks, disk, network, or sleeps. Most user-facing latency is waiting, not computing, so off-CPU analysis often explains slowness that on-CPU profiling completely misses.

Why does latency explode near 100% utilization?

Queueing theory: as a resource approaches full, even tiny bursts have nowhere to go, so queue length — and therefore wait time — shoots up non-linearly. It's why capacity planning targets ~70-80%, not 99%.

10 Key takeaways

A methodology replaces guessing with a repeatable procedure — the most valuable skill in performance work.
Avoid the anti-methods: street-light, random change, blame-shifting, habitual checklists.
The USE method — Utilization, Saturation, Errors for every resource — is the exhaustive first sweep.
Characterize the workload (who/why/what/how) before tuning; often the work itself is the problem.
Drill down from symptom to cause; use latency analysis to spend effort on the biggest time slice.
Remember off-CPU time, static tuning, and baselines; keep the queueing curve in mind.

// chapter cheatsheetUSE method by resource

CPU

util: mpstat -P ALL 1Per-CPU %usr/%sys/%idle. Catch one pinned core.

sat: vmstat 1 → rRun-queue length > CPU count = CPU saturation.

err: dmesg; perf countersRare, but throttling / MCE errors show here.

Memory

util: free -m; vmstatAvailable memory; cache is reclaimable.

sat: vmstat 1 → si/soSwapping in/out = memory saturation.

err: dmesg | grep -i oomOOM killer events = out of memory.

Disks

util: iostat -xz 1 → %utilPer-device busy percent.

sat: aqu-sz / awaitQueue length & wait time = saturation.

err: dmesg; smartctlI/O errors, bad sectors.

Network

util: sar -n DEV 1Throughput vs link/limit ceiling.

sat: sar -n EDEV; ss -tiDrops, retransmits, full backlogs.

err: ip -s link; netstat -sErrors, drops, overruns per interface.

the procedure

1. errors → 2. saturation → 3. utilizationPer resource, in that order.

characterize → drill down → latencyUnderstand the work, trace to cause, spend on the biggest slice.

11 Wrapping up

Methods are the backbone of everything that follows. USE gives you breadth, drill-down and latency analysis give you depth, workload characterization keeps you honest about what you're optimizing, and baselines tell you when you're done. To run any of this you need to read the metrics correctly — which means understanding the machine producing them. Next: Operating Systems, the internals behind every number.

Systems Performance Chapter 2 — Methodologies.