// the one-minute version
Systems performance is the study of an entire computer — all software and all hardware, from the application down to the bare metal — working together. It is hard because a request crosses dozens of layers, and the real bottleneck is rarely where you first look. The single most useful metric is latency (how long something took), because it ties directly to user pain and is easy to reason about. Two ways to learn what a system is doing: observability (watch it) and experimentation (poke it and measure). The rest of the book is methods and tools for doing both without guessing.
A user says "the site feels slow." You open a dashboard. CPU is at 40%, memory looks fine, the disk is quiet. Everything is "green" and yet the complaint is real. This is the everyday reality of systems performance: the problem is somewhere in a stack of a dozen layers, the obvious metrics lie by omission, and your job is to find the truth fast. Chapter 1 sets up the whole game — what you are actually analyzing, why it resists easy answers, and the mindset that the rest of the book turns into concrete tools.
01 What "systems performance" actually means
Systems performance is the study of a whole system: the application, the libraries it calls, the system calls into the kernel, the kernel's schedulers and memory manager, the device drivers, and the physical hardware — CPUs, memory, disks, network cards. Most engineers specialize in one slice. Systems performance is the discipline of seeing all the slices at once, because a slow request almost never respects your team's org chart.
The goal is usually one of three things: make something faster (lower latency), let it handle more (higher throughput), or do the same work on less hardware (lower cost). These often trade against each other, and which one matters depends entirely on the business. A trading system cares about the worst-case microsecond; a batch pipeline cares only about total throughput; a startup cares about the cloud bill.
02 Why it is genuinely hard
Performance work has a reputation for being mysterious, and there are real reasons for that.
The stack is deep
One web request touches the app, runtime, libraries, syscalls, kernel, drivers, and hardware. A bottleneck in any layer slows the whole thing, and each layer has its own tools and vocabulary.
It's a system, not a part
Components interact. Fixing the disk can expose a lock. Adding CPUs can make a single-threaded app no faster. Effects are non-local and sometimes counterintuitive.
Bottlenecks move
Solve one and the next appears somewhere else. Performance is a moving target — there is always a limiting resource, only the question of which one.
It can be subjective
"Slow" has no units until you measure. Without latency numbers and a target, you can't tell when you're done, or whether a change helped at all.
03 The full stack you are analyzing
Keep this mental picture. A request enters at the top and travels down through layers until it hits hardware, then results travel back up. Any layer can be the slow one.
Fig 1 — The system stack. Latency added at any layer is latency the user feels. Your job is to find which layer.
This picture is why the book is organized by resource — CPUs, memory, file systems, disks, network — with a chapter of methods first. You learn one repeatable way to interrogate each layer, then apply it wherever the trail leads.
04 Latency: the metric that matters most
If you remember one word from this book, make it latency — the time for an operation to complete. A disk read latency of 2 ms, a request latency of 100 ms, a lock-wait latency of 5 ms. Latency is powerful because it directly maps to suffering (users wait) and because it composes: the request latency is the sum of the latencies of its parts, so you can break a slow request down and find the dominant cost.
Contrast latency with metrics like IOPS, throughput, or utilization. Those are useful, but they are secondary — you can have high IOPS and a happy system, or low IOPS and a miserable one. Latency tells you whether there is a problem; the other metrics help explain why. Gregg's advice: when you can, measure and target latency first.
05 Two ways to know: observability and experimentation
There are exactly two families of activity in performance work.
Observability
Watch the system as it runs, without changing it. Counters, tracing, profiling, logs. This is where you spend most of your time, and the bulk of this book. The golden rule: observation should be safe and low-overhead in production.
Experimentation
Apply a deliberate, controlled load and measure the response — benchmarking, load tests, micro-benchmarks. Powerful for "what if," but easy to get wrong and to mislead yourself (Chapter 12 is a list of those traps).
Most real investigations are observability-led: you watch production, form a hypothesis about the bottleneck, and only then run an experiment to confirm a fix. Leading with benchmarks — "let's see how fast it can go" — tends to produce impressive numbers that have nothing to do with the actual problem.
06 Counters, tracing, and profiling
Observability tools come in three flavors, and knowing which is which saves hours.
Counters are numbers the kernel and apps keep all the time — bytes sent, context switches, page faults. Cheap to read; tools like vmstat and iostat just print them. Tracing records individual events as they happen — every syscall, every disk I/O — giving rich detail at higher cost; strace, ftrace, and BPF live here. Profiling takes samples at intervals (say, the running function 99 times a second) to build a statistical picture of where time goes; this is what produces flame graphs.
07 Utilization, saturation, errors
Chapter 2 formalizes a method around three quantities you will meet for every resource, so meet them now. Utilization is how busy a resource is (a disk 90% busy). Saturation is the work that couldn't be serviced and had to queue (the run-queue length, the I/O wait queue). Errors are failed operations (dropped packets, failed mallocs). A resource that is saturated is actively hurting you, even if utilization isn't quite 100% — queueing is where latency is born.
08 Cloud computing changes the rules
This edition is "Enterprise and the Cloud" for a reason. In the cloud you share hardware with strangers (multi-tenancy), so a noisy neighbor can steal your CPU cycles or disk bandwidth and never show up in your own metrics. You also hit hard limits — the provider caps your IOPS or network, and you saturate a ceiling you can't see in top. And instances are disposable, which changes the economics: sometimes the fix is "add another node," sometimes that just multiplies a per-node inefficiency. Cloud performance gets its own chapter (11), but keep the multi-tenant mindset from the start.
common catches & gotchas
- Blaming the first green-ish metric — CPU at 40% does not exonerate the CPU; a single saturated core or a hot thread can bottleneck while the average looks calm. Always check per-CPU and per-thread.
- Trusting averages — The mean smears the tail you care about. Demand percentiles and histograms before you conclude anything about latency.
- Benchmarking first — Running a load test before you understand the workload usually measures the benchmark's flaws, not your system. Observe production first.
- Confusing utilization with saturation — 100% busy is not the alarm; queueing (saturation) is. A resource can be 100% utilized and perfectly healthy if nothing waits.
- Observer effect — Heavy tracing can slow the very thing you measure. Know each tool's overhead before pointing it at production.
- Stopping at the symptom — "CPU is high" is a symptom, not a cause. Keep drilling until you reach the function, query, or config that actually drives it.
09 Questions engineers actually ask
Where do I even start when something is "slow"?
Get a latency number and a target first ("p99 is 800 ms, we want 200"). Then run a quick broad checklist (Chapter 2's USE method and the "60-second" tool tour) to spot the obviously stressed resource, and drill down from there. Never start by guessing a cause.
Is high CPU usage bad?
Not by itself. High utilization with low latency and no saturation is a system doing its job efficiently. CPU is "bad" only when it causes queueing or steals time from work that matters. Look at run-queue length and per-thread breakdown, not just the percentage.
Why not just add more hardware?
Sometimes that's the right call. But if the app is single-threaded, or the bottleneck is a lock or a slow query, more hardware does nothing — or multiplies the waste. Measure first so you scale the resource that's actually limiting you.
What's the difference between tracing and profiling?
Tracing records every event of a type (every syscall) — complete but expensive. Profiling samples at intervals to estimate where time is spent — cheap and great for "where is the CPU going," which is exactly what flame graphs show.
Do I need to know kernel internals?
Enough to read the metrics, yes — which is why Chapter 3 exists. You don't need to write a scheduler, but knowing what a context switch, page fault, or run queue is turns cryptic counters into a clear story.
10 Key takeaways
- Systems performance studies the whole machine — app to hardware — because bottlenecks ignore team boundaries.
- It's hard because the stack is deep, components interact, and the limiting resource keeps moving.
- Latency is the most useful metric: it maps to user pain and breaks down cleanly into parts.
- Two activities only: observability (watch) and experimentation (poke). Observe first, benchmark second.
- Tools are counters (cheap, broad), profiling (where time goes), and tracing (exactly what happened) — use them broad-to-narrow.
- For every resource, watch utilization, saturation, and errors — and never trust the average over the distribution.
- The cloud adds multi-tenancy, hidden limits, and noisy neighbors to every problem.
the 60-second triage (run these in order)
r (run queue) and si/so (swapping).await), queue. The disk truth.the three things to ask every resource
r, await, backlog) — this is where latency is born.mindset
11 Wrapping up
Chapter 1 is the orientation map. The system is the whole stack; the metric is latency; the work is observe-then-experiment; the tools are counters, profiling, and tracing used broad-to-narrow; and every resource gets the same three questions — utilization, saturation, errors. Nothing here is a recipe yet. The recipes start next, with the methodologies that turn this mindset into a repeatable checklist you can run under pressure.