Systems Performance Chapter 12 — Benchmarking.

Benchmarks are how we make decisions — which database, which instance, did this change help — and they are wrong far more often than anyone admits. The danger isn't that they fail; it's that they succeed, producing a clean, plausible number that measures the wrong thing entirely. This chapter is a guide to skepticism: how benchmarks lie, and the one practice — active benchmarking — that turns them from a trap into a tool.

01 Why benchmarks lie

A benchmark produces a number, and numbers feel authoritative. But that number is only as good as the assumption that the benchmark exercised the real thing under realistic conditions — and that assumption is usually false. The benchmark might measure the page cache instead of the disk, a warmed-up best case that production never sees, a single-thread path when production is concurrent, or simply a different workload than yours. The result is precise, reproducible, and meaningless. Gregg's blunt framing: treat every benchmark result as wrong until you've actively proven what it measured.

02 Micro vs macro benchmarks

Two kinds, two failure modes. A micro-benchmark tests one narrow operation (raw disk reads, a single function, syscall latency). It's controllable and repeatable but isolated — fast micro numbers often don't translate to real-app speed because production mixes operations and contends for resources. A macro-benchmark (or application benchmark) drives the whole system with a realistic workload — closer to truth but harder to set up and interpret. Micro-benchmarks answer "how fast is this component in isolation"; only macro-benchmarks answer "how fast will my application be," and confusing the two is a common mistake.

03 Active benchmarking — the core practice

The chapter's central method, and the antidote to lying benchmarks. Passive benchmarking is the usual approach: run the tool, record the number, move on. Active benchmarking means that while the benchmark runs, you analyze the system with the full observability toolkit — is the resource you intended actually the bottleneck? Is CPU pegged, or is it waiting on disk? Is the benchmark even saturating anything? Half the time you discover the benchmark is limited by something irrelevant (its own client, a network hop, a cache) and the number is garbage. Active benchmarking both validates the result and teaches you where the real limit is.

04 The sins of benchmarking

A catalog of how they go wrong. No warmup: measuring cold caches, an un-JITed runtime, or empty buffers gives a worst case the real system never hits — or, run the other way, a tiny dataset cached entirely in RAM gives a best case that doesn't scale. Wrong target: a "disk benchmark" that fits in the page cache measures RAM. Ignoring variance: one run is noise; you need many to see the spread. Single-number reporting: an average hides the tail that matters. Unrealistic workload: uniform random keys when production is skewed, or one giant request when production sends thousands of small ones.

05 Workload realism

A benchmark is only as useful as its resemblance to your real load. Production workloads have skew (a few hot keys get most traffic), mixed operations (reads and writes interleaved, not pure), realistic concurrency (many clients, not one), realistic data sizes (datasets bigger than cache), and arrival patterns (bursty, not uniform). A benchmark that gets these wrong measures a system that doesn't exist. The closer you can get to replaying or modeling actual production traffic, the more the result means. When in doubt, prefer a flawed macro-benchmark of real-ish load over a pristine micro-benchmark of something you don't actually do.

06 Statistics: distributions, not points

One number is a lie of omission. Run the benchmark many times and you get a distribution — and that's the truth. Report the mean with a measure of variance (standard deviation), and for latency always report percentiles (p50, p99, p99.9) and the max, because the tail is what users feel and what averages bury. High variance between runs is itself a finding: it means something uncontrolled is interfering (a noisy neighbor, thermal throttling, background jobs), and the "result" isn't stable enough to trust. Don't compare two single numbers; compare two distributions.

07 Benchmarking for regression detection

One of the most valuable uses: catching performance regressions before they reach production. Run a consistent benchmark in CI on each change and compare against a baseline — a sudden 20% latency increase flags a bad commit while it's cheap to fix. The keys are consistency (same hardware, same workload, controlled environment) and statistical rigor (enough runs to distinguish a real regression from noise). This is where even an imperfect benchmark earns its keep: as long as it's consistent, relative changes are meaningful even if the absolute number isn't perfectly realistic.

08 A benchmarking checklist

(1) Define what you're measuring and why — what decision rides on it? (2) Make the workload as realistic as you can (size > cache, real concurrency, skew). (3) Warm up appropriately, then measure steady state. (4) Actively observe during the run — is the intended resource the bottleneck? (iostat, mpstat, etc.). (5) Run many times; compute mean, variance, and percentiles. (6) Control the environment (no background jobs, pinned hardware, watch for cloud steal/throttling). (7) Report the distribution and the conditions, not a lone number. Skip any step and you've probably produced a confident lie.

09 Questions engineers actually ask

10 Key takeaways

• Most benchmarks are wrong — and dangerously plausible. Distrust every result until verified.
• Active benchmarking — observe the system while it runs — is the core practice that catches lies and finds the real bottleneck.
• Micro benchmarks test isolated ops; only macro benchmarks predict application performance.
• Avoid the sins: no/too-much warmup, measuring the cache, ignoring variance, single averages, unrealistic workloads.
• Make the workload realistic — skew, mix, concurrency, data bigger than cache.
• Report distributions (mean + variance + percentiles), never a lone number; run many times.
• Consistent benchmarks in CI catch regressions early — relative change is meaningful even if absolute isn't perfect.

11 Wrapping up

Benchmarking is less about running tools and more about disciplined skepticism: active observation, realistic workloads, and distributions over single numbers. Do it well and you get decisions you can trust; do it lazily and you get confident fiction. The remaining chapters return to the deep observability tools that make active benchmarking — and all serious analysis — possible. First, the standard Linux profiler. Next: perf.