Benchmarking Basics — what the numbers actually mean.

TL;DR — A benchmark is a controlled measurement that turns "feels fast" into a number you can compare. But the number only means something if you know what it measured, how, and against what. This part lays the foundation — kinds of benchmarks, the metrics that matter (throughput vs latency, p50 vs p99), the traps that produce lies, and a glossary you can return to for the rest of the series.

What a benchmark actually is

A benchmark is a repeatable test that puts a system under a defined load and records how it responds. That's it. The point is comparison — chip A vs chip B, instance X vs instance Y, before a change vs after. The hard part isn't running the test; it's making the comparison fair and the result meaningful.

A number with no context — "it scored 9,200" — is useless. Useful is: "9,200 multi-core Geekbench 6 on an 8-vCPU Graviton4 instance, Ubuntu 24.04, us-east-1, median of 5 runs." Now someone can reproduce it and compare.

The two families: synthetic vs real-world

Every benchmark falls on a spectrum between two poles.

• Synthetic benchmarks stress one thing in isolation — integer math (CoreMark), memory bandwidth (STREAM), crypto (OpenSSL). Clean, repeatable, easy to compare. But they don't tell you how your app behaves.
• Real-world / application benchmarks run an actual workload — compile the Linux kernel, serve 10k HTTP requests, run a Postgres query mix. Messier and harder to reproduce, but they predict production.

The mistake is trusting one pole alone. A CPU that wins CoreMark can lose your build because your build is memory-bound. Synthetic tells you why; real-world tells you whether. You want both.

Throughput vs latency — the metric that trips everyone

Two systems can have identical throughput and wildly different latency. They measure different things:

• Throughput — how much work per unit time (requests/sec, tokens/sec, builds/hour). What you want for batch jobs, data pipelines, CI fleets.
• Latency — how long one operation takes (ms per request). What you want for user-facing services.

You can raise throughput by batching, which often raises latency. Optimizing one can hurt the other. Decide which your workload cares about before you benchmark, or you'll optimize the wrong number.

Why averages lie — percentiles

"Average latency 20ms" hides the users having a bad time. If 1% of requests take 2 seconds, the average barely moves, but those users churn. Report percentiles:

• p50 (median) — the typical experience.
• p95 / p99 — the tail. The 99th percentile is the slowest 1%. At scale that 1% is millions of requests.

A flat p50 with a spiking p99 means inconsistency — often noisy neighbours or GC pauses. The tail is usually where the real problem lives.

Single-core vs multi-core

Two different questions:

• Single-core — how fast is one thread? Decides compile latency, single-threaded DB sections, anything serial. Driven by clock speed × IPC.
• Multi-core — how much total work across all cores? Decides parallel builds, web servers, batch. Driven by core count × per-core × how well it scales.

Scaling is rarely linear — 16 cores rarely give 16× the throughput because of shared caches, memory bandwidth limits, and lock contention (Amdahl's law). A multi-thread scaling test measures exactly this falloff.

The traps that produce lies

Most bad benchmarks aren't fraud — they're sloppy. The usual suspects:

• No warm-up. The first runs hit cold caches, JIT compilation, and cloud burst credits. Measure steady state, discard the warm-up.
• One run. Cloud performance varies run-to-run (neighbours, throttling). Run 5–10 times, report median + spread, not a single number.
• Apples to oranges. Different region, OS, kernel, compiler flags, or instance size = invalid comparison. Change one variable at a time.
• Ignoring CPU steal. On shared/burstable instances the hypervisor can take your core. High steal = your number isn't about the chip.
• Synthetic-only. Winning CoreMark ≠ winning your workload.
• Ignoring cost. The fastest instance is irrelevant if it's 3× the price. Always divide by dollars.

A benchmark you can't reproduce is an anecdote. Record the instance type, region, OS, kernel, tool version, and run count — every time.

Performance per dollar — the only ranking that pays

At fleet scale, raw speed is a vanity metric. What matters is score ÷ price. An instance that's 10% slower but 40% cheaper wins almost every real decision. This is why ARM and Oracle show up at the top of value charts even when they don't top the speed charts — and it's the thread running through this whole series.

Glossary — terms for the rest of the series

vCPU

A virtual CPU — usually one hardware thread (half a physical core on SMT/hyperthreaded x86). The unit most clouds bill and size by.

OCPU (Oracle)

One full physical core (= 2 vCPU-equivalent threads). OCI prices per OCPU, so 1 OCPU looks pricey but delivers ~2 vCPUs of work.

IPC

Instructions per cycle. How much work a core does each clock tick. Higher IPC × higher clock = faster single-thread.

Throughput

Work completed per unit time (req/s, ops/s, tokens/s).

Latency

Time for a single operation to complete (ms).

p50 / p95 / p99

Percentiles. p99 = the slowest 1% of operations — the tail that hurts at scale.

Geekbench 6

Popular cross-platform CPU benchmark giving single- and multi-core scores. Quick, comparable, but synthetic.

sysbench

Scriptable benchmark for CPU, memory, and database (OLTP) workloads. A DevOps staple.

CoreMark

Tiny synthetic integer CPU benchmark from EEMBC. Isolates core compute.

SPEC CPU

The industry-standard (paid) CPU suite — real application kernels. The gold standard, rarely run casually.

STREAM

The standard memory-bandwidth benchmark (copy/scale/add/triad). Dominated by memory channels × DDR speed.

UnixBench

Old but enduring composite system benchmark (mixed CPU, process, syscall).

fio

Flexible I/O tester — the standard for disk/storage throughput and latency.

CPU steal

Time your vCPU was ready but the hypervisor gave the physical core to someone else. High steal = noisy neighbours.

NUMA

Non-Uniform Memory Access. On multi-socket machines, reaching another socket's RAM is slower.

Warm-up

Initial runs that fill caches / JIT / burst credits. Excluded from the measured steady state.

Geomean

Geometric mean — the right way to average ratios across a benchmark suite (one giant result can't dominate).

Baseline / regression

A reference measurement; a regression is a measured slowdown against it.

Noisy neighbour

Another tenant on the same host stealing shared resources, hurting your consistency.

Burstable instance

Cheap instances (AWS t-class etc.) that allow brief CPU bursts on credits, then throttle. Bad for sustained benchmarks.

Perf-per-dollar

Benchmark score ÷ hourly price. The metric that decides real fleets.

What's next

With the vocabulary in hand, Part 2 applies it: a full cited comparison of CPU and memory performance across AWS, Azure, GCP, and OCI in 2026 — who wins per vCPU, per GB, and per dollar.