// the one-minute version
Disks have three numbers: IOPS (operations/sec), throughput (MB/sec), and latency (ms/op) — and latency is the one users feel. Random I/O is far slower than sequential, brutally so on spinning disks. Read iostat -xz 1 properly: await is per-I/O latency, aqu-sz is queueing (saturation), and %util is busy time — but %util is a trap on SSDs and virtual disks that serve many I/Os in parallel, where 100% doesn't mean full. Judge a disk by its latency and queue, not by %util.
The disk is the slowest thing most requests touch, so when it's the bottleneck, it really hurts — and it's also the resource people most often misread, because the famous %util column means something different than it did twenty years ago. This chapter teaches you to read storage metrics for what they actually say in a world of SSDs, RAID, and cloud volumes, so you can tell a busy-but-healthy disk from one that's drowning.
01 HDD vs SSD: different physics
A spinning disk (HDD) moves a physical head to a track and waits for the platter to rotate under it. That mechanical seek + rotation is why random I/O is so slow — each random read may cost ~5-10 ms of pure mechanical delay, capping a HDD at a few hundred random IOPS. A solid-state disk (SSD) has no moving parts: random access is nearly as fast as sequential, delivering tens of thousands of IOPS at sub-millisecond latency. This single difference reshapes every storage decision — a layout that's catastrophic on HDD (random small reads) may be perfectly fine on SSD.
02 The three metrics: IOPS, throughput, latency
Keep them distinct. IOPS is operations per second — what matters for many small random I/Os (databases, metadata). Throughput (bandwidth, MB/s) is what matters for large sequential transfers (streaming, backups, big scans). Latency is time per operation — what the application waits for. They trade off: you can have high IOPS of tiny ops with low throughput, or high throughput of big ops with low IOPS. And a disk can hit its IOPS or throughput ceiling and then latency explodes. As always, latency is the metric tied to user pain; the other two explain the limit you hit.
03 Random vs sequential
The most important access-pattern distinction. Sequential I/O reads/writes adjacent blocks — the disk (or its read-ahead, or the SSD controller) streams them efficiently. Random I/O jumps around — on a HDD every jump is a seek; even on SSDs random small I/O has more overhead than large sequential. Workloads that look identical in total bytes can differ 10-100× in time purely by pattern. This is why "the disk does 500 MB/s" is meaningless without "doing what" — 500 MB/s sequential and 5 MB/s random can be the same drive.
04 Reading iostat correctly
The core tool. Run iostat -xz 1 (the -x for extended, 1 for per-second, and ignore the first sample — it's a since-boot average). The columns that matter:
| Column | Means |
|---|---|
r/s, w/s | read & write IOPS |
rkB/s, wkB/s | read & write throughput |
r_await, w_await | latency per I/O (ms) — the key number |
aqu-sz | average queue length — saturation |
%util | fraction of time the device had I/O in flight |
Read it as: await tells you if I/O is slow, aqu-sz tells you if work is piling up, r/s+w/s and the kB rates tell you the load shape. That trio — latency, queue, load — diagnoses a disk far better than the single number everyone fixates on.
05 The %util trap
On an old single HDD, %util near 100% meant "saturated" because it could do one I/O at a time. Modern devices — SSDs, RAID arrays, cloud volumes — serve many I/Os in parallel, so %util only means "at least one I/O was in flight," which can be true at a small fraction of real capacity. A cloud SSD can sit at 100% %util while handling a tenth of its IOPS limit, or be nearly full at 60%. Treat %util as "is the device doing anything," not "how full is it."
%util = 100% does not mean a modern disk is saturated. Because SSDs and virtual volumes process I/O concurrently, 100% just means the queue was never empty — it could be at 5% or 95% of true throughput capacity. Engineers raise alarms on "disk 100% utilized" that are completely false. Judge saturation by rising await (latency climbing under load) and growing aqu-sz (queue building), not by %util. On these devices, %util is the most misleading column in the table.06 Queueing and saturation
When I/O arrives faster than the device drains it, requests queue — and queue time adds directly to latency. aqu-sz (average queue size) is your saturation signal: a consistently high queue means demand exceeds the device's ability to keep up, and await will climb with it. This is the queueing curve from Chapter 2 made concrete: as you push a disk toward its limit, latency stays flat, then rises sharply as the queue grows. The fix is fewer/larger I/Os, a faster device, more parallelism (more disks), or caching to avoid the I/O entirely.
Fig 1 — Push a disk past the knee and queueing makes latency rise non-linearly. aqu-sz and await climbing together = saturation.
07 The I/O scheduler and merging
Between the block layer and the device sits an I/O scheduler that orders and merges requests. On HDDs, ordering reduces seeks; merging adjacent I/Os into one larger I/O cuts overhead. Linux options include mq-deadline, bfq, and none (often best for fast NVMe SSDs, where reordering just adds latency). The scheduler also enforces some fairness between processes. For most modern NVMe, none is the right choice; for HDDs or mixed latency-sensitive workloads, deadline/bfq earn their keep. It's a knob worth knowing exists when a fast device underperforms.
08 Latency outliers and a workflow
Averages hide the disk's worst behavior — and storage is prone to nasty outliers (an SSD pausing for garbage collection, a HDD retrying a weak sector, a cloud volume throttling). A p99.9 disk latency 50× the average is common and can dominate tail request latency. So: (1) iostat -xz 1 — read await, aqu-sz, the load shape (ignore %util's headline). (2) Use BPF biolatency for a true latency histogram, not just an average — that's where outliers show. (3) biosnoop to catch individual slow I/Os with their process and block. (4) Check whether the I/O is even necessary (cache misses from Chapter 8) before blaming the device.
await) and how long the on-ramp queue is (aqu-sz). A road can have a car on it 100% of the time and still be moving freely, or be jammed solid — you only know by measuring travel time and queue length, not occupancy.common catches & gotchas
- Alarming on %util=100% — On SSDs/RAID/cloud volumes that serve I/O in parallel, 100% just means "never idle," not "full." Judge by await and aqu-sz.
- Reading iostat's first line — It's a since-boot average. Always take an interval and ignore sample one.
- Ignoring random vs sequential — The same byte count can be 100× slower random than sequential, especially on HDD. "500 MB/s" is meaningless without the pattern.
- Trusting average latency — Storage outliers (SSD GC, throttling, retries) make p99.9 dominate tail latency. Use a histogram (
biolatency). - Wrong I/O scheduler — Reordering on fast NVMe adds latency;
noneis often best there, deadline/bfq for HDDs. - Blaming the disk for cache misses — If the file system's cache is too small (Chapter 8), the fix is more RAM/better hit rate, not a faster disk.
09 Questions engineers actually ask
My disk shows 100% util — is it maxed out?
Probably not. On SSDs, RAID, and cloud volumes that handle many I/Os concurrently, %util only means at least one I/O was in flight — it can read 100% well below capacity. Check await (is latency rising?) and aqu-sz (is the queue growing?) to judge real saturation.
What's the difference between IOPS and throughput?
IOPS counts operations per second (matters for small random I/O like databases); throughput measures MB/s (matters for large sequential transfers). A disk can be IOPS-bound on tiny ops or bandwidth-bound on big ones — they're different ceilings, and you can hit either.
Why is random I/O so much slower than sequential?
On HDDs, each random access needs a mechanical seek and rotation (~ms); sequential streams adjacent blocks with none. On SSDs the gap is smaller but real (large sequential still beats small random). Always characterize the pattern — it can change effective speed by 10-100×.
How do I see the worst disk latency, not the average?
Use a latency histogram — BPF biolatency shows the full distribution including the tail. biosnoop traces individual slow I/Os with the responsible process and block. Averages hide the outliers (GC pauses, throttling) that often dominate tail request latency.
Which I/O scheduler should I use?
For fast NVMe SSDs, none is usually best — reordering just adds latency. For HDDs or latency-sensitive mixed workloads, mq-deadline or bfq reduce seeks and provide fairness. Test with your workload; the default isn't always optimal for fast storage.
10 Key takeaways
- Disks have IOPS, throughput, and latency — latency is what users feel; the others explain the ceiling.
- Random I/O is far slower than sequential, brutally so on HDDs — always characterize the pattern.
- Read
iostat -xz 1byawait(latency),aqu-sz(queue/saturation), and the load shape — skip the first sample. - %util is a trap on parallel devices (SSD/RAID/cloud): 100% ≠ full. Judge by latency and queue.
- Queueing makes latency explode past the knee;
aqu-szrising withawaitis saturation. - Storage outliers dominate tail latency — use a histogram (
biolatency), not the average.
the core view
r_await/w_awaitPer-I/O latency (ms) — the key number.aqu-szQueue length = saturation.r/s w/s rkB/s wkB/sIOPS & throughput — load shape.%util"In use," not "full," on SSD/cloud.latency distribution (outliers)
who & what
device info & tuning
11 Wrapping up
Disks reward reading the right columns: latency and queue over the seductive but misleading %util, the access pattern over raw byte counts, and a histogram over an average that buries the outliers. With CPU, memory, file systems, and disks covered, one major resource remains — the one that connects machines together and adds a whole new dimension of latency. Next: the Network.