// the one-minute version
At the block layer, BPF shows what iostat can't: the full latency distribution and individual slow I/Os. biolatency prints a histogram of block-I/O latency — revealing the tail (outliers) an average hides. biosnoop traces each I/O with process, block, size, and latency — catching specific slow operations. biotop ranks processes by disk I/O. bitesize/biopattern show I/O sizes and random-vs-sequential. biostacks attributes latency to the code path. Use the histogram for the tail, biosnoop for the culprits.
The disk is the slowest thing most requests touch, and storage is prone to nasty outliers — an SSD pausing for garbage collection, a cloud volume throttling, a drive retrying a weak sector. Averages bury exactly those outliers, which is why iostat's mean latency can look fine while 1% of I/O takes 50×. BPF traces the block layer directly, giving you the distribution, the individual slow I/Os, and the code behind them. This chapter is the disk-I/O toolkit — and it pairs with the file-system layer above (Chapter 8).
01 The block layer and what BPF adds
Below the file system sits the block layer, which queues and dispatches I/O to devices. iostat reads its summary counters — average latency, IOPS, queue. BPF instead traces the block-I/O events (issue and completion) directly, so it can compute the full latency distribution, time each individual I/O, and capture the process and stack behind it. The difference is distribution vs average and per-I/O vs aggregate: where iostat says "average 5 ms," BPF shows the histogram with a 50 ms tail and names which process and code path caused it.
biolatency shows the full distribution; biosnoop shows the actual slow I/Os. When a disk "looks fine on average" but something is slow, get the histogram — the average is lying by omission.02 biolatency: the latency histogram
The flagship. biolatency traces block-I/O issue and completion and prints a histogram of the latency between them — power-of-two buckets with counts. You instantly see the shape: a tight cluster around the median, plus any tail in the higher buckets. A bimodal distribution (two humps) often reveals two storage behaviors — say, cache hits vs misses, or normal I/O vs a throttled path. Options let you split by read/write (-F), by device, or by I/O flags. This is the first tool to run for any "disk latency" question, because it shows the distribution counters can't.
Fig 1 — A latency histogram shows the fast cluster and the slow tail. The average sits between them, describing neither.
03 biosnoop: per-I/O detail
When the histogram shows a tail, biosnoop finds the culprits. It prints one line per block I/O: timestamp, process, PID, device, sector, size, and latency. Scan for the high-latency lines and you see exactly which process, which block, and how slow — the individual outliers behind the histogram's tail. It's the drill-down from "1% of I/O is slow" to "this process's I/O to this region took 380 ms at this moment." Being per-event, it costs more on busy systems, so filter and time-box — but for catching specific slow I/Os it's unmatched.
04 biotop: who's doing the I/O
biotop is top for disk I/O — a refreshing table of processes ranked by block I/O, showing reads/writes, bytes, and average latency per process. When the disk is busy and you need to know who, this answers it directly, attributing device I/O to processes (something traditional tools struggle with once the page cache and write-back blur the connection between a process and its physical I/O). Start here for "the disk is hammered — by what?", then drop to biosnoop or biolatency for the latency detail.
05 I/O size and pattern: bitesize, biopattern
The access pattern drives disk performance (random is far slower than sequential, especially on HDDs). bitesize shows a histogram of I/O sizes per process — lots of tiny I/Os may mean an app that should be batching, while large I/Os stream efficiently. biopattern (a bpftrace tool) reports the ratio of random vs sequential I/O by watching whether successive block offsets are contiguous. Knowing you're doing 90% random small I/O explains poor throughput that raw bandwidth numbers don't — and points to fixes (larger I/Os, better locality, caching) rather than just "buy a faster disk."
06 Latency by code path: biostacks
Sometimes you need to know why an I/O was issued. biostacks ties block-I/O latency to the kernel stack that initiated it — so you see whether slow I/O came from read-ahead, a sync, a journal write, or a specific code path. This is valuable when the same device shows mixed latency: the stacks separate the fast path from the slow one. Combined with the file-system tools (Chapter 8), you can follow an I/O from the application's read() down through the cache miss to the block-layer latency and the stack that drove it — full-stack disk drill-down.
biolatency measures from I/O issue to completion at the device — but the application may also wait before the I/O is issued: queued in the block layer, blocked on a lock, or stalled in write-back above the block device. So a perfect-looking biolatency (all I/O fast at the device) can coexist with slow application reads, because the time was spent getting to the device, not at it. Measure file-system latency (Chapter 8) for the app's full wait, and use block latency for the device's contribution — they answer different questions.07 Tracepoints and one-liners
The block layer has stable tracepoints — block:block_rq_issue and block:block_rq_complete — which the tools use and you can trace directly in bpftrace. The per-thread (or per-request) timestamp pattern on these gives a custom latency histogram; aggregating by device, by I/O size, or by process flag answers bespoke questions. Because these are tracepoints (not kprobes), the one-liners are stable across kernels — a reason to prefer them. For device-specific latency, lower-level tools like nvmelatency or scsilatency trace the driver layer when you need to separate device time from block-layer time.
08 A disk-I/O analysis workflow
(1) Start with biolatency for the distribution — is there a tail, is it bimodal? (2) If there's a tail, biosnoop to catch the individual slow I/Os with process and block. (3) biotop to see who is generating the I/O. (4) bitesize/biopattern to understand the size and pattern (random small I/O explains poor throughput). (5) biostacks to attribute latency to the code path. (6) Remember to also check file-system latency (Chapter 8) for the app's full wait — block latency is only the device's part. Histogram first, culprits second, pattern and path third.
common catches & gotchas
- Average over histogram — Disk outliers dominate tail latency and the mean hides them. Always start with
biolatency's distribution. - Block latency ≠ app wait —
biolatencymeasures device time; the app may also wait queued/blocked above the device. Pair with file-system latency. - biosnoop floods on busy disks — It's per-event. Filter by device/process and time-box on high-IOPS systems.
- Ignoring the pattern — Poor throughput at low utilization is often random small I/O. Check
bitesize/biopatternbefore blaming the device. - Confusing who-does-I/O — The page cache and write-back blur process→I/O attribution; use
biotop/biosnoopwhich trace the actual block events. - Forgetting device-level tools — For "is it the device or the block layer,"
nvmelatency/scsilatencyseparate driver/device time from block-layer time.
09 Questions engineers actually ask
iostat shows fine average latency but something's slow — what do I do?
Run biolatency for the full distribution — the average hides outliers. You'll likely see a tail (or a second hump) of slow I/Os that the mean smeared away. Then use biosnoop to catch the individual slow I/Os with the responsible process and block.
How do I find which process is hammering the disk?
Run biotop — it's top for block I/O, ranking processes by reads/writes, bytes, and latency. It traces the actual block events, so it attributes I/O correctly even when the page cache and write-back blur the process→I/O link that traditional tools rely on.
My throughput is low but the disk isn't saturated — why?
Often random small I/O. Check bitesize (I/O size histogram) and biopattern (random vs sequential ratio). Lots of tiny random I/Os give poor throughput regardless of the device's bandwidth — the fix is larger I/Os, better locality, or caching, not a faster disk.
Does biolatency show what my application waits for?
Only the device portion — issue to completion at the block device. The app may also wait queued in the block layer, blocked on a lock, or in write-back above it. Pair biolatency with file-system latency tools (Chapter 8) to capture the application's full wait.
How do I know why a slow I/O was issued?
Run biostacks, which ties block-I/O latency to the kernel stack that initiated it — read-ahead, fsync, journaling, or a specific path. On a device with mixed latency, the stacks separate the fast path from the slow one, telling you not just how slow but why.
10 Key takeaways
- BPF traces the block layer for the distribution and individual I/Os that
iostat's averages hide. biolatencyshows the latency histogram — the tail and bimodality that explain timeouts.biosnoopcatches individual slow I/Os with process, block, size, and latency.biotopranks processes by disk I/O;bitesize/biopatternreveal size and random-vs-sequential.biostacksattributes latency to the code path (read-ahead, fsync, journal).- Block latency is only the device's part — pair with file-system latency for the app's full wait.
- Prefer the stable block tracepoints for custom one-liners.
distribution & outliers
who & what
why (code path / device)
custom (stable tracepoints)
remember
11 Wrapping up
For disk I/O, BPF gives you the histogram biolatency, the culprits biosnoop, the who biotop, and the pattern and path — everything the averaged counters omit, with the reminder that device latency is only part of the app's wait. With storage covered top to bottom, the trail leads off the machine entirely. Next: Networking.