// the one-minute version
Measure performance at the file system, not the disk — it's closer to what the application actually experiences. Most reads are served instantly from the page cache (RAM), so a "disk-heavy" app may barely touch the disk. Writes usually go to cache first and flush asynchronously (write-back), which is fast — until an fsync() forces a synchronous wait for durability, the classic latency cliff. The VFS gives every file system a common interface; journaling and metadata operations add their own costs. Trace file-system latency directly; disk metrics miss the cache and the queueing above it.
An app does a read() and the disk graph stays flat — yet the read was slow. Or the disk is hammered but the app feels fine. Both happen constantly, because between your application and the spinning (or flash) media sits the file system and its enormous RAM cache, which absorbs most I/O and reshapes the rest. To understand storage performance you have to start here, one layer above the disk, where the application's truth lives.
01 Why measure at the file system, not the disk
Disk tools like iostat see only I/O that reaches the physical device. But the application's experience is shaped by everything above: the page cache (which may satisfy a read with zero disk I/O), the VFS, locks, write-back timing, and kernel queues. A read can take 50 ms at the application while the disk reports 2 ms — because 48 ms was a cache miss waiting in a queue, a lock, or a write-back stall. File-system latency — measured per operation as the app sees it — is the metric that correlates with user pain. Make it your default.
02 The page cache: most I/O never hits disk
The kernel caches file contents in RAM (the page cache, same memory discussed in Chapter 7). A read that hits the cache returns in microseconds with no disk involvement; a miss goes to the device and is ~1000× slower. This is why an app reading the same hot files repeatedly shows almost no disk I/O — and why a cold cache (after a reboot, or when the working set exceeds RAM) suddenly produces a storm of disk reads and a latency spike. Your effective read performance is mostly a function of cache hit rate.
Fig 1 — Reads check the page cache first. Hits never touch the disk; misses pay the ~1000× penalty. Watch your hit ratio.
03 Write-back: fast writes, deferred cost
By default a write() just copies data into the page cache and returns — the actual disk write happens later, asynchronously, by kernel write-back threads flushing dirty pages. This makes writes feel instant. The deferred cost surfaces two ways: a burst of dirty pages can trigger heavy flushing that competes with reads and stalls the app, and on a crash any unflushed dirty data is lost. Tunables (dirty_ratio, dirty_background_ratio) control how much dirty data accumulates before forced flushing — set too high, you get periodic write storms; too low, you flush constantly.
04 fsync: where durability meets latency
When data must survive a crash — a database commit, a "saved" confirmation — the app calls fsync() (or opens with O_SYNC), forcing a synchronous wait until the data is durably on the device. This bypasses the speed of write-back and exposes the raw disk latency, plus any journaling and cache-flush overhead. fsync is the single most common source of write-latency surprises: an app that writes at memory speed suddenly blocks for milliseconds per commit because each one waits for the platter or flash to confirm.
fsync frequency, and it's invisible in throughput numbers. A database doing one fsync per transaction is bottlenecked by disk commit latency, not bandwidth — batching commits (group commit) or relaxing durability can transform performance. Benchmarks that skip fsync report fantasy numbers; production that fsyncs every write wonders why it's slow. Always know whether the durable path is being exercised.05 The VFS and file-system types
The Virtual File System is a kernel abstraction that gives every file system the same interface (open, read, write), so applications and tools don't care whether they're on ext4, XFS, ZFS, or a network mount. Underneath, file systems differ in performance character: ext4 is the solid general default; XFS excels at large files and parallel I/O; ZFS/btrfs add checksums, snapshots, and copy-on-write at some overhead. The right choice depends on workload — many small files vs few huge ones, read-heavy vs write-heavy, integrity needs vs raw speed.
06 Read-ahead and metadata
Two more behaviors shape latency. Read-ahead (prefetch): when the kernel detects sequential reads it speculatively loads the next blocks, so streaming a large file stays fast — but for random access, read-ahead can waste I/O loading data you never use. Metadata operations — stat, directory listings, file creation/deletion, permission checks — are often overlooked yet can dominate workloads with millions of small files (think a build tree or a mail spool). An app that stats thousands of files per request is doing metadata-bound I/O that no amount of read bandwidth fixes.
07 Journaling and consistency cost
To survive crashes without corruption, most file systems journal: they first write a record of an intended change to a log, then apply it, so a crash mid-update can be replayed or rolled back. This costs extra writes — metadata (and optionally data) is written twice. Modes trade safety for speed: data=ordered (default ext4) journals metadata only; data=journal journals everything (safest, slowest); data=writeback is fastest, least safe. The journal is also why a flurry of small synchronous writes can be surprisingly expensive — each may force journal I/O.
08 A file-system analysis workflow
(1) Measure file-system latency directly with BPF tools (xfsslower, ext4slower, fileslower) — they show slow operations as the app sees them, cache and all. (2) Check the cache hit ratio (cachestat) — low hits mean the working set exceeds RAM and you're disk-bound. (3) Look at dirty pages and write-back (/proc/meminfo Dirty/Writeback) for flush stalls. (4) Profile which operations dominate — is it reads, writes, or metadata (stat storms)? (5) Only then drop to disk metrics (Chapter 9) if the misses are truly hitting the device. File system first, disk second.
common catches & gotchas
- Trusting iostat for app latency — Disk tools miss the page cache and kernel queueing. Measure file-system latency for what the app feels.
- Ignoring fsync — Durable commits expose raw disk latency and journaling cost. The fsync rate, not throughput, often sets write performance.
- Cold-cache surprises — After a reboot or when the working set exceeds RAM, reads suddenly hit disk and latency spikes. Warm caches hide this.
- Metadata blindness — Millions of small
stat/create/delete ops can bottleneck a workload that has trivial read/write bandwidth. - Read-ahead on random I/O — Prefetch helps sequential reads, wastes I/O on random access. Tune or disable for random workloads.
- Benchmarks without fsync — A write benchmark that skips durability reports memory speed, not real write performance. Match production's durability path.
09 Questions engineers actually ask
Why is my app slow when the disk looks idle?
The latency is above the disk — a page-cache miss waiting in a kernel queue, a lock, or a write-back stall. iostat only sees device I/O. Measure file-system operation latency (BPF *slower tools) to see the time the app actually experiences.
What does fsync actually do, and why is it slow?
fsync() forces buffered writes to durable storage and waits for confirmation, so data survives a crash. It's slow because it bypasses fast write-back and exposes raw device commit latency plus journaling overhead. Frequent fsyncs (one per transaction) often dominate write performance.
How do I tell if the page cache is helping?
Check the cache hit ratio (e.g. cachestat from BCC). A high hit rate means most reads are served from RAM and the disk is barely involved. A falling hit rate signals the working set is outgrowing RAM and you're about to become disk-bound.
Which file system should I use?
It depends on the workload. ext4 is a safe general default; XFS shines for large files and parallel I/O; ZFS/btrfs add integrity and snapshots at some cost. Match the file system to your access pattern (file sizes, read/write mix, durability needs) rather than picking by reputation.
My writes are fast in tests but slow in production — why?
Tests probably skip fsync (writing only to cache), while production forces durable commits. The production path pays disk commit and journaling latency per sync. Reproduce by fsyncing in the test, then consider batching commits or tuning durability.
10 Key takeaways
- Measure file-system latency, not disk metrics — it's what the application actually experiences.
- The page cache serves most reads from RAM; your read latency tracks the hit ratio, not disk speed.
- Writes go to cache and flush via write-back; bursts of dirty pages cause flush stalls and crashes lose unflushed data.
- fsync forces durable, synchronous writes — exposing raw disk + journaling latency, the classic write cliff.
- The VFS unifies file systems; ext4/XFS/ZFS differ in performance character — match to workload.
- Read-ahead helps sequential I/O, hurts random; metadata ops can bottleneck small-file workloads.
- Journaling trades extra writes for crash safety; trace the FS path before blaming the disk.
file-system latency (app's truth)
cache effectiveness
activity & ops
tuning knobs
11 Wrapping up
The file system is where storage performance becomes real for the application: the page cache absorbs most reads, write-back hides most writes, and fsync is the cliff where durability meets latency. Measure here first. When the cache truly misses and operations reach the device, you've arrived at the physical layer — the disks themselves, and the art of reading iostat correctly.