// the one-minute version
BPF measures file-system performance as the application experiences it — above the disk, including the page cache. VFS-level latency tools (fileslower, vfsstat) and per-filesystem tools (ext4slower, xfsslower) show slow read/write/open/fsync operations with the responsible process. cachestat/cachetop reveal the page-cache hit ratio — your real read performance. opensnoop/statsnoop catch metadata storms; filetop ranks files by I/O. Trace the file-system layer first; the disk (next chapter) is below it.
An app does a slow read and the disk graph stays flat — because the file system and its huge RAM cache sit between them, absorbing most I/O and reshaping the rest. To diagnose storage performance you start here, one layer above the disk, where the application's truth lives. BPF is ideal for it: it can time each file-system operation as the app sees it, expose the cache hit ratio, and attribute slow ops to exact processes and files. This chapter is that file-system toolkit.
01 Why trace at the file system
Disk tools (iostat) see only I/O that reaches the device — blind to the page cache, the VFS, locks, and kernel queues above it. But those upper layers shape what the application waits for: a read can take 50 ms at the app while the disk reports 2 ms. File-system latency, measured per operation, is the metric that correlates with user pain. BPF measures it directly by tracing VFS and filesystem functions, capturing the operation's duration and the process behind it — exactly the application-level truth disk metrics miss.
fileslower) gives you the number the user actually experiences — which the disk's own metrics can't, because they see only what reaches the device.02 VFS-level latency: fileslower and vfsstat
The VFS (Virtual File System) is the common layer every file system sits behind, so tracing it works regardless of ext4/XFS/ZFS. fileslower shows read/write operations slower than a threshold (e.g. 10 ms) with the process, size, and latency — instantly surfacing the slow file I/O an application is suffering, cache and all. vfsstat summarizes the rate of VFS operations (reads, writes, opens, fsyncs) per second — a quick pulse of file-system activity. Start at the VFS for a filesystem-agnostic view, then drop to the specific filesystem if needed.
03 Per-filesystem latency: ext4slower and friends
For more precision, per-filesystem tools trace that filesystem's own operations. ext4slower (and xfsslower, btrfsslower, zfsslower) show reads, writes, opens, and fsyncs slower than a threshold, attributed to process and file. The inclusion of fsync matters: a database blocked on durable commits shows up here as slow sync operations, pinpointing the durability path as the bottleneck. These tools are the file-system equivalent of biolatency — but measured above the cache, so they capture the latency the application actually pays, not just disk time.
04 The page cache: cachestat and cachetop
Your effective read performance is mostly the page-cache hit ratio — hits return in microseconds from RAM, misses pay the ~1000× disk penalty. cachestat shows page-cache hits, misses, and the hit ratio per second; a falling ratio means the working set is outgrowing RAM and you're about to become disk-bound. cachetop breaks the cache activity down by process. When reads are slow, checking the hit ratio first tells you whether the problem is a cold/too-small cache (fix with RAM or access patterns) or genuinely slow disk underneath (Chapter 9).
Fig 1 — BPF measures file-system latency and cache hit ratio above the disk — the application's actual experience.
05 Metadata and opens: opensnoop, statsnoop
Workloads with millions of small files can be metadata-bound — dominated by open, stat, and directory operations rather than read/write bandwidth. opensnoop traces every file open with the process and path (great for "what files is this touching, and is it failing to find some?"). statsnoop traces stat calls — a process that stats thousands of files per request is doing metadata I/O no read bandwidth fixes. These tools turn an opaque "the app is slow on the file system" into a concrete list of files and a count of metadata operations.
06 Hot files and other angles
More attribution. filetop is like top for files — it ranks files by read/write bytes, showing which files dominate I/O right now. filelife shows short-lived files (created and deleted quickly — a sign of temp-file churn). dcstat/dcsnoop trace the directory-entry cache (dcache) hit ratio — important for path-lookup-heavy workloads. mountsnoop catches mount activity. Each answers a specific question; the skill (from Chapter 4) is recognizing which tool fits the symptom rather than tracing blindly.
cachestat alongside the latency tools: a great-looking ext4slower with a 99% cache hit ratio is measuring RAM, and tells you little about behavior under a cold or oversized working set.07 One-liners and custom tracing
When the canned tools don't fit, bpftrace traces VFS and filesystem functions directly. Time vfs_read/vfs_write with the per-thread timestamp pattern for a custom latency histogram; aggregate vfs_* calls by type with funccount to spot a metadata storm; trace a specific filesystem function to study one operation. Tracepoints exist for some events; many file-system internals require kprobes (with the version-fragility caveat). The point: the ready tools cover the common cases, and bpftrace covers the rest — you're never stuck if you can name the function.
08 A file-system analysis workflow
(1) Measure operation latency as the app sees it — fileslower (VFS) or ext4slower/xfsslower (per-FS), watching for slow reads/writes/fsyncs. (2) Check the cache hit ratio with cachestat — low hits mean disk-bound; high hits mean you're measuring RAM. (3) Identify the dominant operation — reads, writes, or metadata (opensnoop/statsnoop for stat storms). (4) Find hot files with filetop. (5) Only if misses truly hit the device, descend to disk tools (Chapter 9). File system first, disk second — measure where the app feels it.
common catches & gotchas
- Trusting iostat for app latency — Disk tools miss the cache and kernel queueing. Use file-system latency tools for what the app feels.
- Cache-warm illusion — Latency looks great with a hot cache and collapses when it's cold/oversized. Always check the hit ratio with
cachestat. - Ignoring fsync — Slow durable commits show in
ext4sloweras slow fsyncs; that's often the real write bottleneck, not bandwidth. - Metadata blindness — Millions of
stat/openops can bottleneck a workload with trivial read/write bandwidth. Trace withstatsnoop/opensnoop. - Per-event floods —
opensnoop/fileslowerstream per-event; on busy systems filter (-p PID) and raise the latency threshold. - kprobe fragility — Custom filesystem-internal tracing via kprobes can break on kernel upgrades. Prefer tracepoints/ready tools where possible.
09 Questions engineers actually ask
Why is my app slow on file I/O when the disk looks idle?
The latency is above the disk — a cache miss waiting in a kernel queue, a lock, or a write-back stall — which iostat can't see. Measure file-system operation latency with fileslower or ext4slower to see the time the app actually experiences.
How do I know if the page cache is helping?
Run cachestat for the hit/miss ratio per second (and cachetop per process). A high hit ratio means most reads come from RAM; a falling ratio means the working set is outgrowing memory and you're heading toward disk-bound.
My database writes are slow — how do I find the cause?
Run ext4slower (or your filesystem's variant) and watch for slow fsync operations. Durable commits call fsync, which exposes raw disk + journaling latency. Frequent slow fsyncs point at the durability path, fixable by batching commits or tuning durability.
How do I catch a metadata-heavy workload?
Use statsnoop and opensnoop to trace stat and open calls with process and path, and vfsstat to see operation rates. A process doing thousands of stats per request is metadata-bound — no amount of read bandwidth helps; the fix is fewer metadata operations.
Which files are driving my I/O?
Run filetop — it's top for files, ranking them by read/write bytes in real time. It instantly shows which files dominate I/O, turning "the disk is busy" into "this specific file is being hammered by this process."
10 Key takeaways
- Measure file-system latency as the app sees it — above the disk, including the page cache.
fileslower/vfsstatgive a filesystem-agnostic VFS view;ext4slower/xfssloweradd per-FS precision and fsync.cachestat/cachetopreveal the page-cache hit ratio — your real read performance.opensnoop/statsnoopcatch metadata storms;filetopranks hot files.- A warm cache makes latency look great and hides cold/oversized-working-set behavior — always check the hit ratio.
- bpftrace traces VFS/filesystem functions directly when canned tools don't fit.
- Trace the file system first; the disk is the layer below.
latency (app's truth)
page cache
metadata & opens
hot files & dcache
custom
11 Wrapping up
For file systems, BPF measures the latency and cache behavior the application actually experiences — above the disk, where the truth lives. Lead with fileslower/ext4slower and cachestat, watch fsyncs and metadata, and only then descend. That descent is the next chapter: when cache misses truly reach the device. Next: Disk I/O.