// the one-minute version
To read performance metrics you must know what they describe. The kernel manages every resource; apps reach it through system calls that flip the CPU from user mode to kernel mode. The scheduler decides which thread runs when (and creates run-queue latency); virtual memory gives each process its own address space backed by paging; interrupts let devices and the timer seize the CPU. The kernel exposes its state through /proc, /sys, and tracepoints — that's where almost every tool's numbers come from.
A counter like "context switches: 40,000/s" means nothing until you know what a context switch is and what causes one. This chapter is the systems-programming primer that turns cryptic metrics into a story you can follow. You don't need to write kernel code — you need a working model of the kernel's main jobs so that when a tool reports run-queue latency, major faults, or syscall counts, you know exactly what part of the machine is talking to you.
01 The kernel: the program that runs everything
The kernel is the core of the OS — always resident, fully privileged, and in charge of sharing the CPU, memory, storage, and network among every process. It schedules threads, maps memory, drives devices, implements file systems and the TCP/IP stack, and enforces protection. Everything else — your shell, your database, your browser — is a user program that asks the kernel for resources. Most performance metrics are kernel bookkeeping: it counts the work it does on your behalf.
02 User mode, kernel mode, and system calls
The CPU runs in two privilege levels. User mode is where apps live: restricted, can't touch hardware directly. Kernel mode is fully privileged. When an app needs the kernel — to read a file, send a packet, allocate memory — it makes a system call: a controlled doorway that switches to kernel mode, runs vetted code, and returns. Crossing that boundary isn't free; a syscall costs hundreds of nanoseconds to low microseconds, and apps that make millions of tiny syscalls (one byte at a time) pay dearly.
%sys usually means heavy syscall or I/O activity — lots of boundary crossings — and is a strong hint to look at what the app is asking the kernel to do (often: too many small reads/writes, fixable with buffering).03 Processes and threads
A process is a running program with its own address space, file descriptors, and identity (PID). A thread is a unit of execution within a process; one process can have many threads sharing its memory. The kernel actually schedules threads (tasks), not processes. This matters for metrics: "the process uses 100% CPU" might mean one thread pinned on one core, or eight threads each at 12% — wildly different situations that the per-process number alone can't distinguish. Always be ready to drop to per-thread detail.
04 The scheduler and run-queue latency
More runnable threads than CPUs is the normal state. The scheduler (on Linux, historically CFS, now EEVDF) decides which runnable thread gets a CPU and for how long, aiming for fairness and responsiveness. When a thread is ready to run but no CPU is free, it waits on a run queue — and that wait is run-queue latency, pure scheduler-induced delay before your code even starts. A context switch is the act of swapping one thread off a CPU and another on; it has real cost (saving state, cache and TLB disruption), so very high switch rates are themselves a performance signal.
Fig 1 — A thread cycles between running, waiting for a CPU (run-queue latency), and sleeping on I/O or locks (off-CPU). Slowness often lives in the waits, not the running.
05 Virtual memory and paging
Each process sees its own large, private virtual address space; the kernel maps virtual pages to physical RAM through page tables, with the CPU's MMU and TLB doing translation in hardware. Memory is handed out lazily: a malloc reserves virtual space, but physical pages are only attached on first touch — a page fault. A minor fault just wires up a page already in memory (cheap); a major fault must read from disk/swap (expensive, milliseconds). This is why a process's virtual size (VSZ) can dwarf its resident size (RSS) — virtual is what it reserved, resident is what's actually in RAM.
06 Interrupts and the clock
Devices signal the kernel with interrupts — "disk read done," "packet arrived" — so the CPU never has to poll and wait. The kernel's interrupt handler runs briefly, then often defers the heavier work to a softirq/bottom-half. A special timer interrupt (or modern tickless equivalents) gives the kernel a heartbeat for scheduling and timekeeping. High interrupt rates show up as system CPU time and can themselves be a bottleneck on busy network boxes — which is why techniques like interrupt coalescing and NAPI exist.
07 The I/O stack
A single read() travels a long way: through the syscall layer, the Virtual File System (VFS) abstraction, the specific file system (ext4, XFS, ZFS), the page cache (which may satisfy it instantly from RAM), the block layer with its I/O scheduler and queues, the device driver, and finally the disk. Latency can be added at any stage, and crucially the page cache means most "file reads" never touch a disk at all. This layering is why file-system latency and disk latency are different metrics (Chapters 8 and 9) — and why you should usually measure at the file-system layer, closer to the application's truth.
iostat) see only what reaches the device — they're blind to time spent waiting in the kernel's own queues, locks, or the file-system layer above the block device. An app can experience 50 ms of "file read" latency while the disk reports 2 ms, because 48 ms was spent queued or blocked higher up. Measure as close to the application as you can, or you'll exonerate the wrong layer.08 Where the numbers come from
Nearly every tool reads kernel-exposed interfaces. /proc is a virtual filesystem of per-process and system stats (/proc/stat, /proc/PID/status, /proc/meminfo) — this is what top, ps, and vmstat parse. /sys exposes device and kernel-object details. Tracepoints, kprobes, and perf events are dynamic instrumentation points the kernel offers for tracing and profiling (the foundation for perf, ftrace, and BPF in later chapters). Knowing the source helps you trust — or distrust — a number and find a deeper one when the surface tool is too coarse.
common catches & gotchas
- Per-process vs per-thread — "100% CPU" hides whether it's one hot thread or many warm ones. Drop to per-thread (
top -H,pidstat -t) before concluding. - VSZ ≠ RSS — Virtual size is reserved address space; resident size is real RAM. Don't alarm on a big VSZ.
- High %sys isn't automatically bad — But it's a flag: heavy syscall/IO/interrupt work. Find which syscalls (
strace -c, BPF) before judging. - Minor vs major faults — Minor faults are cheap and constant; major faults hit disk and hurt. Watch the major-fault rate, not the total.
- Run-queue latency is invisible in %CPU — A thread waiting for a CPU isn't "using" CPU, yet it's delayed. You need scheduler-latency tools to see it.
- iostat can't see kernel-queue delay — Disk tools miss latency added above the block device. Prefer file-system-level measurement for application truth.
09 Questions engineers actually ask
What's the difference between user time and system time?
User time (%usr) is the CPU spent running your application code. System time (%sys) is the CPU spent in the kernel doing work your app requested via syscalls — I/O, memory management, networking. High system time points at the kernel boundary; high user time points at your own code.
Why are there so many context switches?
Threads switch when they block (I/O, locks), when their time slice expires, or when a higher-priority thread wakes. Lots of small I/O or heavy lock contention produces high switch rates. Each switch costs CPU and disturbs caches, so the rate is itself a signal worth investigating.
What is a page fault — is it an error?
No, it's normal. A page fault is the kernel attaching physical memory to a virtual page on demand. Minor faults are cheap (page already in RAM); major faults read from disk/swap and are slow. Only a high major-fault rate signals a memory problem.
Do I need to know which scheduler Linux uses?
Not the internals, but know that the scheduler can introduce run-queue latency when threads outnumber CPUs, and that scheduler policy/priority (nice, cgroups, CPU limits) affects who runs. That's enough to interpret run-queue and latency metrics.
What is /proc and why do tools read it?
/proc is a virtual filesystem the kernel populates with live statistics about processes and the system. It costs nothing on disk — reading a file there asks the kernel for current numbers. Most classic tools are just friendly front-ends over /proc.
10 Key takeaways
- The kernel manages all resources; its bookkeeping is most of your metrics.
- System calls cross from user to kernel mode at real cost — hence the
%usrvs%syssplit. - The kernel schedules threads; per-process CPU can hide one hot thread — drop to per-thread.
- The scheduler adds run-queue latency when runnable threads outnumber CPUs.
- Virtual memory is lazy: VSZ is reserved, RSS is real; major faults are the costly ones.
- The I/O stack is deep and the page cache intercepts most reads — measure near the app.
- Numbers come from
/proc,/sys, tracepoints — know the source to find a deeper one.
processes & threads
CPU split & scheduling
cs, rContext-switch rate; run-queue length.memory & faults
syscalls & interrupts
kernel state sources
11 Wrapping up
You now have the model behind the metrics: kernel, syscalls and the mode boundary, thread scheduling and run-queue latency, lazy virtual memory and faults, interrupts, and the deep I/O stack with its page cache. Crucially, you know where the numbers come from. With that grounding, the next step is the toolbox itself — the families of observability tools and how to choose the right one for a question.