Systems Performance Chapter 7 — Memory Explained: Virtual vs Resident, Paging, Swap, OOM, NUMA

// the one-minute version

Memory metrics confuse people because virtual (reserved address space, VSZ) is not resident (real RAM, RSS), and "used" memory includes the page cache that the kernel will hand back instantly when needed. The performance danger is paging/swapping: when RAM runs low the kernel reclaims pages, and if an app's hot pages get pushed to disk, every access becomes a slow major fault. Worst case, the OOM killer shoots a process. On big servers, NUMA means remote memory is slower than local. Watch the major-fault and swap rates, not just "used."

"We're out of memory!" — usually we aren't. Linux deliberately uses nearly all RAM for cache, so free almost always shows little "free," and that's healthy. The real questions are subtler: is the kernel having to reclaim pages under pressure, are hot pages getting swapped to disk and faulting back, and is any process actually growing without bound? This chapter untangles the memory numbers so you stop fearing the wrong ones and catch the genuinely dangerous ones.

01 Virtual vs physical memory

Each process gets a large private virtual address space. The kernel maps virtual pages to physical RAM only as they're actually touched (demand paging). So a process can reserve gigabytes virtually (VSZ) while using a fraction physically (RSS, resident set size). VSZ growing means it reserved more address space; RSS growing means it's consuming more real RAM. For "is this eating memory?" you almost always want RSS — and even RSS overcounts shared pages, which is where PSS comes in.

key ideaVSZ is a promise; RSS is the bill. A 20 GB VSZ with 2 GB RSS is a process that could use 20 GB but is actually using 2. Alerting on VSZ produces endless false alarms. Track RSS (or PSS for shared-memory-heavy apps) for real consumption, and the per-process major-fault rate for memory pressure.

02 The page cache: why "free" is always low

The kernel caches file data in otherwise-unused RAM — the page cache. This is why a server with 64 GB shows almost none "free": the rest is cache, holding recently-read files so future reads are instant. Crucially this memory is reclaimable — the moment an app needs RAM, the kernel drops clean cache pages and hands it over. So the number that matters isn't "free," it's available (free + reclaimable cache). Reading "low free = problem" is the single most common memory misinterpretation on Linux.

watch outDon't panic at near-zero "free" memory, and never "fix" it by dropping caches in production. A full page cache is the kernel doing its job — using RAM that would otherwise sit idle to speed up I/O. The meaningful figure is available in free -m / /proc/meminfo: that's what apps can actually get. Low available is the warning sign; low free is normal.

03 Anonymous vs file-backed pages

Two kinds of memory behave very differently under pressure. File-backed pages (program code, mmapped files, page cache) have a home on disk, so the kernel can drop a clean copy for free and re-read it later. Anonymous pages (heap, stack — your malloc'd data) have no disk backing, so to reclaim them the kernel must write them to swap. That's the key asymmetry: reclaiming file pages is cheap; reclaiming anonymous pages requires swap I/O, and getting them back is a slow major fault. When anonymous memory exceeds RAM, you swap, and swapping hot data is where performance falls off a cliff.

Fig 1 — File pages reclaim cheaply; anonymous pages must be swapped. Faulting swapped pages back is what makes a low-memory box crawl.

04 Paging, swapping, and thrashing

Terms to keep straight. Paging is moving individual pages between RAM and disk (normal, constant). Swapping historically meant moving whole processes out; on Linux today people use "swapping" loosely for paging anonymous memory to the swap device. The danger sign is the swap-in/swap-out rate (si/so in vmstat): steady swapping of active data means RAM is too small for the working set. Thrashing is the death spiral — so much swapping that the CPU spends its time waiting on disk and almost no real work gets done. A thrashing box is often best rebooted; it won't recover quickly on its own.

05 The OOM killer

When memory is truly exhausted — no free RAM, swap full or absent, nothing left to reclaim — Linux invokes the Out-Of-Memory killer, which picks a process (by a heuristic "badness" score weighted toward big memory users) and kills it to save the system. From the app's side this is a sudden, unexplained death; from dmesg you'll see a clear "Out of memory: Killed process..." line. In containers, hitting the cgroup memory limit triggers a per-cgroup OOM that kills inside the container while the host has memory to spare — another "mystery crash" that's really a limit.

the catchA container that keeps dying with exit code 137 isn't buggy in the way it looks — 137 means SIGKILL, and the usual cause is the cgroup memory limit triggering the OOM killer. The host has plenty of RAM, your app logs show no error (it was killed mid-instruction, no chance to log), and the only evidence is in dmesg / the cgroup's memory.events. Always check OOM and cgroup limits before assuming an application crash bug.

06 NUMA: not all RAM is equal

On multi-socket servers, memory is divided among CPU sockets — each CPU has local memory it reaches fast and remote memory (attached to another socket) it reaches slower over an interconnect. This is NUMA (Non-Uniform Memory Access). A thread running on socket 0 but allocating memory pinned to socket 1 pays a latency penalty on every access. The kernel tries to keep allocations local, but migrations and bad pinning can scatter a process's memory remotely. On big boxes, NUMA locality can be a double-digit-percent performance factor — and it's invisible unless you look with NUMA-aware tools.

07 Leaks vs growth vs working set

Three different "memory keeps rising" stories. A true leak is memory allocated and never freed — RSS climbs forever until OOM. Growth is legitimate: a cache filling to its configured size, then leveling off. A large working set is the app genuinely needing that much active memory. They look identical on a coarse graph; the tell is the shape over time — a leak never plateaus. To find what leaks, profile allocations (BPF allocation tracing, heap profilers, valgrind in dev) and look for the call stack whose allocations are never matched by frees.

think of it likeA sink filling up. A leak is the drain plugged — water rises until it floods (OOM). Growth is filling the sink on purpose to a set level, then the tap stops. A big working set is just needing a full sink to do the dishes. On a snapshot they all look like "lots of water"; only watching over time — does it plateau or keep rising? — tells you which you've got.

08 A memory analysis workflow

(1) free -m — look at available, not free. (2) vmstat 1 — watch si/so for swapping and free/buff/cache trends. (3) dmesg | grep -i oom — any kills? (4) pidstat -r 1 — per-process RSS and major-fault rate; rising RSS + major faults = pressure. (5) for a suspected leak, graph RSS over hours and allocation-profile the climber. (6) on multi-socket, check numastat for remote-access ratios. Cheap broad checks first, deep allocation tracing only on the real suspect.

common catches & gotchas

Reading "free" instead of "available" — Low free is normal (page cache). Available is what apps can get; that's the number to watch.
Alerting on VSZ — Virtual size is reserved space, not consumption. Track RSS/PSS for real RAM use.
Dropping caches to "free" memory — Pointless and harmful in production; the kernel reclaims cache automatically when needed. You just slow the next reads.
Exit 137 = "app bug" — It's SIGKILL, usually the OOM killer or a cgroup memory limit. Check dmesg and memory.events first.
Ignoring swap rate — A little swap used is fine; a steady si/so rate means the working set exceeds RAM and latency is suffering.
Missing NUMA — On multi-socket boxes, remote memory access silently taxes performance. Check locality with NUMA tools.

09 Questions engineers actually ask

My server shows almost no free memory — is that bad?

No, it's normal and healthy. Linux uses spare RAM for the page cache to speed up I/O, and reclaims it instantly when apps need it. Look at available memory (in free -m), not free. Low available — not low free — is the warning.

What's the difference between VSZ and RSS?

VSZ is the total virtual address space the process has reserved; RSS is how much physical RAM it's actually using right now. VSZ can be huge while RSS is small. For "how much memory is this using," use RSS (or PSS to fairly split shared pages).

Should I add swap or run without it?

A modest swap acts as a safety valve, letting the kernel reclaim cold anonymous pages and avoid premature OOM. But swap is no substitute for RAM — if the active working set exceeds memory, swapping hot data thrashes. Size RAM for the working set; keep some swap for cold pages.

How do I find a memory leak?

Confirm it's a leak (RSS rises and never plateaus over hours, unlike a cache that levels off), then allocation-profile the process — BPF allocation tracers, language heap profilers, or valgrind in dev — to find the call stack whose allocations are never freed.

My container crashes with no error log — why?

Very likely the cgroup memory limit triggering an OOM kill (exit 137 / SIGKILL). The process is killed instantly with no chance to log. Check dmesg, the cgroup's memory.events for oom_kill, and raise the limit or reduce usage.

10 Key takeaways

VSZ (reserved) ≠ RSS (real RAM); track RSS/PSS, not virtual size.
Low free is normal — the page cache uses spare RAM and is reclaimable; watch available instead.
File-backed pages reclaim cheaply; anonymous pages need swap, and faulting them back is slow.
Steady swap-in/out means the working set exceeds RAM; runaway swapping is thrashing.
The OOM killer (and cgroup limits → exit 137) kills processes under exhaustion — check dmesg.
NUMA makes remote memory slower; locality matters on multi-socket servers.
Distinguish a leak (never plateaus) from growth or a large working set by watching the shape over time.

// chapter cheatsheetmemory analysis

overview

free -mWatch available, not free.

cat /proc/meminfoFull breakdown: MemAvailable, Cached, SwapFree, Dirty.

vmstat 1 → si/soSwap-in/out rate = real pressure.

per-process

pidstat -r 1RSS + minor/major fault rate per process.

ps -o pid,vsz,rss,comm --sort=-rssTop RSS consumers (ignore VSZ for "usage").

cat /proc/PID/smaps_rollupPSS — fair share of shared memory.

OOM & limits

dmesg | grep -i oom"Out of memory: Killed process..." events.

cat .../cgroup/memory.eventsoom_kill count — container OOMs (exit 137).

leaks & NUMA

graph RSS over hoursNever plateaus → leak; levels off → cache.

memleak (BCC) / heap profilerFind the un-freed allocation stack.

numastat -m · numactl --hardwareRemote vs local memory access.

11 Wrapping up

Memory rewards knowing which numbers to trust: available over free, RSS over VSZ, fault and swap rates over absolute "used." The page cache is your friend, swap is a safety valve that becomes a cliff, and OOM/cgroup limits explain most mystery crashes. The page cache also bridges to the next layer — most file reads are satisfied from it, not the disk. Next: File Systems.

Systems Performance Chapter 7 — Memory.