// the one-minute version
For memory, BPF traces the events behind the counters. Page faults (where a process grows RSS) are traced by stack, revealing which code is consuming memory. memleak tracks allocations not freed and reports the leaking stack. oomkill catches OOM-killer events with context. Swapping and reclaim — swapin, vmscan, drsnoop — show memory pressure as it bites. Growth tools brkstack/mmapsnoop show heap/mmap expansion by stack. Use page-fault and memleak stacks to find what grows; reclaim/swap tools to see pressure; oomkill for the crash.
Counters tell you memory is filling up and maybe swapping; they don't tell you which code is responsible. BPF does — by tracing the allocation and fault events and capturing the stack that caused each one. That turns "RSS keeps climbing" into "this exact call path allocates and never frees," and "the box is swapping" into "these pages, from this process, at this rate." This chapter is the BPF memory toolkit: finding the code behind the growth, the leak, and the pressure.
01 Where memory analysis with BPF fits
Recall the counter view (from Systems Performance): watch available not free, RSS not VSZ, and the major-fault and swap rates. BPF picks up where those leave off. When a counter says "RSS is growing" or "swap-in rate is high," BPF answers the next question — which code, which allocations, which pages. It does this by tracing the kernel events behind memory: page faults, allocator calls, reclaim, OOM. The everyday tools find the problem; BPF attributes it to a stack you can fix.
02 Page faults: where RSS grows
A process's resident memory grows when it page-faults in new pages (the kernel attaching physical memory on first touch). By tracing the page-fault tracepoint and capturing the stack, you see exactly which code paths are growing the process — far more useful than a total RSS number. A bpftrace one-liner on tracepoint:exceptions:page_fault_user aggregating by ustack produces a "memory growth flame graph": width = pages faulted in per stack. When RSS climbs, this shows you the responsible code without guessing.
03 Hunting leaks: memleak
The dedicated leak finder. memleak traces allocation calls (malloc/free, or kernel kmalloc/kfree) and tracks which allocations are never freed, then periodically reports the stacks with outstanding allocations — ranked by bytes. The call path holding the most un-freed memory is your leak suspect. It works on user-space allocators (with uprobes) and kernel allocations. Unlike heavyweight tools, memleak runs on a live process with manageable overhead, so you can attach it to a production service that's slowly bloating and get the leaking stack directly.
memleak distinguishes a true leak from legitimate growth only by what's still outstanding over time — but a cache filling to its configured size also shows large outstanding allocations that are not a leak. Watch the trend: a real leak's outstanding bytes keep climbing and never plateau; a cache levels off. Also, tracing every allocation has overhead proportional to allocation rate — sample or target a specific size/stack on very hot allocators rather than tracing all of them.04 OOM kills: oomkill
When memory is exhausted, the kernel's OOM killer terminates a process — and from the app's side it's a sudden, unexplained death (exit 137 / SIGKILL). oomkill traces the OOM-killer event and prints when it fired, which process was killed, and context (the triggering process, memory state). This catches the moment that's otherwise only a terse dmesg line, and in containers it surfaces cgroup-limit OOMs that kill inside a container while the host has memory to spare. When a service mysteriously restarts, oomkill running in the background tells you immediately if the OOM killer did it.
05 Swapping and reclaim
Under memory pressure the kernel reclaims pages, and if anonymous memory must go, it swaps. BPF traces each stage. swapin shows processes faulting pages back from swap — the slow major faults that make a low-memory box crawl, attributed by process. vmscan traces the kernel's page-reclaim scanner, showing reclaim activity and latency. drsnoop traces direct reclaim — when an allocation must itself stop and reclaim memory before proceeding, adding latency right in the allocation path. Together they reveal memory pressure as latency the application actually pays, not just a swap counter.
Fig 1 — Match the BPF tool to the memory question: growth, leak, kill, pressure, or allocation source.
06 Heap and mmap growth
Two more attribution angles. brkstack traces brk() calls (heap expansion) with stacks — showing which code grows the heap. mmapsnoop traces mmap() calls — large memory regions a process maps in, which can be a major RSS contributor (mmapped files, big buffers). These complement page-fault tracing: page faults show pages actually touched, while brk/mmap show the reservations that precede them. For a process whose memory is climbing, combining "what's reserved" (brk/mmap) with "what's resident" (page faults) gives the full story of how it grows.
07 A caution on overhead
Memory events can be extremely high-frequency — a busy allocator does millions of mallocs per second. Tracing every allocation with uprobes adds overhead proportional to that rate, and can meaningfully slow a hot path. The defenses are the usual ones: aggregate in-kernel (count/histogram by stack, don't print per-event), filter to a specific process or allocation size, and prefer sampling where the tool supports it. Page-fault tracing is generally lighter than allocation tracing because faults are far less frequent than allocations. Know which you're tracing and at what rate before pointing it at production.
memleak, malloc uprobes) has overhead proportional to allocation rate — and some applications allocate millions of times per second. On such a hot allocator, naive full-allocation tracing can slow the application enough to distort the very behavior you're measuring (and annoy production). Page-fault and OOM tracing are far cheaper because those events are rarer. Before tracing all allocations, estimate the rate; if it's high, sample, filter to a size/stack, or trace page faults instead — they attribute RSS growth without touching every malloc.08 A memory analysis workflow
(1) Confirm with counters: is available memory low, RSS climbing, or swap-in rate high? (2) For growth, trace page faults by stack (or brkstack/mmapsnoop for reservations) to find the responsible code. (3) For a suspected leak, run memleak and watch which stack's outstanding bytes never plateau. (4) For pressure, use swapin/vmscan/drsnoop to see reclaim and swap latency hitting the app. (5) For mysterious restarts, keep oomkill running to catch OOM/cgroup kills. (6) Mind allocation-tracing overhead — sample or filter on hot allocators. Counters find it; BPF stacks name it.
common catches & gotchas
- Allocation-tracing overhead — Tracing every
malloccosts in proportion to allocation rate. Sample/filter on hot allocators, or trace page faults instead. - Leak vs cache — Large outstanding allocations may be a cache at its limit, not a leak. Watch the trend: a leak never plateaus.
- VSZ vs RSS again —
brk/mmapshow reservations (virtual); page faults show resident (real RAM). Don't confuse reserving with using. - Missing the OOM — A service that "just restarts" may be OOM-killed (exit 137). Run
oomkillin the background to catch it; check cgroup limits in containers. - Broken allocation stacks — Without frame pointers/symbols, leak stacks are
[unknown]. Build with frame pointers; install debuginfo. - Tracing swap as the cause — Swap activity is a symptom of the working set exceeding RAM; the fix is usually finding the growth/leak, not tuning swap.
09 Questions engineers actually ask
How do I find which code is using all the memory?
Trace page faults by stack (e.g. bpftrace on page_fault_user aggregating ustack) — since RSS grows when pages fault in, the fault stacks show exactly which code paths consume memory. Render as a flame graph and read the widest tower. For reservations, add brkstack/mmapsnoop.
How does memleak find a leak?
It traces allocations and frees, tracks which allocations remain outstanding, and periodically reports the stacks holding the most un-freed bytes. The stack whose outstanding bytes keep climbing (and never plateau) is the leak. It runs on live user-space or kernel allocators.
My container keeps dying with exit 137 — how do I confirm OOM?
Run oomkill to catch OOM-killer events with the killed process and context, and check the cgroup's memory.events for oom_kill. Exit 137 is SIGKILL, and a cgroup memory-limit OOM is the usual cause — the host can have free RAM while the container hits its limit.
Won't tracing every allocation slow my app?
It can, on hot allocators — overhead scales with allocation rate, which can be millions per second. Sample, filter to a process or allocation size, or trace page faults instead (far less frequent than allocations) to attribute RSS growth without touching every malloc.
What's the difference between swapin, vmscan, and drsnoop?
swapin shows pages being faulted back from swap (slow major faults). vmscan traces the kernel's reclaim scanner activity. drsnoop traces direct reclaim — when an allocation must stop and reclaim memory itself, adding latency in the allocation path. Together they show memory pressure as application-visible latency.
10 Key takeaways
- BPF attributes memory behavior to code — counters find the problem, BPF stacks name it.
- Page faults by stack show which code grows RSS — a memory-growth flame graph.
memleakreports stacks with outstanding allocations — the never-plateauing one is the leak.oomkillcatches OOM/cgroup kills (exit 137) that otherwise look like mystery restarts.swapin/vmscan/drsnoopreveal reclaim and swap as latency the app pays.brkstack/mmapsnoopattribute heap/mmap reservations (vs page faults' resident growth).- Allocation tracing overhead scales with rate — sample/filter on hot allocators, or trace faults instead.
what's growing (attribution)
leaks
OOM
oom_kill count for container limits.pressure & swap
overhead rule
11 Wrapping up
For memory, BPF answers "which code?" — page faults and brk/mmap for growth, memleak for leaks, oomkill for the crash, and the reclaim/swap tools for pressure felt as latency — while you mind the allocation-rate overhead. Counters find it; BPF stacks name it. Next we follow the data down to storage, starting with the file-system layer. Next: File Systems.