Systems Performance Chapter 6 — CPUs Explained: Utilization, IPC, Caches, Flame Graphs

// the one-minute version

CPU performance is two questions: are the CPUs busy (utilization), and is work queueing for them (saturation, seen in run-queue length and load average). But "busy" hides quality: a CPU at 100% might be doing useful work or stalling on memory. IPC (instructions per cycle) and the cache hierarchy reveal whether cycles are productive — a low IPC means the CPU is waiting on RAM, not computing. To find what burns CPU, use a flame graph. To find why each cycle is slow, use hardware counters (PMCs).

"CPU is at 90%" is where most people stop and where the interesting questions begin. Ninety percent of what? Doing useful work, or spinning while it waits for memory? On one core or spread evenly? With work queued behind it or not? This chapter goes past the headline percentage into how CPUs actually spend cycles, so you can tell a healthy busy system from a sick one that happens to look the same on a dashboard.

01 CPU fundamentals

A modern chip has multiple cores, and often each core runs two hardware threads (hyperthreads / SMT) that share the core's execution units. The OS sees each hyperthread as a "CPU," but two hyperthreads on one core are not two full cores — they share resources and a busy pair may deliver well under 2× a single thread. The clock (GHz) sets cycles per second, but cycles aren't instructions: how many instructions complete per cycle is IPC, and that depends heavily on whether data is in cache.

key ideaClock speed is not performance. A 3 GHz CPU doing 0.3 IPC (stalling on memory) does less real work than a 2 GHz CPU at 2.0 IPC. Utilization tells you the CPU is occupied; IPC tells you whether that occupation is productive. Both numbers together are the real story — utilization alone is half a metric.

02 The cache hierarchy and the memory wall

CPUs are vastly faster than main memory, so they hide the gap with caches. L1 is tiny and per-core (~1 ns), L2 larger (~4 ns), L3 shared and bigger (~15 ns), and main memory is ~100 ns — roughly 100× slower than L1. When data isn't cached, the core stalls, waiting dozens of cycles doing nothing. This is the memory wall: for many workloads the CPU isn't compute-bound, it's memory-bound, burning utilization on stalls. Cache-friendly data layout (locality) can speed code several-fold without changing a single line of logic.

Fig 1 — Miss in cache and the CPU waits on RAM for ~100 cycles. Locality, not GHz, often decides real speed.

03 Utilization vs saturation

For CPUs, utilization is the percent of time non-idle. Saturation is runnable threads waiting for a CPU — the run-queue length beyond the number of CPUs. A box can be 100% utilized with no saturation (fully used, nothing waiting — efficient) or show saturation while utilization is below 100% if work is unevenly spread. Saturation is what adds run-queue latency to your requests, so it, not raw utilization, is the alarm.

04 Load averages — and what they really mean

The famous 1/5/15-minute numbers from uptime. On Linux, load average counts threads that are runnable or in uninterruptible sleep (usually disk I/O) — so it's not purely a CPU metric. A load of 4.0 on a 4-CPU box means "about fully busy"; the same 4.0 on a 16-CPU box means lots of idle capacity. Always divide load by CPU count, and remember the Linux twist: high load can come from blocked-on-disk threads, not CPU demand at all.

watch outLoad average is widely misread. It is not a percentage, not CPU-only (on Linux it includes uninterruptible I/O sleepers), and meaningless without the CPU count. "Load is 8, that's high!" is unanswerable until you ask: on how many CPUs, and is it CPU or disk? Use it as a coarse trend, then confirm with per-CPU utilization and run-queue tools.

05 Scheduling, priorities, and limits

The scheduler shares CPUs among threads by policy. Nice values and priorities bias who runs; real-time classes can preempt normal work. In containers and clouds, cgroup CPU limits cap how much CPU a group may use — and when a container hits its quota, the kernel throttles it, injecting latency that looks like a mystery stall because the host has idle CPUs. CPU throttling under cgroup limits is one of the most common "why is my container slow when the node is idle?" puzzles.

the catchA container can be throttled to a crawl while the underlying node shows plenty of free CPU. The cgroup quota (e.g. CPU limit of 1 core) caps the container regardless of host capacity, and the kernel enforces it by stalling the container at quota — visible only in cgroup throttling stats (nr_throttled, throttled_time), not in host-level top. If a containerized app is slow but the node looks idle, check throttling before anything else.

06 Finding what burns CPU: flame graphs

When utilization is high and you want the culprit, profile and build a CPU flame graph. Sample stacks at, say, 99 Hz across all CPUs, collapse them, and render: the widest towers are where cycles go, and the leaf of each tower is the actual hot function. This single picture replaces hours of guessing — you can see at a glance that 60% of CPU is in JSON parsing, or a regex, or a hot loop. It's the fastest path from "CPU is high" to "this function is why."

07 Finding why cycles are slow: PMCs and IPC

Flame graphs show where CPU is spent; Performance Monitoring Counters show why each cycle is (un)productive. perf stat reads hardware counters for cycles, instructions, cache misses, branch mispredicts — and computes IPC. A high IPC (e.g. >1.0) means the CPU is doing real work; a low IPC (e.g. <0.5) means it's stalling, usually on cache/memory misses. Two apps at 100% CPU with IPC 2.0 vs 0.3 are in completely different situations: one is compute-bound (optimize the algorithm), the other memory-bound (fix data locality).

think of it likeA worker clocked in for a full 8-hour shift (100% utilization) might assemble 200 widgets or 20, depending on whether parts are at hand or they keep walking to the far warehouse for each one. IPC is the widgets-per-hour. Two fully-utilized workers can have wildly different output — and the slow one isn't lazy, they're stalled waiting on supplies (memory). You fix that by moving the parts closer (cache locality), not by yelling "work harder."

08 A CPU analysis workflow

(1) uptime / load for a coarse read (divide by CPU count). (2) mpstat -P ALL to see per-CPU balance and the %usr/%sys split — one hot core, or all even? (3) vmstat's r for run-queue saturation. (4) pidstat to find the process, top -H for the hot thread. (5) a flame graph to find the hot function. (6) perf stat for IPC to learn if it's compute- or memory-bound. (7) in containers, check cgroup throttling. Broad to narrow, every time.

common catches & gotchas

Utilization without IPC — 100% busy can be productive work or pure memory stalls. Check IPC with perf stat before deciding how to optimize.
Misreading load average — Not a percentage, includes disk-blocked threads on Linux, meaningless without CPU count. Divide and confirm.
Hyperthreads counted as full cores — Two SMT threads on one core aren't 2× a core. Capacity planning that assumes they are will over-promise.
Ignoring cgroup throttling — A container can stall at its CPU quota while the node is idle. Check throttled_time for "slow app, idle host" mysteries.
Average hides a hot core — 50% overall can be one core pinned at 100% (single-threaded bottleneck). Always look per-CPU.
Optimizing GHz instead of locality — For memory-bound code, faster clocks barely help; better data layout (cache locality) can be a multiplier.

09 Questions engineers actually ask

Is 100% CPU bad?

Not necessarily. 100% with no run-queue saturation and good IPC is a system efficiently doing work. It's "bad" only when work is queueing (latency rises) or the cycles are wasted on stalls (low IPC). Check saturation and IPC before alarming.

What does load average actually measure?

The average number of threads runnable or in uninterruptible sleep (disk I/O on Linux) over 1/5/15 minutes. It's a demand indicator, not a percentage, and includes I/O-blocked threads — so high load doesn't always mean CPU-bound. Always interpret relative to CPU count.

My app is at 100% CPU — how do I find the cause?

Profile it into a flame graph to see the hot functions, then run perf stat for IPC. High IPC → compute-bound, optimize the algorithm/hot path. Low IPC → memory-bound, improve cache locality or reduce memory traffic.

Why is my container slow when the host has free CPU?

Almost always cgroup CPU throttling. The container hit its CPU quota/limit, so the kernel stalls it even though the node is idle. Check the cgroup's nr_throttled and throttled_time, and raise or remove the limit if appropriate.

Should I disable hyperthreading?

It depends on the workload. SMT helps when threads stall a lot (filling idle execution slots) and can hurt latency-sensitive or cache-heavy workloads that contend for shared core resources. Measure both ways; there's no universal answer.

10 Key takeaways

CPU health is utilization (busy?) plus saturation (queued?) — saturation, via run queue, is the latency alarm.
Busy isn't productive: IPC reveals whether cycles do work or stall on the memory wall.
The cache hierarchy is ~100× faster than RAM; locality often beats clock speed.
Load average isn't a percentage, includes disk-blocked threads on Linux, and needs the CPU count to interpret.
cgroup throttling can stall a container while the host is idle — check it for mystery slowness.
Use a flame graph to find what burns CPU and PMCs/IPC to find why cycles are slow.

// chapter cheatsheetCPU analysis

utilization & saturation

uptimeLoad avg 1/5/15 — divide by CPU count.

mpstat -P ALL 1Per-CPU %usr/%sys/%idle — find a hot core.

vmstat 1 → rRun-queue length > #CPUs = saturation.

pidstat 1Per-process CPU over time — who's burning it.

top -HPer-thread — single-threaded bottleneck?

what burns CPU (profiling)

perf record -F 99 -ag -- sleep 30Sample all CPUs for a flame graph.

... | stackcollapse-perf.pl | flamegraph.plRender the CPU flame graph.

perf topLive hot-function view.

why cycles are slow (PMCs)

perf stat -d -p PIDCycles, instructions, IPC, cache misses.

IPC > 1 = compute-bound; < 0.5 = memory-boundPicks your optimization direction.

containers

cat /sys/fs/cgroup/.../cpu.statnr_throttled, throttled_usec — quota stalls.

11 Wrapping up

CPUs reward looking past the headline number: utilization tells you they're busy, saturation tells you work is waiting, and IPC tells you whether "busy" means productive or stalled. The memory wall is a recurring theme — and it leads naturally to the resource on the other side of it. Next: Memory, where virtual vs resident, paging, swapping, and the OOM killer await.

Systems Performance Chapter 6 — CPUs.