// the one-minute version
Most performance wins live in the application, not the kernel. The big levers: do less work (caching, buffering, avoiding needless syscalls), pick the right concurrency model (thread pools, event loops, non-blocking I/O), and avoid lock contention that serializes threads. To find the cost, split analysis into on-CPU (where the CPU burns — use flame graphs) and off-CPU (where threads block and wait — locks, I/O, sleeps). Real latency usually hides in the off-CPU waiting, which naive CPU profiling never shows.
When a service is slow, the cause is far more often in its own code and design than in the hardware beneath it. A missing cache, a chatty syscall loop, a single global lock — these throttle systems that have CPU, memory, and disk to spare. This chapter is about analyzing the application as the first-class performance target it is: how to make it do less, how to make it wait less, and how to see exactly where its time goes.
01 Setting an objective first
Before tuning, define the goal in numbers: a target latency (p99 < 200 ms), a throughput (10k req/s), or a resource budget (under 2 cores). Without a target you can't tell whether a change helped or when to stop. The objective also picks your strategy — a latency goal sends you hunting tail outliers and blocking time; a throughput goal sends you after per-request CPU cost and lock contention. Same app, different optimization path depending on what you actually promised.
02 The cheapest win: do less work
The fastest operation is the one you never perform. Three classic techniques.
Caching
Keep results of expensive work (query results, rendered fragments, computed values) so repeats are free. The biggest lever in most apps — but invites staleness and invalidation bugs.
Buffering
Batch small operations into big ones. Writing 1 MB in 4 KB chunks is 256 syscalls; buffering makes it a handful. Each boundary crossing has fixed cost; amortize it.
Avoiding syscalls
Every syscall crosses into the kernel. Apps that read one byte at a time, or stat a file in a hot loop, drown in %sys. Batch, cache, or use bigger reads.
%sys) in an application is a tell: the app is making a flood of syscalls. Trace which ones (strace -c briefly, or BPF) and you'll usually find tiny, repeated I/O that buffering collapses into a fraction of the calls. Doing less work beats doing the same work faster, almost every time.03 Concurrency models
How an app handles many requests at once shapes its whole performance profile. Thread-per-request is simple but each thread costs memory and scheduling; thousands of them thrash the scheduler. Thread pools bound that — a fixed set of workers pull from a queue — but the pool size becomes a critical tuning knob (too small starves throughput, too large oversubscribes CPUs). Event-driven / non-blocking models (one thread, an event loop, async I/O) scale to huge connection counts cheaply but make CPU-bound work and blocking calls dangerous (one slow handler stalls everyone).
04 Locks and contention
Shared data needs synchronization, and locks are the usual tool — but a lock is a serialization point. When many threads want the same lock, they queue, and the parallelism you paid for evaporates. A single hot global lock can cap a 32-core machine at the throughput of one core. Symptoms: high context-switch rates, threads spending time off-CPU in lock waits, CPUs not fully utilized despite a backlog. Fixes: shrink critical sections, shard the lock (per-bucket instead of global), use lock-free structures, or read-mostly tricks like RCU.
Fig 1 — Contention turns parallel work serial. Threads 2 and 3 burn wall-clock time waiting, not computing — invisible to on-CPU profiling.
05 On-CPU vs off-CPU analysis
This is the chapter's most important distinction. On-CPU analysis asks "what is the CPU executing?" — found by profiling the running stacks (flame graphs). It catches hot loops, expensive functions, busy work. Off-CPU analysis asks "why are threads not running?" — they're blocked on disk, network, locks, or sleeping. For most request-serving apps, the wall-clock latency is dominated by off-CPU waiting, so an on-CPU-only investigation can show a nearly idle flame graph while users wait seconds. You need both lenses.
06 Flame graphs
The flame graph is the signature visualization. Each box is a function; the box's width is how much time (sampled) was spent in it and its children; stacks go bottom (entry) to top (leaf). Wide boxes near the top are where the CPU actually is — read it like a heat map, scan for the widest towers. The same visualization works for off-CPU time (width = time blocked) and even allocations. Once you can read flame graphs, "where's my time going?" becomes a five-second glance instead of an afternoon.
07 Runtime and language effects
Managed runtimes add their own performance physics. Garbage collection (Java, Go, Node) reclaims memory automatically but introduces pauses — stop-the-world GC can add latency spikes that look mysterious until you correlate them with collection cycles. JIT compilation means code is slow at startup then fast once hot (cold-start latency). Interpreters and dynamic languages spend cycles on dispatch overhead. None of this is wrong, but it means your tail latency may be the GC's, not your logic's — so profile with the runtime's awareness (GC logs, runtime-specific tools) alongside system tools.
08 A workflow for app analysis
Put it together: (1) set a target; (2) check the USE basics so you're not chasing the app when the disk is dying; (3) profile on-CPU with a flame graph to find hot code; (4) profile off-CPU to find blocking — locks, I/O waits, downstream calls; (5) check %sys for a syscall storm fixable by buffering; (6) for managed runtimes, correlate with GC/JIT. Each step either finds the bottleneck or rules out a layer, so you converge instead of guessing.
common catches & gotchas
- On-CPU tunnel vision — A clean CPU flame graph doesn't clear the app; the latency may be entirely off-CPU (locks, I/O). Always profile blocking time too.
- Wrong thread-pool size — Too small idles cores while requests queue; too large thrashes the scheduler. Size to the work type, then measure.
- Chatty syscalls — Tiny per-byte reads/writes inflate
%sys. Buffer and batch; one big call beats hundreds of small ones. - One global lock — A single hot lock caps a many-core box at one-core throughput. Shard it or shrink the critical section.
- Blaming hardware for GC pauses — Periodic latency spikes in managed runtimes are often stop-the-world GC, not the disk or network. Check GC logs before adding hardware.
- Cache without invalidation — Caching is the biggest win and the biggest correctness risk. A stale cache serves wrong data; plan invalidation up front.
09 Questions engineers actually ask
My CPU profile looks empty but requests are slow. Why?
Because the time is off-CPU — threads blocked on locks, disk, or a downstream service. On-CPU profiling only sees running code. Run an off-CPU analysis (BPF off-CPU profiling, or wakeup/latency tools) to see where threads wait, and you'll find the missing time.
How big should my thread pool be?
For CPU-bound work, roughly the number of cores (more just causes contention). For I/O-bound work, higher, because threads spend most of their time blocked — size it so enough are runnable to keep cores busy. There's no universal number; measure utilization and queueing and adjust.
How do I know if lock contention is my problem?
Signs: high context-switch rate, CPUs not fully used despite a request backlog, and off-CPU profiles showing time in lock/futex waits. Lock-profiling tools (and BPF) can attribute wait time to specific locks. The fix is usually to shrink or shard the contended lock.
Should I optimize the app or add hardware?
Profile first. If the bottleneck is a hot function, a global lock, or a syscall storm, hardware won't help (and may multiply waste). If you're genuinely CPU- or I/O-saturated doing necessary work, scaling is valid. The app analysis tells you which world you're in.
How do I read a flame graph quickly?
Scan for the widest towers — width is time. The leaf (top) of a wide tower is where the CPU actually is. Ignore tall-but-thin stacks; they're rare paths. Click to zoom into a subtree. Width matters, depth is just call nesting.
10 Key takeaways
- Most performance wins are in the application: do less work, wait less, design concurrency well.
- Set a numeric objective first — it picks your strategy and tells you when you're done.
- Caching, buffering, fewer syscalls are the cheapest, biggest levers; high
%sysmeans a syscall storm. - Choose the right concurrency model and size thread pools to the work type.
- A single hot lock serializes parallel threads — shrink or shard it.
- Analyze on-CPU (flame graphs) and off-CPU (blocking) — real latency usually hides in the waits.
- Managed runtimes add GC/JIT effects; correlate tail spikes with collection cycles.
on-CPU profiling (flame graphs)
off-CPU / blocking time
syscalls & locks
threads & CPU split
managed runtimes
11 Wrapping up
The application is where you'll spend most of your tuning effort, and the on-CPU/off-CPU split is the lens that keeps you honest about where its time really goes. But applications run on physical resources, and sometimes they truly are the limit. Starting with the most scrutinized one, the next chapters analyze each in depth — first the CPUs.