BPF Performance Tools Chapter 13 — Applications Explained: Request Latency, Locks, Off-CPU

// the one-minute version

At the application level, BPF traces what the app actually does: request latency (via USDT or uprobes on request functions), lock contention (where threads serialize), and — most importantly — off-CPU time, since real latency is usually waiting, not computing. offcputime shows where threads block; offwaketime shows who woke them, linking the blocked thread to its blocker. Plus thread/signal tracing and database query latency (mysqld_qslower, dbslower). Profile on-CPU and off-CPU; the waits are where the time hides.

An application is where most performance problems live and where users feel them. BPF lets you trace the app's own behavior — individual requests, the locks threads fight over, the moments they block and wait — without modifying or restarting it. The recurring lesson from the whole series applies most sharply here: a clean CPU profile does not mean a fast app, because the latency is usually in the waiting. This chapter is the BPF toolkit for applications, centered on the off-CPU analysis that finds hidden waits.

01 Tracing requests and latency

The unit that matters to users is the request, and BPF can time it. If the app exposes USDT probes for request start/end (many servers and databases do), you attach to those for clean, stable per-request latency. Otherwise, uprobes on the request-handling function entry/return, with the per-thread timestamp pattern, give a latency histogram. From there you can break a slow request into its parts — time in the handler, time waiting on a query, time blocked on a lock — and attack the dominant slice. This is latency analysis (Chapter 3) applied at the application's own boundaries.

key ideaMeasure latency at the application's own boundaries — the request, the query, the transaction — because that's what users experience and what decomposes cleanly into parts. USDT probes give stable per-request timing where they exist; uprobes plus the timestamp pattern work everywhere else. Once you can time a request, you can attribute its latency to handler code, downstream calls, locks, or I/O — and fix the biggest piece.

02 The central lesson: off-CPU time

For most request-serving apps, wall-clock latency is dominated by waiting — blocked on locks, disk, network, or downstream services — not by on-CPU computation. On-CPU profiling (Chapter 6's profile) only sees running code, so a slow app can show a nearly idle CPU flame graph. offcputime is the complement: it records how long threads spend off-CPU and the stacks that led there, revealing exactly where they block. A thin on-CPU profile plus a fat off-CPU profile is the unmistakable signature of a waiting-bound application — and off-CPU is where you'll spend most application-analysis effort.

03 Who woke me: offwaketime

Off-CPU tells you a thread blocked and where; it doesn't tell you what unblocked it. offwaketime closes that gap by linking the blocked thread's stack to the waker's stack — the thread (or kernel path) that woke it. This is powerful for lock and producer-consumer problems: you see not just "thread A waited on this lock" but "thread A waited, and thread B (doing this) woke it" — revealing the contention relationship. Chains of wakeups can be followed to find the root of a latency that propagates across threads, turning a mysterious stall into a visible dependency.

Fig 1 — offcputime shows the blocked stack; offwaketime adds the waker, revealing the dependency behind the wait.

04 Lock contention

Shared locks serialize threads, capping parallelism (the application lesson from Systems Performance). BPF finds contention from several angles: offcputime/offwaketime show time blocked in futex/lock waits with the contending stacks; user-space lock tracing (uprobes on pthread_mutex_lock and friends) can time lock-hold and lock-wait durations directly. The symptoms — high context-switch rates, CPUs idle despite a request backlog, off-CPU time concentrated in lock waits — all point here. Once you've identified the hot lock and the code holding it, the fixes are the familiar ones: shrink the critical section, shard the lock, or go lock-free.

05 Threads, signals, and I/O

More application internals. Thread creation and lifecycle can be traced (a storm of short-lived threads is its own problem). Signals — which can interrupt and disrupt timing — are traceable with tools like killsnoop/signals, useful when a process behaves oddly under signals it's receiving. Application I/O can be profiled to see how much time goes to reads/writes versus compute. The theme is that BPF can attach to almost any application behavior the app or libraries expose, so when a question is specific ("how often does this thread pool grow," "who's signaling this process"), there's usually a probe and a one-liner for it.

06 Database query latency

A concrete, high-value example: databases. Tools like mysqld_qslower (and dbslower, dbstat) trace SQL query execution via the database's USDT probes, showing queries slower than a threshold with their text and latency — directly surfacing the slow queries hurting your application, on a live server, without enabling the database's own (heavier) slow-query log. This is drill-down across a boundary: a slow request traced to a slow database call, then to the specific query. The same approach generalizes to any application that exposes USDT probes for its key operations.

the catchThe most expensive application-analysis mistake is concluding "it's not the app" from a clean on-CPU profile. If a request spends 5 ms computing and 95 ms blocked on a lock, a downstream call, or a slow query, the CPU flame graph is 5% of the story and looks nearly empty — and engineers routinely stop there and blame infrastructure. The latency is entirely in the off-CPU time they never measured. Always pair on-CPU profiling with offcputime (and offwaketime for the waker) on a slow application. The clean CPU profile isn't an all-clear — for a waiting-bound app, it's the clue that points you to the off-CPU analysis.

07 On-CPU plus off-CPU: the full picture

The complete application profile combines both lenses. On-CPU (profile → flame graph) finds hot code — expensive functions, busy loops, parsing. Off-CPU (offcputime → off-CPU flame graph) finds blocking — locks, I/O, downstream waits. Rendered as two flame graphs side by side, they account for where a thread's wall-clock time goes: running vs waiting. For most services the off-CPU graph is wider and more revealing. The discipline is to always look at both before drawing a conclusion, because each alone tells only half the story — and the half that matters is usually the one people skip.

08 An application analysis workflow

(1) Time the request (USDT or uprobe + timestamp pattern) to confirm and quantify the latency. (2) Profile on-CPU for hot code. (3) Profile off-CPU with offcputime — usually where the latency is. (4) If blocking is on locks or producer-consumer, use offwaketime to find the waker and the contention relationship. (5) Trace downstream calls (e.g. mysqld_qslower) to attribute latency across boundaries to specific queries. (6) Fix the dominant slice and verify against the request-latency metric. On-CPU and off-CPU, every time; the waits are where the time hides.

common catches & gotchas

"Clean CPU profile = fine" — For waiting-bound apps it's the opposite: the clue to do off-CPU analysis. Always pair on-CPU with offcputime.
Ignoring the waker — offcputime shows the wait; offwaketime shows who caused it. For lock/producer-consumer issues you need the waker.
Broken app stacks — Missing frame pointers/symbols make off-CPU stacks [unknown]; for JIT/interpreted, see Chapter 12. Fix stacks first.
Per-event off-CPU overhead — Off-CPU tracing fires on every block; on very high context-switch rates it costs. Filter to a process and time-box.
Heavy DB slow-query logs — The database's own slow-query log can add overhead; BPF tools like mysqld_qslower trace queries with less, on a live server.
Optimizing the wrong slice — Without timing the request's parts, you may optimize 5% on-CPU while 95% is a lock. Decompose latency before fixing.

09 Questions engineers actually ask

My app is slow but the CPU profile looks empty — what now?

That's the signature of off-CPU (waiting) latency. Run offcputime to see where threads block, with stacks, and render it as an off-CPU flame graph. For lock or producer-consumer waits, add offwaketime to find what woke the thread. The missing time is almost always in the waits.

What's the difference between offcputime and offwaketime?

offcputime shows how long threads are blocked and the stack where they blocked. offwaketime additionally links that blocked stack to the waker's stack — the thread or path that woke it. The first finds the wait; the second finds who caused it, which is what you need for contention analysis.

How do I find lock contention?

Look for off-CPU time concentrated in futex/lock waits (via offcputime/offwaketime), plus symptoms like high context switches and idle CPUs despite a backlog. User-space lock tracing (uprobes on pthread_mutex_lock) can time lock hold/wait directly. Then shrink, shard, or remove the hot lock.

How do I find slow database queries?

Use mysqld_qslower (or dbslower/dbstat), which trace query execution via the database's USDT probes and show queries slower than a threshold with their text and latency — on a live server, with less overhead than the database's own slow-query log.

Do I need to modify or restart my app to trace it?

No — that's a core BPF benefit. uprobes attach to existing functions and USDT probes to existing static tracepoints in the running process, with no recompile or restart. You can attach offcputime, request timing, or query tracing to a live production app and detach when done.

10 Key takeaways

Trace latency at the application's boundaries — requests, queries — via USDT or uprobes + the timestamp pattern.
Real latency is usually off-CPU (waiting), not on-CPU — a clean CPU profile is the clue, not the all-clear.
offcputime shows where threads block; offwaketime adds who woke them.
Lock contention shows as off-CPU time in lock waits, idle CPUs with a backlog, high context switches.
Database tools (mysqld_qslower) attribute latency across boundaries to specific queries.
Always profile on-CPU and off-CPU together — each tells half the story.
BPF traces a live app with no recompile or restart.

// chapter cheatsheetapplications with BPF

request latency

bpftrace usdt:/app:req__start / req__donePer-request latency via USDT.

uprobe:handler { @s[tid]=nsecs } / uretprobe { @=hist(nsecs-@s[tid]) }Request latency via uprobes.

off-CPU (the key analysis)

offcputime -p PID 30Where threads block, with stacks.

offwaketime -p PID 30Blocked stack + the waker — contention.

profile -p PIDOn-CPU flame graph — pair with off-CPU.

locks & threads

uprobe:/lib/libpthread:pthread_mutex_lockTime lock hold/wait directly.

killsnoopSignals sent to/by processes.

databases

mysqld_qslower PID 10Queries slower than 10ms with text.

dbslower · dbstatGeneric DB query latency/stats.

11 Wrapping up

For applications, BPF times requests at the app's own boundaries and — crucially — reveals the off-CPU waiting where most latency lives, with offwaketime exposing the threads behind a stall and DB tools attributing latency to queries. Profile on-CPU and off-CPU together, always. Beneath the application sits the kernel itself, and the next chapter turns BPF's lens inward. Next: Kernel.

BPF Performance Tools Chapter 13 — Applications.