BPF Performance Tools Chapter 14 — Kernel Explained: Wakeups, Work Queues, Mutexes, Interrupts

// the one-minute version

When the bottleneck is the kernel itself, BPF traces its internals. Key targets: the scheduler (run-queue and wakeup latency), work queues and deferred work (where the kernel does background processing), kernel locks (mutexes, spinlocks — contention inside the kernel), and interrupts (hardirqs/softirqs). Use stable tracepoints where they exist (scheduler, IRQ, workqueue) and kprobes for everything else — with the version-fragility caveat. Kernel tracing is the deepest layer; most problems are solved above it, but when they're not, BPF is how you see inside.

Most performance problems live in your application or in how it uses resources — but occasionally the kernel itself is the bottleneck: a scheduler decision adding wakeup latency, a contended kernel lock serializing CPUs, interrupt handling eating cycles, a work queue backing up. These are nearly impossible to see with traditional tools. BPF, attached to the kernel's own tracepoints and functions, makes them visible. This chapter is the kernel-internals toolkit — the deepest layer, for when the answer is inside the kernel.

01 When the kernel is the target

You reach kernel tracing after the higher layers come up empty — the app profile is clean, off-CPU points into kernel waits, the resource tools show the kernel doing something costly. Signs the kernel itself is involved: high system CPU (%sys) not explained by your syscalls, latency in scheduler or wakeup paths, contention on kernel locks, or interrupt-heavy behavior. BPF can attach to kernel functions and tracepoints to measure these directly. It's the most specialized tracing in the book — and the most powerful, because it sees what nothing else can.

key ideaKernel tracing is a last resort by depth, not by importance. Most problems are solved in the app or resource layers above, so you don't start here. But when off-CPU analysis points into kernel waits, or system time is unexplained, BPF on kernel tracepoints/functions is the only way to see the cause. Prefer stable tracepoints (scheduler, IRQ, workqueue) to keep tools robust across kernel versions.

02 Scheduler and wakeup latency

The scheduler decides which thread runs when, and two latencies matter. Run-queue latency (Chapter 6's runqlat) is the wait for a CPU. Wakeup latency is subtler — the time between a thread being woken (an event it waited on occurred) and it actually running. High wakeup latency means the scheduler isn't getting the woken thread onto a CPU promptly, hurting latency-sensitive work. BPF traces the scheduler tracepoints (sched:sched_wakeup, sched:sched_switch) to measure both, and tools like wakeuptime connect a wakeup to its cause. This is where mysterious scheduling delays become visible numbers.

03 Work queues and deferred work

The kernel defers a lot of work — interrupt bottom-halves, background maintenance — to work queues and kernel threads (kworker). When this deferred work backs up or runs slowly, it can delay the events your application depends on, and it's nearly invisible to application-level tools (it's the kernel doing things on everyone's behalf). BPF traces workqueue tracepoints to see what work is queued, how long it waits, and how long it runs. A kworker thread consuming surprising CPU, or work-queue latency spiking, points at kernel background processing as the issue — something you'd never find without tracing the kernel directly.

Fig 1 — When the kernel is the bottleneck, BPF traces its scheduler, locks, interrupts, work queues, and memory directly.

04 Kernel locks: mutexes and spinlocks

The kernel uses its own locks to protect shared data, and just like application locks, contended kernel locks serialize CPUs and add latency. A hot kernel mutex or spinlock can cap scalability on a many-core box — a workload that should scale linearly flattens because all CPUs queue on one lock. Spinlocks are especially costly because waiting CPUs spin (burn cycles) rather than sleep. BPF can trace lock functions to measure hold and wait times and identify the contended lock and the code path holding it. This is advanced — kernel-lock contention is rarer than application-lock contention — but when scalability mysteriously plateaus, it's a prime suspect.

05 Interrupts

Hardware devices interrupt the CPU to signal events, and the kernel services these in hard IRQ handlers (immediate) and deferred soft IRQs (the heavier follow-up). On busy systems — especially high-packet-rate network boxes — interrupt servicing can consume significant CPU, showing as system time with no obvious process behind it. hardirqs and softirqs (from Chapter 6) measure time spent in each, by type. A network softirq dominating CPU, or interrupt latency spiking, points at the interrupt path — and at mitigations like interrupt coalescing or spreading IRQs across CPUs. BPF makes this otherwise-invisible CPU consumption concrete.

06 Kernel memory and other internals

The kernel manages its own memory (the slab allocator for kernel objects), and leaks or excessive allocation there are a real failure mode — kernel memory isn't swappable and a kernel leak can exhaust the system. BPF can trace kernel allocation functions (kmem tracepoints, kmalloc/kfree) to attribute kernel memory growth to code paths, much like memleak does for user space. Beyond memory, almost any kernel subsystem — VFS, networking stack, block layer internals — is traceable, because BPF can attach to its functions and tracepoints. The kernel is, with BPF, as observable as anything else.

the catchKernel tracing leans heavily on kprobes (attaching to internal kernel functions), and those functions are not a stable interface — they're renamed, inlined away, split, or removed across kernel versions. A kernel one-liner or tool that works perfectly today can silently match nothing (or, worse, match a similarly-named but different function) after a kernel upgrade, giving empty or misleading results with no error. Always prefer the stable tracepoints the kernel deliberately exposes (scheduler, IRQ, workqueue, block, kmem) over kprobes on internal functions, and re-verify any kprobe-based kernel tooling after every kernel change. This fragility is sharpest at the kernel layer because that's where you're most tempted to reach for kprobes on internals.

07 Tracepoints vs kprobes for the kernel

The choice matters most here. Tracepoints are stable, documented kernel events with a committed interface — the scheduler, IRQ, workqueue, block, syscall, and kmem subsystems all expose them. Tools built on tracepoints survive upgrades. kprobes reach any kernel function but target internal names that move. The discipline: list what's available (bpftrace -l 'tracepoint:*'), use a tracepoint if one answers your question, and fall back to kprobes only when nothing else can see what you need — accepting that you'll re-verify after kernel changes. For exploratory kernel work kprobes are invaluable; for durable tooling, tracepoints win.

08 A kernel analysis workflow

(1) Arrive here from above — clean app profile, off-CPU pointing into kernel waits, or unexplained %sys. (2) For scheduling delays, runqlat (run-queue) and wakeup-latency tools. (3) For unexplained system CPU, check interrupts (hardirqs/softirqs) and kworker/work-queue activity. (4) For scalability plateaus on many cores, suspect kernel-lock contention and trace lock hold/wait. (5) For kernel memory growth, trace kmem allocations by stack. (6) Throughout, prefer tracepoints; use kprobes only when necessary and re-verify after upgrades. Deepest layer, last — but decisive when the kernel is the cause.

common catches & gotchas

kprobe fragility — Internal kernel function names change across versions; kprobe tools silently break or mismatch. Prefer tracepoints; re-verify after upgrades.
Starting at the kernel — It's the deepest layer; most problems are above it. Arrive here from off-CPU/system-time evidence, not first.
Unexplained %sys — High system time with no matching syscalls often means interrupts or kernel work-queue activity. Check hardirqs/softirqs and kworker.
Spinlock cost — Spinning CPUs burn cycles while waiting; kernel spinlock contention can flatten many-core scalability invisibly. Trace lock hold/wait.
Kernel memory leaks — Kernel memory isn't swappable; a slab/kmem leak can exhaust the system. Trace kmem allocations by stack.
Inlined functions — kprobes can't attach to inlined kernel functions (they have no entry); a missing probe may mean the function was inlined, not absent.

09 Questions engineers actually ask

When should I trace the kernel itself?

After the layers above come up empty — a clean application profile, off-CPU analysis pointing into kernel waits, or high system CPU you can't explain by your syscalls. Kernel tracing is the deepest, most specialized layer; you arrive here from evidence, not as a first step.

My system CPU is high but my syscalls don't explain it — why?

Often interrupts or kernel background work. Check hardirqs/softirqs for time in interrupt handlers (common on busy network boxes) and look at kworker threads / work-queue activity for deferred kernel work. Both consume system CPU with no obvious application process behind them.

Why did my kernel bpftrace one-liner stop working after an upgrade?

It almost certainly used a kprobe on an internal kernel function whose name changed, was inlined, split, or removed — kprobe targets aren't a stable interface. Switch to a stable tracepoint if one covers your question, and always re-verify kprobe-based kernel tools after a kernel upgrade.

Can kernel locks really be my bottleneck?

Yes, though it's rarer than application-lock contention. A hot kernel mutex or spinlock serializes CPUs and can flatten scalability on many-core systems — spinlocks especially, since waiting CPUs burn cycles. Trace lock hold/wait to find the contended lock when scalability plateaus unexpectedly.

What's wakeup latency?

The time between a thread being woken (the event it waited on happened) and it actually running on a CPU. High wakeup latency means the scheduler isn't promptly placing woken threads, hurting latency-sensitive work. BPF measures it via scheduler tracepoints; wakeuptime links a wakeup to its cause.

10 Key takeaways

Trace the kernel itself when higher layers are clean but off-CPU/system-time points inside it.
The scheduler adds run-queue and wakeup latency — measurable via scheduler tracepoints.
Work queues/kworker do deferred kernel work that can back up invisibly to apps.
Kernel locks (mutexes, spinlocks) contend just like app locks — flattening many-core scalability.
Interrupts (hardirqs/softirqs) can eat system CPU with no obvious process behind them.
Kernel memory (slab/kmem) is traceable and, since it's unswappable, leaks are serious.
Prefer stable tracepoints over fragile kprobes; re-verify kprobe tools after kernel upgrades.

// chapter cheatsheetkernel with BPF

scheduler & wakeups

runqlatRun-queue (CPU-wait) latency histogram.

wakeuptimeWakeup latency, linked to the waker.

-e 'tracepoint:sched:sched_switch { ... }'Custom scheduler tracing (stable tracepoint).

interrupts & work queues

hardirqs · softirqsTime in IRQ handlers — unexplained %sys.

-l 'tracepoint:workqueue:*'Trace deferred work / kworker.

kernel locks & memory

kprobe:mutex_lock / queued_spin_lock_slowpathKernel lock hold/wait (kprobe — fragile).

-l 'tracepoint:kmem:*'Kernel (slab) allocations by stack.

discipline

bpftrace -l 'tracepoint:*'Prefer stable tracepoints over kprobes.

re-verify after every kernel upgradekprobe targets move; tracepoints don't.

11 Wrapping up

The kernel is the deepest target and, with BPF, a fully observable one: scheduler and wakeup latency, work queues, kernel locks, interrupts, and kernel memory all yield to tracepoints and (carefully) kprobes. You arrive here from evidence above, and you lean on stable tracepoints to keep tools durable. The book now turns to the modern deployment realities, starting with containers. Next: Containers.

BPF Performance Tools Chapter 14 — Kernel.