BPF Performance Tools Chapter 16 — Hypervisors Explained: Guest vs Host, Steal, KVM Exits

// the one-minute version

Virtual machines add a hard visibility boundary: a guest can't see the host, and the host can't see inside the guest's kernel. From the guest, the one leaked host signal is CPU steal time (%st) — time your vCPU was ready but the host ran someone else. From the host (if you control it), BPF can trace KVM exits (vmexit) — the costly moments a guest traps into the hypervisor — and attribute resource use per guest. Where you can run BPF depends on which side you own; in public clouds you usually only have the guest, so steal time plus standard in-guest tracing is your toolkit.

Hardware virtualization gives each VM its own kernel behind a hypervisor — strong isolation, but a wall that BPF can't see through from either side. The practical question becomes which side can you run BPF on, and what each side can and can't reveal. In a public cloud you typically only have the guest; on your own infrastructure you may have the host too, which unlocks far more. This short chapter maps the virtualized terrain and the limits BPF hits there.

01 The visibility boundary

A hypervisor (KVM, Xen, etc.) runs guest VMs, each with its own kernel, isolated by hardware virtualization. That isolation is also an observability wall: the guest kernel can't see the host or other guests, and the host can't see inside the guest's kernel (it sees the VM as a process, not the guest's processes). BPF runs in a kernel — so a BPF program in the guest sees the guest's world only, and one on the host sees the host's. Neither crosses the boundary. The first question for any VM analysis is therefore: which kernel can I attach BPF to?

key ideaBPF observes the kernel it runs in, and a VM has two kernels separated by the hypervisor wall. Guest BPF sees guest internals but nothing of the host; host BPF sees the host and the VM-as-a-process but nothing inside the guest. There's no single vantage point that sees both — so what you can learn depends entirely on which side you can run BPF, and in the cloud that's usually only the guest.

02 From the guest: steal time

Inside a guest you have limited insight into the host, but one crucial signal leaks through: CPU steal time (%st in top/mpstat). It's the percentage of time your virtual CPU was runnable but the hypervisor gave the physical CPU to another guest — i.e., the host was oversubscribed and starved you. High steal means your bottleneck is host contention, not your code (Systems Performance covered this). It's the single most important number for guest-side performance, and it's available in the standard tools. BPF in the guest otherwise works normally — it just can't see beyond the VM.

03 From the guest: what BPF can and can't do

In the guest, all the in-VM BPF tooling from earlier chapters works on the guest's own kernel — profiling, off-CPU, syscalls, disk and network tracing of the guest's view. What you can't do from the guest is see the host's contention beyond steal time, the physical device behind your virtual disk, other tenants, or how the hypervisor is scheduling you. So guest-side BPF gives a complete picture of your software and a single leaked signal (steal) about the host. For most cloud users this is the entire toolkit, and it's enough to separate "my problem" from "the host's problem."

Fig 1 — Guest BPF and host BPF each see only their own kernel. Steal time leaks to the guest; KVM exits are visible on the host.

04 From the host: KVM exits

If you control the host (private infrastructure), BPF unlocks far more. The key event is the VM exit (vmexit) — when a guest traps out of guest mode into the hypervisor to handle something it can't do directly (certain instructions, I/O, interrupts). Each exit costs CPU, and frequent exits are a major source of virtualization overhead. BPF on the host can trace KVM exit events (there are KVM tracepoints), count them by reason, and measure their cost — revealing whether a guest is suffering excessive exits and why. This is host-side analysis no guest can do, and it's where you diagnose virtualization overhead at its source.

05 From the host: per-guest attribution

The host also sees each VM as a process (or set of threads), so host-side BPF can attribute physical resource use per guest — which VM is driving the disk, the network, the CPU. This is the noisy-neighbor analysis of the VM world (parallel to containers in Chapter 15): from the host you can see that guest X is saturating the shared device and starving guest Y. Combined with KVM-exit tracing, the host vantage gives a complete operator's view of how guests behave and interfere — the picture cloud providers have and cloud users don't.

the catchThe defining limitation of VM tracing is that in a public cloud you almost never control the host — so the rich host-side analysis (KVM exits, per-guest attribution, true contention) is simply unavailable to you. Your entire BPF toolkit is guest-side, plus the one leaked host signal: steal time. This means some questions are unanswerable from where you sit — you cannot see the noisy neighbor, the physical device, or why the hypervisor scheduled you a certain way. Recognizing this boundary saves wasted effort: when steal time is high, the cause is host contention you can't trace, and the fix is operational (resize, migrate, pick a less crowded instance), not more tracing.

06 Paravirtualization and hardware assists

Some context on why exits matter. Early virtualization trapped many operations (slow). Paravirtualization (the guest is aware it's virtualized and uses hypercalls) and hardware-assisted virtualization (CPU features like Intel VT-x/AMD-V, plus device assists like SR-IOV) drastically cut the exits and overhead. Modern VMs are mostly hardware-assisted, so overhead is low — but not zero, and specific workloads (I/O-heavy, lots of privileged operations) still generate enough exits to matter. When they do, host-side vmexit tracing is how you confirm it. The trend is less overhead over time, but the analysis approach — count and cost the exits — stays the same.

07 A note on nested and modern environments

Reality has gotten more layered: VMs running containers, nested virtualization, and lightweight VMs (Firecracker, etc.) blurring the VM/container line. The principle holds regardless — BPF sees the kernel it runs in, and each virtualization boundary is a wall it can't cross. For a container inside a VM in a cloud, you may have the container and guest view but not the host or hypervisor. The skill is mapping which boundaries exist between you and the resource you're questioning, and recognizing which sit beyond your BPF reach — so you know when the answer requires access (or cooperation) you don't have.

08 A hypervisor analysis workflow

(1) Identify which side you can run BPF on — guest only (typical cloud) or host too (your infrastructure). (2) From the guest, check steal time first — high steal means host contention you can't trace; the fix is operational. (3) Use standard in-guest BPF for everything inside your VM. (4) If you have the host, trace KVM exits (count and cost by reason) to find virtualization overhead, and attribute per-guest resource use to spot noisy neighbors. (5) Recognize the boundary: some questions are unanswerable from the guest. Which kernel you own decides what you can see.

common catches & gotchas

Expecting to trace the host from the guest — BPF can't cross the hypervisor wall. From a cloud guest you get steal time and your own kernel, nothing more.
Ignoring steal time — High %st means the host starved your vCPU; don't profile code. The fix is operational (resize/migrate).
Profiling code for a host problem — When the cause is host contention or a noisy neighbor you can't see, more in-guest tracing won't reveal it.
Assuming zero virtualization overhead — Modern hardware assists cut it, but I/O-heavy or privileged-op-heavy guests still generate costly exits. Confirm with host-side vmexit tracing.
Forgetting which side you own — Host-side analysis (KVM exits, per-guest attribution) needs host access you usually lack in public clouds.
Nested boundaries — VMs-in-VMs and containers-in-VMs add walls; map which boundaries sit between you and the resource you're questioning.

09 Questions engineers actually ask

Can I trace the hypervisor from inside my cloud VM?

No — BPF can't cross the hypervisor wall, and in a public cloud you only have the guest. You get your own kernel's full visibility plus one leaked host signal: CPU steal time. Host-side analysis (KVM exits, contention) requires host access you don't have in the cloud.

What is CPU steal time and why does it matter?

The percent of time your vCPU was runnable but the hypervisor gave the physical CPU to another guest — i.e., the host was oversubscribed and starved you. High steal means your bottleneck is host contention, not your code. It's the most important guest-side performance signal and it's in top/mpstat.

What are KVM exits and how do I see them?

Moments when a guest traps out of guest mode into the hypervisor to handle something it can't do directly (privileged instructions, I/O). Each costs CPU; frequent exits are a major source of virtualization overhead. Trace them with BPF on KVM tracepoints — but only from the host, which you need to control.

My VM is slow and steal time is high — what do I do?

The host is oversubscribed and starving your vCPU — a problem you can't trace or fix from inside the guest. The remedy is operational: resize to a less contended instance type, migrate to another host, or use dedicated/burstable instances with headroom. More in-guest tracing won't help.

Is virtualization overhead still a concern?

Much less than it used to be — hardware assists (VT-x/AMD-V, SR-IOV) and paravirtualization cut the exit overhead dramatically. But it's not zero: I/O-heavy or privileged-operation-heavy guests can still generate enough exits to matter. Confirm with host-side vmexit tracing when you control the host.

10 Key takeaways

A VM has two kernels split by the hypervisor wall; BPF sees only the kernel it runs in.
From the guest, the one leaked host signal is steal time — high steal = host contention you can't trace.
Standard in-guest BPF works on the guest's kernel; it can't see the host, devices, or other tenants.
From the host (if you own it), trace KVM exits (vmexit) for virtualization overhead.
Host-side BPF attributes resource use per guest — the VM noisy-neighbor view.
In public clouds you usually only have the guest — some questions are unanswerable from there; the fix is operational.
Map the boundaries between you and the resource; nested VMs/containers add more walls.

// chapter cheatsheethypervisors with BPF

from the guest (typical cloud)

mpstat -P ALL 1 → %stealHost contention starving your vCPU — check first.

top → st · vmstat 1 → stSteal time at a glance.

standard BPF tools (profile, offcputime, ...)Full visibility into the guest's own kernel.

from the host (your infra only)

bpftrace -l 'tracepoint:kvm:*'KVM tracepoints (exits, etc.).

trace kvm:kvm_exit by reasonCount + cost VM exits — virtualization overhead.

per-VM process attributionWhich guest drives disk/net/CPU — noisy neighbor.

decision

which kernel can I attach BPF to?Guest vs host decides what's visible.

high steal → operational fixResize/migrate; you can't trace the host.

11 Wrapping up

Hypervisors draw a wall BPF can't cross: from a cloud guest you get your own kernel plus steal time; from a host you own, KVM exits and per-guest attribution. Knowing which side you're on tells you which questions are answerable and which need an operational fix. The resource and environment tour is complete — the book closes with the wider ecosystem and the practical traps. Next: Other BPF Tools.

BPF Performance Tools Chapter 16 — Hypervisors.