// the one-minute version
Containers are just processes isolated by namespaces (what they see) and cgroups (what they can use) — so the best place to trace them is the host, where BPF sees everything at once and can attribute activity per container. The big wins: spotting CPU throttling (a container clamped at its cgroup quota while the node is idle) and blk/IO throttling, and seeing noisy neighbors across containers. The hard part is mapping a PID/event back to a container — by cgroup ID or namespace — which trace output doesn't label for you. Trace from the host; attribute by cgroup.
A container looks like a sealed box, but to the kernel it's ordinary processes wearing namespaces and cgroups. That's good news for tracing: from the host, BPF sees every container's activity directly, no per-container agents needed. The challenges are specific — recognizing cgroup-imposed throttling that looks like a mystery stall, seeing how containers interfere, and the genuinely annoying problem of figuring out which container a traced event belongs to. This chapter is BPF for the container world.
01 What a container actually is
There's no "container" object in the kernel — a container is a set of processes isolated by two mechanisms. Namespaces control what a process can see: its own view of PIDs, the filesystem, network interfaces, hostnames. cgroups control what it can use: CPU, memory, I/O limits. Together they make processes feel like they're alone on a machine while sharing the host kernel. Because it's all just processes and kernel mechanisms, the host kernel — and BPF running there — has full visibility into every container at once, which shapes the whole tracing approach.
02 The host vantage point
From the host, the BPF tools you already know work across all containers — execsnoop, biolatency, tcplife, runqlat, offcputime — observing every container's activity in one place. This is powerful for whole-node analysis: you see the busiest container, cross-container interference, and node-wide resource pressure. The catch (next sections) is that the raw output identifies processes by host PID and name, not by container — so you get complete visibility but must do extra work for per-container attribution. Still, "run the tool once on the host, see all containers" is the right mental model.
03 CPU throttling: the cgroup catch
The most important container-specific issue. A container's cgroup can cap its CPU (a quota), and when it hits that cap the kernel throttles it — stalling the container even though the host has idle CPUs. From inside, this is a baffling "we're slow but the node looks empty" mystery. BPF (and the cgroup's cpu.stat with nr_throttled/throttled_time) reveals it directly: the container is being clamped at its quota. This is the same throttling discussed in Systems Performance, and it's the first thing to check when a containerized app is slow while the node is underutilized — the cause is the limit, not the code.
top on the host shows plenty of idle CPU — because the cgroup quota clamps the container regardless of host capacity. The throttling is invisible in host-wide CPU metrics and only shows in the cgroup's own cpu.stat (nr_throttled, throttled_usec) or via BPF tracing the throttle events. Engineers burn hours profiling code that's fine, because the real cause is a CPU limit set too low. Whenever a containerized workload is slow and the node looks idle, check cgroup throttling first — it's the single most common container performance trap.04 Block I/O throttling and other limits
CPU isn't the only throttled resource. cgroups also limit block I/O (IOPS/bandwidth caps), and a container hitting its I/O limit sees added latency that, again, won't appear in host-wide disk metrics — the device may be fine while this container is throttled. BPF can trace the throttling and attribute the added latency to the container. Memory limits trigger per-cgroup OOM (Chapter 7). The pattern across all of them is the same: a per-container limit causes latency that node-level tools can't see, and the fix is recognizing the limit rather than chasing a phantom resource problem. Always check the relevant cgroup's limit and throttle stats.
Fig 1 — From the host, BPF sees all containers; B's throttling is invisible node-wide and only shows in its cgroup stats.
05 Noisy neighbors
Containers share host resources, so one can degrade others — the noisy neighbor problem. A container saturating the disk, the network, the CPU cache, or memory bandwidth can hurt co-located containers that look innocent in their own metrics. The host vantage point is exactly what you need: BPF on the host attributes resource use per container, so you can see that container C is generating the disk I/O that's adding latency to container B. This cross-container view is impossible from inside any single container — only the host (and BPF there) sees the whole picture and can name the neighbor doing the damage.
06 The attribution problem: which container?
The genuine annoyance of container tracing: a trace shows a host PID and process name, but which container is that? The mapping isn't in the trace output. The answer lives in the process's cgroup (the path encodes the container/pod ID) or its namespaces. BPF can read the cgroup ID at the event and key results by it, so tools can aggregate per container — but it takes deliberate work, and correlating a cgroup ID back to a friendly container/pod name requires consulting the runtime (Docker/Kubernetes). This is the recurring friction of container observability: full visibility, but identity requires extra mapping.
07 Per-container tracing techniques
To attribute by container, BPF tools read the cgroup ID at the event and use it as a map key (e.g. run-queue latency per cgroup, I/O per cgroup), giving per-container breakdowns from a single host-side trace. You can also filter to a target cgroup. Some workflows trace from inside a container's namespace when you specifically want that container's view. The kernel exposes cgroup info to BPF programs, so the building blocks exist; the practical step is mapping the numeric cgroup ID to the human container name via the orchestrator. Modern container-aware tooling automates this correlation.
08 A container analysis workflow
(1) Trace from the host with the standard BPF tools — one session sees all containers. (2) If a container is slow while the node looks idle, check CPU throttling (cgroup cpu.stat / throttle tracing) — usually the cause. (3) Check other cgroup limits (block I/O, memory) for added latency invisible node-wide. (4) For cross-container interference, use the host view to find the noisy neighbor generating the contention. (5) Attribute events to containers via cgroup ID, then map to names through the runtime. Trace from the host, suspect limits first, attribute by cgroup.
common catches & gotchas
- CPU throttling = "slow app, idle node" — Check the cgroup's
cpu.stat(nr_throttled) before profiling code. It's the top container trap. - Per-container limits hide latency — Block-I/O and memory cgroup limits add latency invisible in host-wide metrics. Check the relevant cgroup limit.
- Tracing from inside containers — More limited and often needs privileges. Trace from the host for full, deduplicated visibility.
- Unlabeled trace output — Traces show host PIDs, not container names. Attribute via cgroup ID, then map to names through the runtime.
- Missing noisy neighbors — A container's own metrics look fine while a neighbor saturates a shared resource. Only the host view reveals it.
- cgroup v1 vs v2 differences — Limit and stat file layouts differ between cgroup versions; know which your host uses when reading throttle stats.
09 Questions engineers actually ask
Should I trace inside the container or on the host?
On the host. Containers share the host kernel and are just namespaced/cgrouped processes, so one BPF session on the host sees every container's activity — no per-container agents, no privileges inside each container. Tracing from inside is more limited; use it only when you specifically want a single container's narrow view.
My container is slow but the node has free CPU — why?
Almost certainly cgroup CPU throttling — the container hit its CPU quota and the kernel clamped it regardless of host capacity. Check the cgroup's cpu.stat for nr_throttled/throttled_usec, or trace the throttle events with BPF. Raise or remove the limit; don't profile code that's fine.
How do I tell which container a traced event belongs to?
Read the process's cgroup ID (or namespaces) — the cgroup path encodes the container/pod identity. BPF can capture the cgroup ID at the event and key results by it for per-container breakdowns. To get a friendly name, map the numeric cgroup ID back through the container runtime (Docker/Kubernetes).
How do I find a noisy neighbor?
From the host, where BPF attributes resource use per container. A container saturating disk, network, or CPU cache can degrade co-located ones whose own metrics look fine. The host view shows which container generates the contention — impossible to see from inside any single container.
Do the standard BPF tools work with containers?
Yes — run them on the host and they observe all containers at once (execsnoop, biolatency, runqlat, offcputime, etc.). The only added work is attributing events to specific containers via cgroup ID and watching for cgroup-imposed throttling that node-level metrics hide.
10 Key takeaways
- A container is processes isolated by namespaces (what they see) and cgroups (what they use).
- Trace from the host — one BPF session sees every container, no per-container agents.
- CPU throttling (cgroup quota) clamps a container while the node looks idle — the top container trap.
- Other cgroup limits (block I/O, memory) add latency invisible in host-wide metrics.
- The host view reveals noisy neighbors degrading co-located containers.
- The hard part is attributing events to a container — via cgroup ID, then mapped to names through the runtime.
- Standard BPF tools work across containers; suspect limits first for "slow app, idle node."
trace from the host (sees all)
CPU throttling (the top trap)
nr_throttled, throttled_usec — quota stalls.other limits
attribution (which container)
11 Wrapping up
Containers are processes in disguise, so the host is your vantage point and a single BPF session sees them all — with cgroup throttling the trap to check first and per-container attribution the friction to manage. One layer of virtualization remains, where even the host's view has limits. Next: Hypervisors.