Systems Performance Chapter 11 — Cloud Computing Explained: Noisy Neighbors, Steal, Limits

// the one-minute version

In the cloud you share hardware with strangers, which breaks two assumptions. First, a noisy neighbor can steal CPU, disk, or network from you — visible as CPU steal time (%st) but often otherwise invisible in your guest metrics. Second, you hit hard limits the provider imposes (IOPS caps, network caps, burst credits that deplete), so you saturate a ceiling that doesn't appear in top. Containers add cgroup limits (CPU throttling, memory OOM). The cloud rule: your guest can't see the host, so always suspect limits and neighbors before your own code.

Everything in the previous chapters assumed you owned the machine. The cloud quietly removes that assumption: your "server" is a slice of someone else's hardware, sharing CPUs, disks, and NICs with tenants you can't see, under limits you didn't set. So a new class of mystery appears — the app is slow, every guest metric looks fine, and the cause is entirely outside your virtual machine. This chapter is about diagnosing performance when you don't control, and can't fully see, the hardware underneath.

01 Multi-tenancy and the noisy neighbor

Cloud economics come from packing many tenants onto shared physical hosts. Most of the time isolation works, but resources like the last level of CPU cache, memory bandwidth, disk, and network are genuinely shared, and a greedy co-tenant — the noisy neighbor — can degrade your performance without touching your VM. Your CPU usage looks normal, your code is unchanged, yet latency rises, because someone else on the same box is saturating a resource you both depend on. This is the defining cloud performance problem: a cause that lives outside your visibility boundary.

key ideaIn the cloud, "my metrics all look fine but the app is slow" is not a paradox — it's the signature of a host-level cause (a noisy neighbor or a provider limit) that your guest simply cannot see. The first instinct should flip from "what did my code do?" to "what is the host or the limit doing to me?" That mindset shift saves hours of fruitless in-VM investigation.

02 CPU steal time

The one host-level signal that does leak into the guest. Steal time (%st in top/mpstat) is the percentage of time your virtual CPU was ready to run but the hypervisor gave the physical CPU to someone else. High steal means the host is oversubscribed — your VM is waiting for a CPU it nominally "has." If your app is slow and you see meaningful %st, the bottleneck is contention on the physical host, not your code. It's the cloud's most important and most overlooked metric, and it's right there in the standard tools if you know to look.

watch outSteal time is easy to miss because it sits in a column most people ignore. A VM at "50% CPU" with 30% steal is actually being starved — it wants more CPU but the host won't give it. Don't optimize your code when %st is high; the fix is a less crowded host (resize, migrate, or move to dedicated/burstable-with-headroom instances). Always scan the steal column before concluding anything about cloud CPU performance.

03 Hard limits and burst credits

Providers cap your resources, and you can saturate a cap while every "busy" metric looks calm. A volume rated for 3,000 IOPS will queue and add latency at 3,001 — but your disk %util might not scream it. Network has per-instance bandwidth and packet-per-second caps. Most insidiously, many instance and volume types use burst credits: you get high performance until a credit balance depletes, then you're throttled to a low baseline. An app that's fast for an hour then mysteriously slow has very likely exhausted its CPU or IOPS burst credits.

the catchBurst-credit throttling produces a textbook "it was fine, now it's slow, nothing changed" mystery — because from inside the guest, nothing did change. The CPU/disk wasn't upgraded or downgraded; your credit balance silently ran to zero and the provider clamped you to baseline. It's invisible in standard in-VM tools and only shows in the provider's own metrics (e.g. CPUCreditBalance, BurstBalance). When performance falls off a cliff after a period of load, check credit balances before anything inside the machine.

04 Hypervisors vs containers

Two virtualization models with different performance profiles. Hardware virtualization (hypervisors: KVM, Xen, etc.) gives each guest its own kernel and strong isolation, at the cost of some overhead and a hard visibility boundary (the guest can't see the host). OS virtualization (containers) shares the host kernel and isolates via namespaces and cgroups — far lighter, faster to start, denser, but isolation is weaker and "limits" are cgroup limits rather than virtual hardware. Many clouds layer them: containers inside VMs. Knowing which layer you're in tells you which limits and blind spots apply.

Fig 1 — VMs each run a kernel behind a hypervisor (strong isolation, host hidden); containers share the host kernel and are bounded by cgroups (light, weaker isolation).

05 The guest's observability blind spots

Inside a VM, your tools see your world, not the host's. You can't see the neighbor's workload, the host's true CPU contention (beyond steal), or how close you are to a provider limit. Even basic numbers can mislead: CPU counts and memory may be presented as if dedicated, while actually contended. The practical consequence is that in-guest analysis has a ceiling — past it you must consult provider-side metrics (CloudWatch, Stackdriver, Azure Monitor) for steal, credits, limit utilization, and host events. The most important cloud skill is knowing when the answer is outside the box.

06 Scaling: more nodes vs better nodes

The cloud's headline feature is elastic horizontal scaling — add instances under load, remove them after. It's powerful but not a cure-all. If each node is inefficient (a hot lock, a bad query, low IPC), scaling out just multiplies the waste and the bill. Autoscaling also reacts on a delay, so a sudden spike can overwhelm before new nodes warm up. And some bottlenecks are shared (a single database, a rate-limited dependency) that no amount of app nodes fixes. The discipline from earlier chapters still applies: make one node efficient first, then scale the efficient thing.

think of it likeHiring more cashiers (scaling out) speeds up a busy store — unless every cashier is slow because the price scanner is broken (per-node inefficiency), or there's one shared key to the register everyone waits for (a shared bottleneck). Then you've just paid more people to stand in the same jam. Fix the scanner and the shared key first; then more cashiers actually help.

07 Orchestration and overhead

Container orchestration (Kubernetes and friends) adds its own performance considerations. Scheduling decisions place your pod on a node that may already be busy; resource requests and limits you set become the cgroup CPU/memory bounds that throttle or OOM you (Chapters 6 and 7); overlay networks add hops and latency; sidecars and service meshes add per-request proxy overhead. None of this is wrong, but each layer is a place latency hides. When a containerized app is slow, the cause may be your limits, the node's contention, the network overlay, or the mesh — not the application at all.

08 A cloud analysis workflow

(1) Check steal time (%st in mpstat/top) — host CPU contention? (2) Check cgroup limits — CPU throttling (cpu.stat) and memory OOM (memory.events) from Chapters 6/7. (3) Compare against provider limits and credits (IOPS/network caps, burst balances) in the cloud console — saturating a ceiling or out of credits? (4) Only after ruling out host/limit causes, run the normal in-guest USE analysis on your own code. The cloud inverts the usual order: suspect the environment first, your application second.

common catches & gotchas

Ignoring steal time — High %st means the host starved your vCPU. Don't tune code; fix the crowding. Scan the steal column first.
Burst-credit cliffs — "Fast then suddenly slow, nothing changed" is usually depleted CPU/IOPS credits. Check the provider's credit-balance metric.
Hidden hard limits — You can saturate an IOPS or network cap while in-VM %util looks calm. Compare against the provider's documented limits.
Scaling out inefficiency — More nodes multiply a per-node waste and the bill. Make one node efficient before autoscaling.
Trusting only in-guest tools — The guest can't see the host or limits. Consult provider-side metrics for the full picture.
Forgetting the orchestration layers — cgroup limits, overlay networks, and service meshes each add latency. The app may be innocent.

09 Questions engineers actually ask

My app is slow but all my metrics look fine — what now?

That's the classic cloud signature of a host-level cause. Check CPU steal time (%st), cgroup throttling/OOM, and the provider's limit and burst-credit metrics. The bottleneck is likely a noisy neighbor or a hit ceiling outside your VM's visibility, not your code.

What is CPU steal time?

The percent of time your virtual CPU was runnable but the hypervisor gave the physical core to another tenant. High steal means the host is oversubscribed and starving your VM. It appears as %st in top/mpstat — the one host signal that reaches the guest.

Why was my instance fast for an hour then suddenly slow?

Almost certainly burst-credit exhaustion. Many instance/volume types give high performance until a credit balance depletes, then throttle to a low baseline. It's invisible inside the VM — check the provider's CPUCreditBalance/BurstBalance metric.

Should I scale up (bigger node) or out (more nodes)?

First make a single node efficient — scaling out an inefficient node just multiplies waste and cost. Scale out for genuinely parallelizable load with efficient nodes; scale up when a single instance is the unit of work or you're hitting per-instance limits. And watch for shared bottlenecks (one database) that neither fixes.

Containers or VMs for performance?

Containers are lighter (shared kernel, fast start, dense) but isolation is weaker and you're bounded by cgroup limits. VMs give stronger isolation at more overhead and a harder visibility boundary. Many stacks combine both. Pick by isolation needs and density; know which limits apply to whichever you're in.

10 Key takeaways

The cloud shares hardware — a noisy neighbor can degrade you invisibly from outside your VM.
Steal time (%st) is the key leaked signal: high steal = host CPU contention, not your code.
Hard limits (IOPS/network caps) and depleting burst credits cause "fine then suddenly slow" mysteries.
Containers add cgroup CPU throttling and memory OOM; VMs add a hard host-visibility boundary.
In-guest tools have a ceiling — consult provider-side metrics for steal, credits, and limit utilization.
Scaling out only helps with efficient nodes and no shared bottleneck — tune one node first.
Cloud inverts the order: suspect the environment and limits first, your application second.

// chapter cheatsheetcloud analysis

host contention (in-guest)

mpstat -P ALL 1 → %stealvCPU starved by the host — the key tell.

top → st fieldSteal time at a glance.

vmstat 1 → stSteal column in the CPU section.

container limits

cat /sys/fs/cgroup/cpu.statnr_throttled, throttled_usec — CPU quota stalls.

cat /sys/fs/cgroup/memory.eventsoom_kill — memory-limit kills (exit 137).

systemd-cgtopLive per-cgroup resource use.

provider-side (cloud console / CLI)

CPUCreditBalance / CPUCreditUsageBurstable CPU credits — cliff cause.

BurstBalance (EBS) / VolumeQueueLengthDisk credits + queueing at the cap.

NetworkPacketsIn/Out vs instance capHitting the network ceiling?

workflow

steal → cgroup limits → provider limits → codeSuspect environment first, app last.

11 Wrapping up

The cloud rewrites the first rule of analysis: before you suspect your own code, suspect the host and the limits, because half the causes now live outside your visibility. Steal time, burst credits, hidden caps, and cgroup bounds are the new usual suspects. With environments covered, the book turns to the activity that produces most of the misleading performance numbers in our industry — and how to do it right. Next: Benchmarking.

Systems Performance Chapter 11 — Cloud Computing.