// the one-minute version
Network performance is dominated by latency (RTT), not bandwidth — most slow network apps are round-trip-bound, not throughput-bound. Each TCP connection costs a handshake (one RTT before any data), so connection reuse matters enormously. Packet loss triggers retransmits that crater throughput because TCP backs off. Throughput is capped by the bandwidth-delay product and socket buffer sizes. Watch retransmit rate, RTT, and per-connection state with ss and tcpdump/BPF — not just interface byte counts.
"Add more bandwidth" is the network equivalent of "add more CPU" — sometimes right, usually not the actual problem. Most network slowness is about round trips and loss, not pipe size: a chatty protocol over a 50 ms link is slow no matter how fat the pipe, and a 0.1% loss rate can halve throughput. This chapter builds the model of how packets actually move so you can diagnose real network latency instead of throwing bandwidth at it.
01 The stack and where time goes
Data travels down a layered stack — application, socket, TCP (or UDP), IP, the network device and driver — then across the wire and back up the far side. Each layer adds a little processing, but the dominant cost is usually the wire time: the round-trip between machines. On a local network an RTT might be 0.1 ms; across a continent, 50-80 ms; intercontinental, 150 ms+. Those are physical limits (speed of light in fiber), and no tuning beats them — which is why reducing round trips is the biggest network lever you have.
02 Latency vs bandwidth — and why RTT rules
Two different things people conflate. Bandwidth is the pipe's capacity (Gbps). Latency is how long one trip takes (RTT, ms). For bulk transfer, bandwidth matters; for the interactive request/response that defines most apps, latency rules. An app that makes 20 sequential round trips to render a page pays 20×RTT — on a 50 ms link that's a full second of pure waiting, independent of bandwidth. Worse, that cost is invisible in throughput graphs. The fix is fewer round trips: batch requests, pipeline, cache, keep connections open.
03 The TCP handshake and connection cost
Opening a TCP connection takes a three-way handshake (SYN, SYN-ACK, ACK) — a full round trip before any data is sent. Add TLS and it's more round trips for the crypto handshake. So every new connection costs at least one RTT of pure setup. Apps that open a fresh connection per request (no keep-alive, no pooling) pay that tax constantly, and on a high-latency link it dominates. Connection reuse (HTTP keep-alive, connection pools) and modern protocols (HTTP/2 multiplexing, TLS session resumption, TCP Fast Open) exist precisely to amortize or eliminate this cost.
Fig 1 — The TCP handshake spends a round trip on setup. New-connection-per-request multiplies this tax; keep-alive amortizes it.
04 Packet loss and retransmits
TCP guarantees delivery, so a lost packet must be retransmitted — and TCP interprets loss as congestion, so it also slows down (shrinks its congestion window). The double hit means even a tiny loss rate disproportionately wrecks throughput: 1% loss can cut a long-distance TCP flow's throughput by more than half. Retransmits are therefore one of the most important network health metrics. A rising retransmit rate points to congestion, a flaky link, an overloaded receiver, or a misbehaving middlebox — and it explains throughput that mysteriously falls short of the available bandwidth.
05 The bandwidth-delay product and buffers
To keep a pipe full, you must have enough data "in flight" to cover the round trip — that amount is the bandwidth-delay product (BDP = bandwidth × RTT). A 1 Gbps link with 50 ms RTT needs ~6 MB in flight to saturate. TCP can only have as much in flight as the smaller of the send and receive socket buffers (and the congestion window) allow — so if buffers are smaller than the BDP, you cannot fill the pipe no matter what. On high-latency, high-bandwidth ("long fat") links, undersized socket buffers are a classic, invisible throughput ceiling. Auto-tuning handles most cases, but limits and container defaults can still bite.
06 Congestion control and bufferbloat
Congestion control (Reno, CUBIC, BBR) is the algorithm TCP uses to find the right sending rate without overwhelming the network. The choice matters: BBR often outperforms loss-based algorithms on lossy or high-latency paths. The flip side is bufferbloat: oversized buffers in routers and devices that absorb bursts but add huge queueing latency — a download can balloon ping times from 20 ms to 500 ms, ruining interactive traffic sharing the link. Modern queue management (fq_codel, CAKE) fights bufferbloat by keeping queues short. It's the network's version of the "100% util but latency exploding" story.
07 NIC offloads, interrupts, and TIME_WAIT
A few system-level realities. Network cards offload work from the CPU — segmentation offload (GSO/TSO), checksum offload, and interrupt coalescing (batching interrupts so a busy NIC doesn't drown the CPU). On high-packet-rate boxes, per-packet interrupt and softirq cost shows up as system CPU, and techniques like RSS (spreading interrupts across CPUs) matter. On the connection side, after closing, TCP holds connections in TIME_WAIT for a couple of minutes to handle stray packets — and a server churning short connections can pile up tens of thousands of TIME_WAIT sockets, exhausting ports. The fix is again connection reuse, not (usually) the dangerous tcp_tw_recycle tweaks.
TIME_WAIT sockets looks alarming but is usually normal for a busy client making many short-lived connections — it's TCP correctly preventing old packets from corrupting new connections. The real problem it signals is the connection churn itself: you're paying a handshake per request. The fix is keep-alive / pooling to reuse connections, not aggressive sysctl hacks (like the long-removed tcp_tw_recycle) that broke connectivity behind NAT. Treat high TIME_WAIT as a symptom of churn, not a bug to suppress.08 A network analysis workflow
(1) Is it latency or throughput? Measure RTT (ping, ss -ti) and count round trips in the app's protocol. (2) Check retransmits (ss -ti, nstat, netstat -s) — loss explains shortfall vs link speed. (3) Per-connection state and buffers with ss -timep — undersized windows? stuck sends? (4) Interface saturation and errors with sar -n DEV/EDEV (drops, overruns). (5) For deep questions, trace the stack with BPF (tcplife, tcpretrans, tcpconnect) — connection lifetimes, retransmit sources, who's connecting. Start at latency/loss, descend to the stack only as needed.
common catches & gotchas
- Buying bandwidth for a latency problem — Round-trip-bound apps don't get faster with a wider pipe. Reduce round trips (batch, pipeline, keep-alive, cache).
- Ignoring retransmits — A tiny loss rate caps TCP throughput far below link speed. Check the retransmit rate before blaming capacity.
- Undersized socket buffers — On long-fat links, buffers smaller than the bandwidth-delay product make full throughput impossible. Verify auto-tuning isn't capped.
- New connection per request — Each pays a handshake RTT (plus TLS). Reuse connections; it's often the single biggest network win.
- Panicking at TIME_WAIT — Usually normal churn, not a bug. Fix the churn with pooling; avoid dangerous tw sysctls.
- Only watching interface bytes — Throughput graphs hide RTT, loss, and per-connection stalls. Use
ss -tifor the real per-flow story.
09 Questions engineers actually ask
Is my problem bandwidth or latency?
Measure RTT and count how many sequential round trips a request makes. If you're far from saturating the link but requests are slow, it's latency/round-trips — optimize by batching and reusing connections. If you're maxing the interface on bulk transfers, it's bandwidth (or loss capping it).
Why is throughput far below my link speed?
Usually packet loss (causing retransmits and TCP backoff) or socket buffers smaller than the bandwidth-delay product on a high-latency link. Check the retransmit rate (ss -ti/nstat) and the send/receive window sizes before assuming the link is too small.
I have thousands of TIME_WAIT sockets — is that bad?
Usually it's normal for a busy client opening many short connections — TCP holds them briefly to discard stray packets. The underlying issue is connection churn (a handshake per request). Fix it with keep-alive/pooling, not risky sysctl tweaks.
What is bufferbloat?
Oversized network buffers that absorb traffic bursts but add large queueing latency — a bulk download can spike interactive ping times from 20 ms to hundreds. Modern queue management (fq_codel, CAKE) keeps queues short to prevent it. It's the network's "100% busy but latency exploding" pattern.
Should I switch congestion control to BBR?
Often worth testing, especially on lossy or high-latency paths where loss-based algorithms (CUBIC) underperform. BBR models the path's bandwidth and RTT rather than treating every loss as congestion. Measure both with your real traffic — results vary by path.
10 Key takeaways
- Network performance is usually latency (RTT)-bound, not bandwidth-bound — reduce round trips first.
- Every new TCP connection costs a handshake RTT (more with TLS); reuse connections.
- Packet loss → retransmits → backoff; even small loss craters throughput. Watch the retransmit rate.
- Filling a pipe needs in-flight data ≥ the bandwidth-delay product; small socket buffers cap throughput.
- Congestion control choice (BBR vs CUBIC) and bufferbloat shape real-world latency.
- NIC offloads/interrupts affect CPU at high packet rates; TIME_WAIT usually means connection churn, not a bug.
- Diagnose with
ss -ti,nstat, and BPF (tcpretrans,tcplife) — not just interface byte counts.
latency & per-connection
loss & errors
throughput & interfaces
tracing the stack (BPF)
packets & tuning
11 Wrapping up
The network adds a dimension the local resources don't: physical round-trip latency you can't tune away, only design around. Count round trips, watch retransmits, reuse connections, and size buffers to the bandwidth-delay product. With all five core resources covered, the book shifts to the environment most systems now run in — where you share hardware with strangers and hit limits you can't see. Next: Cloud Computing.