BPF Performance Tools Chapter 10 — Networking Explained: tcplife, tcpretrans, tcpconnlat

// the one-minute version

BPF traces the network stack per-connection and per-event, far beyond interface byte counts. tcplife logs every connection's lifetime, bytes, and endpoints — one line per connection, low overhead. tcpconnlat shows connection (handshake) latency; tcpretrans traces each retransmit with its source — pinpointing packet loss. tcptop ranks connections by throughput; tcpconnect/tcpaccept watch new connections live. gethostlatency catches slow DNS. Use these to diagnose latency (RTT, handshakes, DNS) and loss — not just bandwidth.

Interface counters tell you bytes in and out. They can't tell you that connections are churning (a handshake per request), that 0.5% of packets retransmit and crater throughput, or that a DNS lookup stalls some requests for a second. BPF traces the TCP/IP stack at the event and connection level, so you see lifetimes, handshake latency, retransmits, and resolution stalls — the things that actually drive network latency. This chapter is the BPF networking toolkit, complementing the model from Systems Performance.

01 What BPF adds over interface counters

Traditional network tools show throughput and aggregate errors per interface. BPF traces the stack itself — sockets, TCP state changes, retransmits, the connect path — so it can attribute behavior to individual connections and processes: which process opened this connection, how long it lived, how many bytes it moved, how long its handshake took, whether it retransmitted. Because network performance is usually latency-bound (round trips), not bandwidth-bound, this per-connection, per-event view is exactly what you need — the byte counts rarely explain the slowness.

key ideaNetwork slowness is usually about round trips and loss, not pipe size — and those are invisible in interface throughput graphs. BPF surfaces them: connection lifetimes and handshake latency (round-trip cost), retransmits (loss), and DNS latency (a hidden round trip). When "the network is slow" but bandwidth looks fine, these per-event tools find the real cause where counters can't.

02 Connection lifetimes: tcplife

One of the most useful network tools. tcplife prints a single line per TCP connection when it closes: the process, source/destination addresses and ports, bytes sent and received, and the connection's duration. Because it summarizes per connection (not per packet), it's low-overhead and production-safe, yet it gives a rich picture: which services talk to which, how much data moves, and — crucially — connection churn. A flood of very short-lived connections is the signature of missing keep-alive (a handshake per request), one of the most common network inefficiencies, and tcplife makes it obvious.

03 Connection latency: tcpconnlat

Every new TCP connection costs a handshake — a round trip before any data (more with TLS). tcpconnlat measures that connection-establishment latency per connection, showing how long the handshake took to each destination. High connect latency points at a distant or overloaded endpoint, or network issues on the path. Combined with tcplife's churn view, it quantifies the cost of not reusing connections: if you're opening thousands of connections and each handshake is 50 ms, that's where your latency went. The fix — connection pooling/keep-alive — becomes obviously worth it.

04 Retransmits: tcpretrans

TCP retransmits lost packets and slows down (shrinking its window), so even a small loss rate disproportionately wrecks throughput. tcpretrans traces each retransmit as it happens, showing the connection and TCP state — pinpointing which connections suffer loss and roughly where. A steady stream of retransmits explains throughput that falls short of link speed (a loss-and-latency problem, not a capacity one). Because it traces the actual retransmit event, it's far more precise than an aggregate retransmit counter: you see the specific flows and can correlate with a destination, a path, or a time.

Fig 1 — Match the BPF tool to the network question: connections, handshake latency, loss, new-connection activity, or DNS.

05 Throughput and new connections: tcptop, tcpconnect, tcpaccept

More live views. tcptop is top for TCP — connections ranked by throughput right now, attributed to processes. tcpconnect traces outbound connection attempts (who's connecting where), and tcpaccept traces inbound accepts (who's connecting to you). These are invaluable for understanding traffic relationships and catching unexpected connections — a service reaching an endpoint it shouldn't, a burst of new inbound connections, a client opening far more connections than expected. Per-connection attribution turns "there's network traffic" into "this process is connecting to that host this many times."

06 DNS and resolution latency: gethostlatency

A frequently-missed source of latency. Name resolution (DNS) happens before many connections, and a slow or cache-missing lookup can stall a request for a second or more — invisible in TCP metrics because the connection hasn't even started. gethostlatency traces the resolver library calls (getaddrinfo, gethostbyname) and shows their latency per process. When some requests are intermittently slow and TCP looks clean, a DNS stall is a prime suspect — and gethostlatency catches it directly, often revealing a misconfigured resolver or an uncached lookup hitting a slow upstream.

the catchA huge amount of "mysterious" network latency lives outside TCP entirely — in DNS resolution that happens before the connection. The request is slow, tcplife and tcpretrans look perfectly clean (because the TCP part was fast), and every network dashboard is green — yet a name lookup quietly stalled for 1+ seconds on a cache miss. TCP-focused tools can't see it because it precedes the socket. When intermittent latency doesn't show up in connection or retransmit traces, check gethostlatency before assuming the network is fine — resolution is a hidden round trip that no TCP metric captures.

07 Socket and stack-level tracing

For deeper questions, BPF traces sockets and lower stack functions. Tools and one-liners cover socket I/O rates, TCP window and buffer behavior, SYN backlog (tcpsynbl — is the accept queue overflowing under connection floods?), and Nagle/delayed-ACK interactions. The kernel exposes stable TCP tracepoints (e.g. tcp:tcp_retransmit_skb) that the tools build on and you can trace directly. When a network question is bespoke — "what's the distribution of send sizes from this process," "how full is the SYN backlog" — bpftrace on the socket/TCP layer answers it where no canned tool exists.

08 A network analysis workflow

(1) Is it latency or throughput? Most often latency — check round trips. (2) tcplife for connection lifetimes and churn — lots of short connections means missing keep-alive. (3) tcpconnlat for handshake latency to quantify per-connection cost. (4) tcpretrans for loss — explains throughput below link speed. (5) gethostlatency for hidden DNS stalls when TCP looks clean. (6) tcptop/tcpconnect/tcpaccept to see who talks to whom. (7) Drop to socket/TCP tracepoints for bespoke questions. Per-connection and per-event, not just interface bytes.

common catches & gotchas

Watching only interface bytes — Throughput graphs hide round trips, loss, and per-connection stalls. Use tcplife/tcpretrans for the real per-flow story.
Missing DNS latency — Resolution precedes TCP, so TCP tools can't see it. Check gethostlatency when latency is intermittent and TCP looks clean.
Ignoring connection churn — A flood of short-lived connections (a handshake per request) is a top inefficiency. tcplife makes it obvious; the fix is keep-alive/pooling.
Aggregate retransmit counters — A total retransmit count doesn't say which flows. tcpretrans traces each one with its connection.
Per-event floods — Some tools stream per-event; on high-connection-rate systems filter and time-box. tcplife is per-connection (cheaper) by design.
kprobe fragility — Prefer stable TCP tracepoints over kprobes for custom tracing; re-verify after kernel upgrades.

09 Questions engineers actually ask

How do I see what connections my server is actually making?

Run tcplife for a per-connection log (process, endpoints, bytes, duration) as connections close, and tcpconnect/tcpaccept for live outbound/inbound connections. Together they show exactly who talks to whom, how much, and for how long — far more than interface counters.

My throughput is below link speed — how do I find out why?

Usually packet loss. Run tcpretrans to trace each retransmit with its connection — a steady stream means loss is throttling TCP (it backs off on loss). That explains a shortfall the bandwidth number can't. Combine with tcpconnlat if handshake latency is also a factor.

Some requests are randomly slow but TCP looks fine — what's going on?

Likely DNS. Name resolution happens before the connection, so a slow or cache-missing lookup stalls the request invisibly to TCP tools. Run gethostlatency to trace resolver-call latency per process — it often reveals a misconfigured resolver or uncached lookups hitting a slow upstream.

How do I spot connection churn?

tcplife shows each connection's duration. A flood of very short-lived connections means you're opening a new connection (and paying a handshake) per request — missing keep-alive or pooling. The fix is connection reuse, often the single biggest network win.

Is tcplife safe to run in production?

Yes — it summarizes per connection (one line at close), not per packet, so overhead is low even on busy servers. That's a deliberate design: rich per-connection visibility without the cost of per-packet tracing. Per-event tools like tcpretrans need more care on high-rate systems.

10 Key takeaways

BPF traces the network stack per-connection and per-event, far beyond interface byte counts.
Network slowness is usually round trips and loss, not bandwidth — exactly what these tools surface.
tcplife logs connection lifetimes/bytes (and reveals churn) cheaply; tcptop ranks throughput.
tcpconnlat measures handshake latency; tcpretrans traces each retransmit (loss).
tcpconnect/tcpaccept watch new connections; gethostlatency catches hidden DNS stalls.
Much "mystery" latency is DNS, invisible to TCP tools — check it when TCP looks clean.
Drop to stable TCP tracepoints for bespoke socket/stack questions.

// chapter cheatsheetnetworking with BPF

connections

tcplifePer-connection: process, endpoints, bytes, duration. Spot churn.

tcptoptop for TCP — connections by throughput.

tcpconnect · tcpacceptNew outbound / inbound connections, live.

latency & loss

tcpconnlatTCP handshake (connect) latency per connection.

tcpretransEach retransmit with connection + state — pinpoint loss.

tcpsynblSYN backlog — accept queue overflow under floods.

DNS (the hidden round trip)

gethostlatencyResolver-call latency — catch slow DNS.

custom (stable tracepoints)

tcp:tcp_retransmit_skbTrace retransmits in bpftrace.

-l 'tracepoint:tcp:*' / 'tracepoint:sock:*'List network tracepoints.

11 Wrapping up

For networking, BPF reveals what counters hide: connection lifetimes and churn, handshake latency, retransmit-level loss, and the DNS stalls that no TCP metric shows. Lead with tcplife, quantify handshakes and loss, and never forget to check resolution. The resource chapters end here; the book now turns to specialized domains, starting with using BPF for security observability. Next: Security.

BPF Performance Tools Chapter 10 — Networking.