eBPF Observability for Real-Time Database Troubleshooting

TL;DR — When a database is slow, the cause usually hides below the application: disk I/O, the network stack, the CPU run queue, or the kernel scheduler. eBPF lets you ask the kernel exactly what's happening — with nanosecond precision and near-zero overhead — without patching the kernel, loading a module, or restarting the DB. This is the one-stop guide: how eBPF actually works (verifier, JIT, maps, probe types), a repeatable troubleshooting methodology, then five tools in depth (biolatency, fileslower, tcpretrans, runqlat, profile) — each with exact commands, both healthy and unhealthy output, how to read the histogram, and the fix when it's red. Plus overhead/safety, continuous profiling, more tools to grow into, an FAQ, and a glossary. Real output throughout from a live PostgreSQL + OpenTelemetry box.

Why eBPF for database troubleshooting

Picture the 2 a.m. page. The application dashboard is red: p99 latency on checkout queries jumped from 8 ms to 600 ms. You open your usual tools. The query planner looks identical to yesterday. CPU sits at 30%. Memory is fine. Connections are nowhere near the pool limit. Everything says "healthy" — and yet the database is crawling. Where is the time going?

This is the single most common shape of a hard database incident: the slowness is real, but it lives in a layer your normal tools can't see. top, iostat, vmstat, and even most APM agents report averages over seconds. They are great at telling you that something is slow. They are nearly useless at telling you why, because the "why" is almost always a tail event — a single block I/O that took 80 ms, a TCP retransmit on the replication socket that cost a 200 ms timeout, a thread that sat 4 ms in the run queue waiting for a CPU that a noisy neighbour was hogging. Average those events into a one-second bucket and they vanish.

A database is, from the kernel's point of view, just a process that does an enormous amount of three things: disk I/O (reading pages, writing the WAL, checkpointing), network I/O (client connections, replication streams), and CPU + scheduling (executing queries, sorting, hashing). Every one of those operations passes through the Linux kernel. If you can watch the kernel do them, in real time, with per-event resolution, you can find the bottleneck directly instead of guessing. That is exactly what eBPF gives you.

What eBPF actually is

eBPF (extended Berkeley Packet Filter) lets you run small, sandboxed programs inside the Linux kernel without modifying kernel source code and without loading a kernel module. You write a tiny program, attach it to a kernel event (a function entry, a tracepoint, a timer tick), and the kernel runs your program every time that event fires. Your program collects data — a timestamp, a latency, a stack trace — and stashes it in a shared data structure that user space can read.

The name is a historical accident. The original BPF was a packet-filtering virtual machine from 1992 (the thing behind tcpdump). "Extended" BPF generalised that little VM so it can run programs attached to almost anything in the kernel, not just network packets. Today eBPF is the foundation of modern Linux observability, networking (Cilium), and security (Falco, Tetragon).

Why it's safe to run on production

The thing that makes eBPF special — and the reason you can run it on a live database server — is the combination of three guarantees:

The verifier. Before the kernel accepts your program, an in-kernel verifier proves it will terminate (no unbounded loops), never dereferences a bad pointer, and never reads uninitialised memory. A program that could crash the kernel is rejected at load time. This is why a buggy eBPF script gives you an error, not a kernel panic.
The JIT. Once verified, the program is compiled by a just-in-time compiler to native machine code, so it runs at roughly the speed of compiled C — not interpreted. The per-event cost is typically tens of nanoseconds.
Maps. eBPF programs can't allocate memory or call arbitrary kernel functions. They communicate through maps — kernel-managed key/value data structures (hash maps, histograms, per-CPU arrays). Your kernel-side probe writes to a map; your user-space tool reads it. This is how a histogram of disk latency gets out of the kernel and onto your screen.

Net effect: near-zero overhead and no restart. You attach to a running PostgreSQL, watch it, and detach — the database never knows you were there.

Probe types — where you can attach

You don't need all of these to start, but knowing the vocabulary makes every tool's source readable:

Probe type	Fires when…	Example
kprobe	a kernel function is entered	`kprobe:vfs_read`
kretprobe	a kernel function returns (lets you measure duration)	`kretprobe:vfs_read`
tracepoint	a stable, named kernel event fires (preferred — won't break across kernels)	`tracepoint:block:block_rq_complete`
uprobe / uretprobe	a user-space function runs (e.g. a PostgreSQL or libc function)	`uprobe:/usr/lib/postgresql/...:exec_simple_query`
USDT	a statically-defined trace point baked into an app binary	`usdt:...:query__start`
profile / interval	on a timer, every N Hz (for sampling)	`profile:hz:99`

kprobe vs tracepoint, why it matters later: kprobes attach to raw kernel function names, which the compiler can rename or inline between kernel versions — that's the source of the cannot attach kprobe errors you'll hit on newer kernels. Tracepoints are a stable kernel ABI and survive upgrades. When a BCC tool fails, the bpftrace fallback often uses a tracepoint instead.

Environment for this guide: Ubuntu 24.04 · bpfcc-tools 0.29.1 · bpftrace 0.20.2. Target server 10.100.43.121 running PostgreSQL + an OpenTelemetry collector. Traced June 3, 2026.

Two toolkits, one foundation

You almost never write raw eBPF bytecode. Two front-ends do the heavy lifting:

Toolkit	What it is	Best for
BCC (`bpfcc-tools`)	Python-based tools that compile C eBPF behind the scenes; rich, well-labelled histograms	interactive deep dives, ready-made tools
bpftrace	an awk-like one-line language for ad-hoc tracing scripts	fast one-shots, and the fallback when BCC can't attach

Rule of thumb: reach for the BCC tool first (nicer output), drop to bpftrace when BCC fails or when you need a custom one-liner the toolkit doesn't ship.

Fig 1 — Five probe points across the kernel, one per resource dimension: storage, files, network, scheduling, CPU.

A methodology, not random poking

Tools without a method waste time. The discipline that turns five commands into a diagnosis is Brendan Gregg's USE method: for every resource, check Utilization, Saturation, and Errors. "Saturation" is the key word — it's the queueing that happens when a resource is over-committed, and it's invisible to utilization-only tools. A CPU can be 40% utilized and still have threads waiting in the run queue because the work arrives in bursts. eBPF measures saturation directly.

Map the five tools onto the four resources a database leans on:

Resource	Utilization	Saturation	eBPF tool
Disk	busy %	I/O latency tail	`biolatency`, `fileslower`
Network	throughput	retransmits / drops	`tcpretrans`
CPU	cycles used	run-queue wait	`runqlat`, `profile`
Scheduler	—	off-CPU wait	`runqlat`, `offcputime`

The workflow for any incident:

Establish a baseline. Run each tool on a healthy box (or off-peak) so you know what "normal" looks like. The whole back half of this guide is a baseline you can borrow.
Sweep top to bottom. Disk → files → network → scheduling → CPU. Five minutes, five commands.
Follow the red. The first tool that shows a long tail or a non-zero error count is your lead. Drill in.
Confirm the fix. Apply one change, re-run the same tool, prove the tail shrank. Never change two things at once.

Installation & prerequisites

Ubuntu / Debian:

# BCC tools (biolatency, runqlat, profile, ...)
sudo apt update
sudo apt install -y bpfcc-tools python3-bpfcc
# bpftrace (tcpretrans.bt, runqlat.bt, ...)
sudo apt install -y bpftrace
# verify
biolatency-bpfcc --version
bpftrace --version

RHEL / CentOS / Amazon Linux:

sudo dnf install -y bcc-tools bpftrace
# tools land in /usr/share/bcc/tools/ — add to PATH:
export PATH=$PATH:/usr/share/bcc/tools

Verify kernel support:

uname -r                        # need 4.9+ for BCC, 4.17+ for bpftrace
ls /sys/kernel/debug/tracing     # debugfs must be mounted
# confirm BTF is present (makes modern tools portable across kernels):
ls /sys/kernel/btf/vmlinux

You need root (or CAP_BPF + CAP_PERFMON on newer kernels). On a managed/cloud host you usually have this on the VM but not inside an unprivileged container — trace from the host, or run the tracing container privileged with /sys mounted.

On Ubuntu 24.04 the BCC tools install with a -bpfcc suffix (biolatency-bpfcc, runqlat-bpfcc). On RHEL-family distros they're plain biolatency, runqlat under /usr/share/bcc/tools/. Adjust the command names to match your box.

01 · Disk I/O latency — `biolatency`

What it does: traces every block-device I/O request from issue to completion and buckets the durations into a power-of-two histogram (microseconds). It hooks the block layer (block_rq_issue / block_rq_complete tracepoints), so it sees the real device latency below the filesystem and page cache — exactly the number that matters for slow WAL flushes, slow checkpoints, and cold-page reads.

Why a database cares: PostgreSQL commits are gated on fsync of the WAL. If the device that holds pg_wal develops a latency tail — a failing SSD, a saturated EBS volume hitting its IOPS ceiling, a noisy multi-tenant SAN — every commit waits. Throughput collapses while CPU stays low, because the database is blocked on the disk, not busy.

# 5 intervals of 1 second each
sudo biolatency-bpfcc 1 5
# split by disk and by read/write flag:
sudo biolatency-bpfcc -D -F 1 5
# bpftrace equivalent (no -bpfcc suffix needed)
sudo bpftrace /usr/sbin/biolatency.bt

How to read it: each row is a latency bucket; the bar is how many I/Os landed there. You're reading the shape:

everything under 1 ms (1000 µs) → NVMe/SSD-class, excellent;
a fat band at 1–10 ms → ordinary SSD under load, usually fine;
a second hump at 50–100 ms+ → a latency tail. This is the killer. The average can still look okay while a few percent of I/Os take 100× longer, and those few are the ones blocking commits.

Healthy output from our server:

Tracing block device I/O... Hit Ctrl-C to end.
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
        ...
       128 -> 255        : 0        |                                        |
       256 -> 511        : 9        |****************************************|
       512 -> 1023       : 2        |********                                |

All I/O clustered in 256–1023 µs (0.25–1 ms). Zero operations crossed 1 ms. The bars are short — few operations total, so the disk is idle most of the time. Single, tight hump, no tail.

✅ Verdict: storage healthy — NVMe/SSD-class latency.

What an unhealthy result looks like (a saturated cloud volume):

     usecs               : count     distribution
       256 -> 511        : 410      |***********                             |
       512 -> 1023       : 980      |****************************************|
      1024 -> 2047       : 220      |********                                |
     ...
     65536 -> 131071     : 74       |***                                     |   <- 65-131 ms tail
    131072 -> 262143     : 31       |*                                       |   <- 131-262 ms tail

❌ A clear bimodal shape: a main hump at sub-ms plus a second cluster at 65–262 ms. That tail is what your p99 query latency is made of.

The fix path when disk is red:

Confirm it's the WAL device: re-run with -D to split by disk, and pair with fileslower (next section) to see which process/file.
Cloud: you've likely hit the volume's provisioned IOPS/throughput cap — bump the tier (gp3 IOPS, io2), or move pg_wal to a dedicated faster volume.
On-prem: check the device with smartctl for a dying SSD; check the RAID controller's battery-backed cache (a dead BBU silently disables write-back and tanks fsync latency).
Reduce the I/O: tune checkpoint_timeout / max_wal_size so checkpoints aren't bursty, and confirm full_page_writes behaviour after a checkpoint.

02 · Slow file operations — `fileslower`

What it does: surfaces individual file read/write operations slower than a threshold (default 10 ms), with the PID, command, and latency for each. Where biolatency tells you the device is slow, fileslower tells you which process and which operation ate the time — at the VFS layer, so it captures the full cost including any blocking, not just the raw device.

Why a database cares: this is the tool that catches slow WAL writes, slow journal flushes, and fsync stalls — the events that block a single commit and never show up in pg_stat_statements because the time is spent in the kernel, not in the query executor.

# BCC version: show file reads/writes slower than 10 ms
sudo timeout 15 fileslower-bpfcc 10

# bpftrace inline version (works on all kernels)
sudo timeout 15 bpftrace -e '
kprobe:vfs_read  { @s[tid]=nsecs; @c[tid]=comm; }
kretprobe:vfs_read /@s[tid]/ {
  $l=(nsecs-@s[tid])/1000000;
  if($l>10){ printf("PID:%-6d COMM:%-20s OP:READ   LAT:%d ms\n",pid,@c[tid],$l); }
  delete(@s[tid]); delete(@c[tid]);
}
kprobe:vfs_write { @s[tid]=nsecs; @c[tid]=comm; }
kretprobe:vfs_write /@s[tid]/ {
  $l=(nsecs-@s[tid])/1000000;
  if($l>10){ printf("PID:%-6d COMM:%-20s OP:WRITE  LAT:%d ms\n",pid,@c[tid],$l); }
  delete(@s[tid]); delete(@c[tid]);
}'

How to read it: what you're hunting for is a pattern of who and which op:

postgres / mysqld / mongod doing slow writes → WAL / redo-log / fsync bottleneck. Cross-check with biolatency.
postgres doing slow reads → cold pages being faulted from disk; could mean shared_buffers is too small or a sequential scan is thrashing the cache.
java / python with frequent slow reads → app reading a large config/log file, often on a hot path it shouldn't be.

Healthy output from our server: zero file operations exceeded 10 ms across the entire 15-second window. The only active processes were otelcol-contrib (lightweight metric reads) and the tracing tools themselves. postgres had no slow file ops at all.

What an unhealthy result looks like:

Tracing file ops slower than 10 ms...
PID:1843   COMM:postgres             OP:WRITE  LAT:72 ms
PID:1843   COMM:postgres             OP:WRITE  LAT:118 ms
PID:1850   COMM:postgres             OP:WRITE  LAT:64 ms

❌ Repeated multi-tens-of-ms writes from postgres → the WAL flush is stalling. Combined with a biolatency tail this is a storage problem; with a clean biolatency it points higher up — an overloaded filesystem, an fsync storm from too-frequent commits, or synchronous_commit hammering a slow device.

kernel gotchaOn kernels 5.15+ the BCC fileslower-bpfcc can fail with cannot attach kprobe because the compiler inlined the target function. The bpftrace inline version above is functionally identical and always works — it attaches to vfs_read/vfs_write directly and computes the delta itself.

✅ Verdict (our box): no slow file operations — filesystem responding instantly.

03 · TCP retransmissions — `tcpretrans`

What it does: hooks the kernel TCP stack and prints every retransmitted packet with source, destination, port, and TCP state. A retransmit means a packet was sent, not acknowledged in time, and sent again — each one costs at least one retransmission timeout (often 200 ms+), which is an eternity for a database.

Why a database cares: this is the go-to for the "random slowness" class of incidents. Replication lag, connection drops, and app-to-database query timeouts that appear intermittent are very often a small but steady rate of retransmits on a specific socket. A streaming replica that retransmits a few packets a minute will silently fall behind; a connection pool that sees retransmits on its query sockets will show sporadic 200 ms+ outliers that look like "the database hiccupped" but are really the network.

# bpftrace version (recommended)
sudo timeout 30 bpftrace /usr/sbin/tcpretrans.bt
# BCC version
sudo timeout 30 tcpretrans-bpfcc

Output format: TIME PID LADDR:LPORT RADDR:RPORT STATE. The state column is the diagnosis:

State	Meaning
`ESTABLISHED`	active connection dropping packets mid-stream — serious; this is replication/query traffic
`SYN_SENT`	can't even complete the handshake — firewall, dropped SYNs, or server overloaded/down
`CLOSE_WAIT` / `FIN_WAIT`	connection teardown issues — often app not closing sockets cleanly

Healthy output from our server: zero TCP retransmissions across the window. Every connection — SSH, database client traffic, telemetry — delivered packets on the first attempt. Clean RTT, no packet loss between this node and its peers.

What an unhealthy result looks like:

TIME     PID    LADDR:LPORT              RADDR:RPORT          STATE
04:12:09 1843   10.100.43.121:5432       10.100.43.140:51120  ESTABLISHED
04:12:09 1843   10.100.43.121:5432       10.100.43.140:51120  ESTABLISHED
04:12:11 1843   10.100.43.121:5432       10.100.43.155:43880  ESTABLISHED

❌ Retransmits on port 5432 (PostgreSQL) to replica/client IPs, in ESTABLISHED state → real packet loss on live database traffic. Next steps: check the NIC for errors (ethtool -S eth0 | grep -i err), check for a saturated link or a flapping switch port, and look at ss -ti on the affected socket for the retransmit count and RTT variance.

For production monitoring: run for 60–120 seconds during peak load, not 15. Retransmits are rare-but-costly, so a short window can miss them. Even 1–2 retransmissions per minute on a replication port warrants investigation — that's exactly where silent lag is born.

✅ Verdict (our box): network healthy — no retransmissions.

04 · Run-queue / scheduling latency — `runqlat`

What it does: measures how long threads wait in the CPU run queue after they're runnable but before the scheduler actually puts them on a CPU. This is pure saturation: it's the time your query spends ready-to-run but stuck waiting for a core. It buckets those waits into a microsecond histogram.

Why a database cares: this metric reveals the slowness that average CPU utilization hides completely. On shared/cloud infrastructure it's the direct fingerprint of a noisy neighbour — another tenant on the same physical host stealing cycles. A box can read 40% CPU and still have query threads waiting milliseconds to be scheduled because the work is bursty or because the hypervisor is overcommitted. runqlat is the only one of these tools that quantifies that wait.

# BCC version — 5 intervals of 1 second
sudo runqlat-bpfcc 1 5
# bpftrace version
sudo timeout 15 bpftrace /usr/sbin/runqlat.bt

How to read the histogram:

0–7 µs dominant → a CPU is almost always free the instant a thread wants to run. Healthy. ✅
100–1000 µs dominant → CPU moderately loaded; threads wait tens-to-hundreds of microseconds. Watch it.
1000 µs+ (1 ms+) present and growing → saturation. Threads wait a full millisecond or more for a core. Your tail latency is being manufactured here. ❌

Healthy output from our server (3 of 5 intervals):

     usecs               : count     distribution
         0 -> 1          : 126      |******************************          |
         2 -> 3          : 166      |****************************************|
         4 -> 7          : 18       |****                                    |
         8 -> 15         : 6        |*                                       |
        16 -> 31         : 1        |                                        |
        32 -> 63         : 1        |                                        |
     ... (interval 3 burst) ...
         0 -> 1          : 1082     |****************************************|
         2 -> 3          : 500      |******************                      |
         4 -> 7          : 31       |*                                       |

The overwhelming majority of scheduling delays sit at 0–3 µs. One lone outlier at 32–63 µs — negligible. The interval-3 spike to 1082 hits at 0–1 µs, meaning a burst of many short-lived threads all scheduled instantly (the OTel collector scraping metrics). Volume went up; wait time did not. That's the signature of healthy load.

✅ Verdict: no CPU saturation — scheduling is instant, no noisy neighbours.

What an unhealthy result looks like (noisy neighbour / overcommit):

     usecs               : count     distribution
         0 -> 1          : 90       |*****                                   |
       ...
      1024 -> 2047       : 540      |**********************                  |  <- 1-2 ms waits
      2048 -> 4095       : 980      |****************************************|  <- 2-4 ms waits
      4096 -> 8191       : 210      |********                                |

❌ The mass has shifted to the 1–4 ms buckets — threads routinely wait milliseconds for a CPU. On a VM this usually means the host is overcommitted; confirm with steal time (%st in top/mpstat). Fixes: move to a dedicated/larger instance, pin the DB to specific cores, raise its scheduling priority, or evict the noisy co-tenant workload.

05 · CPU flame profile — `profile`

What it does: samples the full stack (kernel + user) of every running thread at a fixed frequency — 99 Hz by convention. The odd number is deliberate: sampling at exactly 100 Hz can fall into lock-step with kernel timers that also fire at 100 Hz, biasing your samples; 99 Hz dodges that. Aggregate the samples and the stacks that appear most often are where CPU time is actually spent.

Why a database cares: when runqlat says the CPU is saturated, profile tells you by what. Is it your queries doing real work (sorts, hashes, decompression)? Spinlock contention inside the database? A regex in a trigger? TLS handshakes? The flame graph answers "where did the cycles go" with stack-level precision, and it's the input to the classic flame-graph SVG.

# profile a specific PID at 99 Hz for 30s
sudo profile-bpfcc -F 99 -p <PID> 30

# system-wide, bpftrace version
sudo timeout 15 bpftrace -e '
  profile:hz:99 { @[comm, kstack] = count(); }
  END { print(@); clear(@); }'

# find the busiest PID first
ps -eo pid,pcpu,comm --sort=-pcpu | head -5

Generate a flame graph (optional but worth it):

git clone https://github.com/brendangregg/FlameGraph
cd FlameGraph
sudo profile-bpfcc -F 99 -f 30 > out.stacks
./flamegraph.pl out.stacks > cpu_flame.svg
# open cpu_flame.svg in a browser; width = time spent

Healthy output from our server (top stacks by sample count):

@[postgres,  do_user_addr_fault -> exc_page_fault -> asm_exc_page_fault ]: 2
@[postgres,  zap_present_ptes -> zap_pte_range -> exit_mmap -> do_exit ]:   3
@[systemd, ]: 3
@[postgres, ]: 41
@[otelcol-contrib, ]: 44
@[swapper/1,  pv_native_safe_halt -> arch_cpu_idle -> cpuidle_idle_call ]: 1429
@[swapper/2,  ... ]: 1451
@[swapper/3,  ... ]: 1458

Process	Samples	What it means
`swapper/0–3`	5793 (97%)	all 4 CPUs idle — the "swapper" is the idle task; this many samples = doing nothing
`otelcol-contrib`	44 (0.7%)	OTel collector, light metric scraping
`postgres`	44 (0.7%)	page-fault on exit (process cleanup) + idle waits
`systemd`	3 (<0.1%)	background housekeeping

The postgres stack shows exit_mmap → do_exit — a short-lived worker process exiting, not a query executing. That's normal PostgreSQL per-connection-process behaviour. otelcol-contrib is the busiest "real" process, and it's the monitoring agent scraping this very box. The headline is that 97% of all samples are the idle task.

✅ Verdict: CPU almost completely idle — 97% of samples are idle/halt states.

What an unhealthy profile reveals: if instead the top stacks were dominated by postgres → ExecSort or postgres → LWLockAcquire → ..., you'd have your answer — a missing index forcing in-memory sorts, or lock contention on a hot buffer. The fix is then a database-level change (add the index, reduce contention, raise work_mem), and you re-profile to confirm the hot stack shrank.

Beyond the five — tools to grow into

The five above cover the common cases. As you go deeper, these are the next ones worth knowing — all ship with BCC/bpftrace:

Tool	What it shows	Database use
`offcputime`	where threads block (off-CPU) and for how long	find what a query waits on — locks, I/O, sleeps
`execsnoop`	every new process exec	spot unexpected fork storms, cron jobs, backup scripts
`opensnoop`	every file open with path + result	catch config/temp-file churn, permission errors
`tcpconnect` / `tcpaccept`	new outbound / inbound TCP connections	map who's connecting to the DB and how often
`tcplife`	connection lifespan + bytes	short-lived connection thrash → add pooling (PgBouncer)
`cachestat`	page-cache hit/miss rate	is the working set in RAM, or thrashing the disk?
`ext4slower` / `xfsslower`	slow filesystem ops by FS type	filesystem-specific version of `fileslower`
`runqlen`	run-queue length over time	complements `runqlat` — depth vs wait

Two database-specific tricks worth calling out:

USDT probes in Postgres/MySQL. If your build was compiled --with-dtrace, you can trace query start/end, transaction commit, and lock waits directly with usdt: probes — query latency from inside the engine, no DB-side logging overhead.
dbslower / dbstat (BCC). These attach to MySQL/PostgreSQL USDT or uprobes to give per-query latency histograms straight from eBPF.

Overhead, safety, and production etiquette

"Near-zero overhead" is the headline, but be precise about it so you can defend running this on a production primary:

Histogram tools are cheap. biolatency, runqlat, biosnoop-style aggregation runs in the low single-digit percent at most, because each event just does a map lookup and a counter bump in the kernel. Safe to run continuously.
Per-event printing tools cost more. Anything that prints a line per event (fileslower with a low threshold, execsnoop on a fork-heavy box) pushes data to user space per event. Keep thresholds sane and windows short.
Sampling scales with frequency. profile -F 99 is light; cranking to -F 9999 multiplies the cost 100×. 99 Hz is plenty for most diagnoses.
Symbol resolution. Stack traces need symbols. For stripped binaries or JITed runtimes (JVM, Node) you may see hex addresses — install debug symbols or use the runtime's perfmap. This affects readability, not safety.

Etiquette: trace in bounded windows (timeout 30 …) rather than leaving a tool running forever, prefer tracepoint-based tools on production (stable, won't break on kernel upgrade), and test any custom bpftrace one-liner on a staging box first — the verifier protects the kernel, but a too-chatty probe can still flood your terminal.

From firefighting to continuous profiling

Everything above is reactive — you run a tool when something breaks. The same eBPF foundation powers always-on profiling so the data is already there when the page fires. Tools like Parca, Pyroscope, and Pixie run eBPF samplers as agents, ship folded stacks to a backend, and give you a time-travel flame graph: "show me what postgres was doing at 04:12 last night." For a database fleet this is the endgame — you stop reproducing incidents and start replaying them. Start with the manual five tools to build intuition; graduate to continuous profiling once you know what the stacks mean.

Full performance summary (our server)

#	Tool	Command	Result	Status
01	Disk latency	`biolatency-bpfcc 1 5`	all I/O in 256–1023 µs	✅
02	Slow file ops	`bpftrace vfs_read/write >10ms`	zero slow ops	✅
03	TCP retransmits	`bpftrace tcpretrans.bt`	zero retransmits	✅
04	Scheduling delay	`runqlat-bpfcc 1 5`	0–3 µs dominant	✅
05	CPU profile	`bpftrace profile:hz:99`	97% idle	✅

Overall: the server is in excellent health — PostgreSQL + an OpenTelemetry collector with generous headroom across all four resource dimensions: storage, network, CPU, and scheduling. That's exactly the kind of clean baseline you want recorded before an incident, so the night it goes red you have something to compare against.

Quick reference — when to reach for which tool

Symptom	Tool
queries slow, disk IOPS high	`biolatency`
slow WAL writes, fsync delays	`fileslower 10`
replication lag, connection drops	`tcpretrans`
CPU at 100%, response time variable	`runqlat` then `profile`
unknown CPU consumer	`profile -F 99`
query waits but CPU is idle	`offcputime`
connection thrash / no pooling	`tcplife`
working set thrashing RAM	`cachestat`
want to know which file is slow	`fileslower 1` + name filter

Kernel compatibility notes

On Ubuntu 24.04 with kernel 6.8+, some BCC tools fail with cannot attach kprobe because certain kernel functions were inlined by the compiler and no longer exist as discrete symbols to hook. The fallback is always the bpftrace .bt script versions, which lean on stable tracepoints:

# if biolatency-bpfcc fails:
sudo bpftrace /usr/sbin/biolatency.bt
# if fileslower-bpfcc fails: use the inline bpftrace script from section 02
# if tcpretrans-bpfcc fails:
sudo bpftrace /usr/sbin/tcpretrans.bt
# if runqlat-bpfcc fails:
sudo bpftrace /usr/sbin/runqlat.bt

Compiler warnings like warning: multiple identical address spaces specified are harmless — they come from the BCC JIT compiler processing kernel headers and do not affect output accuracy. If you see them, your trace is still valid.

BTF & CO-RE: modern eBPF uses BTF (BPF Type Format) and CO-RE ("Compile Once, Run Everywhere") so a tool compiled on one kernel runs on another without re-compiling against local headers. If /sys/kernel/btf/vmlinux exists, you're on the happy path and most "kprobe not found" issues disappear with the newer libbpf-based tools.

FAQ

Does eBPF need a database restart? No. You attach to the running process and detach when done. The DB is never aware.

Is it safe on a production primary? Yes, for the histogram/sampling tools, in bounded windows. The verifier guarantees the kernel can't be crashed; the cost is single-digit percent at most. Be more careful with per-event printers at low thresholds.

Can I run it inside a container? The container needs the capabilities (CAP_BPF/CAP_PERFMON or privileged) and access to /sys. The easiest pattern is to trace from the host, which sees every container's processes anyway.

BCC or bpftrace — which should I learn first? Learn the BCC tools as ready-made instruments (better output), and learn enough bpftrace to read and write one-liners for when BCC can't attach or you need something custom.

Why 99 Hz and not 100? To avoid sampling in lock-step with kernel timers that fire at round frequencies, which would bias which stacks you catch.

It says cannot attach kprobe — broken? No — the function was inlined on your kernel. Use the bpftrace/tracepoint fallback for that tool.

How is this different from APM (Datadog, New Relic)? APM instruments your application and sees query spans. eBPF instruments the kernel and sees what's below the query — the disk, network, and scheduler time the APM span can't break down. They're complementary: APM says "this query is slow," eBPF says "because the WAL device has a 90 ms tail."

Takeaways

Trace below the app. When the DB is slow and the query plan is fine, the answer is in the kernel — eBPF is how you read it live, without a restart.
One tool per dimension, USE method as the map. biolatency = storage, fileslower = file/fsync, tcpretrans = network, runqlat = scheduling, profile = CPU. Sweep all five, get a full health picture in minutes.
Read the shape, hunt the tail. Averages lie; the second hump at high latency is your incident. Healthy = one tight hump near zero.
bpftrace is the safety net. When BCC can't attach a kprobe on a newer kernel, the .bt/tracepoint equivalents always work.
Baseline now, not during the fire. Record a clean run today so the night it goes red, you have something to diff against.
Graduate to continuous profiling. Parca/Pyroscope/Pixie turn these one-shots into always-on history — replay incidents instead of reproducing them.

References

BCC Tools Reference — the full toolkit + per-tool docs.
bpftrace Reference Guide — probe types and one-liners.
Brendan Gregg — Linux Performance — the canonical methodology (USE method).
BPF Performance Tools (book) — the deep reference for every tool here.
ebpf.io — what eBPF is, verifier/JIT/maps explained.

Extra reads

The Complete AMD EPYC "Turin" Tuning Guide — tune the box once eBPF finds the bottleneck.
Benchmark Your Own Workload — prove a fix actually helped.

Traced on a live Ubuntu 24.04 server (PostgreSQL + OpenTelemetry), June 2026. Tool names assume the -bpfcc suffix of Ubuntu/Debian packaging; on RHEL-family distros drop the suffix and look under /usr/share/bcc/tools/. Healthy output shown is from the real box; unhealthy examples are representative of the failure modes described. Always trace on a representative window during real load before drawing conclusions.

eBPF Observability for Real-Time, Kernel-Level Database Troubleshooting.