// the one-minute version
bpftrace is an awk-like language for writing custom BPF tracers in one line. Structure: probe /filter/ { action } — where to attach, an optional condition, and what to do. Attach via probes (kprobe:, tracepoint:, uprobe:, profile:). Read context via builtins (pid, comm, nsecs, arg0, retval, kstack). Aggregate in-kernel with maps (@name) and functions like count(), hist(), sum(). BEGIN/END run setup/teardown. Learn this shape and you can answer almost any custom question in one line.
BCC gives you finished tools; bpftrace gives you a pen. When the question is bespoke — "what's the latency distribution of this specific function, filtered to that process, bucketed my way" — bpftrace answers it in a single line you type and discard. The language is deliberately small and awk-like, so the whole thing fits in your head. This chapter teaches that language: the four or five concepts that let you write any tracer on the spot.
01 The shape: probe, filter, action
Every bpftrace program is one or more blocks of the form probe /filter/ { action }. The probe says where to attach (which event). The optional filter (predicate, in / /) is a condition that must be true for the action to run — e.g. only for a certain PID. The action is what to do when the event fires and the filter passes — print something, update a map. That's the entire grammar. A one-liner is just this on the command line with -e; a script is the same blocks in a .bt file. Internalize this shape and the rest is vocabulary.
Fig 1 — The whole language in one line: attach to an event, optionally filter it, then act — usually aggregating into a map.
02 Probe types
The probe names the event source (Chapter 2's probe types, in bpftrace syntax). kprobe:func / kretprobe:func — entry/return of a kernel function. uprobe:/bin/app:func — a user-space function. tracepoint:category:event — a stable kernel tracepoint (preferred). usdt:/path:probe — an app's static probe. profile:hz:99 — timed sampling at 99 Hz (for flame graphs). interval:s:1 — fire every second (for periodic printing). software:/hardware: — perf software events and PMCs. You can use wildcards (kprobe:tcp_*) and list matches with -l.
03 Builtins: reading context
Inside an action, builtin variables give you the event's context for free. The common ones: pid, tid (process/thread ID), comm (process name), uid, nsecs (timestamp in nanoseconds), cpu, arg0..argN (the probed function's arguments), retval (a kretprobe's return value), and kstack/ustack (the kernel/user stack traces). These are the raw material of every tracer — you filter on them, print them, and key your maps by them. Knowing this handful covers the vast majority of one-liners.
04 Variables and maps
Two kinds of storage. Scratch variables ($x) are local to one probe action — handy for a temporary, like saving a start timestamp. Maps (@name) persist across events and aggregate — this is where the in-kernel summarization happens. A bare @ is a single global; @[key] is a hash keyed by anything (e.g. @[comm] per process, @[pid] per process, @start[tid] per thread for timing). The classic latency pattern: save a start time in @start[tid] on entry, compute nsecs - @start[tid] on return, feed it to a histogram. Maps are the heart of bpftrace.
@start[tid] = nsecs on a function's entry probe, then on its return probe compute the delta and aggregate it — @ns = hist(nsecs - @start[tid]); delete(@start[tid]);. That four-line shape measures the latency of any function or operation, custom-filtered however you like. Master it and you can build a latency tool for anything the kernel exposes.05 Aggregating functions
Maps become summaries through aggregating functions. count() — tally events (@[comm] = count() counts by process). sum(x), avg(x), min(x), max(x) — running totals/stats. hist(x) — a power-of-two histogram (great for latency). lhist(x, min, max, step) — a linear histogram with chosen buckets. These all run in the kernel, so you emit a compact summary, not a flood of events — the overhead win from Chapter 1. bpftrace auto-prints all populated maps when the program ends, so often you just populate a map and quit.
06 Actions and output
Beyond aggregating, actions can print. printf("...", args) formats a line per event (use sparingly — per-event output costs more). print(@map) emits a map on demand. time(), str(ptr) (read a string from a pointer), ksym(addr)/usym(addr) (resolve an address to a symbol), and delete(@map[key]) (free a map entry) round out the common toolkit. The discipline from earlier chapters applies: prefer aggregating into maps over printf-ing every event, so you stay cheap and readable on busy systems.
07 BEGIN, END, and structure
Two special probes bracket a program. BEGIN { ... } runs once at start — print a header, initialize a variable. END { ... } runs once at exit (Ctrl-C) — print final results, clear maps with clear(). A real script combines blocks: a BEGIN banner, one or more event probes that aggregate into maps, maybe an interval:s:1 block to print periodically, and an END to tidy up. This structure scales a one-liner into a readable multi-probe tool while staying far shorter than the equivalent BCC Python+C.
printf on a high-frequency probe — it feels natural ("just print each event") but on a busy system it floods your terminal, drops events when the output buffer overflows, and adds real overhead per event. bpftrace's whole advantage is in-kernel aggregation: a hist() or count() in a map ships one compact summary instead of millions of lines. Reserve printf for low-frequency events or short, filtered captures; for anything hot, aggregate into a map and let bpftrace print the summary at the end.08 From one-liner to tool: a workflow
(1) Identify the event — list probes with bpftrace -l 'tracepoint:syscalls:*' or wildcards. (2) Start with a count: -e 'tracepoint:... { @[comm] = count(); }' to confirm it fires and see who triggers it. (3) Add a filter to narrow (/comm == "nginx"/). (4) For latency, apply the per-thread timestamp pattern with hist(). (5) If it's useful and reusable, promote it to a .bt script with BEGIN/END — or formalize it as a BCC tool. Build up incrementally: count, filter, time, aggregate. That progression turns a vague question into a precise tracer in minutes.
common catches & gotchas
- printf on hot probes — Floods output, drops events, adds overhead. Aggregate into maps (
hist/count) and print the summary instead. - kprobe on a moving target — Dynamic probes break across kernel versions. Prefer
tracepoint:/usdt:when available; re-check after upgrades. - Forgetting to delete map entries — Per-thread timing maps (
@start[tid]) leak if you don'tdelete()them after use; the map grows unbounded. - Unfiltered wide wildcards —
kprobe:*attaches to a huge number of functions with real overhead. Narrow the wildcard. - Broken ustack — User stacks need frame pointers/symbols; otherwise
[unknown]. Same frame-pointer issue as Chapter 2. - Expecting it to print mid-run — Maps print at
ENDby default. Use aninterval:s:1 { print(@); }block for live output.
09 Questions engineers actually ask
What's the basic structure of a bpftrace program?
probe /filter/ { action } — the probe says where to attach, the optional filter is a condition, the action runs when the event fires and the filter passes. One-liners use -e on the command line; scripts put the same blocks in a .bt file. That single shape is the whole language.
How do I measure a function's latency?
Use the per-thread timestamp pattern: on the entry probe save @start[tid] = nsecs; on the return (kretprobe) compute @ = hist(nsecs - @start[tid]) and delete(@start[tid]). That gives a latency histogram for any function, filterable however you like.
What's the difference between $x and @x?
$x is a scratch variable, local to one probe action (e.g. a temporary). @x is a map — it persists across events and aggregates, and bpftrace auto-prints it at the end. Use scratch vars for temporaries, maps for anything you want to summarize or carry between events.
Why is my one-liner flooding the terminal?
You're probably printf-ing on a high-frequency probe. Switch to in-kernel aggregation — count() or hist() into a map — so bpftrace emits one summary instead of a line per event. Reserve printf for low-rate or tightly filtered events.
How do I find what probes are available?
bpftrace -l lists probes, with wildcards: bpftrace -l 'tracepoint:syscalls:*' or 'kprobe:tcp_*'. Prefer tracepoints (stable) over kprobes (version-fragile) when one exists for your question.
10 Key takeaways
- bpftrace is an awk-like language:
probe /filter/ { action }— where, when, what. - Probes:
kprobe/uprobe(dynamic),tracepoint/usdt(stable),profile/interval(timed). Prefer stable ones. - Builtins (
pid,comm,nsecs,arg0,retval,kstack) give event context for free. - Maps (
@name,@[key]) aggregate in-kernel; scratch vars ($x) are local temporaries. - The per-thread timestamp pattern +
hist()measures any function's latency. - Aggregate with
count()/sum()/hist()instead ofprintf-ing hot events — stay cheap. BEGIN/ENDbracket setup/teardown; build tools up incrementally: count → filter → time → aggregate.
structure & probes
builtins (context)
variables & aggregation
the latency pattern
structure blocks
11 Wrapping up
bpftrace fits in your head: one grammar, a handful of builtins, maps for aggregation, and the per-thread timestamp pattern for latency. With BCC for finished tools and bpftrace for custom one-liners, you now have the full toolkit — and the rest of the book is applying it, subsystem by subsystem. We start where most analysis begins, with the processor. Next: CPUs.