← BPF Performance Tools

BOOK NOTES · BPF PERFORMANCE TOOLS · CHAPTER 4

BPF Performance Tools Chapter 4 — BCC.

bpf-performance-toolschapter-4bcctoolstracingfunclatency

// the one-minute version

BCC (BPF Compiler Collection) is a framework plus a big library of ready-to-run BPF tools — you mostly just run them, no coding. Two flavors: single-purpose tools (biolatency, execsnoop, tcplife…) each answering one question well, and multi-tools (trace, argdist, funccount, funclatency, stackcount) that take arguments to trace whatever you point them at. Common options: -p PID to target a process, an interval to repeat, and histogram output. Learn a few dozen single-purpose tools plus the five multi-tools and you can answer most questions.

BCC is where BPF stops being a technology and becomes a toolbox you actually use. It ships dozens of production-grade tools, each the distilled result of an expert asking "how do I see X." You don't need to understand the BPF inside them to get enormous value — you need to know which tool answers which question, and the handful of flexible multi-tools that handle everything the single-purpose ones don't. This chapter is the map of that toolbox.

01 What BCC is

BCC is two things. Underneath, it's a framework: a way to write BPF programs in C with a Python (or Lua/C++) front-end that compiles, loads, and reads results. On top, and far more important to most users, it's a collection of finished tools — over a hundred ready-to-run programs covering CPU, memory, file systems, disk, network, and more. The book references these constantly. You can write your own BCC tools, but the day-to-day value is running the existing ones, which encode years of tracing expertise.

02 Installing and finding the tools

BCC installs from your distribution (apt install bpfcc-tools on Debian/Ubuntu, dnf install bcc-tools on RHEL/Fedora). A quirk to know: packaged tools are often named with a -bpfcc suffix (e.g. execsnoop-bpfcc) and live in /usr/share/bcc/tools. There's a man page and an example file for each tool — invaluable because every tool's output is slightly different. When you don't know which tool to use, browsing that tools directory (or Gregg's tool diagrams) by subsystem is the fastest way to find the right one.

key ideaThe biggest BCC skill isn't programming — it's knowing the catalog. There's almost always a single-purpose tool for your exact question, and finding it (by subsystem in the tools directory, the man pages, or Gregg's labeled diagrams) beats writing anything. Spend your first hour with BCC browsing what exists, not learning to author tools. Recognition is the superpower here.

03 Single-purpose tools

The bulk of BCC: each tool answers one question with polished, documented output. A sampler across subsystems — execsnoop (new processes), opensnoop (file opens), biolatency (disk I/O latency histogram), biosnoop (per-I/O detail), tcplife (TCP connection lifetimes), tcpretrans (retransmits), ext4slower (slow file ops), runqlat (run-queue latency), profile (CPU flame-graph samples), offcputime (off-CPU blocking). Each is reliable and example-backed. These are the tools the resource chapters (6-16) walk through one subsystem at a time.

04 The multi-tools

When no single-purpose tool fits, five flexible multi-tools let you trace arbitrary functions and events by argument — a step toward bpftrace without leaving BCC.

funccount

Count how often a function (or set, with wildcards) is called. "Is this code path even hit, and how often?" funccount 'vfs_*'.

funclatency

Histogram of how long a function takes. Instantly turns "is this function slow?" into a latency distribution.

trace

Print a custom line each time a function is called, with chosen arguments and a filter. Like a programmable, targeted strace.

argdist / stackcount

argdist summarizes a function's arguments or return values into a histogram; stackcount counts the stack traces leading to an event — great for "what's calling this?"

05 Common options and reading output

Most BCC tools share conventions worth memorizing. -p PID targets one process. A trailing interval (and optional count) makes it repeat — biolatency 1 10 prints a histogram every second, ten times. Many tools emit histograms with power-of-two buckets and a little ASCII bar chart — read the bucket ranges and the counts to see the distribution and its tail. Others emit a per-event line with a timestamp, PID, and details. Knowing whether a tool summarizes (histogram) or streams (per-event) tells you its overhead and how to read it.

BCC output styles: histogram vs per-eventhistogram (summary)usecs : count distribution4-7 : 812 |####8-15 : 5403 |########low overhead · see the tailper-event (stream)TIME PID COMM ...one line per occurrencerich detail · cost scaleswith event rate

Fig 1 — Histogram tools aggregate in-kernel (cheap); per-event tools stream each occurrence (detailed, costlier at high rates).

06 Writing your own BCC tool (briefly)

You rarely need to, but it helps to know the shape. A BCC tool is a Python script embedding a chunk of restricted C (the BPF program). The Python loads the C, attaches it to probes, and reads the maps to print output; the C runs in the kernel at each event. It's more verbose than bpftrace — which is why bpftrace exists for quick work — but BCC's Python front-end gives you full programming power for complex tools (argument parsing, rich formatting, multiple probes). The polished single-purpose tools are exactly this, refined.

07 BCC vs bpftrace

The natural question. Use BCC when a finished tool exists for your problem (run it — done) or when you're building a complex, reusable, well-formatted tool. Use bpftrace (next chapter) for quick, custom, throwaway one-liners where writing a Python+C BCC tool would be overkill. In practice you run BCC tools most of the time and drop to bpftrace for the occasional bespoke question. They're complementary, not competing — and many bpftrace one-liners are essentially prototypes of what a BCC tool would formalize.

the catchClassic BCC tools compile their embedded C at runtime, every time you run them — which means they need kernel headers (or LLVM/Clang) present, take a noticeable moment to start, and use memory to compile. On a tiny container or a header-less host, a BCC tool can fail to run for reasons that have nothing to do with your actual question. The modern fix is libbpf/CO-RE tools (Chapter 2's BTF) that compile once and run anywhere — many BCC tools now have libbpf versions. If a BCC tool won't start, suspect missing headers/LLVM before doubting BPF itself.

08 A BCC workflow

(1) Identify the subsystem from your triage (Chapter 3) — disk, CPU, file system, network. (2) Find the single-purpose tool for it in the tools directory / diagrams (biolatency for disk latency, runqlat for scheduler, etc.). (3) Run it with -p PID and an interval as needed; read the histogram or per-event output. (4) If no tool fits, reach for a multi-toolfunclatency for "is this function slow," funccount for "is this hit," stackcount for "what's calling this," trace for custom lines. (5) For anything still bespoke, switch to bpftrace. Recognition first, multi-tools second, custom code last.

common catches & gotchas

  • Not knowing the catalog — There's usually a single-purpose tool for your exact question. Browse the tools directory/diagrams before writing anything.
  • Runtime compile failures — Classic BCC compiles C at start and needs headers/LLVM. Missing them breaks the tool, not BPF. Prefer libbpf/CO-RE versions on minimal hosts.
  • The -bpfcc naming — Packaged tools may be named execsnoop-bpfcc and live in /usr/share/bcc/tools. Don't assume the bare name.
  • Per-event floods — Streaming tools (trace, biosnoop) cost more at high event rates. Filter (-p, predicates) and time-box.
  • Misreading histograms — Buckets are power-of-two ranges; the count is per bucket. The tail (rightmost buckets) is usually what matters, not the mode.
  • Forgetting the man/examples — Each tool's output differs; the _examples.txt and man page explain the columns. Read them when output is unclear.

09 Questions engineers actually ask

How do I find the right BCC tool?

Browse the tools directory (/usr/share/bcc/tools), the per-tool man pages and _examples.txt files, or Brendan Gregg's labeled tool diagrams that map tools to subsystems. There's almost always a single-purpose tool for your exact question — recognition beats authoring.

What are the multi-tools and when do I use them?

funccount (call counts), funclatency (function latency histogram), trace (custom per-call lines), argdist (argument/return histograms), and stackcount (count stacks to an event). Use them when no single-purpose tool fits but you can name the function or event to trace.

Why does a BCC tool take a second to start or fail to run?

Classic BCC compiles its embedded C at runtime, needing kernel headers or LLVM/Clang. That causes startup delay and, if they're missing (common on minimal containers), failures unrelated to your question. Use libbpf/CO-RE tool versions where available to avoid runtime compilation.

Should I use BCC or bpftrace?

Run BCC tools when a finished one exists, or build a complex reusable tool with BCC's Python front-end. Use bpftrace for quick custom one-liners where a full BCC script is overkill. You'll use BCC most of the time and bpftrace for bespoke questions.

How do I read the histogram output?

Each row is a power-of-two bucket (e.g. 8-15 usecs) with a count and an ASCII bar. Scan for where the mass is and especially the rightmost (slowest) buckets — the tail is usually the performance problem the average would hide.

10 Key takeaways

  • BCC is a framework plus a big library of ready-to-run BPF tools — mostly you just run them.
  • The key skill is knowing the catalog: there's usually a single-purpose tool for your exact question.
  • Single-purpose tools (biolatency, execsnoop, tcplife…) each answer one question well.
  • The multi-tools (funccount, funclatency, trace, argdist, stackcount) trace whatever you point them at.
  • Know the options: -p PID, intervals, and histogram vs per-event output (and its overhead).
  • Classic BCC compiles at runtime (needs headers/LLVM); prefer libbpf/CO-RE on minimal hosts.
  • Use BCC for finished/complex tools, bpftrace for quick custom one-liners.
// chapter cheatsheetBCC tools

install & find

apt install bpfcc-tools (or dnf bcc-tools)Install the tool library.
ls /usr/share/bcc/toolsBrowse the catalog by name.
man execsnoop · cat *_example.txtPer-tool docs + sample output.

single-purpose staples

execsnoop · opensnoopNew processes / file opens.
biolatency · biosnoopDisk latency histogram / per-I/O.
runqlat · profile · offcputimeSched latency / on-CPU / off-CPU.
tcplife · tcpretransConnection lifetimes / retransmits.

the five multi-tools

funccount 'vfs_*'Count calls to matching functions.
funclatency vfs_readHistogram of a function's duration.
trace 'do_sys_open "%s", arg2'Custom per-call line with args + filter.
argdist / stackcountArg/return histograms / count stacks to an event.

common options

-p PID · trailing interval [count]Target a process; repeat every N sec.

11 Wrapping up

BCC turns BPF into a hundred-plus ready answers plus five flexible multi-tools — and the winning move is recognizing which one fits, not writing code. Keep the order: single-purpose tool, then multi-tool, then bpftrace. For that last step — the quick, custom, throwaway question — you need the language built for it. Next: bpftrace.

← prev: Chapter 3next: Chapter 5 →
© cvam — written in plaintext, served warm