← BPF Performance Tools

BOOK NOTES · BPF PERFORMANCE TOOLS · CHAPTER 12

BPF Performance Tools Chapter 12 — Languages.

bpf-performance-toolschapter-12languagesstacksusdtjit

// the one-minute version

How hard a language is to trace depends on how it runs. Compiled (C/C++, Go, Rust): straightforward — uprobes on functions, stacks work with frame pointers and symbols. JIT-compiled (Java, Node): code is generated at runtime, so you need a symbol map (e.g. perf-map-agent) to name those addresses, plus USDT probes the runtime exposes (GC, method events). Interpreted (Python, bash): the CPU runs the interpreter, not your code, so native stacks show interpreter frames — you need language-specific USDT/uprobe tools to see your functions. Fix symbols and stacks first; everything else follows.

BPF can trace any program — but "trace your code" means very different work depending on whether the language is compiled, JIT-ed, or interpreted. The wall everyone hits is the same: stacks full of hex addresses or interpreter internals instead of your function names. Solving that — symbols and stack walking, per language — is most of the battle. This chapter is the per-language playbook: what works out of the box, what needs a symbol map, and what needs the runtime's own probes.

01 Three ways code runs, three tracing problems

The tracing difficulty follows the execution model. Compiled languages produce native machine code with (ideally) symbols — BPF's uprobes and stack walkers work directly. JIT runtimes compile to native code at runtime, so the machine code exists but has no static symbol table — you must supply a map from address to method name. Interpreted languages never compile your code to native at all; the CPU executes the interpreter loop, so a native stack shows the interpreter's frames, not yours. Knowing which model you're in tells you exactly which problem you'll face.

key ideaThe CPU runs native machine code, and BPF stacks are native stacks. For compiled code that is your code (with symbols). For JIT code it's your code but unnamed (needs a symbol map). For interpreted code it's the interpreter, not your code (needs the runtime's own probes to recover your functions). The execution model dictates the tracing approach — match the technique to how the language reaches the CPU.

02 Compiled languages: C, C++, Go, Rust

The easy case. Native binaries can be traced with uprobes on any function, and stacks work — provided two things: frame pointers (so the stack can be walked) and symbols (so addresses resolve to names). The recurring pitfall (Chapter 2) is binaries built with -fomit-frame-pointer, which breaks walking; rebuild with frame pointers or use DWARF/LBR unwinding. Go has its own quirks (goroutines, its own stack management) but is broadly traceable. Once symbols and frame pointers are sorted, compiled-language profiling and function tracing with BPF works exactly as the CPU and application chapters described.

03 JIT languages: Java, Node.js

JIT runtimes compile bytecode to native code on the fly, so the methods are native — but their addresses aren't in any static symbol table, so a flame graph shows raw hex for your hottest methods. The fix is a symbol map: a helper (e.g. perf-map-agent for Java, Node's --perf-basic-prof) writes a /tmp/perf-PID.map file mapping JIT addresses to method names, which BPF/perf read to label stacks. With that in place you get full Java/Node flame graphs. You also need frame pointers preserved through the JIT (e.g. the JVM's -XX:+PreserveFramePointer) or stacks break — a JVM-specific gotcha.

Why your function names go missingcompilednative + symbols✓ stacks work(need frame pointers)JITnative, no symbols→ symbol map(perf-map-agent)interpretedCPU runs interpreter→ runtime USDT/ language tools

Fig 1 — Compiled code names resolve directly; JIT needs a symbol map; interpreted needs the runtime's own probes.

04 Interpreted languages: Python, Ruby, bash

The hardest case. Your Python function never becomes native code — the CPython interpreter executes bytecode in a big eval loop, so a native stack shows PyEval_EvalFrameEx and friends, not your functions. To see your code you need language-specific instrumentation: many runtimes ship USDT probes (Python's function__entry, Ruby's method probes) that fire on language-level events, and there are tools that walk the interpreter's own frame structures. bashreadline, for instance, traces shell commands by hooking bash's readline. Without these, BPF can still tell you the interpreter is hot, but not which of your functions — a partial answer.

05 The u* tools for runtimes

BCC ships a family of language-runtime tools (prefixed u) driven by USDT probes the runtimes expose. ucalls counts method calls (or summarizes their latency) by language. uflow traces method entry/exit as a flow (a call trace). ugc traces garbage-collection events and their duration — invaluable for correlating latency spikes with GC pauses (the runtime effect from Systems Performance). uobjnew tracks object allocation by type; uthreads traces thread creation; ustat summarizes runtime stats. They work across Java, Python, Ruby, Node, and more — wherever the USDT probes exist — giving language-aware visibility without per-language custom code.

06 GC: a common culprit

Managed runtimes reclaim memory automatically, and garbage collection pauses are a frequent, mysterious source of tail latency — a request that's normally 20 ms suddenly takes 300 ms because a stop-the-world GC ran. ugc traces GC events and durations, letting you confirm whether a latency spike lines up with a collection. This is the BPF way to test the GC hypothesis directly rather than guessing from runtime logs. Combined with allocation tracing (uobjnew), you can connect what allocates heavily to when the GC pauses — pointing at the code creating the garbage.

the catchThe single most common reason a language flame graph is useless is unresolved stacks — and the cause differs by language, which trips people up. Compiled: missing frame pointers (-fomit-frame-pointer). JIT: missing symbol map (no perf-map-agent / --perf-basic-prof) and/or the JIT not preserving frame pointers (e.g. without -XX:+PreserveFramePointer on the JVM). Interpreted: there are no native frames for your code at all — you need USDT/runtime tools, and a native profiler will never show your functions no matter what you fix. Diagnose by execution model: don't try to "fix symbols" on an interpreted runtime, and don't expect a JIT flame graph without a symbol map.

07 USDT probes and custom tracing

USDT (User Statically-Defined Tracing) probes are the bridge for non-native code. Many runtimes and applications ship them — stable, named events like "method entry," "GC start," "query executed" — that BPF can attach to with predictable semantics across versions (unlike uprobes on internal functions). List them with bpftrace -l 'usdt:/path/to/runtime:*' or tplist. When the u* tools don't cover your question, a bpftrace one-liner on the right USDT probe often does, giving language-level visibility (which method, which query) without grappling with the interpreter's internals or the JIT's symbol gaps.

08 A per-language workflow

(1) Identify the execution model — compiled, JIT, or interpreted. (2) Compiled: ensure frame pointers + symbols, then uprobe/profile as normal. (3) JIT: set up a symbol map (perf-map-agent / --perf-basic-prof) and preserve frame pointers in the runtime, then profile for flame graphs. (4) Interpreted: use USDT probes and the u* tools (or interpreter-aware tools) — don't expect native stacks to show your code. (5) For managed runtimes, check ugc for GC-pause latency. (6) Reach for USDT one-liners when the canned tools don't fit. Match technique to model; fix stacks first.

common catches & gotchas

  • Wrong fix for the model — Trying to "fix symbols" on an interpreted runtime is futile; you need USDT/runtime tools. Diagnose by execution model first.
  • JIT without a symbol map — Java/Node flame graphs show hex for your methods until you add perf-map-agent / --perf-basic-prof.
  • JIT without preserved frame pointers — Even with a symbol map, the JVM needs -XX:+PreserveFramePointer or stacks break.
  • Compiled with omitted frame pointers — Native stacks collapse; rebuild with frame pointers or use DWARF/LBR.
  • Expecting interpreter natives to be your code — They're the eval loop, not your functions. Use USDT/u* tools.
  • Missing USDT support — Some runtime builds ship without USDT probes; the u* tools then can't see language events. Check with tplist/bpftrace -l.

09 Questions engineers actually ask

Why does my Java/Node flame graph show hex instead of method names?

JIT runtimes compile methods to native code at runtime, so their addresses aren't in any static symbol table. Add a symbol map — perf-map-agent for Java, --perf-basic-prof for Node — which writes address-to-name mappings that BPF/perf read. Also ensure the JVM runs with -XX:+PreserveFramePointer so stacks walk.

Why does my Python profile show interpreter functions, not my code?

Because CPython runs your code as bytecode in an eval loop — the CPU executes the interpreter, so native stacks show interpreter frames. To see your functions, use Python's USDT probes or interpreter-aware tools (and the u* tools). A native profiler alone will never show your Python functions.

How do I confirm GC is causing latency spikes?

Run ugc, which traces garbage-collection events and durations for the runtime. Correlate its pause times with your latency spikes — if a 300 ms request lines up with a stop-the-world GC, you've confirmed the cause. Add uobjnew to find what's allocating the garbage.

Which languages are easiest to trace with BPF?

Compiled ones (C/C++, Go, Rust) — native code with symbols, traced directly by uprobes once frame pointers are present. JIT (Java/Node) needs a symbol map; interpreted (Python/Ruby/bash) needs runtime USDT probes. The execution model sets the difficulty.

What are the u* tools?

BCC's language-runtime tools driven by USDT probes: ucalls (method calls/latency), uflow (method flow), ugc (GC events), uobjnew (object allocation), uthreads (thread creation), ustat (runtime stats). They give language-aware visibility across Java, Python, Ruby, Node, etc., wherever USDT probes exist.

10 Key takeaways

  • Tracing difficulty follows the execution model — compiled, JIT, or interpreted.
  • Compiled: uprobe/profile directly, but need frame pointers + symbols.
  • JIT (Java/Node): need a symbol map (perf-map-agent / --perf-basic-prof) and preserved frame pointers.
  • Interpreted (Python/bash): native stacks show the interpreter, not your code — use USDT/runtime tools.
  • The u* tools (ucalls, uflow, ugc, uobjnew...) give language-aware visibility via USDT.
  • ugc confirms GC-pause latency — a common mysterious tail-latency cause.
  • Diagnose missing stacks by model; fix symbols/stacks before anything else.
// chapter cheatsheettracing languages

compiled (C/C++/Go/Rust)

build -fno-omit-frame-pointerMake native stacks walkable.
uprobe:/bin/app:func · profile -p PIDTrace functions / flame graph directly.

JIT (Java / Node)

java: perf-map-agent + -XX:+PreserveFramePointerSymbol map + walkable stacks.
node: --perf-basic-profWrite /tmp/perf-PID.map for symbols.

interpreted (Python/Ruby/bash)

bpftrace -l 'usdt:/path/to/python:*'List runtime USDT probes.
bashreadlineTrace shell commands via readline.

the u* runtime tools (USDT)

ucalls · uflowMethod call counts/latency; call flow.
ugcGC events + duration — confirm pause latency.
uobjnew · uthreads · ustatAllocations / threads / runtime stats.

diagnose by model

tplist -p PIDList USDT probes a process exposes.

11 Wrapping up

Tracing a language is mostly about recovering your function names from the way the code reaches the CPU: symbols and frame pointers for compiled, a symbol map for JIT, runtime USDT for interpreted — with ugc and friends giving language-aware insight. Get the stacks right and the rest of the toolkit applies. Next we move up to whole-application tracing — requests, threads, and locks. Next: Applications.

← prev: Chapter 11next: Chapter 13 →
© cvam — written in plaintext, served warm