// the one-minute version
Ftrace is the kernel's own tracer — built in, no packages, controlled through the tracefs filesystem at /sys/kernel/tracing. Its headline features: the function tracer (which kernel functions ran) and function_graph (a call tree with timing — great for "what's slow in the kernel"). It also enables tracepoints and can aggregate with hist triggers. You rarely poke tracefs by hand; front-ends like trace-cmd and Gregg's perf-tools wrap it. Low overhead, always available — the tracer to reach for when you can't install anything.
Sometimes you're on a locked-down box where you can't install BCC, can't build bpftrace, can't add a single package — and you still need to see inside the kernel. Ftrace is the answer. It has been in Linux for over a decade, lives entirely in the kernel, and is driven by reading and writing plain files. It's less glamorous than BPF, but it's always there, and for a class of questions — especially "which kernel function is taking the time" — it's the fastest path to an answer.
01 What Ftrace is
Ftrace (function tracer) is a tracing framework compiled into virtually every Linux kernel. It can trace kernel function execution, enable tracepoints and kprobes, measure latencies, and aggregate counts — all without any user-space agent. You control it by writing to and reading from files under /sys/kernel/tracing (the tracefs mount). Because it's part of the kernel, there's nothing to install and nothing to keep in sync with kernel versions — a major advantage on restricted production systems where adding software is hard or forbidden.
echo and cat. It won't do everything BPF does, but "the tool you can actually run right now" beats "the better tool you can't install" every time an incident is live.02 The tracefs interface
Everything is files. echo function > current_tracer turns on the function tracer; cat trace reads the output; echo 0 > tracing_on pauses; echo > trace clears the buffer. Key files: available_tracers (what's supported), set_ftrace_filter (limit to specific functions), set_ftrace_pid (limit to a process), and the events/ directory tree (every tracepoint, toggled by writing 1 to its enable). It's clunky by hand but completely scriptable, and understanding the file layout demystifies what the front-end tools are doing under the hood.
03 The function tracer
The simplest mode: record every kernel function as it's entered. Useful to see what the kernel is doing during an event, but on a busy system it produces a firehose — so you almost always filter (via set_ftrace_filter) to a subsystem or a few functions, and limit to a PID. Even filtered, the function tracer answers questions like "is the kernel actually calling into this driver?" or "what path does this syscall take?" that are otherwise hard to see. Pair it with filtering and a short capture window to keep the volume sane.
04 function_graph: the call tree with timing
The most loved Ftrace mode. function_graph traces both function entry and exit, so it can show a nested call tree with the duration of each call — like a flame graph in text, for the kernel. Output looks like indented C with microsecond timings beside each function, and a +/! marker highlights unusually slow calls. This is the go-to when a kernel operation is mysteriously slow: trace it with function_graph and the timing column shows exactly which nested call ate the milliseconds. It's drill-down (Chapter 2) applied inside the kernel.
Fig 1 — function_graph shows the kernel call tree with per-call durations. The slow leaf jumps out — drill-down without leaving the terminal.
05 Tracepoints and events via Ftrace
Ftrace is also the simplest way to switch on kernel tracepoints. Under events/ you'll find a directory per subsystem (sched/, block/, net/, syscalls/); write 1 to a tracepoint's enable file and its events stream into trace. Want to see every block I/O issued? Enable events/block/block_rq_issue. Each event carries useful fields (device, sector, bytes), and you can filter on them (e.g. only a particular PID or device). Since tracepoints are stable, these recipes keep working across kernel upgrades.
06 Hist triggers: in-kernel aggregation
A more advanced feature that closes the gap toward BPF. Histogram triggers let Ftrace aggregate event data in the kernel — counting, summing, or bucketing by a field — and emit a summary instead of raw events. For example, you can attach a hist trigger to a tracepoint to build a latency histogram or a count-by-key table without dumping millions of events to user space. It's not as flexible as bpftrace, but it brings the crucial "summarize in-kernel, stay low-overhead" idea to a tool that's already installed everywhere.
07 Front-end tools: trace-cmd and perf-tools
You rarely script tracefs by hand. Two wrappers make Ftrace humane. trace-cmd is the official CLI front-end: trace-cmd record -p function_graph -g some_func captures, and trace-cmd report reads it back — far nicer than echoing into files, with a GUI (KernelShark) for visualization. perf-tools (Brendan Gregg's shell scripts) are thin, dependency-free Ftrace wrappers for common tasks: funccount, funcgraph, funcslower, iolatency, tcpretrans, execsnoop. They're ideal on minimal systems — just shell and Ftrace, no compiler needed.
tracing_on in an odd state, your next trace gives confusing or empty results, and a forgotten function tracer can quietly add overhead system-wide. Always reset before and after: clear the buffer, set current_tracer to nop, and disable any events you enabled. The front-ends (trace-cmd, perf-tools) handle this cleanup for you — another reason to prefer them over raw tracefs.08 An Ftrace workflow
(1) Reach for Ftrace when you can't install BPF/BCC or need a quick kernel-function view. (2) For "which kernel function is slow," use function_graph on the suspect function (funcgraph from perf-tools, or trace-cmd) and read the timing column. (3) For "is this event happening and how often," enable the relevant tracepoint (or use funccount) and watch counts. (4) For latency distributions, use a hist trigger or a perf-tool like iolatency. (5) Always filter (function, PID) and time-box to control volume, and reset state when done.
common catches & gotchas
- Leftover global state — Ftrace is shared and persistent; a forgotten tracer/filter skews the next trace and can add overhead. Reset to
nopand clear when done. - Unfiltered function tracing — Tracing every kernel function on a busy box is a firehose with real cost. Filter to functions/PID and keep windows short.
- Reading raw tracefs by hand — Error-prone and forgets cleanup. Prefer trace-cmd or perf-tools, which manage state for you.
- Expecting BPF flexibility — Ftrace can't run arbitrary in-kernel programs; for complex custom logic use bpftrace (next chapter). Use Ftrace for its strengths: function tracing and timing.
- Buffer overruns — The trace buffer is fixed-size and wraps; long captures lose early events. Size the buffer or use trace-cmd to stream to disk.
- tracing_on left off — If a tool disabled tracing globally, you'll get empty output. Check
tracing_onfirst.
09 Questions engineers actually ask
When should I use Ftrace instead of BPF?
When you can't install anything (locked-down host, no build tools) — Ftrace is already in the kernel. Also when your question is simply "which kernel function ran / how long did it take," where function_graph is the most direct tool. For complex custom aggregation, prefer bpftrace.
What's the difference between the function and function_graph tracers?
The function tracer records each function entry — good for "what ran." function_graph records entry and exit, producing a nested call tree with per-call durations — ideal for "what's slow," since the timing column pinpoints the expensive nested call.
Do I have to echo into files manually?
No, and you shouldn't. Use trace-cmd (the official front-end) or Brendan Gregg's perf-tools shell scripts (funcgraph, funccount, funcslower, etc.). They wrap tracefs, handle filtering, and clean up state — far safer than raw echo/cat.
Is Ftrace safe in production?
Yes, with care. It's low-overhead when filtered, but unfiltered function tracing on a busy system is costly, and its global state can affect other users. Filter tightly, time-box captures, and reset afterward — or let trace-cmd/perf-tools manage it.
What are hist triggers?
Ftrace's in-kernel aggregation: attach to a tracepoint to count/sum/bucket by a field and emit a summary (e.g. a latency histogram) instead of raw events. It brings BPF's "aggregate in-kernel, stay cheap" idea to the built-in tracer, though with less flexibility.
10 Key takeaways
- Ftrace is the kernel's built-in tracer — always available, controlled via
tracefs, no packages needed. - The function tracer shows which kernel functions run; function_graph adds a timed call tree for "what's slow."
- It enables tracepoints via the
events/tree and aggregates with hist triggers. - Use front-ends — trace-cmd and perf-tools — instead of raw tracefs; they filter and clean up for you.
- Ftrace state is global and persistent — always filter, time-box, and reset to avoid skew and overhead.
- Reach for Ftrace when you can't install BPF or need a fast kernel-function/timing view.
via front-ends (preferred)
raw tracefs (under /sys/kernel/tracing)
always reset when done
11 Wrapping up
Ftrace is the unglamorous workhorse that's always on the box: function_graph for kernel timing, tracepoints by flipping a file, and front-ends to keep it sane. When you can't install anything, it's often the difference between solving an incident and waiting for permissions. For everything more programmable — custom logic, arbitrary aggregation, production-safe deep tracing — the final tool chapter delivers. Next: BPF.