Linux tracing and perf tools

In this 18min presentation, Brendan Gregg show us some great tools and resources for linux debugging, tracing, and profiling. He starts by introducing his own tools, all based in ftrace and perf, and then shows how BPF works and the front-end tools provided by BCC.  In this post I copy-pasted the tools’ descriptions and the full presentation is available at the LISA16 conference website.

Brendan Gregg‘s Perf-tools

A miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events (aka the “perf” command). Both ftrace and perf are core Linux tracing tools, included in the kernel source. Your system probably has ftrace already, and perf is often just a package add (see Prerequisites).

These tools are designed to be easy to install (fewest dependencies), provide advanced performance observability, and be simple to use: do one thing and do it well. This collection was created by Brendan Gregg (author of the DTraceToolkit).

Many of these tools employ workarounds so that functionality is possible on existing Linux kernels. Because of this, many tools have caveats (see man pages), and their implementation should be considered a placeholder until future kernel features, or new tracing subsystems, are added.

These are intended for Linux 3.2 and newer kernels. For Linux 2.6.x, see Warnings.

BPF Compiler Collection (BCC)

BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.

eBPF was described by Ingo Molnár as:

One of the more interesting features in this cycle is the ability to attach eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to kprobes. This allows user-defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively.

BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. It is suited for many tasks, including performance analysis and network traffic control.


A dynamic tracer for Linux that lets you:

  • Extract arbitrary data, i.e register values, function arguments, stack/heap data, stack traces.
  • Perform in-kernel aggregations on arbitrary data.

ply follows the Little Language approach of yore, compiling ply scripts into Linux BPF programs that are attached to kprobes and tracepoints in the kernel. The scripts have a C-like syntax, heavily inspired by dtrace(1) and by extension awk(1).

The primary goals of ply are:

  • Expose most of the BPF tracing feature-set in such a way that new scripts can be whipped up very quickly to test different hypotheses.
  • Keep dependencies to a minimum. Right now Flex and Bison are required at build-time, leaving libc as the only runtime dependency. Thus, ply is well suited for embedded targets.

For a more complete documentation and language reference, see



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s