BPF Performance Tools (book)

New tools developed for this book colored red.

Draft copy

This is the official site for the book BPF Performance Tools: Linux System and Application Observability, published by Addison Wesley (2019). This book can help you get the most out of your systems and applications, helping you improve performance, reduce costs, and solve software issues. Here I'll describe the book, link to related content, and list errata.

The book has been drafted and is available as an electronic rough cut on Safari (prior to final editing), and will be a physical book later in 2019. The publisher's page for it is on InformIT, and it is on Amazon.com. ISBN-13: 9780136554820. It is over 700 pages.

As an example new tool from the book, readahead.bt provides a new view of file system read ahead performance: the age of read-ahead pages when they are finally referenced, and unused read-ahead pages while tracing:

# readahead.bt
Attaching 5 probes...
Readahead unused pages: 128

Readahead used page age (ms):
[1]                 2455 |@@@@@@@@@@@@@@@                                     |
[2, 4)              8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)              4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@                         |
[8, 16)             7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     |
[16, 32)            4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[32, 64)               0 |                                                    |
[64, 128)              0 |                                                    |
[128, 256)           384 |@@                                                  |

The book covers many of the existing tools as well, for example, tcplife for efficiently logging TCP session details:

# tcplife
PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
4169  java   40158   6001      7    33 3590.91
4169  java   56940    6101      0     0 2.48
4169  java   6001    49482     0     0 17.94
4169  java   18926   6101      0     0 0.90
4169  java   44530    6001      0     0 2.64
4169  java   44406     6001     11    28 3982.11
34781 sshd   22    41566     5     7 2317.30

The book explains these and over 150 other BPF tools, as well as summarizing over 30 traditional performance analysis tools (top, vmstat, iostat, perf, Ftrace, etc) so that you can use the right tool for the job.

What is BPF?

Berkeley Packet Filter (BPF) is an in-kernel execution engine that processes a virtual instruction set, and has been extended recently (aka eBPF) for providing a safe way to extend kernel functionality. In some ways, eBPF does to the kernel what JavaScript does to websites: it allows all sorts of new applications to be created. BPF is now used for software defined networking, observability (this book), security enforcement, and more. The main front-ends for BPF performance tools are BCC and bpftrace.

Operating Systems

Extended BPF is a built-in Linux kernel technology, added in parts since 3.18. At least Linux 4.9 is necessary to utilize the tools in this book. All Linux distributions can use the BPF tools (Ubuntu, CentOS, Fedora, Red Hat, etc): although the status of BCC and bpftrace varies for each distribution. Some have packages, others still require a build from source. See the install instructions for BCC and bpftrace.

Other operating systems including BSD (where BPF originated) are not covered in this book. As extended BPF is being ported elsewhere, a future edition of this book may cover more than Linux.


This book is primarily for engineers, developers, and support staff in enterprise and cloud environments. No programming is required, unless you want to, as you can use this book as either:

This book is also useful for students as a way to learn system internals in an interactive way: you can run and develop tools to examine the workings of the system.


Over 150 BPF tools are covered in the book, for performance analysis, troubleshooting, and other uses (e.g., security forensics). These tools provide observability for CPUs, memory, disks, file systems, networking, languages, applications, containers, hypervisors, security, and the Linux kernel. To explain how to analyze different languages, three types of execution are studied: compiled, JIT-compiled, and interpreted, using C, Java, and the bash shell as examples. The same approaches can be applied to other languages, and a summary for Node.js, C++, and Golang are included.

To cover all these targets, many new tools needed to be developed for this book. The diagram on the top right shows these new tools colored red. The source to these is included in the book, and can also be found here:

The /originals directory contains an as-is snapshot of the published tools, and /updated contains those tools plus updated versions.

Table of Contents

Related Content


<will be placed here>

Thanks to all the reviewers, and to Deirdré Straughan for editing another one of my books!

Last updated: 20-Aug-2018