bpftrace Cheat Sheet

This page is a bpftrace cheat sheet that you can print out for reference, and is from the bpftrace programming section of my eBPF Tools page. There is another bpftrace cheat sheet as Appendix B in BPF Performance Tools: Linux System and Application Observability.

Syntax

probe[,probe,...] /filter/ { action }

The probe specifies what events to instrument, the filter is optional and can filter down the events based on a boolean expression, and the action is the mini program that runs.

Here's hello world:

# bpftrace -e 'BEGIN { printf("Hello eBPF!\n"); }'

The probe is BEGIN, a special probe that runs at the beginning of the program (like awk). There's no filter. The action is a printf() statement.

Now a real example:

# bpftrace -e 'kretprobe:vfs_read /pid == 181/ { @bytes = hist(retval); }'

This uses a kretprobe to instrument the return of the sys_read() kernel function. If the PID is 181, a special map variable @bytes is populated with a log2 histogram function with the return value retval of sys_read(). This produces a histogram of the returned read size for PID 181. Is your app doing lots of 1 byte reads? Maybe that can be optimized.

Probe Types

These are libraries of probes which are related. The currently supported types are (more will be added):

Alias	Type	Description
t	tracepoint	Kernel static instrumentation points
U	usdt	User-level statically defined tracing
k	kprobe	Kernel dynamic function instrumentation (standard)
kr	kretprobe	Kernel dynamic function return instrumentation (standard)
f	kfunc	Kernel dynamic function instrumentation (BPF based)
fr	kretfunc	Kernel dynamic function return instrumentation (BPF based)
u	uprobe	User-level dynamic function instrumentation
ur	uretprobe	User-level dynamic function return instrumentation
s	software	Kernel software-based events
h	hardware	Hardware counter-based instrumentation
w	watchpoint	Memory watchpoint events
p	profile	Timed sampling across all CPUs
i	interval	Timed reporting (from one CPU)
	iter	Iterator tracing over kernel objects
	BEGIN	Start of bpftrace
	END	End of bpftrace

Dynamic instrumentation lets you trace any software function in a running binary without restarting it. However, the functions it exposes are not considered a stable API, as they can change from one software version to another, breaking the bpftrace tools you develop. Try to use the static probe types wherever possible, as they are usually best effort stable.

Variable Types

Variable	Description
@name	global
@name[key]	hash
@name[tid]	thread-local
$name	scratch

Variables with a '@' prefix use BPF maps, which can behave like associative arrays. They can be populated in one of two ways:

• variable assignment: @name = x;
• function assignment: @name = hist(x);

There are various map-populating functions as builtins that provide quick ways to summarize data.

Builtin Variables

Variable	Description
pid	Process ID
tid	Thread ID
uid	User ID
username	Username
comm	Process or command name
curtask	Current task_struct as a u64
nsecs	Current time in nanoseconds
elapsed	Time in nanoseconds since bpftrace start
kstack	Kernel stack trace
ustack	User-level stack trace
arg0...argN	Function arguments
args	Tracepoint arguments
retval	Function return value
func	Function name
probe	Full probe name
$1...$N	Positional parameters
cgroup	Default cgroup v2 ID

Builtin Functions

Function	Description
printf("...")	Print formatted string
time("...")	Print formatted time
join(char *arr[])	Join array of strings with a space
str(char *s [, int length])	Return string from s pointer
buf(void *p [, int length])	Return a hexadecimal string from p pointer
strncmp(char *s1, char *s2, int length)	Compares two strings up to length
sizeof(expression)	Returns the size of the expression
kstack([limit])	Kernel stack trace up to limit frames
ustack([limit])	User-level stack trace up to limit frames
ksym(void *p)	Resolve kernel address to symbol
usym(void *p)	Resolve user-space address to symbol
kaddr(char *name)	Resolve kernel symbol name to address
uaddr(char *name)	Resolve user-space symbol name to address
ntop([int af,]int|char[4:16] addr)	Convert IP address data to text
reg(char *name)	Return register value
cgroupid(char *path)	Return cgroupid for /sys/fs/cgroup/... path
time("...")	Print formatted time
system("...")	Run shell command
cat(char *filename)	Print file content
signal(char[] sig | int sig)	Send a signal to the current task
override(u64 rc)	Override a kernel function return value
exit()	Exits bpftrace
@ = count()	Count events
@ = sum(x)	Sum the value
@ = hist(x)	Power-of-2 histogram for x
@ = lhist(x, min, max, step)	Linear histogram for x
@ = min(x)	Record the minimum value seen
@ = max(x)	Record the maximum value seen
@ = stats(x)	Return the count, average, and total for this value
delete(@x[key])	Delete the map element
clear(@x)	Delete all keys from the map

There are additional lesser-used functions and capabilities not summarized here. See the bpftrace Reference Guide.