Systems@Scale 2021: Performance Wins with BPF, Getting Started
Talk by Brendan Gregg for Facebook1s Systems@Scale 2021Video: https://www.facebook.com/atscaleevents/videos/536868690830345
Description: "BPF (eBPF) tracing is the superpower that can analyze everything, helping you find performance wins, troubleshoot software, and more. But with many different front-ends and languages, and years of evolution, finding the right starting point can be hard. This talk will make it easy, showing how to install and run selected BPF tools in the bcc and bpftrace open source projects for some quick wins. Think like a sysadmin, not like a programmer."
 | next prev 1/30 |
 | next prev 2/30 |
 | next prev 3/30 |
 | next prev 4/30 |
 | next prev 5/30 |
 | next prev 6/30 |
 | next prev 7/30 |
 | next prev 8/30 |
 | next prev 9/30 |
 | next prev 10/30 |
 | next prev 11/30 |
 | next prev 12/30 |
 | next prev 13/30 |
 | next prev 14/30 |
 | next prev 15/30 |
 | next prev 16/30 |
 | next prev 17/30 |
 | next prev 18/30 |
 | next prev 19/30 |
 | next prev 20/30 |
 | next prev 21/30 |
 | next prev 22/30 |
 | next prev 23/30 |
 | next prev 24/30 |
 | next prev 25/30 |
 | next prev 26/30 |
 | next prev 27/30 |
 | next prev 28/30 |
 | next prev 29/30 |
 | next prev 30/30 |
PDF: Scale2021_BPF_performance_getting_started_2021.pdf
Keywords (from pdftotext):
slide 1:
Performance Wins with BPF Getting Started Brendan Gregg Jun 2021slide 2:
This article is not for beginners BPF has evolved Many docs were true in 2015 but not today … not the best start Newcomers keep re-posting old info as new … out of date This talk is Jun 2021slide 3:
BPF is no longer an acronym BPF is a bytecode and execution environmentslide 4:
How to get quick and easy BPF performance winsslide 5:
Think like a sysadmin Not like a programmerslide 6:
Think like a sysadmin Get it installed everywhere and use it. # apt-get install bcc-tools # PATH=$PATH:/usr/share/bcc/tools # execsnoop # opensnoop # tcplife # ext4slower # biosnoop [...]slide 7:
Think like a sysadmin Get it installed everywhere and use it. # apt-get install bcc-tools # PATH=$PATH:/usr/share/bcc/tools # execsnoop Anything periodic running? crontab? # opensnoop Any misconfigurations? File not found? # tcplife # ext4slower Any unexpected TCP sessions? # biosnoop Any file system I/O slower than 10ms? [...] Any unusual disk access patterns? Outliers?slide 8:
Case Study: BCC biosnoop # iostat -xz 1 Linux 4.15.0-1052-aws (cass-xxx) […] avg-cpu: %user Device: rrqm/s xvda xvdb xvdc xvdd xvde xvdf xvdg md0 12/04/2019 _x86_64_ %nice %system %iowait %steal wrqm/s rkB/s r/s w/s (8 CPU) %idle wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util […]slide 9:
Case Study: BCC biosnoop, cont. # /usr/share/bcc/tools/biosnoop TIME(s) COMM PID perl biosnoop perl perl perl perl perl perl perl perl perl perl perl perl perl perl perl […] DISK xvdc xvda xvde xvde xvdb xvdc xvdb xvdg xvdd xvdg xvdg xvdg xvdg xvdc xvdc xvdg xvde T SECTOR R 610822184 R 269480 R 610737856 R 377704624 R 732825200 R 732953880 R 377707064 R 732998328 R 733127392 R 732868048 R 732906896 R 610744920 R 377706520 R 610951744 R 732858664 R 732937416 R 610853240 BYTES LAT(ms)slide 10:
Case Study: BCC biosnoop, cont. # ps -ef | grep perl root 3285 3274 root 7755 7748 root 11366 11359 root 15054 15049 root 19675 19670 root 23937 23930 root 27565 27561 root 28232 28223 root 31913 31907 […] 1 14:16 ? 1 04:16 ? 1 10:16 ? 2 16:16 ? 1 06:16 ? 1 12:16 ? 2 18:16 ? 1 02:16 ? 1 08:15 ? 00:04:24 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:10:20 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:06:39 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:03:07 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:08:53 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:05:30 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:00:28 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:11:43 /usr/bin/perl /apps/…ec2rotatelogs.pl 00:07:40 /usr/bin/perl /apps/…ec2rotatelogs.plslide 11:
Many more tools to try! bcc tools bpftrace tools (from my book, open source) Solve >gt;90% of perf issues with canned observability (tracing) tools This is BPF observability in one picslide 12:
Print on your office/home wall, use as a checklist Suspected disk issue? Try these first:slide 13:
Print on your office/home wall, use as a checklist Suspected disk issue? Try these first: Then walk up the stack: file system tools, VFS tools, syscall tools, etc.slide 14:
The future of BPF perf observability … is GUIs. The end user may not even know it’s BPF. Tool output, visualized This GUI is in development by Susie Xia, Netflixslide 15:
Example real-time BPF observability UI Client Web Server GUI GUI Webpage Server Instance agent Server bcc tools Application Instance agent bcc tools Application Netflix Vector (now retired) uses this model Instance agent pcp pmdabcc https://github.com/Netflix/vectorslide 16:
Example real-time BPF observability UI #2 Client GUI Server Instance agent Web Server GUI Webpage bpftrace tools Server bpftrace Application Instance agent bpftrace Application Netflix FlameCommander UI (not yet open source)slide 17:
Think like a sysadmin Please try to use bcc/bpftrace tools as-is and fetch regular updates Many tools are sandcastles, and require frequent rebuilding to match kernel changes Fortunately many companies and engineers maintain these versions Netflix Facebook bcc & bpftrace tools Isovalent (many more)slide 18:
Sysadmins sometimes program shell scripting awk sed one-liners bpftrace tools bpftrace one-linersslide 19:
Think like a programmer if You have a real-world problem that tools don’t solve You’re a BPF-based startup You’re debugging your own code* You’re doing networking/security/etc. You really want to learn BPF internals * JIT-ed runtimes like Java are currently complex to traceslide 20:
Recommended tracing front-ends I want to run some tools bcc, bpftrace I want to hack up some new tools bpftrace I want to spend weeks developing a BPF product bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs New, lightweight, CO-RE & BTF based Requires LLVM; becoming obsolete / special-use onlyslide 21:
Recommended tracing front-ends I want to run some tools Unix analogies bcc, bpftrace /usr/bin/* I want to hack up some new tools bpftrace bash, awk I want to spend weeks developing a BPF product bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs New, lightweight, CO-RE & BTF based C, C++ Requires LLVM; becoming obsolete / special-use onlyslide 22:
bpftrace example # readahead.bt Attaching 5 probes... Readahead unused pages: 128 Readahead used page age (ms): @age_ms: [1] 2455 |@@@@@@@@@@@@@@@ [2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ [8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ [32, 64) 0 | [64, 128) 0 | [128, 256) 384 |@@slide 23:
bpftrace example
#!/usr/local/bin/bpftrace
kprobe:__do_page_cache_readahead
{ @in_readahead[tid] = 1; }
kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
kretprobe:__page_cache_alloc
/@in_readahead[tid]/
@birth[retval] = nsecs;
@rapages++;
Fits on one slide!
kprobe:mark_page_accessed
/@birth[arg0]/
@age_ms = hist((nsecs - @birth[arg0]) / 1000000);
delete(@birth[arg0]);
@rapages--;
END
printf("\nReadahead unused pages: %d\n", @rapages);
printf("\nReadahead used page age (ms):\n");
print(@age_ms); clear(@age_ms);
clear(@birth); clear(@in_readahead); clear(@rapages);
slide 24:BCC libbpf tool example # ./opensnoop PID COMM 27974 opensnoop redis-server […] FD ERR PATH 0 /etc/localtime 0 /proc/1482/stat # ldd opensnoop linux-vdso.so.1 (0x00007ffddf3f1000) libelf.so.1 =>gt; /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f9fb7836000) libz.so.1 =>gt; /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9fb7619000) libc.so.6 =>gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fb7228000) /lib64/ld-linux-x86-64.so.2 (0x00007f9fb7c76000) # ls -lh opensnoop opensnoop.stripped -rwxr-xr-x 1 root root 645K Feb 28 23:18 opensnoop -rwxr-xr-x 1 root root 151K Feb 28 23:33 opensnoop.stripped 151 Kbytes for a stand-alone BPF program! (Note: A static bpftrace/BTF + scripts will also have a small average tool size)slide 25:
PSA CONFIG_DEBUG_INFO_BTF=y E.g., Ubuntu 20.10, Fedora 30, and RHEL 8.2 have itslide 26:
BPF Future: Event-based Applications User-mode Applications Kernel-mode Applications (BPF) U.E. Scheduler Kernel Kernel Events Hardware Events (incl. clock)slide 27:
A New Type of Software Execution User model defined Compilation Security Failure mode Resource access User task yes any user based abort syscall, fault Kernel task static none panic direct BPF event yes JIT, CO-RE verified, JIT error message restricted helpersslide 28:
Take Away To get started with BPF performance wins, think like a sysadmin: 1. Install BCC & bpftrace tools 2. Run them 3. Get some winsslide 29:
References This is also where I recommend you go to learn more: https://github.com/iovisor/bcc/blob/master/docs/tutorial.md https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md BPF Performance Tools, Addison Wesley 2020 Systems Performance 2nd Edition, Addison Wesley 2021 http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html http://www.brendangregg.com/ebpf.html https://ebpf.io/what-is-ebpfslide 30:
Thanks BPF: Alexei Starovoitov (Facebook), Daniel Borkmann (Isovalent), David S. Miller (Red Hat), Jakub Kicinski (Facebook), Yonghong Song (Facebook), Martin KaFai Lau (Facebook), John Fastabend (Isovalent), Quentin Monnet (Isovalent), Jesper Dangaard Brouer (Red Hat), Andrey Ignatov (Facebook), and Stanislav Fomichev (Google), Linus Torvalds, and many more in the BPF community BCC: Brenden Blanco (VMware), Yonghong Song, Sasha Goldsthein (Google), Teng Qin (Facebook), Paul Chaignon (Isovalent), Vicent Martí (PlanetScale), and many more in the BCC community bpftrace: Alastair Robertson (Yellowbrick Data), Dan Xu (Facebook), Bas Smit, Mary Marchini (Netflix), Masanori Misono, Jiri Olsa, Viktor Malík, Dale Hamel, Willian Gaspar, Augusto Mecking Caringi, and many more in the bpftrace community