Systems Performance 2nd Ed.



BPF Performance Tools book

Recent posts:
Blog index
About
RSS

Systems@Scale 2021: Performance Wins with BPF, Getting Started

Talk by Brendan Gregg for Facebook1s Systems@Scale 2021

Video: https://www.facebook.com/atscaleevents/videos/536868690830345

Description: "BPF (eBPF) tracing is the superpower that can analyze everything, helping you find performance wins, troubleshoot software, and more. But with many different front-ends and languages, and years of evolution, finding the right starting point can be hard. This talk will make it easy, showing how to install and run selected BPF tools in the bcc and bpftrace open source projects for some quick wins. Think like a sysadmin, not like a programmer."

next
prev
1/30
next
prev
2/30
next
prev
3/30
next
prev
4/30
next
prev
5/30
next
prev
6/30
next
prev
7/30
next
prev
8/30
next
prev
9/30
next
prev
10/30
next
prev
11/30
next
prev
12/30
next
prev
13/30
next
prev
14/30
next
prev
15/30
next
prev
16/30
next
prev
17/30
next
prev
18/30
next
prev
19/30
next
prev
20/30
next
prev
21/30
next
prev
22/30
next
prev
23/30
next
prev
24/30
next
prev
25/30
next
prev
26/30
next
prev
27/30
next
prev
28/30
next
prev
29/30
next
prev
30/30

PDF: Scale2021_BPF_performance_getting_started_2021.pdf

Keywords (from pdftotext):

slide 1:
    Performance Wins with BPF
    Getting Started
    Brendan Gregg
    Jun 2021
    
slide 2:
    This article is
    not for beginners
    BPF has evolved
    Many docs were true in 2015
    but not today
    … not the best start
    Newcomers keep re-posting
    old info as new
    … out of date
    This talk is Jun 2021
    
slide 3:
    BPF is no longer an
    acronym
    BPF is a bytecode and
    execution environment
    
slide 4:
    How to get quick and easy
    BPF performance wins
    
slide 5:
    Think like a
    sysadmin
    Not like a
    programmer
    
slide 6:
    Think like a sysadmin
    Get it installed everywhere and use it.
    # apt-get install bcc-tools
    # PATH=$PATH:/usr/share/bcc/tools
    # execsnoop
    # opensnoop
    # tcplife
    # ext4slower
    # biosnoop
    [...]
    
slide 7:
    Think like a sysadmin
    Get it installed everywhere and use it.
    # apt-get install bcc-tools
    # PATH=$PATH:/usr/share/bcc/tools
    # execsnoop
    Anything periodic running? crontab?
    # opensnoop
    Any misconfigurations? File not found?
    # tcplife
    # ext4slower
    Any unexpected TCP sessions?
    # biosnoop
    Any file system I/O slower than 10ms?
    [...]
    Any unusual disk access patterns? Outliers?
    
slide 8:
    Case Study: BCC biosnoop
    # iostat -xz 1
    Linux 4.15.0-1052-aws (cass-xxx)
    […]
    avg-cpu:
    %user
    Device: rrqm/s
    xvda
    xvdb
    xvdc
    xvdd
    xvde
    xvdf
    xvdg
    md0
    12/04/2019 _x86_64_
    %nice %system %iowait
    %steal
    wrqm/s
    rkB/s
    r/s
    w/s
    (8 CPU)
    %idle
    wkB/s avgrq-sz avgqu-sz
    await r_await w_await
    svctm
    %util
    […]
    
slide 9:
    Case Study: BCC biosnoop, cont.
    # /usr/share/bcc/tools/biosnoop
    TIME(s)
    COMM
    PID
    perl
    biosnoop
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    […]
    DISK
    xvdc
    xvda
    xvde
    xvde
    xvdb
    xvdc
    xvdb
    xvdg
    xvdd
    xvdg
    xvdg
    xvdg
    xvdg
    xvdc
    xvdc
    xvdg
    xvde
    T SECTOR
    R 610822184
    R 269480
    R 610737856
    R 377704624
    R 732825200
    R 732953880
    R 377707064
    R 732998328
    R 733127392
    R 732868048
    R 732906896
    R 610744920
    R 377706520
    R 610951744
    R 732858664
    R 732937416
    R 610853240
    BYTES
    LAT(ms)
    
slide 10:
    Case Study: BCC biosnoop, cont.
    # ps -ef | grep perl
    root
    3285 3274
    root
    7755 7748
    root
    11366 11359
    root
    15054 15049
    root
    19675 19670
    root
    23937 23930
    root
    27565 27561
    root
    28232 28223
    root
    31913 31907
    […]
    1 14:16 ?
    1 04:16 ?
    1 10:16 ?
    2 16:16 ?
    1 06:16 ?
    1 12:16 ?
    2 18:16 ?
    1 02:16 ?
    1 08:15 ?
    00:04:24 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:10:20 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:06:39 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:03:07 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:08:53 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:05:30 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:00:28 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:11:43 /usr/bin/perl /apps/…ec2rotatelogs.pl
    00:07:40 /usr/bin/perl /apps/…ec2rotatelogs.pl
    
slide 11:
    Many more
    tools to try!
    bcc tools
    bpftrace tools
    (from my book, open source)
    Solve >gt;90% of perf
    issues with canned
    observability
    (tracing) tools
    This is BPF
    observability
    in one pic
    
slide 12:
    Print on your
    office/home
    wall, use as a
    checklist
    Suspected disk issue?
    Try these first:
    
slide 13:
    Print on your
    office/home
    wall, use as a
    checklist
    Suspected disk issue?
    Try these first:
    Then walk up the
    stack: file system tools,
    VFS tools, syscall tools,
    etc.
    
slide 14:
    The future of BPF perf observability
    … is GUIs. The end user may not even know it’s BPF.
    Tool output, visualized
    This GUI is in development by Susie Xia, Netflix
    
slide 15:
    Example real-time BPF observability UI
    Client
    Web Server
    GUI
    GUI
    Webpage
    Server
    Instance
    agent
    Server
    bcc
    tools
    Application
    Instance
    agent
    bcc
    tools
    Application
    Netflix Vector (now retired)
    uses this model
    Instance agent pcp pmdabcc
    https://github.com/Netflix/vector
    
slide 16:
    Example real-time BPF observability UI #2
    Client
    GUI
    Server
    Instance
    agent
    Web Server
    GUI
    Webpage
    bpftrace tools
    Server
    bpftrace
    Application
    Instance
    agent
    bpftrace
    Application
    Netflix FlameCommander UI
    (not yet open source)
    
slide 17:
    Think like a sysadmin
    Please try to use bcc/bpftrace tools as-is and fetch regular updates
    Many tools are sandcastles, and require frequent rebuilding to match kernel changes
    Fortunately many companies and engineers maintain these versions
    Netflix
    Facebook
    bcc & bpftrace
    tools
    Isovalent
    (many more)
    
slide 18:
    Sysadmins sometimes program
    shell scripting
    awk
    sed one-liners
    bpftrace tools
    bpftrace one-liners
    
slide 19:
    Think like a programmer if
    You have a real-world problem that tools don’t solve
    You’re a BPF-based startup
    You’re debugging your own code*
    You’re doing networking/security/etc.
    You really want to learn BPF internals
    * JIT-ed runtimes like Java are currently complex to trace
    
slide 20:
    Recommended tracing front-ends
    I want to run some tools
    bcc, bpftrace
    I want to hack up some new tools
    bpftrace
    I want to spend weeks developing a BPF product
    bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs
    New, lightweight,
    CO-RE & BTF based
    Requires LLVM; becoming
    obsolete / special-use only
    
slide 21:
    Recommended tracing front-ends
    I want to run some tools
    Unix analogies
    bcc, bpftrace
    /usr/bin/*
    I want to hack up some new tools
    bpftrace
    bash, awk
    I want to spend weeks developing a BPF product
    bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs
    New, lightweight,
    CO-RE & BTF based
    C, C++
    Requires LLVM; becoming
    obsolete / special-use only
    
slide 22:
    bpftrace example
    # readahead.bt
    Attaching 5 probes...
    Readahead unused pages: 128
    Readahead used page age (ms):
    @age_ms:
    [1]
    2455 |@@@@@@@@@@@@@@@
    [2, 4)
    8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
    [4, 8)
    4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@
    [8, 16)
    7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    [16, 32)
    4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@
    [32, 64)
    0 |
    [64, 128)
    0 |
    [128, 256)
    384 |@@
    
slide 23:
    bpftrace example
    #!/usr/local/bin/bpftrace
    kprobe:__do_page_cache_readahead
    { @in_readahead[tid] = 1; }
    kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
    kretprobe:__page_cache_alloc
    /@in_readahead[tid]/
    @birth[retval] = nsecs;
    @rapages++;
    Fits on one slide!
    kprobe:mark_page_accessed
    /@birth[arg0]/
    @age_ms = hist((nsecs - @birth[arg0]) / 1000000);
    delete(@birth[arg0]);
    @rapages--;
    END
    printf("\nReadahead unused pages: %d\n", @rapages);
    printf("\nReadahead used page age (ms):\n");
    print(@age_ms); clear(@age_ms);
    clear(@birth); clear(@in_readahead); clear(@rapages);
    
slide 24:
    BCC libbpf tool example
    # ./opensnoop
    PID
    COMM
    27974 opensnoop
    redis-server
    […]
    FD ERR PATH
    0 /etc/localtime
    0 /proc/1482/stat
    # ldd opensnoop
    linux-vdso.so.1 (0x00007ffddf3f1000)
    libelf.so.1 =>gt; /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f9fb7836000)
    libz.so.1 =>gt; /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9fb7619000)
    libc.so.6 =>gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fb7228000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9fb7c76000)
    # ls -lh opensnoop opensnoop.stripped
    -rwxr-xr-x 1 root root 645K Feb 28 23:18 opensnoop
    -rwxr-xr-x 1 root root 151K Feb 28 23:33 opensnoop.stripped
    151 Kbytes for a stand-alone BPF program!
    (Note: A static bpftrace/BTF + scripts will also have a small average tool size)
    
slide 25:
    PSA
    CONFIG_DEBUG_INFO_BTF=y
    E.g., Ubuntu 20.10, Fedora 30, and RHEL 8.2 have it
    
slide 26:
    BPF Future: Event-based Applications
    User-mode
    Applications
    Kernel-mode
    Applications (BPF)
    U.E.
    Scheduler
    Kernel
    Kernel
    Events
    Hardware Events (incl. clock)
    
slide 27:
    A New Type of Software
    Execution User
    model
    defined
    Compilation
    Security
    Failure
    mode
    Resource
    access
    User
    task
    yes
    any
    user
    based
    abort
    syscall,
    fault
    Kernel
    task
    static
    none
    panic
    direct
    BPF
    event
    yes
    JIT,
    CO-RE
    verified,
    JIT
    error
    message
    restricted
    helpers
    
slide 28:
    Take Away
    To get started with BPF performance wins,
    think like a sysadmin:
    1. Install BCC & bpftrace tools
    2. Run them
    3. Get some wins
    
slide 29:
    References
    This is also where I recommend you go to learn more:
    https://github.com/iovisor/bcc/blob/master/docs/tutorial.md
    https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
    BPF Performance Tools, Addison Wesley 2020
    Systems Performance 2nd Edition, Addison Wesley 2021
    http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
    http://www.brendangregg.com/ebpf.html
    https://ebpf.io/what-is-ebpf
    
slide 30:
    Thanks
    BPF: Alexei Starovoitov (Facebook), Daniel Borkmann (Isovalent), David S. Miller (Red
    Hat), Jakub Kicinski (Facebook), Yonghong Song (Facebook), Martin KaFai Lau
    (Facebook), John Fastabend (Isovalent), Quentin Monnet (Isovalent), Jesper Dangaard
    Brouer (Red Hat), Andrey Ignatov (Facebook), and Stanislav Fomichev (Google), Linus
    Torvalds, and many more in the BPF community
    BCC: Brenden Blanco (VMware), Yonghong Song, Sasha Goldsthein (Google), Teng Qin
    (Facebook), Paul Chaignon (Isovalent), Vicent Martí (PlanetScale), and many more in the
    BCC community
    bpftrace: Alastair Robertson (Yellowbrick Data), Dan Xu (Facebook), Bas Smit, Mary
    Marchini (Netflix), Masanori Misono, Jiri Olsa, Viktor Malík, Dale Hamel, Willian Gaspar,
    Augusto Mecking Caringi, and many more in the bpftrace community