Systems Performance 2nd Ed.



BPF Performance Tools book

Recent posts:
Blog index
About
RSS

eBPF Summit 2020: Performance Wins with BPF, Getting Started

Keynote by Brendan Gregg for the eBPF Summit 2020.

Video: https://www.youtube.com/watch?v=wyfhjr_ufag

Description: "How to get started finding performance wins using the BPF (eBPF) technology. This short talk covers the quickest and easiest way to find performance wins using BPF observability tools on Linux."

next
prev
1/24
next
prev
2/24
next
prev
3/24
next
prev
4/24
next
prev
5/24
next
prev
6/24
next
prev
7/24
next
prev
8/24
next
prev
9/24
next
prev
10/24
next
prev
11/24
next
prev
12/24
next
prev
13/24
next
prev
14/24
next
prev
15/24
next
prev
16/24
next
prev
17/24
next
prev
18/24
next
prev
19/24
next
prev
20/24
next
prev
21/24
next
prev
22/24
next
prev
23/24
next
prev
24/24

PDF: eBPF2020_performance_getting_started.pdf

Keywords (from pdftotext):

slide 1:
    Performance Wins with BPF
    Getting Started
    Brendan Gregg
    Oct 2020
    
slide 2:
    This article is
    not for beginners
    … not the best start
    … out of date
    BPF has evolved
    Many docs were true
    in 2014, 2015, etc.,
    but not today.
    (PS. Google search did better)
    
slide 3:
    BPF is no longer
    an acronym
    BPF is a bytecode
    and execution
    environment
    
slide 4:
    How to get quick and easy
    BPF performance wins
    
slide 5:
    Think like a
    sysadmin
    Not like a
    programmer
    
slide 6:
    Think like a sysadmin
    Get it installed everywhere and use it.
    # apt-get install bcc-tools
    # PATH=$PATH:/usr/share/bcc/tools
    # execsnoop
    # opensnoop
    # tcplife
    # ext4slower
    # biosnoop
    [...]
    
slide 7:
    Think like a sysadmin
    Get it installed everywhere and use it.
    # apt-get install bcc-tools
    # PATH=$PATH:/usr/share/bcc/tools
    # execsnoop
    Anything periodic running? crontab?
    # opensnoop
    Any misconfigurations? File not found?
    # tcplife
    # ext4slower
    Any unexpected TCP sessions?
    # biosnoop
    Any file system I/O slower than 10ms?
    [...]
    Any unusual disk access patters? Outliers?
    
slide 8:
    Case Study: BCC biosnoop
    # iostat -xz 1
    Linux 4.15.0-1052-aws (cass-xxx)
    […]
    avg-cpu:
    %user
    Device: rrqm/s
    xvda
    xvdb
    xvdc
    xvdd
    xvde
    xvdf
    xvdg
    md0
    […]
    12/04/2019 _x86_64_
    %nice %system %iowait
    %steal
    wrqm/s
    rkB/s
    r/s
    w/s
    (8 CPU)
    %idle
    wkB/s avgrq-sz avgqu-sz
    await r_await w_await
    svctm
    %util
    
slide 9:
    Case Study: BCC biosnoop, cont.
    # /usr/share/bcc/tools/biosnoop
    TIME(s)
    COMM
    PID
    perl
    biosnoop
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    perl
    […]
    DISK
    xvdc
    xvda
    xvde
    xvde
    xvdb
    xvdc
    xvdb
    xvdg
    xvdd
    xvdg
    xvdg
    xvdg
    xvdg
    xvdc
    xvdc
    xvdg
    xvde
    T SECTOR
    R 610822184
    R 269480
    R 610737856
    R 377704624
    R 732825200
    R 732953880
    R 377707064
    R 732998328
    R 733127392
    R 732868048
    R 732906896
    R 610744920
    R 377706520
    R 610951744
    R 732858664
    R 732937416
    R 610853240
    BYTES
    LAT(ms)
    
slide 10:
    Case Study: BCC biosnoop, cont.
    # ps -ef | grep perl
    root
    3285 3274
    root
    7755 7748
    root
    11366 11359
    root
    15054 15049
    root
    19675 19670
    root
    23937 23930
    root
    27565 27561
    root
    28232 28223
    root
    31913 31907
    […]
    1 14:16 ?
    1 04:16 ?
    1 10:16 ?
    2 16:16 ?
    1 06:16 ?
    1 12:16 ?
    2 18:16 ?
    1 02:16 ?
    1 08:15 ?
    00:04:24 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:10:20 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:06:39 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:03:07 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:08:53 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:05:30 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:00:28 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:11:43 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    00:07:40 /usr/bin/perl /apps/nflx-ec2rotatelogs/bin/nflx-ec2rotatelogs.pl
    
slide 11:
    Many more
    tools to try!
    bcc tools
    bpftrace tools
    from my book,
    all open source
    Solve >gt;90% of perf
    issues with canned
    observability
    tools
    
slide 12:
    The future of BPF perf observability
    … is GUIs. The end user may not even know it’s BPF.
    Tool output, visualized
    This GUI is in development by Susie Xia, Netflix
    
slide 13:
    Sysadmins sometimes program
    shell scripting
    awk
    sed one-liners
    bpftrace tools
    bpftrace one-liners
    
slide 14:
    Think like a programmer if
    You have a real-world problem that tools don’t solve
    You’re a BPF-based startup
    You’re debugging your own code*
    You’re doing networking/security/etc.
    You really want to learn BPF internals
    * JIT-ed runtimes like Java are currently complex to trace
    
slide 15:
    Performance tool languages
    bpftrace
    Concise, like pseudocode. Start here!
    BCC
    Python/C interface
    libbpf/C interface
    etc.
    * JIT-ed runtimes like Java are complex to trace
    WARNING: requires LLVM;
    May become obsolete /
    special-use only
    New, lightweight,
    CO-RE & BTF based
    
slide 16:
    bpftrace example
    # readahead.bt
    Attaching 5 probes...
    Readahead unused pages: 128
    Readahead used page age (ms):
    @age_ms:
    [1]
    2455 |@@@@@@@@@@@@@@@
    [2, 4)
    8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
    [4, 8)
    4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@
    [8, 16)
    7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    [16, 32)
    4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@
    [32, 64)
    0 |
    [64, 128)
    0 |
    [128, 256)
    384 |@@
    
slide 17:
    bpftrace example
    #!/usr/local/bin/bpftrace
    kprobe:__do_page_cache_readahead
    { @in_readahead[tid] = 1; }
    kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
    kretprobe:__page_cache_alloc
    /@in_readahead[tid]/
    @birth[retval] = nsecs;
    @rapages++;
    Fits on one slide!
    kprobe:mark_page_accessed
    /@birth[arg0]/
    @age_ms = hist((nsecs - @birth[arg0]) / 1000000);
    delete(@birth[arg0]);
    @rapages--;
    END
    printf("\nReadahead unused pages: %d\n", @rapages);
    printf("\nReadahead used page age (ms):\n");
    print(@age_ms); clear(@age_ms);
    clear(@birth); clear(@in_readahead); clear(@rapages);
    
slide 18:
    BCC libbpf tool example
    # ./opensnoop
    PID
    COMM
    27974 opensnoop
    redis-server
    […]
    FD ERR PATH
    0 /etc/localtime
    0 /proc/1482/stat
    # ldd opensnoop
    linux-vdso.so.1 (0x00007ffddf3f1000)
    libelf.so.1 =>gt; /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f9fb7836000)
    libz.so.1 =>gt; /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9fb7619000)
    libc.so.6 =>gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fb7228000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9fb7c76000)
    # ls -lh opensnoop opensnoop.stripped
    -rwxr-xr-x 1 root root 645K Feb 28 23:18 opensnoop
    -rwxr-xr-x 1 root root 151K Feb 28 23:33 opensnoop.stripped
    151 Kbytes for a stand-alone BPF program!
    (Note: A static bpftrace/BTF + scripts will also have a small average tool size)
    
slide 19:
    PSA
    CONFIG_DEBUG_INFO_BTF=y
    E.g., Ubuntu 20.10 has it
    
slide 20:
    BPF Future: Event-based Applications
    User-mode
    Applications
    Kernel-mode
    Applications (BPF)
    U.E.
    Scheduler
    Kernel
    Kernel
    Events
    Hardware Events (incl. clock)
    
slide 21:
    A New Type of Software
    Execution User
    model
    defined
    Compilation
    Security
    Failure
    mode
    Resource
    access
    User
    task
    yes
    any
    user
    based
    abort
    syscall,
    fault
    Kernel
    task
    static
    none
    panic
    direct
    BPF
    event
    yes
    JIT,
    CO-RE
    verified,
    JIT
    error
    message
    restricted
    helpers
    
slide 22:
    Take Away
    To get started with BPF performance wins,
    think like a sysadmin:
    1. Install BCC & bpftrace tools
    2. Run them
    3. Get some wins
    
slide 23:
    References
    This is still
    generally true
    http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
    https://github.com/iovisor/bcc
    https://github.com/iovisor/bpftrace
    https://ebpf.io/what-is-ebpf
    
slide 24:
    Thanks
    BPF: Alexei Starovoitov, Daniel Borkmann, David S. Miller, Linus Torvalds,
    BPF community
    BCC: Brenden Blanco, Yonghong Song,
    Sasha Goldsthein, BCC community
    bpftrace: Alastair Robertson, Mary Marchini,
    Dan Xu, Bas Smit, bpftrace community
    https://ebpf.io