sar, benchmarking, tuning:
Images license: creative commons Attribution-ShareAlike 4.0.
This page links to various Linux performance material I've created, including the tools maps on the right. These show: Linux observability tools, Linux benchmarking tools, Linux tuning tools, and Linux sar. For more diagrams, see my slide decks below.
- perf_events: perf one-liners, examples, visualizations.
- perf-tools: perf analysis tools using perf_events and ftrace (github).
- ktap: one-liners, examples, and scripts.
- Flame Graphs: using perf, SystemTap, and ktap.
- Linux Performance Analysis in 60,000 Milliseconds shows the first ten commands to use in an investigation (video, PDF). Written by myself and the performance engineering team at Netflix (2015).
- My post Performance Tuning Linux Instances on EC2 includes the tunables we're using at Netflix (2015).
- Posts about eBPF and bcc: Linux eBPF, bcc: Taming Linux 4.3+ Tracing Superpowers, tcpconnect and tcpaccept for Linux (bcc) Linux eBPF Stack Trace Hack (bcc) Linux eBPF Off-CPU Flame Graph (bcc) (2015-6).
- My lwn.net article Ftrace: The Hidden Light Switch shows a use case for Linux ftrace (Aug, 2014).
- Posts about ftrace-based perf-tools: iosnoop for Linux, iosnoop Latency Heat Maps, opensnoop for Linux, execsnoop for Linux, tcpretrans, Page Cache Hit Ratio, uprobe: User-Level Dynamic Tracing, uprobe: User-Level Dynamic Tracing, Hacking Linux USDT (2014-5).
- Posts about perf-based perf-tools: perf Hacktogram.
- Posts about perf_events: perf CPU Sampling, perf Static Tracepoints, perf Heat Maps, perf Counting, perf Kernel Line Tracing, perf off-CPU Time Flame Graphs, Linux Profiling at Netflix, Java Mixed-Mode Flame Graphs (PDF) (2014-5).
- The blog post strace Wow Much Syscall discusses strace(1) for production use, and compares it to advanced tracing tools (2014).
- USE Method: Linux Performance Checklist; also see the USE Method page for the description of this methodology.
- Systems Performance: Enterprise and the Cloud (Prentice Hall, 2013) uses Linux distributions as the primary example.
In recommended viewing order:
At Velocity 2015, I gave a 90 minute tutorial on Linux performance tools, summarizing performance observability, benchmarking, tuning, static performance tuning, and tracing tools. I also covered performance methodology, and included some live demos. This should be useful for everyone working on Linux systems.
This was similar to my SCaLE11x and LinuxCon talks, however, with 90 minutes I was able to cover more tools and methodologies, making it the most complete tour of the topic I've done. I also posted about it on the Netflix Tech Blog.
At the Southern California Linux Expo (SCaLE 14x), I gave a talk on Broken Linux Performance Tools. This was a follow-on to my earlier Linux Performance Tools talk originally at SCaLE11x (and more recently at Velocity as a tutorial). This broken tools talk was a tour of common problems with Linux system tools, metrics, statistics, visualizations, measurement overhead, and benchmarks. It also includes advice on how to cope (the green "What You Can Do" slides).
I'll post the video soon.
At (SCaLE 13x), I gave a talk on Linux Profiling at Netflix using perf_events (aka "perf"), covering CPU profiling and a tour of other features. This talk covered gotchas, such as fixing stack traces and symbols when profiling Java and Node.js.
In a post about this talk, I included the interactive CPU flame graph SVG I was demonstrating.
At USENIX LISA 2014, I gave a talk on the new ftrace and perf_events tools I've been developing: the perf-tools collection on github.
The perf-tools collection, inspired by my earlier DTraceToolkit, provides advanced system performance analysis tools for Linux. Each tool has a man page and example file. They are unstable and unsupported, and they currently use shell scripting, hacks, and the ftrace and perf_events kernel tracing frameworks. They should be developed (and improved) as the Linux kernel acquires more tracing capabilities (eg, eBPF).
In a post about this talk, I included some more screenshots of these tools in action.
At SCaLE 12x, I gave the keynote on What Linux can learn from Solaris performance and vice-versa. This drew on my experience analyzing Linux vs SmartOS performance, and my work for the Systems Performance book. SmartOS is an OS based on the illumos kernel, which is fork of the OpenSolaris kernel.
I've never seen a good talk comparing performance features of both, I suspect in part because it's hard to know them both in enough depth, and also hard to choose from the many differences which should be highlighted.
This presentation also contains ponies. Lots of ponies. These are the unofficial mascots for DTrace, perf_events, SystemTap, ktap, and LTTng, and were designed by the same person (Deirdré) who designed the original (and popular) DTrace ponycorn.
Other resources (not by me) I'd recommend for the topic of Linux performance:
- Linux Instrumentation: slides from a great talk in June 2010 by Ian Munsie, which summarizes the different Linux tracers very well. If you're trying to understand all the tracers and frameworks, this is worth studying (keeping in mind it's from 2010).