Flame Graphs


CPU Flame Graph

Flame graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately. They can be generated using my open source programs on github.com/brendangregg/FlameGraph, which create interactive SVGs. See the Updates section for other implementations.

The following pages (or posts) introduce different types of flame graphs:

The example on the right is a portion of a CPU flame graph, showing MySQL codepaths that are consuming CPU cycles, and by how much.

Summary

The x-axis shows the stack profile population, sorted alphabetically (it is not the passage of time), and the y-axis shows stack depth. Each rectangle represents a stack frame. The wider a frame is is, the more often it was present in the stacks. The top edge shows what is on-CPU, and beneath it is its ancestry. The colors are usually not significant, picked randomly to differentiate frames.

This visualization is fully explained in my ACMQ article The Flame Graph, also published in Communications of the ACM, Vol. 59 No. 6.

Also see my CPU Flame Graphs page, and the presentation below.

Operating Systems

Flame graphs can be generated from any profile data that contains stack traces, including from the following profiling tools:

Once you have a profiler that can generate meaningful stacks, converting them into a flame graph is usually the easy step.

Presentation

My talk for USENIX/LISA13, titled Blazing Performance with Flame Graphs, explains the visualization and summarizes different types:

These slides can also be downloaded as a PDF.

Origin

I invented flame graphs when working on a MySQL performance issue and needed to understand CPU usage quickly and in depth. The regular profilers/tracers had produced walls of text, so I was exploring visualizations. I first traced CPU function calls and visualized it using Neelakanth Nadgir's time-ordered visualization for callstacks, which itself was inspired by Roch Bourbonnais's CallStackAnalyzer and Jan Boerhout's vftrace. These look similar to flame graphs, but have the passage of time on the x-axis. But there were two problems: the overhead of function tracing was too high, perturbing the target, and the final visualization was too dense to read when spanning multiple seconds. I switched to timed sampling (profiling) to solve the overhead problem, but since the function flow is no longer known (sampling has gaps) I ditched time on the x-axis and reordered samples to maximize frame merging. It worked, the final visualization was much more readable. Neelakanth and Roch's visualizations used completely random colors to differentiate frames. I thought it looked nicer to narrow the color palette, and picked just warm colors initially as it explained why the CPUs were "hot" (busy). Since it resembled flames, it quickly became known as flame graphs.

I described more detail of the original performance problem that led to flame graphs in my ACMQ/CACM article (link above). The flame graph visualization is really an adjacency diagram with an inverted icicle layout, which I used to visualize profiled stack traces.

Updates

Flame graphs were released in Dec 2011. Not long afterwards (updated in 2012):

More Flame Graph news (updated Apr 2013):

More Flame Graph news (updated Aug 2013):

More Flame Graph news (updated Jan 2014):

More Flame Graph news (updated Jun 2014):

More Flame Graph news (updated Nov 2014):

More Flame Graph news (updated May 2015):

More Flame Graph news (updated Jul 2015):

More Flame Graph news (updated Aug 2015):

More Flame Graph news (updated Oct 2015):

More Flame Graph news (updated Dec 2015):

More Flame Graph news (updated Apr 2016):

More Flame Graph news (updated Jun 2016):

More Flame Graph news (updated Jul 2016):

More Flame Graph news (updated Sep 2016):

More Flame Graph news (updated Feb 2017):

Thanks to everyone who has written about flame graphs, developed them further, and shared their results! I'll update this page from time to time with more news.


Last updated: 17-Feb-2017