Systems Performance 2nd Ed.

BPF Performance Tools book

Recent posts:
Blog index

Cloud Analytics: first video

24 Jan 2011

I originally posted this at

The video for the Cloud Analytics talk that Bryan Cantrill and I recently gave was streamed live on ustream and available online immediately after. It's now available edited and in hires.

Just before the talk, I took screenshots of each of the demos I wanted to give, to refer to if needed. I didn't end up using them in the talk as we were able to demo everything live. I've included them here to supplement the video. They show the same heatmaps we demoed, but at a different time of day with some different features visible. Watch the video for the descriptions of these heatmaps, what they are showing, and why they are useful.

These screenshots show our prototype Joyent Cloud Analytics visualizing performance on production systems. The interface (buttons and controls) that surround these heatmaps will be much improved over time. Right now, we experimenting with data collection (thanks mainly to DTrace) and how that data can be visualized (mostly heatmaps, but we are doing other graph types too).

The axes for these heat maps are:

  • x-axis: time
  • y-axis: latency (usually; some heatmaps use sub-second offset, file descriptor offset, etc).
  • z-axis (color saturation): number of events at that time/latency range
  • 4-axis (color hue): breakdown of another axis; eg: application, event type, smartmachine, etc.

System Call Latency by Syscall Type

System Call Latency by Application

System Call Latency by Smartmachine

System Call Latency for httpd by Syscall Type

System Call Latency for mysqld by Syscall Type

System Call Latency for mysqld, by I/O Syscall and Filesystem Type

VFS Latency by ZFS Call

VFS Latency by ZFS Call (nsecs)

CPU Dispatcher Queue Latency, by Application

MySQL Query Latency by Type

MySQL Query Latency by CPU and Filesort State

System Call Sub-Second Execution Time by Application

System Call Sub-Second Execution Time by Application

Read/Write Syscall by File Descriptor Offset