ACM Applicative 2016: System Methodology
Video: https://youtu.be/eO94l0aGLCA?t=3m37sSystem Methodology - Holistic Performance Analysis on Modern Systems
Description: "Traditional systems performance engineering makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. For modern systems, we can choose the metrics, and can choose ones we need to support new holistic performance analysis methodologies. These methodologies provide faster, more accurate, and more complete analysis, and can provide a starting point for unfamiliar systems.
Methodologies are especially helpful for modern applications and their workloads, which can pose extremely complex problems with no obvious starting point. There are also continuous deployment environments such as the Netflix cloud, where these problems must be solved in shorter time frames. Fortunately, with advances in system observability and tracers, we have virtually endless custom metrics to aid performance analysis. The problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.
System methodologies provide a starting point for analysis, as well as guidance for quickly moving through the metrics to root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, and more.
This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. The focus is on single systems (any operating system), including single cloud instances, and quickly locating performance issues or exonerating the system. Many methodologies will be discussed, along with recommendations for their implementation, which may be as documented checklists of tools, or custom dashboards of supporting metrics. In general, you will learn to think differently about your systems, and how to ask better questions."
 | next prev 1/57 |
 | next prev 2/57 |
 | next prev 3/57 |
 | next prev 4/57 |
 | next prev 5/57 |
 | next prev 6/57 |
 | next prev 7/57 |
 | next prev 8/57 |
 | next prev 9/57 |
 | next prev 10/57 |
 | next prev 11/57 |
 | next prev 12/57 |
 | next prev 13/57 |
 | next prev 14/57 |
 | next prev 15/57 |
 | next prev 16/57 |
 | next prev 17/57 |
 | next prev 18/57 |
 | next prev 19/57 |
 | next prev 20/57 |
 | next prev 21/57 |
 | next prev 22/57 |
 | next prev 23/57 |
 | next prev 24/57 |
 | next prev 25/57 |
 | next prev 26/57 |
 | next prev 27/57 |
 | next prev 28/57 |
 | next prev 29/57 |
 | next prev 30/57 |
 | next prev 31/57 |
 | next prev 32/57 |
 | next prev 33/57 |
 | next prev 34/57 |
 | next prev 35/57 |
 | next prev 36/57 |
 | next prev 37/57 |
 | next prev 38/57 |
 | next prev 39/57 |
 | next prev 40/57 |
 | next prev 41/57 |
 | next prev 42/57 |
 | next prev 43/57 |
 | next prev 44/57 |
 | next prev 45/57 |
 | next prev 46/57 |
 | next prev 47/57 |
 | next prev 48/57 |
 | next prev 49/57 |
 | next prev 50/57 |
 | next prev 51/57 |
 | next prev 52/57 |
 | next prev 53/57 |
 | next prev 54/57 |
 | next prev 55/57 |
 | next prev 56/57 |
 | next prev 57/57 |
PDF: ACMApplicative2016_SystemMethodology.pdf
Keywords (from pdftotext):
slide 1:
ACM Applicative 2016 Jun,  2016  System  Methodology  Holis0c  Performance  Analysis  on  Modern  Systems  Brendan Gregg Senior Performance Architectslide 2:
Apollo LMGC performance analysis CORE Â SET Â AREA Â VAC Â SETS Â ERASABLE Â MEMORY Â FIXED Â MEMORY Âslide 4:
Background Âslide 6:
History  • System Performance Analysis up to the '90s: – Closed source UNIXes and applications – Vendor-created metrics and performance tools – Users interpret given metrics • Problems – Vendors may not provide the best metrics – Often had to infer, rather than measure – Given metrics, what do we do with them? # ps alx F S UID 3 S 1 S 1 S […] PID PPID CPU PRI NICE 0 30 0 30 ADDR WCHAN TTY TIME CMD 4412 ? 186:14 swapper 46520 ? 0:00 /etc/init 46554 co 0:00 –shslide 7:
Today  1. Open source Operating systems: Linux, BSDs, illumos, etc. Applications: source online (Github) 2. Custom metrics Can patch the open source, or, Use dynamic tracing (open source helps) 3. Methodologies Start with the questions, then make metrics to answer them Methodologies can pose the questions Biggest problem with dynamic tracing has been what to do with it. Methodologies guide your usage.slide 8:
Crystal  Ball  Thinking Âslide 9:
An#-Ââ€Methodologies Âslide 10:
Street  Light  An#-Ââ€Method  1. Pick observability tools that are – Familiar – Found on the Internet – Found at random 2. Run tools 3. Look for obvious issuesslide 11:
Drunk  Man  An#-Ââ€Method  • Drink Tune things at random until the problem goes awayslide 12:
Blame  Someone  Else  An#-Ââ€Method  1. Find a system or environment component you are not responsible for 2. Hypothesize that the issue is with that component 3. Redirect the issue to the responsible team 4. When proven wrong, go to 1slide 13:
Traffic  Light  An#-Ââ€Method  1. Turn all metrics into traffic lights 2. Open dashboard 3. Everything green? No worries, mate. • Type I errors: red instead of green – team wastes time • Type II errors: green instead of red – performance issues undiagnosed – team wastes more time looking elsewhere Traffic lights are suitable for objective metrics (eg, errors), not subjective metrics (eg, IOPS, latency).slide 14:
Methodologies Âslide 15:
Performance  Methodologies  • For system engineers: – ways to analyze unfamiliar systems and applications • For app developers: – guidance for metric and dashboard design Collect your own toolbox of methodologies System Methodologies: – Problem statement method – Functional diagram method – Workload analysis – Workload characterization – Resource analysis – USE method – Thread State Analysis – On-CPU analysis – CPU flame graph analysis – Off-CPU analysis – Latency correlations – Checklists – Static performance tuning – Tools-based methodsslide 16:
Problem  Statement  Method  1. What makes you think there is a performance problem? 2. Has this system ever performed well? 3. What has changed recently? software? hardware? load? 4. Can the problem be described in terms of latency? or run time. not IOPS or throughput. 5. Does the problem affect other people or applications? 6. What is the environment? software, hardware, instance types? versions? config?slide 17:
Func0onal  Diagram  Method  1. Draw the functional diagram 2. Trace all components in the data path 3. For each component, check performance Breaks up a bigger problem into smaller, relevant parts Eg, imagine throughput between the UCSB 360 and the UTAH PDP10 was slow… ARPA  Network  1969 Âslide 18:
Workload  Analysis  • Begin with application metrics & context • A drill-down methodology Workload  • Pros: – Proportional, accurate metrics – App context Applica0on    Libraries  System  • Cons: – App specific – Difficult to dig from app to resource System  Calls  Kernel  Hardware  Analysis Âslide 19:
Workload  Characteriza0on  • Check the workload: who, why, what, how – not resulting performance Workload  Target  • Eg, for CPUs: Who: which PIDs, programs, users Why: code paths, context What: CPU instructions, cycles How: changing over timeslide 20:
Workload  Characteriza0on:  CPUs  Who Why top CPU  sample  flame  graphs  How What monitoring  PMCs Âslide 21:
Resource  Analysis  • Typical approach for system performance analysis: begin with system tools & metrics Workload  • Pros: – Generic – Aids resource perf tuning Applica0on    Libraries  System  • Cons: – Uneven coverage – False positives System  Calls  Kernel  Hardware  Analysis Âslide 22:
The  USE  Method  • For every resource, check: Utilization: busy time Saturation: queue length or time Errors: easy to interpret (objective) Starts with the questions, then finds the tools Eg, for hardware, check every resource incl. busses:slide 23:
http://www.brendangregg.com/USEmethod/use-rosetta.htmlslide 24:
Apollo Guidance Computer CORE Â SET Â AREA Â VAC Â SETS Â ERASABLE Â MEMORY Â FIXED Â MEMORY Âslide 26:
USE  Method:  SoZware  • USE method can also work for software resources – kernel or app internals, cloud environments – small scale (eg, locks) to large scale (apps). Eg: • Mutex locks: – utilization Ã ïƒ lock hold time – saturation Ã ïƒ lock contention – errors Ã ïƒ any errors X  • Entire application: – utilization Ã ïƒ percentage of worker threads busy – saturation Ã ïƒ length of queued work – errors Ã ïƒ request errors Resource  U0liza0on  (%) Âslide 27:
RED  Method  • For every service, check that: Request rate Error rate Duration (distribution) Metrics  Database  are within SLO/A Another exercise in posing questions from functional diagrams Load  Balancer  Web  Proxy  Web  Server  User  Database  Payments  Server  Asset  Server  By Tom Wilkie: http://www.slideshare.net/weaveworks/monitoring-microservicesslide 28:
Thread  State  Analysis  State transition diagram Identify & quantify time in states Narrows further analysis to state Thread states are applicable to all appsslide 29:
TSA: Â eg, Â Solaris Âslide 30:
TSA: Â eg, Â RSTS/E Â RSTS: DEC OS from the 1970's TENEX (1969-72) also had Control-T for job statesslide 31:
TSA:  eg,  OS  X  Instruments:  Thread  States Âslide 32:
On-Ââ€CPU  Analysis  1. Split into user/kernel states CPU  U0liza0on  Heat  Map  – /proc, vmstat(1) 2. Check CPU balance – mpstat(1), CPU utilization heat map 3. Profile software – User & kernel stack sampling (as a CPU flame graph) 4. Profile cycles, caches, busses – PMCs, CPI flame graphslide 33:
CPU  Flame  Graph  Analysis  1. Take a CPU profile 2. Render it as a flame graph 3. Understand all software that is in >gt;1% of samples Discovers issues by their CPU usage - Directly: CPU consumers - Indirectly: initialization of I/O, locks, times, ... Narrows target of study to only running code - See: "The Flame Graph", CACM, June 2016 Flame  Graph Âslide 34:
Java  Mixed-Ââ€Mode  CPU  Flame  Graph  • eg, Linux perf_events, with: • Java –XX:+PreserveFramePointer • Java perf-map-agent Java  GC  JVM  Kernel Âslide 35:
CPI  Flame  Graph  • Profile cycle stack traces and instructions or stalls separately • Generate CPU flame graph (cycles) and color using other profile • eg, FreeBSD: pmcstat red  ==  instruc0ons  blue  ==  stalls Âslide 36:
Off-Ââ€CPU  Analysis  Analyze off-CPU time via blocking code path: Off-CPU flame graph Often need wakeup code paths as well…slide 37:
Off-Ââ€CPU  Time  Flame  Graph  directory  read  from  disk  ï¬le  read  from  disk  fstat  from  disk  path  read  from  disk  pipe  write  Trace blocking events with kernel stacks & time blocked (eg, using Linux BPF) Off-Ââ€CPU  0me  Stack  depth Âslide 38:
Wakeup  Time  Flame  Graph  Who did the wakeup: … can also associate wake-up stacks with off-CPU stacks (eg, Linux 4.6: samples/bpf/offwaketime*)slide 39:
Chain  Graphs  Associate more than one waker: the full chain of wakeups With enough stacks, all paths lead to metal An approach for analyzing all off-CPU issuesslide 40:
Latency  Correla0ons  1. Measure latency histograms at different stack layers 2. Compare histograms to find latency origin Even better, use latency heat maps • Match outliers based on both latency and timeslide 41:
Checklists:  eg,  Linux  Perf  Analysis  in  60s  1. uptime 2. dmesg | tail 3. vmstat 1 4. mpstat -P ALL 1 5. pidstat 1 6. iostat -xz 1 7. free -m 8. sar -n DEV 1 9. sar -n TCP,ETCP 1 10. top load  averages  kernel  errors  overall  stats  by  0me  CPU  balance  process  usage  disk  I/O  memory  usage  network  I/O  TCP  stats  check  overview  http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.htmlslide 42:
Checklists:  eg,  Neklix  perfvitals  Dashboard  1.  RPS,  CPU  2.  Volume  3.  Instances  4.  Scaling  5.  CPU/RPS  6.  Load  Avg  7.  Java  Heap  8.  ParNew  9.  Latency  10.  99th  0le Âslide 43:
Sta0c  Performance  Tuning:  eg,  Linux Âslide 44:
Tools-Ââ€Based  Method  1. Try all the tools! May be an anti-pattern. Eg, OS X:slide 45:
Other  Methodologies  Scientific method 5 Why's Process of elimination Intel's Top-Down Methodology Method Rslide 46:
What  You  Can  Do Âslide 47:
What  you  can  do  1. Know what's now possible on modern systems – Dynamic tracing: efficiently instrument any software – CPU facilities: PMCs, MSRs (model specific registers) – Visualizations: flame graphs, latency heat maps, … 2. Ask questions first: use methodologies to ask them 3. Then find/build the metrics 4. Build or buy dashboards to support methodologiesslide 48:
Dynamic  Tracing:  Efficient  Metrics  Eg, tracing TCP retransmits Kernel  Old way: packet capture tcpdump  1.  read  2.  dump  buffer  Analyzer  1.  read  2.  process  3.  print  ï¬le  system  send  receive  disks  New way: dynamic tracing Tracer  1.  conï¬gure  2.  read  tcp_retransmit_skb() Âslide 49:
Dynamic  Tracing:  Measure  Anything  Those are Solaris/DTrace tools. Now becoming possible on all OSes: FreeBSD & OS X DTrace, Linux BPF, Windows ETWslide 50:
Performance  Monitoring  Counters  Eg, FreeBSD PMC groups for Intel Sandy Bridge:slide 51:
Visualiza0ons  Eg, Disk I/O latency as a heat map, quantized in kernel:slide 52:
USE  Method:  eg,  Neklix  Vector  CPU:  u0liza0on  Network:  u0liza0on  Memory:  u0liza0on  Disk:  load  satura0on  satura0on  load  satura0on  u0liza0on  satura0on Âslide 53:
USE  Method:  To  Do  Showing what is and is not commonly measured U  S  E  U  S  E  U  S  E  U  S  E  U  S  E  U  S  E  U  S  E  U  S  E  U  S  E  U  S  E  U  S  E Âslide 54:
CPU  Workload  Characteriza0on:  To  Do  Showing what is and is not commonly measured Who Why top,  htop perf record -g flame  Graphs  How What monitoring  perf stat -a -dslide 55:
Summary  • It is the crystal ball age of performance observability • What matters is the questions you want answered • Methodologies are a great way to pose questionsslide 56:
References  &  Resources  USE Method – http://queue.acm.org/detail.cfm?id=2413037 – http://www.brendangregg.com/usemethod.html TSA Method – http://www.brendangregg.com/tsamethod.html Off-CPU Analysis – http://www.brendangregg.com/offcpuanalysis.html – http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html – http://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html Static Performance Tuning, Richard Elling, Sun blueprint, May 2000 RED Method: http://www.slideshare.net/weaveworks/monitoring-microservices Other system methodologies – Systems Performance: Enterprise and the Cloud, Prentice Hall 2013 – http://www.brendangregg.com/methodology.html – The Art of Computer Systems Performance Analysis, Jain, R., 1991 Flame Graphs – http://queue.acm.org/detail.cfm?id=2927301 – http://www.brendangregg.com/flamegraphs.html – http://techblog.netflix.com/2015/07/java-in-flames.html Latency Heat Maps – http://queue.acm.org/detail.cfm?id=1809426 – http://www.brendangregg.com/HeatMaps/latency.html ARPA Network: http://www.computerhistory.org/internethistory/1960s RSTS/E System User's Guide, 1985, page 4-5 DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD, Prentice Hall 2011 Apollo: http://www.hq.nasa.gov/office/pao/History/alsj/a11 http://www.hq.nasa.gov/alsj/alsj-LMdocs.htmlslide 57:
ACM Applicative 2016 Questions? http://slideshare.net/brendangregg http://www.brendangregg.com bgregg@netflix.com @brendangregg Feb  Jun,  2016  2016 Â