# vmstat 1
kthr
r b w
Solaris Performance:
Introduction
Brendan Gregg
Sun Microsystems
May 2007
memory
swap
free
page
mf pi po fr de
0 0 0 4596848 120908 0
0 0 0 4411920 48652 14
0 0 0 4411576 48316 80 476
0 0 0 4411576 48316 37 240
0 0 0 4411196 48004 45 467
0 0 0 4411196 48004
2 0 0 4410852 47728 23 236
1 0 0 4410852 47728
4 0 0 4410504 47448 23 235
3 0 0 4410208 47220 23 237
3 0 0 4410208 47220
3 0 0 4410208 47220
3 0 0 4410208 47224
3 0 0 4410208 47224
2 0 0 4410648 47596
3 0 0 4410696 47644
2 0 0 4410696 47648
0 0 0 4411384 48204
kthr
r b w
memory
swap
page
free
mf pi po fr de
0 0 0 4411736 48488
0 0 0 4412088 48840 37 239
0 0 0 4411752 48572 23 234
0 0 0 4411752 48576 23 237
1 0 0 4411408 48300

Solaris Performance: Introduction
• This presentation is an introduction to the field of
Solaris performance.
• These slides cover:
>gt; Solaris Performance Features
– Top Features
– Solaris
– Solaris 10
>gt; Solaris Performance Observability
– By-Layer Strategy
– 3-Metric Strategy
– System Components

Performance Matters
• How performance helps you:
1. Shipped performance features
– Solaris can do more with less
2. Tune performance features
– Solaris tunables, library features, compiler optimisation, ...
3. Manage resources
– Get the best ROI
4. Solve performance issues
– Solaris has outstanding performance observability

Solaris Performance Features
• Solaris is a mature operating system with numerous
performance features
• Top performance features are,
>gt; CPU and Memory Scaleability
>gt; 64-bit Support
>gt; Fully Preemptive Kernel
>gt; Resource Management
>gt; Compiler Technology
>gt; Observability

CPU and Memory Scaleability
• Sun bet on SMP in early 90's
>gt; Symmetric Multi Processing: user and kernel work
distributed across all CPUs - best scaleability
• Per-CPU dispatcher queues
• Thread CPU affinity
• Processor sets and interrupt masking
• CMP and CMT support and optimisations
• Memory locality aware
• Kernel page relocation - for hot plug and DR

64-Bit Support
• Since Solaris 7 (October 1998)
• Originally for SPARC, now also AMD64 and IA-64
Fully Preemptive Kernel
• Allows Real Time scheduling class

Resource Management
• Standard tools: pbind, ulimit
• Processor sets, pools
• IPQoS - IP Quality of Service (network priorities)
• SRM - Solaris Resource Manager
• Zones + SRM = Containers
• FSS - Fair Share Schedular
• Resource Controls
>gt; CPU shares
>gt; Max threads, CPU time, file descriptors, ...

Compiler Technology
• Sun Studio compiler optimises for SPARC, x86
• Both gcc and cc can be used (try both and see)
• Java VM - hotspot compiler

Observability
• DTrace
• Microstate Accounting - prstat -mL
• kstat - vmstat, mpstat, ...
• procfs - ps, prstat, truss, ...
• PICs - cpustat/cputrack, busstat

Solaris Performance Feature List
• Scaleability
• Reliability
• Fully preemptive
kernel
• Real-Time
scheduling class
• Cyclic page cache
• Inode cache
• UFS buffer cache
• DNLC
• 64-bit support
• direct I/O
• cpustat/cputrack
• truss/apptrace
• libumem
• lgroups
• TCP MDT
• cyclics
• processor sets
• kstat
• procfs
• SNMP
• DISM
• NCA
• MPSS
• MPO
• rcapd
• SRM

Solaris 10 Performance Feature List
• DTrace
• ZFS
• Zones
• FireEngine - faster TCP/IP
• SMF - faster boot
• CMT, Niagara
• Numerous performance improvements
(many found using DTrace)

Status
• Just Covered,
>gt; Solaris Performance Features
– Top features
– Solaris
– Solaris 10
• Next up,
>gt; Solaris Performance Observability
– By-Layer Strategy
– 3-Metric Strategy
– System Components

Solaris Performance Observability
• Solaris provides numerous performance tools;
the trick is knowing what questions to ask performance analysis strategy

By-Layer Strategy
• All software stack layers are observable
>gt; locate latency regardless of location
Dynamic Languages
The Software Stack
User Executable
Libraries
Syscall Interface
Memory
allocation
Kernel
File Systems
Device Drivers
Scheduler
Hardware

By-Layer Strategy
• For an application transaction, is the latency,
>gt; In the application code?
– e.g., bad scaleability architecture
>gt; In library code?
– e.g., synchronisation locks
>gt; In syscalls?
– e.g., disk or network I/O
>gt; In devices?
– e.g., memory bus latency
• Solaris observability tools can provide the answers
>gt; especially DTrace

3-Metric Strategy
• For every system component, look for,
1.Utilisation
2.Saturation
3.Errors

System Components
How do you measure utilisation, saturation and
errors for these?
CPUs
Memory
Busses
Memory
System Busses
Disks
Net
* Your Architecture
Will Vary
Simple diagram, simple question, this should be
easy to answer.

System CPU
• Load average = overall utilisation + saturation
$ uptime
2:30pm
up 39 day(s), 12:40,
5 users,
load average: 0.07, 0.07, 0.11
>gt; printed by uptime, prstat
>gt; 1, 5 and 15 minute averages.
>gt; Divide load average by CPU count,
– value gt; 1.0
suggests saturation
>gt; Useful for an initial impression, then move onto other
tools like vmstat and mpstat

System CPU
• vmstat - utilisation and saturation as metrics
$ vmstat 1
kthr
memory
r b w
swap free re
0 0 0 4592308 120572 0
2 0 0 4349740 48280 10
0 0 0 4349756 48320 0
[...]
page
disk
mf pi po fr de sr cd s0 -- -3 0 0 0 0 5 30 -1 0 0
28 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
faults
cpu
cs us sy id
967 5343 861 2 1 97
602 1253 791 55 0 45
608 1059 723 50 1 49
>gt; first line is summary since boot
>gt; kthr:r = saturation, total threads on the run queues (but
sampled at a low rate)
>gt; cpu:us + cpu:sy = utilisation, CPU user and system time

System CPU
• mpstat - utilisation by-CPU
$ mpstat 1
CPU minf mjf xcal
0 108
CPU minf mjf xcal
CPU minf mjf xcal
0 175
[...]
intr ithr
607 338
intr ithr
451 323
intr ithr
620 328
csw icsw migr smtx
csw icsw migr smtx
csw icsw migr smtx
srw syscl
0 2580
0 2762
srw syscl
srw syscl
usr sys
usr sys
usr sys
wt idl
0 96
0 97
wt idl
0 14
0 86
wt idl
0 16
0 82
• Classic performance problem - under utilised CPUs
due to poor threading architecture

System CPU
• Solaris 10 FMA detects and can automatically
respond to CPU errors
• fmadm faulty - what faults currently exist
• fmstat -m cpumem-retire - raw statistics
$ fmstat -m cpumem-retire
NAME VALUE
auto_flts 0
bad_flts 0
cpu_blfails 0
cpu_blsupp 0
cpu_fails 0
cpu_flts 0
cpu_supp 0
nop_flts 0
[...]
DESCRIPTION
auto-close faults received
invalid fault events received
failed cpu blacklists
cpu blacklists suppressed
cpu faults unresolveable
cpu faults resolved
cpu offlines suppressed
inapplicable fault events received

System Memory
• vmstat - swap and physical memory utilisation
and saturation
$ vmstat 1
kthr
memory
r b w
swap free re
0 0 0 4592236 120548 0
0 0 0 4350572 48096 18
0 0 0 4350572 48124 0
[...]
page
disk
faults
cpu
mf pi po fr de sr cd s0 -- -in
cs us sy id
3 0 0 0 0 5 30 -1 0 0 967 5342 861 2 1 97
30 0 0 0 0 0 0 0 0 0 687 1114 781 0 1 99
0 0 0 0 0 0 0 0 0 0 6206 37271 11979 3 12 85
>gt; swap - free virtual memory (RAM + disk based swap)
>gt; free - available physical memory (RAM)
>gt; page:sr - values suggest physical memory saturation
• mdb -k - provides breakdown with ::memstat

System Memory
• Solaris 10 FMA detects and can automatically
respond to memory errors
• For example, blacklisting a page of RAM that has
had too many (correctable) ECC errors
• fmadm faulty - what is currently faulted
• fmstat -m cpumem-retire - raw statistics

System Disks
• iostat - disk utilisation, saturation, errors
$ iostat -xnmpz 5
r/s
[...]
w/s
extended device statistics
kr/s
kw/s wait actv wsvc_t asvc_t
0.0 0.0 0.0
0.0 0.0 0.0
0.1 0.0 0.0
21.8 0.0 0.0
0.0 0.0 0.0
%b device
0 c0t0d0
0 c1t0d0s0
0 c1t0d0s1
1 c1t0d0s3 (/)
0 c1t0d0s4
>gt; first output is summary since boot
>gt; %b - percent busy, a measure of utilisation
>gt; wait - transactions waiting, a measure of saturation
• iostat -E - error summaries

System Network
• kstat - network utilisation, saturation, errors
$ kstat -n nge0 10
module: nge
name:
nge0
brdcstrcv
brdcstxmt
collisions
crtime
ierrors
ifspeed
ipackets
[...]
instance: 0
class:
net
>gt; output includes byte counts, various errors
• netstat and nicstat (opensource) provide
useful summaries of network stats

System Busses
• Measuring utilisation, saturation and errors is
hard, but usually still possible with some effort
>gt; cpustat - measure CPU Performance Instrumentation
Counters (PICs)
– PICs for cache activity, memory bus activity, instruction events
>gt; cputrack - CPU PICs for a process
>gt; busstat - On some SPARC systems, provides
hardware bus PICs

Processes
• Apart from performance observability by-system,
also examine performance observability by-process.
• prstat -mL - useful microstates by thread
$ prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
557 brendan 7.9 0.3 0.0 0.0 0.0 0.0 91 0.5 579 141 2K 96 Xorg/1
828 brendan 0.6 0.4 0.0 0.0 0.0 0.0 95 4.1 434 299 2K
0 ssh/1
830 brendan 0.2 0.0 0.0 0.0 0.0 0.0 99 0.3 36 11 160
0 gnome-termin/1
788 brendan 0.1 0.1 0.0 0.0 0.0 0.0 100 0.0 58
0 910
0 dtwm/1
1437 brendan 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 44
2 297
0 prstat/1
791 brendan 0.0 0.0 0.0 0.0 0.0 0.0 100 0.3
7 11 129
0 dtterm/1
[...]
• DTrace - measure custom microstates
>gt; in terms of application activity, across all software layers

Further Observability
• Much more can be observed and analysed on
Solaris
>gt; DTrace is its own field of study
• “You don't miss what you never had”
>gt; Once you start exploring Solaris observability, other
OSes won't feel the same again

References
• http://www.solarisinternals.com
>gt; Latest Solaris Performance Slides
>gt; Performance wiki
• The “Solaris Performance and Tools” book,
http://www.sun.com/books/catalog/solaris_perf_tools.xml
• Performance Community,
http://www.opensolaris.org/os/community/performance

Ctrl-D
Brendan Gregg
brendan@sun.com