USE Method: SmartOS Performance Checklist

The USE Method provides a strategy for performing a complete check of system health, identifying common bottlenecks and errors. For each system resource, metrics for utilization, saturation and errors are identified and checked. Any issues discovered are then investigated using further strategies.

This is an example of a USE-based metric list for use within a SmartOS SmartMachine (Zone), such as those provided by the Joyent Public Cloud. These use the illumos kernel, and so this list should also be mostly relevant for OmniOS Zones, and to a lesser degree (due to some missing features) Solaris Zones. This is primarily intended for users of the zones. For the system administrators of the physical systems (via the Global Zone), also see the Solaris checklist, which has greater visibility.

Cloud limits (software resource controls) are listed first, as they are usually encountered before the physical limits.

Cloud Limits

These cover CPU, memory, disk I/O (file system), and network.

CPU caputilizationsm-cpuinfo (previously jinf -c); raw counters: kstat -p caps::cpucaps_zone*:, "usage" == current CPU used, "value" == CPU cap
CPU capsaturationuptime load averages are zone-aware; per-process: prstat -mLc 1, "LAT"; rough counter: kstat -p caps::cpucaps_zone*:above_sec
CPU caperrorsN/A
Memory caputilizationsm-meminfo rss for main memory (previously jinf -m); sm-meminfo swap for virtual memory; zonememstat, "RSS" vs "CAP"; prstat -Z, zone "RSS", "SIZE" (VM); raw counters: kstat -p memory_cap:::, "rss" vs "physcap", "swap" vs "swapcap"
Memory capsaturationzonememstat, increasing "NOVER" (# over) and "POUT" (paged out); per-process: prstat -mLc 1, "DFL"; some raw counters: kstat -p memory_cap:::anonpgin
Memory caperrorsDTrace failed malloc()s; raw counters: kstat -p memory_cap:::anon_alloc_fail
FS I/O throttleutilizationN/A - it kicks in only when needed (see saturation)
FS I/O throttlesaturationvfsstat, "d/s" (delays/sec), and magnitude of "del_t" (average delay time, us)
FS I/O throttleerrorsN/A
FS capacityutilizationdf -h, "used" / "size"
FS capacitysaturationonce it's full, ENOSPC
FS capacityerrorsDTrace errno for FS syscalls; /var/adm/messages file system full messages
Network caputilizationdladm show-linkprop -p maxbw for max bandwidth (if set); dladm show-link -s -i 1 net0, for current throughput; nicstat can also show throughput
Network capsaturationnot available from within a zone (need to DTrace mac_bw_state & SRS_BW_ENFORCED)
Network caperrorsN/A

Storage devices (disks) are not listed, since limits for storage I/O are imposed at the file system layer.

Physical Resources

Since Zones are OS-Virtualization (OS partitioning), the physical resources are not emulated or virtualized, and many of the observability tools will show you the entire physical system. This can be both good – you can really understand what's going on, and confusing – why are the resources busy when my system is idle? (it's someone else; you can't see their process address space).

CPUutilizationper-cpu: mpstat 1, "usr" + "sys"; system-wide: vmstat 1, "us" + "sy"; per-process: prstat -c 1 ("CPU" == recent), prstat -mLc 1 ("USR" + "SYS"); per-kernel-thread: not available from within a zone
CPUsaturationsystem-wide: vmstat 1, "r"; per-process: prstat -mLc 1, "LAT"
Memory capacityutilizationsystem-wide: vmstat 1, "free" (main memory), "swap" (virtual memory); per-process: prstat -c, "RSS" (main memory), "SIZE" (virtual memory)
Memory capacitysaturationsystem-wide: vmstat 1, "sr" (bad now), "w" (was very bad); vmstat -p 1, "api" (anon page ins == pain), "apo"; per-process: prstat -mLc 1, "DFL"
Memory capacityerrorsfmdump; DTrace failed malloc()s
Network Interfacesutilizationnicstat (see notes below); kstat (look for physical interface kstats, eg, kstat -p | grep ifspeed to find their names, and then kstat -p ixgbe::mac: for ixgbe interfaces)
Network Interfacessaturationnicstat; kstat for whatever custom statistics are available (eg, "nocanputs", "defer", "norcvbuf", "noxmtbuf"); netstat -s, retransmits
Network Interfaceserrorsnetstat -i, error counters; kstat for extended errors, look in the interface and "link" statistics (there are often custom counters for the card); driver internals not available from within a zone
Storage device I/Outilizationsystem-wide: iostat -xnz 1, "%b"
Storage device I/Osaturationiostat -xnz 1, "wait"
Storage device I/Oerrorsiostat -En; driver internals not available from within a zone
Storage capacityutilizationswap: swap -s; file systems: "df -h"
Storage capacitysaturationonce it's full, ENOSPC
Storage capacityerrorsDTrace errno on FS syscalls; /var/adm/messages file system full messages
Storage controllerutilizationiostat -Cxnz 1, compare to known IOPS/tput limits per-card
Storage controllersaturationlook for kernel queueing: sd (iostat "wait" again)
Storage controllererrors/var/adm/messages; driver internals not available from within a zone
Network controllerutilizationinfer from kstat or nicstat and known controller max tput
Network controllersaturationsee network interface saturation
Network controllererrorskstat for whatever is there; driver internals not available from within a zone
CPU interconnectutilizationnot available from within a zone
CPU interconnectsaturationnot available from within a zone
CPU interconnecterrorsnot available from within a zone
Memory interconnectutilizationnot available from within a zone
Memory interconnectsaturationnot available from within a zone
Memory interconnecterrorsnot available from within a zone
I/O interconnectutilizationnot available from within a zone
I/O interconnectsaturationnot available from within a zone
I/O interconnecterrorsnot available from within a zone

Software Resources

Kernel mutexutilizationnot available from within a zone
Kernel mutexsaturationmpstat "smtx"
Kernel mutexerrorsnot available from within a zone
User mutexutilizationplockstat -H (held time); DTrace plockstat provider
User mutexsaturationplockstat -C (contention); prstat -mLc 1, "LCK"; DTrace plockstat provider
User mutexerrorsDTrace plockstat and pid providers, for EAGAIN, EINVAL, EPERM, EDEADLK, ENOMEM, EOWNERDEAD, ... see pthread_mutex_lock(3C)
Process capacityutilizationkstat, "unix:0:var:v_proc" for system-wide max, system-wide current usage isn't available in a zone, but "unix:0:process_cache:slab_alloc" gives a rough idea; zone: "unix:0:system_misc:nproc" for current zone usage; prctl -n zone.max-processes -i zone ZONE, "privileged/system" for zone max, and "usage" for current usage.
Process capacitysaturationqueueing on pidlinklock in pid_allocate(), as it scans for available slots once the table gets full.
Process capacityerrors"can't fork()" messages
Thread capacityutilizationuser-level: prctl -n zone.max-lwps -i zone ZONE, "privileged/system" for zone max, and "usage" for current zone usage; kernel: limited by system memory - see memory usage.
Thread capacitysaturationthreads blocking on memory allocation - see memory cap usage.
Thread capacityerrorsuser-level: pthread_create() failures with EAGAIN, EINVAL, ...; kernel: not available from within a zone
File descriptorsutilizationsystem-wide (no limit other than RAM); per-process: pfiles vs ulimit or prctl -t basic -n process.max-file-descriptor PID; a quicker check than pfiles is ls /proc/PID/fd | wc -l
File descriptorssaturationI don't think there is any queueing or blocking, other than on memory allocation.
File descriptorserrorstruss or DTrace (better) to look for errno == EMFILE on syscalls returning fds (eg, open(), accept(), ...).

What's Next

See the USE Method for the follow-up strategies after identifying a possible bottleneck. If you complete this checklist but still have a performance issue, move onto other strategies: drill-down analysis and latency analysis.

Also see the Solaris Performance Checklist if you have access to the physical host (global zone).