This is a demonstration of the procsystime tool, which can give details on how processes make use of system calls. Here we run procsystime on processes which have the name "bash", # ./procsystime -n bash Hit Ctrl-C to stop sampling... ^C Elapsed Times for process bash, SYSCALL TIME (ns) setpgrp 27768 gtime 28692 lwp_sigmask 148074 write 235814 sigaction 553556 ioctl 776691 read 857401243 By default procsystime prints elapsed times, the time from when the syscall was issued to it's completion. In the above output, we can see the read() syscall took the most time for this process - 8.57 seconds for all the reads combined. This is because the read syscall is waiting for keystrokes. Here we try the "-o" option to print CPU overhead times on "bash", # ./procsystime -o -n bash Hit Ctrl-C to stop sampling... ^C CPU Times for process bash, SYSCALL TIME (ns) setpgrp 6994 gtime 8054 lwp_sigmask 33865 read 154895 sigaction 259899 write 343825 ioctl 932280 This identifies which syscall type from bash is consuming the most CPU time. This is ioctl, at 932 microseconds. Compare this output to the default in the first example - both are useful for different reasons, this CPU overhead output helps us see why processes are consuming a lot of sys time. This demonstrates using the "-a" for all details, this time with "ssh", # ./procsystime -a -n ssh Hit Ctrl-C to stop sampling... ^C Elapsed Times for process ssh, SYSCALL TIME (ns) read 295392 write 622903 pollsys 1030310531 CPU Times for process ssh, SYSCALL TIME (ns) read 183515 write 534289 pollsys 650729 Syscall Counts for process ssh, SYSCALL COUNT read 12 write 12 pollsys 24 Now we can see elapsed times, overhead times, and syscall counts in one report. Very handy. procsystime also lets us just examine one PID. For example, # ./procsystime -p 1304 Hit Ctrl-C to stop sampling... ^C Elapsed Times for PID 1304, SYSCALL TIME (ns) fcntl 7323 fstat64 21349 ioctl 190683 read 238197 write 1276169 pollsys 1005360640 Here is a longer example of running procsystime on mozilla, # ./procsystime -a -n mozilla-bin Hit Ctrl-C to stop sampling... ^C Elapsed Times for process mozilla-bin, SYSCALL TIME (ns) lxstat 33754 xstat 92812 close 94176 lseek 112718 open64 282248 access 308738 unlink 453839 readv 2780728 ioctl 10298181 read 16333489 write 31447612 yield 37993903 pollsys 15743829056 lwp_park 19384565166 CPU Times for process mozilla-bin, SYSCALL TIME (ns) lxstat 20116 lseek 34313 close 43205 xstat 77725 access 214299 open64 222881 unlink 438337 yield 455264 readv 1564088 ioctl 4385990 read 8891318 lwp_park 15667040 write 24652699 pollsys 26553426 Syscall Counts for process mozilla-bin, SYSCALL COUNT lxstat 2 unlink 2 xstat 2 access 8 close 8 open64 8 lseek 10 yield 32 readv 180 lwp_park 546 write 730 ioctl 840 pollsys 1017 read 1084