iostat (input/output statistics) an utility that reports Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions. The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates. The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.
The first report generated by the iostat command provides statistics concerning the time since the system was booted. Each subsequent report covers the time since the previous report. All statistics are reported each time the iostat command is run. The report consists of a CPU header row followed by a row of CPU statistics. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all processors. A device header row is displayed followed by a line of statistics for each device that is configured.
Below is the sample default output, which displays a single history since boot report for all CPU and Devices.
$ iostat Linux 2.6.18-92.1.22.el5 (dblx131.lab.perumal.org) 05/26/2009 avg-cpu: %user %nice %system %iowait %steal %idle 1.24 0.88 0.55 0.83 0.00 96.51 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 2.89 34.98 52.43 19131771 28671500 dm-0 7.27 34.67 52.11 18960146 28500800 dm-1 0.08 0.31 0.31 168184 170696
The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:
- %user: Show the percentage of CPU utilization that occurred while executing at the user level (application).
- %nice: Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
- %system: Show the percentage of CPU utilization that occurred while executing at the system level (kernel).
- %iowait: Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
- %steal: Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
- %idle: Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices for which statistics are to be displayed may be entered on the command line. Partitions may also be entered on the command line providing that option -x is not used. If no device nor partition is entered, then statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it. If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including those that have never been used. The report may show the following fields, depending on the flags used:
- Device: This column gives the device (or partition) name.
- tps: Indicate the number of transfers per second that were issued to the device.
- Blk_read/s: Indicate the amount of data read from the device expressed in a number of blocks per second.
- Blk_wrtn/s: Indicate the amount of data written to the device expressed in a number of blocks per second.
- Blk_read: The total number of blocks read.
- Blk_wrtn: The total number of blocks written.
- kB_read/s: Indicate the amount of data read from the device expressed in kilobytes per second.
- kB_wrtn/s: Indicate the amount of data written to the device expressed in kilobytes per second.
- kB_read: The total number of kilobytes read.
- kB_wrtn: The total number of kilobytes written.
- MB_read/s: Indicate the amount of data read from the device expressed in megabytes per second.
- MB_wrtn/s: Indicate the amount of data written to the device expressed in megabytes per second.
- MB_read: The total number of megabytes read.
- MB_wrtn: The total number of megabytes written.
- rrqm/s: The number of read requests merged per second that were queued to the device.
- wrqm/s: The number of write requests merged per second that were queued to the device.
- r/s: The number of read requests that were issued to the device per second.
- w/s: The number of write requests that were issued to the device per second.
- rsec/s: The number of sectors read from the device per second.
- wsec/s: The number of sectors written to the device per second.
- rkB/s: The number of kilobytes read from the device per second.
- wkB/s: The number of kilobytes written to the device per second.
- rMB/s: The number of megabytes read from the device per second.
- wMB/s: The number of megabytes written to the device per second.
- avgrq-sz: The average size (in sectors) of the requests that were issued to the device.
- avgqu-sz: The average queue length of the requests that were issued to the device.
- await: The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
- svctm: The average service time (in milliseconds) for I/O requests that were issued to the device.
- %util: Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.
Below are some frequently used examples ;
- Display a continuous device report at two second intervals.
$ iostat -d 2 Linux 2.6.18-92.1.22.el5 (dblx131.lab.perumal.org) 05/26/2009 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 2.90 34.99 52.45 19203049 28782692 dm-0 7.28 34.68 52.14 19031378 28611760 dm-1 0.08 0.31 0.31 168224 170928 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 2.00 0.00 48.00 0 96 dm-0 6.00 0.00 48.00 0 96 dm-1 0.00 0.00 0.00 0 0 .... .... ....
- Display three reports at two second intervals for all devices.
$ iostat -d 2 3 Linux 2.6.18-92.1.22.el5 (dblx131.lab.perumal.org) 05/26/2009 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 2.90 34.98 52.45 19203113 28789580 dm-0 7.28 34.67 52.14 19031442 28618648 dm-1 0.08 0.31 0.31 168224 170928 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 2.01 0.00 48.24 0 96 dm-0 6.03 0.00 48.24 0 96 dm-1 0.00 0.00 0.00 0 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 3.00 0.00 84.00 0 168 dm-0 10.50 0.00 84.00 0 168 dm-1 0.00 0.00 0.00 0 0
- Display three reports of extended statistics at two second intervals for devices hda and hdb.
$ iostat -x hda 2 3 Linux 2.6.18-92.1.22.el5 (dblx131.lab.perumal.org) 05/26/2009 avg-cpu: %user %nice %system %iowait %steal %idle 1.24 0.88 0.55 0.84 0.00 96.49 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util hda 0.13 4.33 0.68 2.22 34.98 52.45 30.14 0.11 37.77 4.11 1.19 avg-cpu: %user %nice %system %iowait %steal %idle 6.47 0.00 1.49 0.50 0.00 91.54 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util hda 0.00 3.98 0.50 3.48 3.98 59.70 16.00 0.01 1.88 1.50 0.60 avg-cpu: %user %nice %system %iowait %steal %idle 0.50 0.00 0.00 0.00 0.00 99.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util hda 0.00 4.00 0.00 1.00 0.00 40.00 40.00 0.00 1.50 1.50 0.15
- Display three reports at two second intervals for device hda and all its partitions (hda1, etc.)
$ iostat -p hda 2 3 Linux 2.6.18-92.1.22.el5 (dblx131.lab.perumal.org) 05/26/2009 avg-cpu: %user %nice %system %iowait %steal %idle 1.24 0.88 0.55 0.84 0.00 96.49 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 2.90 34.98 52.45 19203121 28793028 hda2 7.36 34.97 52.45 19200291 28793024 hda1 0.00 0.00 0.00 2302 4 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.50 0.00 0.00 99.50 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 3.02 0.00 84.42 0 168 hda2 10.55 0.00 84.42 0 168 hda1 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hda 0.00 0.00 0.00 0 0 hda2 0.00 0.00 0.00 0 0 hda1 0.00 0.00 0.00 0 0
Below are some general tips, which you can use while analyzing the output –
- Look for any devices with abnormal high blocks reads (Blk_read/s) and writes (Blk_wrtn/s) per second
- If %util is higher or near 100% utilization then it is a strong indicator for I/O bottleneck
If one of these disks are used by Oracle then you can check the data dictionary tables v$sql to see which SQL statement has generated most read/write and v$segment_statistics to see which object has generated most read/write activity.