iostat and sar disk analysis
In previous searchStorage.com Admin tips, we discussed the commands iostat and sar. Both commands are useful in disk and CPU analysis. The previous tips simply detailed the output of the commands in general terms. This tip will provide specific examples of high disk usage analysis.
Excessive I/O generated requests is illustrated in this example where the disk, sd15, is a magnetic disk used as a log in a high transaction rate environment.
sdo sd15 CPU rps wps util rps wps util us sy wt id 0 0 0 0 90 90 20 40 40 0 0 0 0 0 0 89 19 42 40 0
This is an implementation where more I/O requests are generated each second than a disk can service. The disk utilization of 90% with fully 80% of the CPU used in system mode or waiting for I/O to complete, indicates this. CPU idle time is zero, because of the excessive CPU resources being used to manage the I/O queue or waiting for the I/O to complete.
If sar was used in the same analysis on the log disk sd15 in the previous example, the results should show high avque numbers due to the high CPU resource consumption.
Because the disk speed is the actual limiting factor, adding more CPU does not help, as the additional CPU generates more unserviced I/O requests, and a larger utilization that consumes even more CPU in I/O queue management.
Replacing the log disk with a very high rate transaction device, such as a solid state disk, raises the I/O rate of the limiting factor--the disk speed--and when iostat is again run the following results would theoretically be obtained:
sdo sd16 CPU rps wps util rps wps Util us sy wt id 0 0 0 0 322 17 40 22 5 34 0 0 0 0 300 16 44 23 8 26
Notice that the percentage of CPU waiting drops from 40% to 5% - a 35% reduction, while approximately 200 more log writes are completed each second with only 17% of the disk utilized. Because of this additional speed, I/Os are completed as they are requested, and the CPU no longer queues I/Os. Thus, the percentage of CPU spent in system mode is reduced and CPU usage doubles. In addition, 30% of the CPU is idle and available for more work. Thus, the elimination of the disk "bottleneck" at the log has now made the system capable of significantly more work, while using less CPU resource.
Database applications can vary widely in the ratio of system resources used. In general, however, high disk utilizations, very low or very high CPU idle percentages and large percentages of CPU spent in waiting or system mode, indicate significant unutilized CPU.
This tip was provided to searchstorage.com courtesy of Imperial Technology, manufacturers of diskless servers.
What did you think of this tip? Like it? Hate it? Email and let us know.