Detecting Disk I/O Problems

If your system spends a lot of time idle, even under heavy load, you have the right to suspect disk I/O problems....

Remember that memory problems and I/O problems are often related. Here are some steps that will help to confirm that your system is suffering disk problems:

• Run iostat 5 (sar-d, if your system supports it). Compare that result with your "healthy system" data. Is disk activity much higher than normal?
• Is disk activity distributed fairly among the system's disks?
• If not, are the most active disks also the fastest disks?
• Run sadp (some System V, but not all) to get a seek histogram.
• Is activity concentrated in one area of the disk (good), spread evenly across the disk (tolerable), or in two well-defined peaks at opposite ends (bad)?
• Are you using NFS? Are users reporting slow file access working locally, or are they accessing a remote filesystem? If they are using remote files, look at your network situation. You don't have local disk I/O problems.
• Look at your memory statistics by running vmstat 5 (sar -rwpg). If your system is paging or swapping consistently, you have memory problems, and they are undoubtedly making disk I/O worse than it would be otherwise. Attack your memory problems first.
• If any of these tests indicates a problem, try these approaches:
• Reorganize your filesystems and disks to distribute I/O activity as evenly as possible.
• Use your fastest disk drive and controller for your root filesystem - this will almost certainly have the heaviest activity. Alternatively, if single-file throughput is important, put performance-critical files into one filesystem and use the fastest drive for that filesystem.
• Put performance-critical files on a filesystem with a large block size: 16 KB or 32 KB for BSD, 2 KB for System V.
• For System V.4, use UFS (BSD-style) filesystems instead of the System V (S5) filesystem.
• Increase the size of the buffer cache by increasing BUFPAGES (BSD), NBUF (V.3), or BUFFERS (XENIX, V.2). This may hurt your system's memory performance.
• Rebuild your filesystems periodically to eliminate fragmentation. System V.4 users can use dcopy; other System V and BSD implementations should do a backup, build a new filesystem, and restore.
• dcopy has some options to concentrate heavily used files at the beginning of the filesystems. If sadp indicates that you have a bad seek pattern (two distinct peaks, many long seeks), try using dcopy to fix the situation.

Source: O'Reilly - System Performance Tuning - Help for UNIX System Administrators