If your system spends a lot of time idle, even under heavy load, you have the right to suspect disk I/O problems. Remember that memory problems and I/O problems are often related. Here are some steps that will help to confirm that your system is suffering disk problems:
- Run iostat 5 (sar-d, if your system supports it). Compare that result with your "healthy system" data. Is disk activity much higher than normal?
- Is disk activity distributed fairly among the system's disks?
- If not, are the most active disks also the fastest disks?
- Run sadp (some System V, but not all) to get a seek histogram.
- Is activity concentrated in one area of the disk (good), spread evenly across the disk (tolerable), or in two well-defined peaks at opposite ends (bad)?
- Are you using NFS? Are users reporting slow file access working locally, or are they accessing a remote filesystem? If they are using remote files, look at your network situation. You don't have local disk I/O problems.
- Look at your memory statistics by running vmstat 5 (sar -rwpg). If your system is paging or swapping consistently, you have memory problems, and they are undoubtedly making disk I/O worse than it would be otherwise. Attack your memory problems first.
- If any of these tests indicates a problem, try these approaches:
- Reorganize your filesystems and disks to distribute I/O activity as evenly as possible.
- Use your fastest disk drive and controller for your root filesystem - this will almost certainly have the heaviest activity. Alternatively, if single-file throughput is important, put performance-critical files into one filesystem and use the fastest drive for that filesystem.
- Put performance-critical files on a filesystem with a large block size: 16 KB or 32 KB for BSD, 2 KB for System V.
- For System V.4, use UFS (BSD-style) filesystems instead of the System V (S5) filesystem.
- Increase the size of the buffer cache by increasing BUFPAGES (BSD), NBUF (V.3), or BUFFERS (XENIX, V.2). This may hurt your system's memory performance.
- Rebuild your filesystems periodically to eliminate fragmentation. System V.4 users can use dcopy; other System V and BSD implementations should do a backup, build a new filesystem, and restore.
- dcopy has some options to concentrate heavily used files at the beginning of the filesystems. If sadp indicates that you have a bad seek pattern (two distinct peaks, many long seeks), try using dcopy to fix the situation.
Source: O'Reilly - System Performance Tuning - Help for UNIX System Administrators