One of the most common reasons SAN performance degrades is one or more saturated links. A saturated link is a connection through the SAN between two objects where the instantaneous load on the link exceeds the capacity of the data path. The problem typically snowballs because requests that use the link stack up and keep the link overloaded for longer than the original saturation -- sometimes much longer.
The good news about saturated links is that they are easy to identify with modern storage- management tools. Most such tools will give you reports showing the loads on the various paths through the SAN and many of them have alarm functions that activate when a link saturates.
The cure for a saturated link is to remove some of the load. The two most common methods are to add redundant paths between the objects to split the load and to move the logical volume to another physical storage device that is less heavily loaded.
Dell discusses this and other SAN problems that can be solved by tuning in the context of the version of the EMC Performance Manager for its VisualSAN product in an article titled, "Tuning SANs for peak performance with EMC VisualSAN performance manager". Even though the article is aimed at the VisualSAN/Performance Manager combination, much of the advice is broadly applicable.
Rick Cook has been writing about mass storage since the days when the term meant an 80K floppy disk. The computers he learned on used ferrite cores and magnetic drums. For the last twenty years he has been a freelance writer specializing in storage and other computer issues.