"A little over a year ago, we tried to allocate additional disks to a volume," he remembered. "And we just couldn't pull it off."
"It was very noticeable, all over the company," Schwalbe said. "It was also extremely painful."
Schwalbe declined to confirm the supplier of the SAN, but further digging revealed it was most likely Hewlett-Packard Co.
Almost immediately, Schwalbe started the search for a new storage vendor. Besides NetApp, Schwalbe said he "evaluated their chief competitor," i.e. EMC Corp., but eventually went with a 12 TB FAS 940 for his main shop in Orlando, he said, because the network engineers who had struggled with managing the FC SAN "understood it better. The skill sets required on Network Appliance's products were very appealing compared to managing a [FC] SAN -- they fell more into their primary competencies."
Schwalbe said he preferred the NetApp over EMC because after testing both in his shop, he found NetApp's Data OnTap interfaces easier for his people to work with.
"The OS handles a lot of things that on the competitor's gear, you have to go configure yourself. And I'm not just talking about adding disks -- I'm talking about managing stripes, where the stripes are … NetApp masks all that for you," he said. "So a lower compensated type of person can manage it better just because of the interface. We can manage more with less."
In the interest of simplicity, Schwalbe also had Cisco Systems Inc. help him configure his IP LAN to handle his storage.
"My guys understand IP much better than Fibre Channel. In the end it was...more intuitive for them."
He cautioned, "It's important to note that iSCSI storage needs its own separate VLAN [virtual LAN]. Cisco had to do some port configuration for channeling for us so we could devote NICs [network interface card] to our iSCSI storage."
In the end, Schwalbe said, he was pleasantly surprised at how the IP network turned out -- the new network and upgrading the hardware on his Microsoft Exchange servers actually resulted in a performance boost.
"We thought going to iSCSI was a downgrade," he said. "But we gained performance. It was nice."
Schwalbe said Microsoft Exchange restores that used to take hours could now be performed in minutes -- but that it wasn't just smart networking that made this possible.
Following the recent hurricanes that hit Florida, Schwalbe said, it became clear that their tape-based backup system was not going to work.
"We did DR tests with every one of the hurricanes that came through, in addition to our regular annual test," he said. "What we found was that our plan of shipping tapes off site and then having someone go get them in a disaster didn't take into account that hurricanes are life events, and people can't always go and perform these functions."
As part of his NetApp purchase, then, Schwalbe selected a 16 TB NearStore R200 for replication at a hot site in Atlanta. "The NearStore automates and eliminates the human factor," he said.
Meanwhile, CNL had decided to also upgrade to a new Overland REO 4000 tape library, and in the end, almost without knowing it, found themselves with a tiered storage architecture -- currently, their production data is backed up on the FAS filer four times a day, replicated to disk at the hot site (where it is eventually cycled out) and written on the newer, faster Overland libraries for long-term archiving off site.
Finally, CNL also added security with a DataFort encryption appliance from NetApp's newly acquired subsidiary Decru, sitting between the company's Veritas NetBackup server, processing data on 82 WinTel hosts virtualized by three VMWare servers and their tape backups.
"My CEO had heard, of course, about the high-profile data losses last year and was unwilling to be the next one," Schwalbe said.
He mused, "We started off just looking for scalability -- but along the way we added performance, DR and security, and all of it is simpler to manage."