Here's a classic storage manager's nightmare: Suddenly end users can't access 100GB of critical data because of a bent pin on a SCSI connector. And here's what really makes you wake up in a cold sweat in the middle of the night: There's no backup copy.
That happened at SAS Institute a few years ago and it ignited an effort to revamp how backups were done. Backup at the Cary, NC, software company is serious business - at stake are millions of lines of code and dozens of simultaneous software builds.
Kelly Wyatt, a senior systems programmer at SAS, knows these challenges all too well. Her team, which supports more than 4,000 people involved in maintaining the 47 individual applications within the company's product portfolio, is continually looking for better ways to manage the company's nearly 8TB of live production data.
The immensity of SAS' storage needs made it clear to Wyatt and her team that simply relying on backups by trying to coordinate independent local tape systems was never going to provide adequate manageability. The firm installed filers from Sunnyvale, CA-based Network Appliance, which not only made highly scalable file servers, but also had worked closely with Intelliguard - now part of Legato - to develop the flexible backup management protocol, Network Data Management Protocol (NDMP).
Although initially designed to improve backup of NetApp filers, NDMP has taken on a wider role: Most network-attached storage (NAS) vendors have endorsed the protocol, saying it's the best way to provide plug-and-play connectivity between backup management software and filer-attached tape drives. If you've got to back up NAS boxes, you need to know about NDMP. It's now maturing as a standard, and gaining storage area network (SAN)-like features such as snapshot.
"It's about ensuring that you're backing up everything that needs to be backed up," says Jay Desai, data protection solutions manager with NetApp. "This is a universal backup agent for all applications. If you have a lot of IP investment, this is a complement to that solution just like Fibre Channel [FC] attached tape drives are in a SAN. That's one of the reasons people implement NDMP - to leverage their investment in large tape resources."
NDMP eliminates the need for data to be funneled from remote locations through a central backup server, then dumped into a single backup repository. Initially, it enabled centralized management of multiple local tape drives in each filer. Newer versions allow you to use centralized tape repositories. Built in a client/server model, NDMP has hooks in both backup management software - the client - and virtual state machines built into compliant networked storage devices - the servers.
At SAS, IT uses Veritas' NetBackup to manage backup of 18 NetApp filers, which provide 11.5TB of NAS space used to store data from Windows and Unix systems spread across multiple logical volumes. Some 30 locally attached DLT 7000 drives are involved, with four drives on each of two NetApp filers dedicated to backing up data from non-NetApp file systems.
Wyatt explains the impracticality of the alternative - going to each filer, typing DUMP and then trying to make sense of all the tapes. "We offsite hundreds of tapes a week, and don't want to have to hand label what they are. With NDMP, we are able to very effectively and efficiently back up 8TB of data on 18 filers, and it's really helped us with consolidating data on fewer servers."
NDMP's expanding role
As NDMP has evolved, its expanding scope has forced a redefinition of terminology. A NDMP client is now also referred to as a Data Management Application (DMA), while a NDMP server is now also called a Data Service Provider (DSP). Various DSPs have been defined, including data (e.g., servers with internal storage, NAS devices, SAN-attached storage subsystems), tape (jukeboxes, general servers with shared tape drives or writable CD-R libraries) and SCSI pass-through DSPs. The upcoming NDMP v.5 adds a Translate DSP, including N to M stream multiplexers and data compression.
Building a storage strategy around NDMP can deliver faster performance rather than relying on the indirect paths taken in conventional backup models. This is because NDMP streams - which only carry a small volume of management-related information - direct the data flow between servers and their locally attached storage devices. If you opt for locally attached tape, data itself isn't transmitted over the corporate network, eliminating the congestion that can degrade application performance and push backups perilously close to the edge of their backup windows.
If you're interested in centralizing tape in a large library, NDMP v.2 and v.3 support sophisticated three-way backups. These allow NDMP-compliant filers to direct their data to backup devices that may be connected to a different filer altogether. By combining this configuration with a restore from attached storage, a user can automatically schedule redundant backups of stored data.
For example, data from one filer-attached tape drive can be shuttled to another tape drive using NDMP. Three-way backups allow data to be pushed to tape jukeboxes, from one filer to another - allowing maintenance of online backups instead of pushing data to tape - or from a filer without local tape storage to one that's connected to a library. Data can be sent from multiple filers to a single tape library, or split across multiple, individually attached tape libraries to increase backup performance.
Because it's built around TCP/IP and not the expensive FC connections required for distributed SAN backups, NDMP is an easier and faster approach that delivers similar functionality into the NAS environment. That makes it a godsend for companies that have struggled with the personnel burden of expanding backup tasks as their corporate data grows.
"We have quite a few techniques for heterogeneous backup and recovery on the SAN side," says Mike Adams, product marketing manager for NetBackup with Veritas. "The question is, do you want to have a box that's very easy to manage? People choose a NAS device because they want file sharing that's simple and easy; SAN folks decide they want to do block-level sharing of information. Depending on your business needs, that can be more technology than you want to deploy."
Support for Windows and Linux
Because NDMP is now supported in Windows CIFS (common Internet file systems) and Unix NFS (Network File System), it can back up data from both environments while retaining each one's file attributes and other idiosyncrasies. This capability led Honeywell Satellite Systems (HSS), Glendale, AZ, to implement NDMP along with an upgrade of its tape drives intended to speed backup of its 700GB Auspex NS2000 file server. Although most configurations maximize speed by directing data to locally attached libraries, NDMP's TCP/IP heritage also provides more backup flexibility by allowing aggregation of backup data from multiple NDMP-compliant filers onto libraries that aren't physically attached to them. This allows customers to put backup data where it's needed over standard networks, while still retaining full manageability over those backups from the central backup software.
Wherever backup data is ultimately stored, the NDMP-compliant tape device transmits confirmation of the write-back to the central backup server. At the same time, data is directed straight from the server to the tape library, often as quickly as the tape can take it. This reduces costs and improves the speed of NDMP solutions.
Integrated circuit maker Advanced Micro Devices (AMD) Sunnyvale, CA, boosted backup speeds to 60GB/hour, per filer using Workstation Solutions' NDMP-compliant Quick Restore to back up 15TB of Unix, Linux and Windows data from 27 NetApp filers to six tape libraries, including 35 SCSI-attached tape drives. The move has improved the robustness of the company's backup strategy, paved the way for future growth and eased management of AMD's backup environment.
Vendors jump on NDMP bandwagon
NetApp originally envisioned NDMP as a way to free itself from the need to write proprietary hooks to link its filers with every proprietary management environment on the market. By giving the data servers the ability to proactively notify backup software about the files being backed up, those servers could take a more hands-off approach once the backup or restore had commenced.
This promise was a major part of the reason Legato snapped up Intelliguard for $52 million early in 1999, then rushed NDMP-compliant versions of its products onto the market later that year.
Since then, broader support for NDMP has largely solved the major headache of NAS device interoperability. At least 39 product families from 30 vendors now support the protocol, allowing customers to choose backup software and NAS devices based on their individual characteristics, not compatibility.
NDMP is based around just eight specific operating system interfaces. Connect, configure, data, tape and SCSI provide a direct interface and control over remote tape drive and SCSI devices from NDMP-compliant vendors. Notify, file history and logging interfaces allow backup software to manage file histories of NDMP-based remote backups.
In theory, any company can easily make its products NAS-capable by implementing these interfaces (SDKs for each version of NDMP are available from http://www.ndmp.org/download). This ease of use has allowed many smaller startups to quickly bring devices such as storage appliances into the NAS market without having to first win support and certification from individual backup software vendors.
"One of the things that's key to our success is ensuring that customers don't have to change anything," says Geoff Barrall, chief technology officer of storage appliance maker BlueArc. "NDMP removes all the driver hassles customers used to have."
It certainly seems to have served this purpose: customers generally report NDMP implementation to be relatively painless, although tweaking can be necessary. For example, SAS had to increase the NDMP timeout to allow for often long delays while filers searched through hundreds of gigabytes of files to identify those that had changed. Such issues, however, are small given NDMP's ability to overcome the massive challenge of interoperability.
Lack of standards
Still, all isn't necessarily well in the world of NDMP standards. Because vendors initially installed NDMP for use with their own products, there were many interpretations of what was actually needed to ensure full compliance. And with no formal standards body to set the record straight, there have been reports of inconsistencies between implementations causing headaches for storage managers keen to take advantage of the protocol's flexibility.
Furthermore, not every vendor supports every form of NDMP transfer. One common inconsistency is support for Direct Access Recovery (DAR), a NDMP capability that allows NDMP-compliant servers to address individual files contained within volume backups. DAR support is far from ubiquitous, but its appeal to customers should make it more common in the future.
In recognition of the growing support for NDMP after the release of NDMP v.3 - now common in most implementations of the standard - its development was handed over to the Storage Networking Industry Association (SNIA), which in mid-2000 formed a NDMP working group to refine the details of NDMP v.4. This year has finally seen vendors release products based on that version, which includes a range of enhancements including improved restartability of interrupted backups and support for custom extensions.
First among these extensions is NDMP Snapshot Management, a NetApp add-on that allows NDMP-compliant devices to take and operate on instantaneous backups of their data. This should allow companies to save considerably on hardware and reduce the complexity of managing massive tape archives.
"With this technology you don't have to worry about [backup] window time anymore," says Hieu Le, senior network system administrator with programmable semiconductor manufacturer Altera, San Jose, CA, which is using NDMP-compliant products from CommVault Systems, BlueArc, and Quantum to back up 2TB of data accessed by a staff of more than 600. NDMP's seamless scalability will help Altera expand its backup strategy to meet storage demand expected to top 5TB by the end of the year.
With NDMP v.4 out the door, work has already begun on the next major upgrade, v.5. This time, the standardization is being driven under the auspices of the IETF (Internet Engineering Task Force), a move that was made to help NDMP tap into the organization's extensive knowledge base pertaining to areas such as security, authentication, operation through firewalls and congestion control. An initial specification is expected by mid-2003.
Now that NDMP is firmly entrenched within the panoply of backup vendors, it has emerged as the great unifier: a means of resolving potential incompatibilities and adding greater flexibility to relatively inexpensive and well-understood NAS environments. Coming versions will add even better features, but NDMP's already considerable growth and successful implementation confirms its value for the long run.
|NDMP roadmap: an evolving protocol|