News Stay informed about the latest enterprise technology news and product updates.

IDC: Unstructured data will become the primary task for storage

According to a new IDC Enterprise Disk Storage Consumption Model report released this week, transaction-intensive applications are giving way as the main segment of enterprise data to an expanded range of apps as well as a tendency to create more copies of data and records for business analytics including data mining and e-Discovery.

The report estimates that unstructured data in traditional data centers will eclipse the growth of transaction-based data that until recently has been the bulk of enterprise data processing. While transactional data is still projected to grow at a compound annual growth rate of 21.8%, it’s far outpaced by a 61.7% CAGR predicted for unstructured data in traditional data centers.

“In the very near future, the management and organization of file-based information will become the primary task for many storage administrators in corporate datacenters,” the report reads. “And this shift will have a significant impact on how companies assess storage solutions in terms of systems’ performance, operational efficiency, and file services intelligence.”

The IDC report also builds on research first highlighted in an IDC blog last week concerning the cloud. According to the report, the sharpest growth in storage capacity will come from new organizations described as “content depots.” IDC estimates storage consumption from these organizations will grow at a compound annual growth rate of 91.8% through 2012. Examples of content depots  include the usual cloud suspects: Google, Amazon, Flickr, and YouTube.

These content depots have different IT requirements and infrastructures than traditional enterprise data centers. We’re seeing examples of these new  infrastructures pop up in the market, including systems with logical abstraction between the hardware and software elements; the use of commodity servers as a hardware basis for storage platforms; and the use of clustered file systems.

Some in the industry have compared this “serverization” of storage to the transition between proprietary workstations and PCs in the 1980’s. But IDC analyst Rick Villars says this isn’t a zero-sum game. “This isn’t going to replace traditional IT,” he said. “Ninety-five percent of what people are developing and building in the storage industry today is irrelevant to what the cloud is building. You could take that as a negative, but it also translates into opportunity. These are new market spaces and new storage consumers that weren’t around five years ago.”

There’s been a lot of discussion lately about the role the cloud will play as the global economy softens. There is a difference of opinion between those who see a capital-strapped storage market as an even more conservative and risk-averse one and those who argue the opportunity to avoid capital expenditures will nudge traditional IT applications into the cloud. Still others point out the hurdles to cloud computing that remain, including network scalability and bandwidth constraints.

For example, when it comes to storage applications such as archiving, analyst reports from Forrester Research this year cited  latency in accessing off-site archived messages and searching them for e-discovery as major barriers to adoption for archiving software-as-a-service (SaaS) offerings.

Cloud computing “definitely exposes weaknesses in networking,” Villars said, but “the closest point to the end user is the cloud, if you want to distribute content to end users spread around the world.”

Other challenges include the growing pains major cloud infrastructures such as Amazon’s S3 have experienced over the last 18 months, and the potential risk of putting more enterprise data eggs in one service provider’s cloud data center basket. Villars points out, “I doubt Amazon has had more problems than a typical large enteprise, and they offer backup with geographic distribution for free.”

However, geographic distribution brings with it its own challenges, such as varying regulations among different countries. “There are regulatory problems with Europe,” Villars said. “Laws there say that if you have data on a European customer, yhou can’t move it out of Europe. If you want your cloud provider to spread copies between the U.S., Asia and Europe for global redundancy, that becomes an issue.”