‘To dupe, or not to dupe’ is not a question

Data de-duplication specialist Data Domain’s successful IPO last month was a clear signal that this technology has hit the big time. The IPO occurred amidst a flurry of de-duplication-related announcements.

Quantum, for example, says that later this year it will deliver a system that supports both inline and post- process de-duplication, which would give users an option while icing the controversy between the two approaches. (Quantum got its data de-dupe technology in its acquisition of ADIC, which had acquired de-dupe pioneer Rocksoft.)

Network Appliance and others are extending data de-dupe beyond its traditional role in backup scenarios, into nearline and primary storage devices and applications (see “NetApp extends de-dupe beyond backups,” InfoStor, June 2007, p. 8).

Users will have to wait until early next year to get data de-duplication functionality on EMC’s disk libraries (aka virtual tape libraries, or VTLs), but the company recently announced support for data de-dupe in both its VMware and NAS platforms (see “EMC supersizes VTLs, extends de-dupe,” p. 8).

How hot is the data de-duplication market? Pretty hot. The 451 Group research and consulting firm expects it to grow from $100 million last year to more than $250 million this year. At that growth rate, it could become a $1 billion market by 2009.

The rapid growth is due to the fact that data de-duplication is, for the most part, a no-brainer technology that has immediate appeal to end users. It provides a sharp reduction in required capacity as well as high-speed recovery, with few drawbacks other than having to evaluate the various approaches (inline vs. post-process, hash-based versus byte-level, etc.).

Users also have to watch out for vendors’ insane de-dupe ratio claims. Some vendors, for example, claim a 500x reduction. Reality: Results from a recent end-user survey by the 451 Group indicate that most de-duplication users experience a 15x to 20x reduction in data, although a few achieved greater than 50x. Other respondents experienced data-reduction rates of less than 5x.

Click here to enlarge image

The July issue of InfoStor will include an in-depth look at the data de-duplication market and the various technology approaches, as well as end users’ experiences and expectations.

Dave Simpson

This article was originally published on July 01, 2007