Disk-based WORM eases compliance issues

Compliance Series: Part 3

By Heidi Biggar

Of the more than 25 petabytes (PB) of Centera capacity EMC says it has shipped to date, 7PB comes from sales of the platform's Compliance Edition in the first quarter of this year, according to the company.

This statistic not only reflects a growing market opportunity for EMC in compliance-related markets, but is also reflective of increasing interest among users across varied industries to implement disk-based write-once, read-many (WORM) technologies, such as the Centera Compliance Edition, as part of their regulatory compliance processes.

Besides EMC, disk-based WORM systems are also available from Hewlett-Packard (StorageWorks Reference Information Storage Systems, or RISS), IBM (TotalStorage Data Retention 450), Network Appliance (SnapLock and LockVault), Permabit (Permeon Compliance Vault) and, in the first quarter of 2005, Archivas (Arc).

The Enterprise Strategy Group (ESG) consulting firm expects the worldwide capacity of compliant records stored on disk-based WORM systems to increase at a compound annual growth rate (CAGR) of 172% from 2003 to 2006.

This compares to worldwide capacity of compliant records stored on tape-based systems and optical WORM, which ESG projects will decline 5% and 17% CAGR, respectively, over the forecast period.

ESG also expects e-mail archiving products and content/document management products to play a significant role in users' compliance environments because they can be used in conjunction with disk-based WORM systems to facilitate retrieval, archival, search, workflow, and auditing of stored data.

For example, as noted in the first part of our series of articles on compliance (see "Users receive a compliance wake-up call," InfoStor, September 2004, p. 1), enterprises that have enterprise content management (ECM) systems in place are more likely than those that don't to train staff on records and information management, which illustrates the link between ECM and regulatory compliance.

Overall, ESG expects the aggregate capacity of compliant records in all vertical markets to increase from 376PB in 2003 to 1,644PB in 2006, for a CAGR of 64%. While virtually all industries will be affected by the deluge of compliant records, life sciences is expected to experience the fastest rate of increase, with a CAGR of 86% projected over the period, according to ESG.

The government sector (78% CAGR), financial services (74%), and healthcare (52%) industries are also expected to see significant growth in compliance-related capacity as a result of DoD 5015.2, SEC 17a 3-4, and HIPAA, respectively (see figure).

Click here to enlarge image

Numbers aside, regulatory compliance is relatively new to most organizations, which makes implementing processes and choosing products difficult. What works for one organization may not work for another due to the size of the organization, IT budget, corporate culture, type and quantity of data that needs to be retained, etc.

For example, although EMC's Centera Compliance Edition may be an appropriate compliance system for Memorial Hermann Healthcare System, a provider of healthcare in the greater Houston and southeast Texas communities, it may not be appropriate for mid-tier investment management firms such as Essex Investments, in Boston.

"We looked at [EMC's Centera and HP's RISS], but they were too big and too expensive," says Randy Wilson, assistant vice president of IT at Essex Investments. "We had no need for 2.4TB of storage space."

After extensive testing, Essex decided to implement Permabit's Permeon Compliance Vault (running CommVault's QiNetix DataArchiver software) because of its scalability, features, and "user feel," according to Wilson, who adds that Per-mabit was a lot easier to work with than some of the larger vendors in this space.

According to the International Data Corp. (IDC) research firm, Essex's experience is typical of small and medium-sized businesses (SMBs), which "can face specific challenges [such as limited resources, both human and financial, or integration with existing applications] when dealing with compliance regulations" (see IDC Opinion #31543, July 2004)

Additionally, IDC says that "many existing compliance solutions are often inadequate for SMBs" and that "cooperation between storage hardware and software providers is essential for delivering cost-effective compliance solutions."

Essex implemented Permabit's Permeon Compliance Vault early this year and is currently testing Permabit Replication, a new feature that allows users to keep a second WORM copy of e-mail data on an off-site Compliance Vault for disaster-recovery purposes (in compliance with the SEC's "second-copy" rule)

While EMC appears to be working on a mid-tier version of Centera Compliance Edition, Network Appliance has been strengthening its reach on the high-end. In September, the company introduced NetApp LockVault Compliance Software, which is intended to help large companies bring unstructured data types (e.g., flat files, PowerPoint files, Excel spreadsheets) under the compliance umbrella. NetApp SnapVault Compliance Software is designed for semi-structured and structured data, including e-mail, which accounts for as much as 50% of all user data, according to ESG.

Like EMC, Network Appliance says it is seeing good traction with its disk-based compliance systems. It claims to have shipped more than 2PB of compliant storage related to sales of its SnapVault Compliance Software.

"Our large customers shared a common a challenge: They had a lot of data that was sitting outside e-mail that needed to be retained to meet compliance requirements, and they were looking for a way to manage the data that wouldn't require them to deploy another ECM application," according to Michael Marchi, senior director of compliance and ILM solutions, at Network Appliance.

"As they saw it, ECM applications were potentially cost-prohibitive [from a deployment and training perspective] and risky since they relied heavily on users to classify data for retention. What they were looking for instead was a product that could give them a picture of what their systems looked like on a daily basis, so if the SEC came calling, they could easily and quickly [recover] files at any point in time," says Marchi.

NetApp LockVault allows users to deploy a single architecture for both backup and compliance. After the initial full backup, LockVault does nightly snapshots (of changed blocks only).

Data is backed up to NetApp NearStore devices, where it is stored in "locked" volumes with pre-set retention periods. (Retention periods are set on a server-by-server basis, not at the individual file level.) A Compliance Journal keeps track of data changes between snapshots, and like the backup volumes, journal volumes are saved in a WORM volume on a backup target.

LockVault Compliance Software supports Linux, Windows, and Unix servers and can be used with NetApp SnapMirror software for WORM-to-WORM remote replication.

Despite its backup capabilities, Net-App says the architecture is specifically designed for regulatory-compliance purposes. "We wouldn't expect a company doing regular backups to buy this product for non-compliance purposes," says Marchi.

HP and IBM also offer disk-based WORM systems for compliance purposes. In fact, one of HP's customers-NetBank-is currently using HP's RISS system for e-mail-compliance purposes and has plans to bring other unstructured data types, such as Word documents and tiff images, onto the system in the future.

"Our objective is to put anything on the RISS system that is compliance-related-that is, any data that could potentially be recalled over some period of time," says Chip Register, a senior vice president and CTO for NetBank.

Register expects the cost savings from HP's RISS technology to be significant, especially in terms of the time spent retrieving files for compliance purposes. "We won't have to pull tapes from off-site and then build the restores," says Register, "and as an added benefit we'll no longer have to put limits on Exchange."

HP's RISS is an "all-in-one" archive-and-retrieval system for indexing reference information based largely on technologies acquired through the company's acquisition of Persist Technologies. The system can currently be used for both e-mail and Word document retention and can be integrated with hardware and software from HP partners ADIC, CaminoSoft, Grau Data Storage, Orchestria, Pegasus Disk Technologies, and Princeton Softech.

This article was originally published on November 01, 2004