CASE STUDIES: How users address compliance

Hint: It's often an afterthought

By Michele Hope

—Given the amount of vendor hype surrounding compliance, you would think IT managers spend (or should spend) most of their time sorting out how best to comply with the latest spate of federal and state regulations, or how to prepare their data to be ready for the threat of looming lawsuits.

Yet, a glimpse into the practices of IT managers tends to reveal a more-balanced emphasis on compliance as just one piece—and often an adjunct one, at that—to many of the more-pressing storage issues they face, such as mounting storage costs, double-digit annual data growth, a quest for more-efficient operations management, and their organization's efforts to successfully back up and recover critical data according to pre-defined service levels.

That's not to say such users ignore compliance issues. It's more that the solutions they currently use for compliance are often deployed first for some other reason. The fact that they are now being used (or soon will be used) for compliance is seen by many as an added bonus to an already-proven technological investment.

E-mail archiving solutions
In several cases, managing the storage required for their company's growing volume of e-mail became the impetus for initial change. Compliance requirements were introduced later in the process. Deploying more-advanced backup-and-archival solutions for the job tended to introduce not just a way to shrink the number of costly e-mail production servers required, but also a way to usher such users into a more compliance-oriented IT framework.

A case in point is Cedars-Sinai Medical Center (CSMC) in Los Angeles. According to Jim Brady, the medical center's senior e-mail administrator and messaging architect, CSMC had begun to suffer the aftereffects of an early decision to provide Exchange users with ongoing access to old e-mail, regardless of its date or frequency of access.

It didn't take long before message stores began growing at 100% per year. CSMC soon found itself needing to buy more and more costly Exchange servers, which were soon filled to maximum capacity.

Like many organizations facing the same problem, Brady first tried "going the PST route" for the group of power users whose mailboxes tended to use the most storage space. That proved too problematic and time-consuming for IT to troubleshoot users' subsequent PST access issues.

Brady's next move was to get the e-mail storage off the medical center's Exchange servers by consolidating it into a Fibre Channel SAN. That move didn't work out as planned, either. As Brady puts it, "We constantly had to upgrade the disks and swap them out. Plus, we were still reaching sizes that were way too large."

Brady subsequently focused on how to reduce the size of Cedar-Sinai's growing message stores prior to an upcoming migration to Microsoft Exchange Server 2003. This is when Brady began looking seriously at e-mail archiving vendors. He soon honed in on Symantec's Enterprise Vault (marketed at the time by KVS), which quickly edged out competitors at the time due to what Brady saw as its perceived ease of use for those trying to locate previously archived messages and attachments.

CSMC now boasts an IBM-based 4Gbps Fibre Channel SAN infrastructure that supports Exchange, Oracle, and SQL Server. An IBM DS-4300—incorporating both SCSI and SATA drives—houses the Symantec Enterprise Vault e-mail archive.

All e-mails older than three months are now archived automatically to Enterprise Vault. A small archive icon appears in the user's mailbox, which makes it easy for him or her to access and view previously archived messages. Links to archived attachments also are easy for users to navigate. The number of Exchange servers has gone from 15 to 5, with as much as 75% of the total e-mail storage capacity soon to be handled by the Enterprise Vault archive.

HIPAA versus future legal action
Because Enterprise Vault's core functionality also offers the ability to perform basic searches of the contents of archived e-mail, the system proved useful for a recent lawsuit and e-discovery request related to 20 mailboxes. "It just so happened all the mailboxes had been archived with Enterprise Vault, going back five years," says Brady. "Even though we didn't have some of Enterprise Vault's more compliance-specific pieces set up yet, such as Compliance Accelerator or Exchange Journal Archiving, we were able to find a lot of the e-mails they were looking for doing just a Google-type search off-site from a Web interface."

According to Brady, responding to potential litigation requests is a much more immediate risk to safeguard against than the oft-touted HIPAA legislation aimed at protecting patients' personal data. "HIPAA doesn't really have any teeth to it, especially in the area of compliance regarding electronic communications. The fine you'd get is $25,000 if you're totally flagrant, and the worst you could get is $250,000," says Brady. "It hasn't been a motivating factor in this case. ?There are very few regulatory things they can really do to a large organization that has things halfway in place."

Brady says a more likely risk for the medical center—and a large portion of healthcare organizations—is the prospect of getting hit with a class-action or defamation lawsuit. For these scenarios, he believes Enterprise Vault addresses the medical center's needs. CSMC has already begun to implement several of Enterprise Vault's more-compliance-centric options, including acquiring licenses for 1,000 PST Migrator options, and a handful of licenses for options such as Exchange Journal Archiving, Compliance Accelerator, and Discovery Accelerator. Brady's goal with these newer Enterprise Vault options is to start involving members of the medical center's human resources and legal teams to "test the waters" for discovery, internal, or external compliance auditing, and the development of future retention policies.

Brady consults closely with corporate compliance personnel, the medical center's legal counsel, and the chief security officer. Yet, despite the medical center's forward-thinking approach to archiving and electronic communications, he's first to admit the process of developing e-mail retention policies around compliance is still very much in the infancy stage. "We're still at the stage where we're trying to define how long we want to keep e-mails and the kind of mechanisms we want to use to go back and do discovery," he says.

Corporate governance compliance
An organization that knows all too well the impact that e-mail communications can have on a case is Stevensdrake, a law firm based in Crawley, England. The firm spends a large portion of its time performing debt collection and recovery for large credit card and motor finance company clients.

It is that type of work, in fact, that drives much of the firm's current compliance needs, according to Gavin Pickering, one of the firm's partners. Some would even go so far as to equate much of what Pickering terms "compliance" for Stevensdrake under a broader category of corporate governance.

"Our compliance issues are both internal and external, but relate more to a risk point of view than a specific regulation," says Pickering. "E-mail was a big driver for this. We need to keep a record of all our e-mails to protect ourselves from either claims against clients or claims among our employees. We also have to back up all of our documents and servers. For some of our big debt recovery clients, we need to demonstrate that we can recover."

In fact, Pickering sees a trend where more and more clients will require their prospective legal providers to prove they won't lose service or access to records for any length of time if a disaster impacts their main offices.

Pickering found his compliance solution by first solving another headache that faced the firm: E-mail storage and archiving for the firm's employees had become increasingly unmanageable with an older Novell GroupWise application.

As he contemplated a move to Microsoft Exchange, Pickering and his team evaluated a few suppliers that might have been able to offer an alternative to the firm's e-mail backup/archive dilemma. The law firm's prior IT supplier introduced them to CommVault's Galaxy and QiNetix software platform. After seeing how easily the application could help them store and recover both e-mail messages and attachments, Pickering decided to move forward with implementation.

The CommVault QiNetix suite is now set up to perform incremental backups of the firm's data to a disk library each night, as well as an auxiliary copy to tape the next day which is then stored off-site. On the weekend, the firm also performs one full backup, which is also sent first to disk, before being copied to tape. Policies are set up to keep two weeks' worth of backup data on disk, with another two weeks of backup data on tape.

Currently, the CommVault system is used to retain approximately 2GB of e-mail and 53GB of migrated e-mail attachments. Regardless of whether the backup data resides on disk or tape, Pickering and his team have grown to appreciate how easy it is to restore any e-mail or document through the system. "The system just looks back over the library and gets you the file exactly as you want," he says.

According to Brian Brockaway, senior director of product management at CommVault, what Pickering is referring to is the indexing functionality built into the solution that automatically tracks backup data as it ages, including its current storage location. "If you do a search and the data is no longer on disk, it automatically looks into the tape, finds the data or e-mails, and pulls them back online again," Brockaway explains. This doesn't require data to first be restored from tape before it can be accessed.

How well has the solution satisfied the needs of the firm? "E-mail recovery is amazingly quick and easy—as easy as going back over your old in-box from a year ago," says Pickering.

Backing into SOX compliance
Randy Eck is another end user who deployed a storage management software tool for one purpose before realizing how well it could also be adapted for use in compliance. A senior systems engineer performing storage management duties under contract to a major US airline, Eck is accustomed to managing and backing up about 65TB of mainframe z/OS data, along with approximately 4TB per day of open systems data derived largely from the airline's database applications.

Using Symantec's NetBackup to run a variety of incremental backup jobs at various times of the day and week, Eck found himself looking for more than what NetBackup offered when it came to assessing the overall health of his backup environment. He decided to try Tek-Tools' BackupProfiler for NetBackup after seeing a demo of its centralized dashboard and real-time backup job success and failure reporting.

"We could take a quick look at the dashboard and determine how well our backups performed overnight, and whether we needed to look at any issues immediately," says Eck.

When talk turned to how best to comply with some of Sarbanes-Oxley's backup-related requirements, however, Eck was first to admit that his team pretty much stumbled on the prospect of using Tek-Tools' BackupProfiler to address some of the airline's SOX compliance needs.

Among the SOX backup requirements they had to prove was that the airline performed adequate daily backups of servers running financial applications, or those storing financial system data. They also had to prove the airline was maintaining adequate off-site storage for this financial data and that any source code associated with financial applications had been adequately protected.

Internal and external audits are conducted regularly at the airline, a factor that began to eat into the time of a storage team engineer who had to sit with the auditor and manipulate a handful of utilities and NetBackup commands to show the successful backup of certain data sets on randomly selected days.

In an effort to free IT time on this task and automate the audit process, Eck starting thinking of ways to use BackupProfiler's historical database of backup information and built-in reporting functionality to satisfy SOX audit requests. "I determined we could just run a report at the start of the month that shows the previous month's [backup job] history. Then we put it in the central repository where all the SOX-compliant documents need to go," Eck explains. "Now, auditors can go straight into that directory and look at the specific report they need. By automating the reporting, it took us out of the loop for the time needed to sit down with the auditor."

The various applications and management teams just let Eck know up-front when a server is associated with a SOX-targeted financial application or financial data. From there, he and his team maintain a monthly report for each application, with backup information related to the various file systems or database servers associated with it.

Eck's advice for others facing these types of issues: "Identify what it is that you need to comply with and how you need to report that. Then, if possible, find a way to automate it. Find a tool that can help you report on those issues in a proven, repeatable manner."

Archiving platform addresses regulation
Another healthcare environment currently girding itself for the impact of current and future compliance legislation is the University Hospitals Leuven, one of Belgium's largest hospitals with a strong heritage in medical research.

According to Reinoud Reynders, IT manager for infrastructure and operations, the hospital had decided early on that its "internal compliance" policy would involve keeping virtually every piece of data forever, so that as much data as possible could be made available for medical research. "We now have electronic patient records going back more than 20 years," says Reynders, a policy that has helped the hospital keep up with the mounting regulations Belgium has begun to impose on healthcare organizations with regard to retention of patient data.

Much of this type of data is now required to be kept for more than 30 years. Legislation regarding retention of digital X-ray images within a PACS system is also sure to follow soon as well, according to Reynders.

But Reynders is confident that his current storage and archiving infrastructure will allow him to handle any new compliance requirements. A long-time Network Appliance customer, the hospital already had a few NetApp FAS clustered storage systems in use to support its primary application databases such as SQL Server and Sybase, where Reynders boosts performance by storing data and database logs on separate NetApp systems to accommodate the high transaction rates required by 1,500 simultaneous users.

When the hospital began its digital X-ray imaging PACS project four years ago, it decided from the start to both store and archive initial PACS images online, directly on ATA disk systems—specifically, NetApp's NearStore R200 and R50 storage systems, which store 50TB of digital cardiology and radiology images (which grow by about 15TB each year).

In terms of compliance, Reynders readily admits: "There's no regulation on PACS currently in Belgium, but we know this will come." He adds that, "With our NetApp systems, we are ready for that. We can easily activate WORM functionality on those systems. We don't do that today because we're not required to do so. But, when a regulation requires it, we'll be ready. We can easily transform our normal volumes we have today into WORM volumes."

Reynders also sees NetApp's optional WORM functionality (available via NetApp's SnapLock software option) being useful in the future for compliance restrictions related to e-mail. A Symantec Enterprise Vault customer, the hospital archives e-mail for nearly 6,000 Microsoft Exchange mailboxes onto the NetApp NearStore platform. Primary Exchange mailbox stores, which amount to almost 200GB, are housed on a NetApp FAS 3050 cluster. Reynders estimates the Enterprise Vault archive will likely grow to approximately 1TB by year-end.

This type of platform mix suits Reynders for a couple of reasons. "All of our users will have an unlimited mailbox, and I have an Exchange Server I can restore very fast," he explains. "Secondly, when I want to initiate journaling and WORM functionality on my archive, I can do it very easily."

Michele Hope is a freelance writer covering enterprise storage and networking. She can be reached at mhope@thestoragewriter.com.

This article was originally published on December 08, 2006