Applications with structured data, such as financial reporting and ERP, may require special approaches to data storage, retention, and protection.
By Michael Casey
Many IT managers are concerned about the legal and regulatory compliance issues posed by electronic documents and e-mail. However, they may not realize that these risks and requirements also apply to the structured data in their accounting and ERP applications. With the advent of new technology, companies can meet these requirements using approaches that minimize the cost of compliance.
New laws, evolving regulatory interpretations, harsher penalties, and more-vigorous enforcement are motivating enterprises to keep more-detailed data and to store it for longer periods. In addition, some regulations require specific capabilities to ensure the availability, privacy, security, and integrity of the stored data. Taken together, these requirements threaten to dramatically increase storage costs and even to require the replacement or upgrade of existing storage infrastructure.
Corporate executives and IT managers face a dilemma: Divert scarce resources to cover increased management and storage costs, or accept an increased risk of regulatory enforcement actions and penalties. The dilemma is sharpened by recent high-profile scandals, fines, and prosecutions in industries such as securities trading and pharmaceuticals, and by looming compliance deadlines imposed by reporting and certification requirements under new laws and regulations such as the Sarbanes-Oxley Act.
In response to this dilemma, IT managers and storage administrators are evaluating new approaches to data management and storage, such as deployment of low-cost ATA disk arrays for extended storage of retained documents and archived data. The appropriate solutions depend on the industry and also on the types of electronic records and data that must be stored and managed. The compliance dilemma applies to both structured and unstructured data.
Structured and unstructured data
Many articles and analyst reports on compliance have focused on unstructured data such as text documents, e-mail messages, and medical images. (See Sorting through regulatory compliance hype, InfoStor.) This focus on unstructured data reflects the recent increase in regulatory scrutiny of electronic messages and documents. Enterprises have adopted electronic document technologies in place of traditional hardcopy formats, such as paper and film, to improve operational efficiency, customer satisfaction, and time to market. In response, government agencies have been formulating new rules to recognize and regulate the use of electronic documents and records. And companies are scrambling to respond to the new rules.
However, while developing compliance initiatives for unstructured data, companies must not overlook the impact of the new rules on structured data. This includes the data created and stored by financial accounting and ERP applications from vendors like SAP, Oracle, and PeopleSoft, and by industry-specific or custom applications such as securities trading systems and pharmaceutical research databases. These applications typically run on database management systems from IBM, Microsoft, Oracle, or Sybase. The table shows examples of structured data applications in a number of compliance environments.
Compliance environments and structured data applications
Government regulations for record retention and data protection generally do not distinguish between structured and unstructured data, so the same storage requirements typically apply to both types of data within a given industry and geography. However, structured data environments allow enterprises to apply additional strategies and tools for managing data and for controlling the impact of regulatory compliance on storage cost.
Storage requirements: the SEC rules
In general, legal and regulatory compliance rules do not require specific storage technologies, but many regulations do require specific capabilities to ensure the confidentiality, integrity, and availability of electronic records.
For example, many regulatory environments require companies to retain records for two to 10 years or more and to retrieve records quickly at a regulator's request. Other regulations require systems to keep secure audit trails of changes and deletions or to prevent changes or modifications to archived data. Some privacy laws require the ability to completely delete certain records at the end of the required retention period or to test and certify the application software without exposing private data during the testing process.
The US Securities and Exchange Commission (SEC) is driving much of the corporate concern about compliance, both through its enforcement of rules focused narrowly on the securities-trading industry and through its interpretation of other requirements such as the Sarbanes-Oxley Act that apply broadly to publicly traded companies.
Narrow and deep: SEC Rule 17
The securities trading industry now has some of the most stringent regulatory requirements for record retention and data storage, particularly under SEC Rule 17 for broker-dealer operations. Major storage vendors have designed their "compliant-storage" offerings to satisfy these SEC requirements, so an understanding of Rule 17 will help illuminate the reasons for specific product features.
SEC rules and interpretations were initially focused on the creation and retention of hardcopy records (paper or microfiche). However, hardcopy records and manual processes do not scale to support the speed and volume of today's global markets and trading operations. The industry has been moving to electronic processing of stock trades to improve back-office efficiency and to reduce trade settlement times and trade-failure risks.
The SEC has responded with informal guidance and official rule changes to recognize and regulate the use of electronic documents and records. However, it requires capabilities that simulate the permanence and "auditability" of paper records. In particular, Rule 17a-4 requires electronic data storage technology to enforce permanent recording that is "non-erasable and non-rewritable." This has effectively limited the choices to optical disk technologies such as WORM (write once, read many) and CD-R and to specialized WORM tape approaches such as StorageTek's VolSafe.
However, optical disk transfer rates and access methods do not readily scale to support very high trading volumes, and special drives and media can create technology migration risks and costs. So the industry has been looking for cost-effective alternatives that can leverage widely available disk storage technology.
In response to securities industry needs and technology innovations, the SEC has recently begun to accept WORM-like magnetic-disk storage systems for compliant records storage. The SEC interpretive release issued in May 2003 allows securities firms to store electronic records on magnetic disk, if the storage system meets certain requirements. The new SEC guidance requires that the software or firmware that makes each record non-rewritable must reside inside an integrated storage system (effectively the media controller), not in a separate application server.
The most prominent storage product in this class is EMC's Centera Compliance Edition, which uses a content-addressed storage (CAS) technique to establish a unique fingerprint that can be used to verify the integrity of each record. Centera also supports deletion of records after the required retention periods expire--a requirement under various data privacy laws.
Other vendors are also introducing products to address these requirements. For example, Network Appliance has introduced a SnapLock function on its filers, enabling users to support both WORM and non-WORM functionality in one flexible architecture. The SnapLock software can be added to existing NetApp filers, so enterprises can use this approach to minimize purchase of new hardware.
Many of the emerging compliance-oriented disk solutions--including EMC's Centera and NetApp's NearStore systems with SnapLock--use low-cost ATA disk drives. This helps reduce the cost of storing reference data and transaction archives for extended retention periods. By providing affordable online retention of required records, such products also support fast retrieval for internal audit and external reporting.
Broad and shallow: Sarbanes-Oxley
In addition to its detailed requirements for broker-dealers under Rule 17, the SEC has defined broadly applicable rules under the Sarbanes-Oxley Act for all companies that are publicly traded in US securities markets. While these rules do not currently require specific storage capabilities, they can impact storage requirements by increasing the amount of data that companies retain for internal audit and external reporting.
The Sarbanes-Oxley Act requires corporate executives to certify the adequacy of their internal processes and controls for financial reporting. For firms with market capitalization above $75 million, the deadline for CEO/CFO certification of internal controls is June 2004. This approaching deadline will motivate enterprises to take a closer look at their systems and procedures for records retention and data management. Since most large companies rely on financial accounting and ERP software for revenue and expense reporting, these structured data applications are due for increased scrutiny as companies prepare to meet the next round of Sarbanes-Oxley compliance deadlines.
Database consolidation can help companies reduce compliance risks with structured data applications by enabling or improving the consistent application of compliance policies across the enterprise. Benefits include greater accuracy, as well as easier and faster reporting and audit processes.
By merging multiple application or database instances into one global database, an organization can also increase efficiency and reduce costs. For example, IT managers at Tektronix found that maintaining separate accounting and ERP systems for 27 different countries led to increased data management complexity. The IT staff was required to handle an increasing volume of special reports and audit inquiries from various government authorities.
To improve efficiency and to support rapid response to inquiries, Tektronix consolidated the accounting and ERP systems for all 27 countries into one database instance. As a result, they could more quickly and efficiently support their worldwide legal and tax reporting requirements and audit inquiries.
While global consolidation also required the application server to manage more data, Tektronix was able to limit the impact on application performance and storage costs by moving to an application-aware archiving approach.
As reported in the July issue of InfoStor (see Database growth leads to archiving trend), Tektronix implemented an archiving solution using OuterBay's Application Data Management (ADM) software to reduce the size of the consolidated worldwide database for its financial and ERP applications.
"With archiving we can keep a minimal amount of data in our production environment, making performance much better," explains Lois Hughes, senior business systems analyst at Tektronix. "To comply with laws and regulations in specific countries, we must retain business transaction records for three to six years, or more. To maintain speed and flexibility of access, we keep the data for closed periods online, but in a separate archive. OuterBay Live Archive allows our users to read the archived data from the original application, without slowing the performance of the production database or multiplying its storage costs." This capability allows any user to respond quickly to an audit inquiry anywhere in the world, by running standard reports from the appropriate Oracle application module. It also eliminates the need for IT staff to retrieve offline data and run special reports.
This archiving approach also reduces the cost of storage for non-production copies of the data, such as test and development instances, which now contain only a small amount of active data.
When considering the storage impact of compliance requirements, don't overlook the risks and opportunities presented by the structured data in applications like financial reporting and ERP systems. These applications may require special approaches to data storage, retention, and protection. They may also benefit from approaches designed and optimized for structured data, such as database consolidation and application-aware archiving solutions.
Michael Casey is a principal analyst at Contoural Inc. (www.contoural.com), an independent storage consulting services firm in Los Altos, CA, and a former Gartner analyst.