Clustered databases offer a variety of advantages, as well as choices and challenges.
By Ashish Nadkarni
You’ve heard the rumblings in the hallway: The databases are getting overloaded, and more projects are coming along. There is talk of upgrading Oracle to 10g RAC (Real Application Clusters). Then one day it’s official: You’re asked to support the database administrator (DBA) in implementing 10g RAC. But this is not your ordinary database upgrade: You hear terms like ASM, CRS, OCR, Vote-disk, etc., and you begin to wonder: Am I supporting a database or a whole new concept?
Having implemented these databases and having gone through the pain of supporting one, I can tell you that never in my experience as a systems administrator have I ever had an application or a database cross the lines of demarcation among systems, storage, clustering, and high availability like 10g RAC does. But convergence is inevitable. So here are some guidelines to assist systems/storage administrators in their decision to implement 10g RAC. Disclaimer: I am not a DBA, nor do I claim to be an expert in 10g RAC.
Do you need RAC?
The concept of implementing a database cluster (and we are not talking active/passive clustering here) may at first appear to solve all your database-related problems, but in reality it may become a support quagmire, especially if the DBAs are new to the RAC concept and you are supporting an environment that is radically different from the rest of the database environments in the company. So before diving into the details, make sure your team spends enough time looking at alternatives, such as simply increasing the processing power of your existing systems instead of clustering a lot of servers and creating a distributed database.
Which OS platform?
A common misconception is that RAC is meant to be implemented on cheap Linux servers, thereby creating a “database farm.” That is not the case: 10g RAC is supported on various platforms, including Solaris, AIX, and HP-UX. If you don’t have Linux in your environment your plans to implement RAC should not be a reason to implement Linux. Your choice of platform should be based primarily on your comfort level and skills.
One caveat is that Oracle may not release patches and updates for all platforms at the same time. That is not to say that point patches will not be available should you escalate any issues to Oracle. Another thing to keep in mind is that Oracle recently switched its primary development platform from Linux to Solaris. Oracle is also a member of an IBM initiative on future AIX developments. That should give you a clue as to which operating systems will see 10g first. From a systems perspective this is the most expensive investment you will be making, so choose wisely.
Which clustering software?
For those who are used to clustering Oracle databases using third-party clustering software-such as Symantec’s Veritas Cluster Server, Sun Cluster, IBM HACMP, or HP MC/ServiceGuard-this may come as news: Oracle now has a cluster engine-called Cluster Ready Services (CRS)-built into the 10g suite. This engine is particularly important when implementing RAC if you decide to also implement service names (see below). This is where you may experience your first issue of contention between the DBA and systems administration teams.
Basically, Oracle pushes CRS as a replacement for third-party clustering products. However, the issue is how stable CRS really is. Most other clustering products are in their third or fourth generation and are very stable in non-RAC environments, and some of these products have been upgraded to include RAC support as well. For example, Symantec’s Veritas Storage Foundation suite for 10g RAC is based on the same clustering platform that all its other clustering packages are based on. If your environment has a lot of (non-RAC) database clusters, this is a very important design decision.
Using third-party clustering engines will allow you to maintain a single, consistent framework across all your database instances. Additionally, these clustering products offer platform/application integration and high-availability options that CRS does not yet provide. In short, CRS is a bundled clustering engine that does not quite compare to the functionality offered by more-mature clustering technologies.
Which volume management and file system software?
Oracle now bundles Automated Storage Manager (ASM) in 10g. ASM performs the task of host-based volume management without the need for another volume manager on the system. From DBAs’ perspective, this is ideal since they don’t have to depend on systems administrators for storage to be made available to the database; as soon as the storage is provisioned and made available to the system (in the form of raw devices), ASM can start using it. Although this sounds like a value proposition because you don’t need extra volume management software for managing database objects, here are some points to consider:
- ASM is an Oracle-only product, so for managing non-Oracle applications and other operating system file systems, you will still require third-party volume management;
- ASM is only bundled in 10g, so non-10g Oracle implementations will require volume management;
- ASM is a 1.0 version; its performance and stability in large database environments are largely unknown and untested. However, 10g release 2 includes significant changes to ASM; and
- ASM does not provide true visibility into how the database objects are distributed across devices presented to it, which means that troubleshooting performance issues or recovering from crashes may be challenging.
As with CRS, ASM is ideal in scenarios where budget constraints limit the purchase of third-party volume management software. However, if you have a large installation of third-party volume management software and a mature storage provisioning process, take a long hard look at ASM. Its adoption should be performed in a planned manner because it’s a new product.
What are some of the 10g dependencies?
SCSI-3 reservations and IO Fencing- The whole idea behind a 10g RAC implementation is that all nodes share access to all database objects. No matter what volume management or clustering software is used, SCSI reservation is a requirement to accomplish this.
If you are migrating from an older instance or have older storage arrays that you are planning to use for the new RAC environment, chances are that either the storage arrays are not configured to handle SCSI reservations or they do not support it at all. Most modern arrays (especially if you are using Fibre Channel) support SCSI-3 persistent reservations (SCSI-3 PR). Some clustering engines, such as Veritas’ SFRAC, use SCSI-3 persistent group reservation (SCSI-3 PGR) to implement a technique known as IO Fencing (sometimes called Disk Fencing), which is a method by which nodes can be stopped from accessing shared storage if they are not in a cluster. It allows the cluster engine to “eject a node” from the cluster, thereby protecting shared data.
SCSI-3 PGR will soon replace all other forms of SCSI reservation technologies, such as SCSI-2 reservations. It is highly advisable to choose a stack that uses SCSI-3 PGR when implementing RAC.
OCR and vote-disk-Oracle 10g RAC uses the concept of a vote-disk to keep all nodes in the cluster in sync. It also uses a registry (Oracle Cluster Registry, or OCR) to maintain cluster information for all nodes. These two entities are normally configured on raw devices. However, if you are using a clustered volume manager and/or file system as a part of an external clustering suite, there are certain benefits to configuring CRS to use a file system to store them. The main benefit is that, unlike raw devices, file systems can be composed of an underlying structure that has software, hardware, and I/O path redundancy. The other benefit is that manipulating records for taking backups or creating multiple copies is much easier.
Service names and CRS-Oracle introduced the concept of service names in 9i that allow you to add a layer of software virtualization over the traditional database instance. There are a variety of benefits to using service names, including dynamic load balancing and transfer between nodes, logical segregation of database traffic based on organizational or application needs, and the ability to provide application-transparent resource pooling.
In 10g RAC, Oracle introduced the requirement that dynamic management of service names require that the database be registered to or with CRS. CRS then is ultimately responsible for online, offline, and fail-over, and monitoring of all databases objects (such as database instances, listeners, virtual IP addresses and, of course, service names)-very similar to how a third-party cluster agent would function. This presents a problem with managing 10g RAC clusters using third-party cluster software:
You can have either CRS or the third-party cluster agent managing databases objects, but not both. Also, once a database is “registered” with CRS, it is almost impossible to have any other external component manage it.
A simple “meet-in-the-middle” approach that is recommended by vendors such as Symantec is to have their products provide the high availability and performance infrastructure to manage CRS and then in turn let CRS manage all the database components. Think about it this way: You have an application-CRS-that needs to be clustered, and you cluster it using a third-party cluster product. The primary advantage of this approach is that you are now complementing CRS in areas where its own capabilities are meager and creating a “best-of-both-worlds scenario.” The biggest disadvantage of this approach is cost: The main reason why Oracle is bundling CRS and all other “goodies” in its database suite is to have a “one-stop shop” for everything needed to manage the database, but whether they are really mature and stable or not is a different story. Adding third-party cluster software adds cost.
Heartbeats and messaging-In the Oracle world, heartbeats are also known as “interconnects.” They are very similar to the heartbeats used by other clustering engines-a private and/or dedicated network that is exclusively used to keep all nodes in sync. CRS uses heartbeats for “cache fusion”-in other words, to ensure writes and reads performed by one node are known and acknowledged by all the other nodes-primarily to prevent data corruption.
There is one major shortcoming, however: In release 1 of 10g there was no facility to provide high availability for heartbeats using either multiple IP or network interfaces or providing a network interface fail-over mechanism (like Solaris IP Multipathing or Veritas MultiNIC). As a result, if you are implementing a pure RAC, you have to depend on the operating system to provide this type of high availability. If you need to aggregate ports (as with Solaris Trunking or HP-UX APA) you may incur additional licensing costs.
By default, Oracle uses the UDP protocol for messaging. Some clustering software, such as Sun Cluster and Veritas SFRAC, provides libraries that allow you to replace UDP with their proprietary protocols for this purpose. There is a lot of debate as to whether these are more stable than UDP. Oracle recommends against replacing UDP, but the company supports both configurations. Veritas SFRAC also provides special agents to monitor the heartbeat interfaces using its own messaging protocols to add an additional layer of protection.
Port aggregation-The limitations posed by CRS with respect to network high availability or port aggregation apply to virtual IP addresses as well. CRS handles fail-over of the virtual IP addresses between nodes, but assumes no responsibility for providing local high availability for any network failure. If you have a node that is well configured, there should be no reason to fail an entire database instance over for a single network failure. Therefore, it is absolutely necessary to provide network high availability using methods described above.
Replication-If you are planning to replicate your RAC environment for disaster-recovery purposes, be aware that Oracle’s vehicle of choice for this is Data Guard. If you have already invested in another replication product, such as EMC’s SRDF or something similar, you may have to integrate it yourself or use third-party cluster agents because Oracle does not provide integration tools.
Data Guard in 10g has some worthwhile improvements over previous versions and deserves evaluation. However, your choice of which replication product to use will largely depend on whether you are planning to replicate just the 10g cluster or a lot of non-Oracle applications, because Data Guard only works with Oracle.
In conclusion, Oracle 10g is a good suite with all the necessary pieces to get RAC configured “out of the box.” However, some of the newer components such as CRS, ASM, etc., are still relatively new and may require a few revisions to make them really robust. For now, it may make sense to go with a hybrid approach using best-of-breed technologies from various vendors, including Oracle, for what they do best.
Ashish Nadkarni is a senior consultant with GlassHouse Technologies (www.glasshouse.com).