veritas volume replication and oracle databases

A S O L U T I O N S W H I T E P A P E R

from VERITAS Software Corporation

VERITAS volume replication technology, and how it enhances disaster tolerance in Oracle databases

VE

RIT

AS

Vol

ume

Rep

licat

ion

an

d O

racl

e D

atab

ases

VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia

3

Abstract Remote replication of online disks and volumes is emerging as the technique of

choice for protecting enterprise data against environmental disasters. Recognizing

this, VERITAS has embarked on aggressive campaign to build disaster recovery solu-

tions around the VERITAS Volume Replicator for major UNIX platforms.

Oracle Corporation, the leading supplier of relational database technology for the en-

terprise, recognizes the importance of disaster recoverability as well, and has intro-

duced its own replication technology that works cooperatively with the volume repli-

cation technologies of other vendors such as VERITAS. In addition, Oracle has insti-

tuted an audited testing program to verify that its technology interoperates correctly

with other vendors’ volume replication technologies for database disaster recovery.

VERITAS is proud to be the first vendor of volume replication technology to success-

fully complete the Oracle Storage Compatibility Program (OSCP) suite of disaster

recoverability tests.

This paper describes the capabilities of the VERITAS Volume Replicator, and how it

interoperates with Oracle’s standby database replication technology to provide robust

disaster recoverability for enterprise Oracle databases.


5

Contents I. ONLINE STORAGE FOR THE ENTERPRISE.................................................................................... 7

ENTERPRISE DATA STORAGE REQUIREMENTS ............................................................................................ 7 THE SOLUTION: RELATIONAL DATABASES AND LOGICAL VOLUMES ........................................................ 7 THE VERITAS VOLUME MANAGER .......................................................................................................... 8

II. DATA REPLICATION......................................................................................................................... 13 THE NATURE OF DATA REPLICATION ....................................................................................................... 13 THE VERITAS VOLUME REPLICATOR ..................................................................................................... 13 SYNCHRONOUS AND ASYNCHRONOUS REPLICATION................................................................................ 16

III. VOLUME REPLICATION AND ORACLE ..................................................................................... 23 ORACLE DATABASE REPLICATION TECHNIQUES ...................................................................................... 23 HYBRID REPLICATION OF ORACLE DATABASES ....................................................................................... 24 COMPARISON OF ORACLE REPLICATION TECHNIQUES.............................................................................. 25

IV. SUMMING UP ..................................................................................................................................... 29 VERITAS TOTAL DATA STORAGE MANAGEMENT .................................................................................. 29 WHY THE VERITAS SOLUTION? ............................................................................................................. 29

Figures FIGURE 1: REPLICATION WITH VERITAS VOLUME REPLICATOR ................................................................. 13 FIGURE 2: VERITAS VOLUME REPLICATOR DATA FLOW ............................................................................ 16 FIGURE 3: SYNCHRONOUS REPLICATION TIME LINE ..................................................................................... 17 FIGURE 4: APPLICATION PERFORMANCE WITH SYNCHRONOUS AND ASYNCHRONOUS REPLICATION............ 18 FIGURE 5: IN-BAND CONTROL MESSAGE TIME LINE..................................................................................... 21 FIGURE 6: RECIPROCAL DISASTER RECOVERY .............................................................................................. 22

Tables TABLE 1: VOLUME TYPES SUPPORTED BY THE VERITAS VOLUME MANAGER.............................................. 8 TABLE 2: COMPARISON OF STANDBY DATABASE HYBRID AND PURE VOLUME REPLICATION ...................... 26


7

I. Online Storage for the Enterprise

High-performance continuous access to online data

Enterprise Data Storage Requirements

The Requirement: Continuously Accessible, Trustworthy Data

Continuous access to data has become an operational necessity for enterprises. Cus-

tomers, suppliers, and even employees expect to be able to access enterprise informa-

tion resources all the time. Downtime, whether due to hardware or software failure,

the need to back data up, or because an upgrade is necessary, is no longer an option.

For managers and administrators responsible for information services data storage

management strategies, the challenges of continuous data availability are formidable:

➨ Maintaining the integrity of enterprise data while meeting increasingly com-plex, rapidly changing application needs.

➨ Providing continuous access to data, even if applications, systems, or entire data centers become incapacitated.

➨ Providing I/O performance that scales with enterprise requirements.

➨ Containing management cost as the amount of data increases.

The Solution: Relational Databases and Logical Volumes

Technology offers two complementary answers to enterprise needs for correct, con-

tinuously available, scalable, manageable data: relational database management sys-

tems and logical storage volumes.

For data integrity, relational database management systems such as Oracle provide:

➨ enterprise-wide application-independent data formats,

➨ data item integrity through constraints and user-definable triggered procedures,

➨ transactional integrity through logging and commitment/rollback techniques,

➨ high I/O performance, through intelligent use of massive cache memories,

➨ manageability through extensive management toolsets.

Relational database management systems do an outstanding job of keeping data error-

free and accessible at high performance levels. To do their job optimally, however,


8

they need storage they can rely on. Oracle is more robust, performs better, and is

more economical to own in the long run if the files or raw storage that hold its user

data and metadata are likewise robust, high performing, and flexibly manageable.

Logical volume management technology aggregates disks, and presents the database

management system with online storage units that are functionally disk-like, but have

superior availability, performance, and manageability. Software packages such as the

VERITAS Volume Manager provide logical volume management. RAID controllers

also provide a level of logical volume management. Wherever it runs, volume man-

agement software has two basic responsibilities:

➨ It protects data against loss or loss of access due to disk or channel hardware failures. Mirroring and RAID are the two techniques commonly employed.

➨ It maps disk-like volume block address spaces to disk blocks for optimal data access performance. Data striping is the most common mapping technique.

The VERITAS Volume Manager

The VERITAS Volume Manager runs as an operating system layer between I/O driv-

ers and the file system or database management system, VERITAS Volume Manager

enhances storage availability, performance, and manageability by presenting parts of

disks, entire disks, or groups of disks as logical volumes. Table 1 indicates the types

of volumes supported by the VERITAS Volume Manager in Solaris, HP-UX, and

Windows server operating system environments.

Volume Type Disk Failure Tolerance

(Data Availability)

Performance (relative to single disk)

Enterprise IT Application

Mirrored (two or more copies)

(“RAID 1”)

Very high High read performance Comparable write performance

Small (single-disk) critical files

Striped and Mirrored

(n-way)

Very high Very high read performance High write performance

Large (multi-disk) critical files

Striped with Parity

(“RAID 5”)

High High read performance Low write performance

Important “read-mostly” data

Striped without Parity

(“RAID 0”)

Lower than single disk High read and write performance

Limited to easily replaceable performance critical data

Concatenated (capacity

aggregated)

Lower than single disk Comparable read and write performance

Limited to easily replaceable non-performance critical data

Simple Same as single disk Same as single disk Small, easily replaceable non-performance critical data

Table 1: Volume Types Supported by the VERITAS Volume Manager


9

RAID array control software exploits specialized hardware for best RAID array per-

formance. Host-based volume managers tend to offer easier manageability because

they run in richer software environments that are more closely integrated with operat-

ing systems, file systems, and applications. For example, VERITAS Volume Manager

manageability features include:

➨ Three (or more) way mirroring. This is useful when a frozen image of opera-tional data is required for backup or application testing. With three mirrored copies, one can be used for backup or testing while live applications continue to run with failure tolerant data.

➨ Snapshots, or point-in-time frozen images of volume data for ad hoc backup, query, testing, or reporting purposes.

➨ Online volume expansion coupled with file system expansion, enabling an ad-ministrator to handle unplanned storage needs without application disruption.

➨ Heterogeneous device support. VERITAS Volume Manager aggregates (mirror, stripe, concatenate or RAID) both physical disks of different types and the vir-tual disks presented by RAID array controllers.

Volume Manager Distance Limitations

Volume managers provide the robust, high-performance storage substrate that file

systems and databases need. They keep data available if disks, channels, or in cluster

environments, servers fail…as long as everything is in one computer room. RAID

arrays and volume managers are designed around highly reliable local channels that

communicate with all disks at about the same speed.

When writing to a mirrored volume, for example, a volume manager typically issues

simultaneous write commands to all disks. Application response and error recovery

algorithms both assume that all commands to devices complete in roughly the same

amount of time. Writing to a RAID volume requires more commands, but again, algo-

rithms assume that no one command takes markedly longer than any other to execute.

Similarly, all disks and channels are assumed to be equally reliable. When a disk

command fails, the volume manager retries it. If the retries fail, the volume manager

adopts permanent failure mode algorithms for the volume. For example, when one

disk of a mirrored volume fails, the volume manager transparently redirects all I/O

requests to the volume’s other disk(s).

The assumption of reliable channels and equidistant devices is valid for computer

rooms where interconnects are short and reliable. In terms of response time, all stor-

age devices are equidistant from all host computers. Business needs, however, have


10

created requirements for maintaining identical copies of data in situations where

channels are not reliable and where response time is not the same for all devices.

➨ Enterprises need to recover quickly from fire, flood, vandalism, power grid failure, software failure, and other events that can incapacitate entire data cen-ters. The need to recover from disasters has led organizations to establish widely separated recovery data centers. Recovery data centers are outside an expected disaster radius, and contain sufficient equipment to run critical appli-cations if the primary data center is incapacitated. For a disaster recovery site to be effective, it must have current data for critical applications.

➨ It is often useful to reuse operational and historical data without affecting online applications, for example to test new applications or to discover trends by mining data. Applications that reuse data typically require dedicated com-puters and storage, but they also need access to recent operational data.

➨ Distributed operations sometimes require that data be published from a central creation point to multiple distributed usage sites. Price lists, product specifica-tions, and web pages all fall into this category. These data must be identical throughout an enterprise.

These applications all require that data be copied between systems while it is in use,

often over links that are less reliable and have lower performance than I/O channels.

If conventional mirroring were to be used for these purposes:

➨ performance of the enterprise’s online applications would be throttled by the I/O time to write data to the remote location, and,

➨ online application availability would be reduced, since there would necessarily be an application outage each time the link between local and remote data failed momentarily.

Data Replication Technology

Data replication technology overcomes these limitations and meets business needs by

mirroring data over long distances and unreliable links. Unlike mirroring, replication

technology assumes that:

➨ writing data to remote devices can take significantly longer than writing to local devices, and,

➨ remote links are less reliable than local channels, and in particular, are subject to momentary recoverable outages.

Data replication can be configured to eliminate remote write time from application re-

sponse time by allowing I/O to be buffered locally for later delivery in case of mo-

mentary network overload or outage. Alternatively, a system administrator can con-


11

figure replication to maintain strict synchronization between local and remote sites.

The following sections describe data replication as implemented by the VERITAS

Volume Replicator, and how the VERITAS Volume Replicator can be used to en-

hance the disaster recoverability and utility of Oracle databases.


13

II. Data Replication

Protecting data against disaster

The Nature of Data Replication

Whether the purpose is reuse or disaster recovery, the mechanism of data replication

is the same. A primary replication manager intercepts live data updates at the data

processing site, and distributes them to one or more secondary replication managers

at remote sites, where they are stored persistently. Secondary replication managers

maintain a defined level of synchronization with the primary replication manager.

Replication is designed for links that are too slow to permit mirroring, or for which

transient overloads and failures are expected. Primary replication managers can queue

data updates for delayed delivery to secondary sites if necessary.

The VERITAS Volume Replicator

The VERITAS Volume Replicator, illustrated in Figure 1, is an adjunct to the

VERITAS Volume Manager. The VERITAS Volume Replicator keeps the contents

of up to 32 secondary volume groups synchronized with the contents of a primary

volume group that holds live data being processed by applications. The VERITAS

Volume Replicator maintains the integrity of replicated volumes in case of network or

system outages without significantly degrading application performance.

r

Secondary Server S1

File System /fs File A.DAT

Volume V1

File B.DAT

File System /oracle

File

Volume W1

File

Storage Replicator

Volume Manager

Secondary Server S2


Volume V2

File B.DAT

File System /oracle

File

Volume W2

File

Storage Replicator

Volume Manager

SRVM propagates application updates to volume groups at secondary sites

Primary Server P

Volume Group G

Volumes may be simple,

concatenated, striped or mirrored

Application Updates Application

Updates Application Write

Oracle RDBMS

File System /fs

File A.DAT

Volume V

File B.DAT

File System /oracle

File T1.ORA

Volume W

File T2.ORA

Storage Replicator

Volume Manager

Volume Group G1

Volume Group G2

Update Update

Update

Update Update

Update Update Update

Update

Figure 1: Replication with VERITAS Volume Replicator


14

In Figure 1, applications run on the primary server, accessing data on volumes in

Volume Group G. Applications may access data directly through a file system (e.g.,

/fs), or through a database management system that uses either files or raw volumes

(e.g., /oracle). In this example, VERITAS Volume Replicator replicates updates

from Volume Group G to volume groups G1 and G2 on secondary servers S1 and S2.

Figure 1 illustrates four key points about VERITAS Volume Replicator:

➨ Volume replication mirrors block updates to volume contents, whether the vol-umes’ blocks contain files used directly by applications, file systems containing databases, or raw storage used directly by a database management system.

➨ Volume replication has a single source (the primary site) and one or more tar-gets (secondary sites). Applications run at the primary site. Secondary repli-cated volumes cannot be accessed by applications while replication is active.

➨ Data are replicated over an arbitrary network. Replication does not require spe-cial hardware or dedicated network connections (although VERITAS recom-mends dedicated network links in most instances for predictable response). In principle, data can be replicated across transcontinental distances. Distance lim-its are a function of application performance needs and affordability.

➨ Although VERITAS Volume Replicator maintains identical contents, secon-dary volumes need not be of the same type as primary volumes. For example, a 3-way mirrored volume at the primary site might be replicated to 2-way mir-rored volumes, or even to non-failure tolerant simple secondary volumes.

Overview of Volume Replication with VERITAS Volume Replicator

VERITAS Volume Replicator replicates each update to a primary volume on all cor-

responding secondary volumes, regardless of the source of the request. VERITAS

Volume Replicator preserves update ordering to ensure that secondary volume con-

tents are in a correct state at all points during a sequence of updates.

VERITAS Volume Replicator is unaware of I/O request interdependencies. It is im-

possible for a file system or database management system at a secondary site to de-

termine whether data or metadata updates are in progress. This is why secondary rep-

licated volumes cannot be used by applications during replication. The after-the-fact

usage characteristic of volume replication makes it most suitable for two general ap-

plication categories:

➨ data reuse: Application testing and data mining both require access to opera-tional and recent historical data. Obviously, testing cannot be done with opera-tional data. Data mining typically requires dedicated computing and I/O re-sources, so it, too, needs its own copy of operational data.


15

➨ disaster recovery: A disaster recovery site is a secondary data center located far enough from the primary data center that it can continue operating if a disaster occurs. Primary site volume groups containing operational data can be repli-cated to a disaster recovery site. If a primary site disaster occurs, applications can be restarted at the disaster recovery site, and processing can resume using the (current) replicated data.

Both of these application classes have the characteristic that data at the secondary site

is used by applications after replication stops, rather than while replication is occur-

ring.1

VERITAS Volume Replicator and the VERITAS Foundation

Since VERITAS Volume Replicator replicates volume contents, VERITAS Founda-

tion Suite features are available for replicated volumes at both primary and secondary

sites, including:

➨ availability enhancements such as mirroring,

➨ general-purpose performance enhancements such as data striping, and,

➨ database-specific performance enhancements such as Quick I/O for Databases.

Policy-Based Management

VERITAS Volume Replicator is policy-based. System administrators set replication

policies to meet business requirements. Replication policies include:

➨ which volumes at the primary site are to be replicated,

➨ which volumes at secondary sites are to be replication targets,

➨ the degree of synchronization between primary and secondary sites, and,

➨ how temporary failures such as network outages are handled.

Once policies have been set, replication is automatic, requiring no system administra-

tor intervention until an exceptional event such as a site disaster actually occurs.

VERITAS Volume Replicator and the VERITAS Volume Manager

From a technical standpoint, VERITAS Volume Replicator is closely integrated with

the VERITAS Volume Manager. The executable images for both products are dis-

tributed together, although they are enabled by separate license keys. VERITAS Vol-

ume Replicator requires that VERITAS Volume Manager be installed and enabled.

1 With VERITAS Volume Replicator, secondary replicated volumes can be mirrored. Mirrors can be broken off at instants when

primary site applications are known to be quiescent, and mounted for application use at the secondary site. In this case, the bro-ken mirror of replicated data is used after replication stops, but replication of the operational copy of data resumes as soon as the mirror has been broken off and applications are re-enabled.


16

Synchronous and Asynchronous Replication

VERITAS Volume Replicator at the primary site logs all write requests to replicated

volumes in a Storage Replicator Log (SRL) before sending them to secondary sites.

The SRL is VERITAS Volume Replicator’s mechanism for riding through transitory

network outages and overloads.

Primary Server Oracle

RDBMS


Volume V

File B.DAT

Secondary Server

Application Updates Application

Updates Application Updates Application

Updates

File System /oracle

File T1.ORA

Volume W

File T2.ORA

File System /oracle

File T1.ORA

Volume W

File T2.ORA


Volume V

File B.DAT

Secondary SRL

Volume Ls

Storage Replicator

Volume Manager

Storage Replicator

Volume Manager

Primary SRL

Volume Lp

Figure 2: VERITAS Volume Replicator Data Flow

If a network link fails, VERITAS Volume Replicator at the primary site continues to

log updates to replicated volumes. After the network recovers, logged updates are

sent to affected secondary sites. Secondary sites with functioning network links are

updated as write requests occur at the primary site.

The behavior of VERITAS Volume Replicator during a network outage is determined

by whether the system administrator has set the volume group to replicate synchro-

nously or asynchronously.

Synchronous Replication

Replicated data at a secondary site is current if volume group contents are identical to

their primary site counterparts. If data at a secondary site is to always be current, rep-

lication must be synchronous. Each update must be logged persistently at the primary

site and received by all secondary sites before it is considered complete.

When replicating synchronously, VERITAS Volume Replicator maintains primary

and secondary site data synchronization. An application write request to a synchro-

nously replicated volume is considered complete as soon as the update is:

➨ logged at the primary site, and,

➨ transmitted to and acknowledged by all secondary sites.


17

Secondary sites confirm updates in two stages. The first confirmation acknowledges

receipt of the update. The second, indicating that the primary site need no longer keep

the update in its SRL, is sent when data is on disk at the secondary site.

Figure 3 is a time line for synchronous replication, showing actions that can occur

concurrently. An update to a synchronously replicated volume takes longer than an

update to a non-replicated volume by a round trip message time to the most distant (in

time) secondary site.

Application may proceed when data is

written to primary disk and received by secondary site

Analyze request

Write data to primary log

Time

Send data to secondary site

Signal completion to application

Event durations are not shown to scale

write to disk at primary site and transmission to secondary site may be concurrent

Write data to disk at secondary site

These can be concurrent

Confirm write to primary site

Application need not wait for write

to disk at secondary sites

Write data to primary site disk

Confirm receipt to primary site

Figure 3: Synchronous Replication Time Line

With the replication algorithm illustrated in Figure 3, data from write requests for

which completion has been signaled are secure against:

➨ primary site disaster, because a copy exists at each secondary site, and,

➨ secondary site or communications link failures, because all updates are logged at the primary site.

Synchronous Replication and Link Failure

With synchronous replication, an application write request is not complete until the

volume updates it implies have been acknowledged by secondary sites. This means

that application writes cannot complete if network link fails. VERITAS Volume Re-

plicator allows a system administrator to set one of three link outage policies for each

replicated volume group:

➨ signal write failure to applications,

➨ abandon replication, reverting to local-only operation, or,

➨ switch to asynchronous replication (described below).

Signaling write failure to applications effectively transfers responsibility for dealing

with the outage to them. Abandoning replication entirely requires that a new base line

copy of data be created at affected secondary sites before replication can recom-

mence. Switching to asynchronous replication allows applications to continue operat-


18

ing, and accumulates updates locally for later transmission to secondary sites. In prac-

tice, most VERITAS Volume Replicator installations set up to switch to asynchro-

nous replication when a link or secondary site failure occurs.

Asynchronous Replication

Application I/O peaks, network overloads, or simply a large number of secondary

sites can elongate I/O response from synchronously replicated volume undesirably.

For these situations VERITAS Volume Replicator supports asynchronous replication.

When replication is asynchronous, an application write completes as soon as

VERITAS Volume Replicator has logged the udpate in the volume group’s SRL.

Transmission and writing to secondary volumes is concurrent with continued applica-

tion execution. Figure 4 illustrates the timing differences between synchronous and

asynchronous replication.

Time

Asynchronous replication reduces application response time by this amount

Synchronous replication

Asynchronous replication

Write data to primary site log



Signal completion to application

Application may proceed when data has

been written to primary disk and

received by secondary it

Event durations are not shown to scale


These can be concurrent


Application need not wait for write

to disk at secondary sites



Primary site signals completion

to application


Write data to primary site log

Analyze request

Write data to log at secondary site




Application may proceed when data has been written

to log at primary site

None of these affect application response

Analyze request

Figure 4: Application Performance with Synchronous and Asynchronous Replication

As Figure 4 illustrates, asynchronous replication reduces application write response

time. The more important consequence of asynchronous replication, however, is that

it enables applications to continue to execute without performance degradation during

momentary network outages and overloads.

Asynchronous Replication and Network Loading

VERITAS Volume Replicator sends asynchronous updates to secondary sites as

quickly as network and secondary site loading permit. If a network overload is tran-

sient, updates queued at the primary SRL are eventually sent, and secondary site data

becomes current. If the network overload is chronic, the number of unsent writes

grows, and eventually, the SRL fills. When this occurs, VERITAS Volume Replicator

implements one of several policies described below. Asynchronous replication pro-


19

tects against short-term network overload, but it is not a substitute for inadequate

steady state network bandwidth.

Limiting Exposure with Asynchronous Replication

The advantages of asynchronous replication are:

➨ better application response compared to synchronous replication (as Figure 4 il-lustrates),

➨ application ride-through of momentary network outages and overloads, and,

➨ recoverability from secondary site crashes.

The disadvantage is that there can be (usually) brief periods of time during which data

on secondary volumes are not completely current. If a secondary computer crashes or

a communication link fails with updates logged at the primary site but not yet trans-

mitted to one or more secondary sites, secondary site data may not be current. Up-

dates in the primary site SRL are transmitted to the secondary site and written after

recovery. If the primary site suffers an unrecoverable disaster, and its SRL is de-

stroyed, secondary site recovery may start with consistent but not-quite-current data.

To limit this exposure, a system administrator can limit the number of updates by

which secondary sites are permitted to be out of date. When the limit is exceeded, ap-

plication write requests stall (no completion signal is given) until the number of un-

sent updates has fallen below a permissible threshold.

Although it does not guarantee absolute data currency, asynchronous replication is of-

ten required for performance reasons. With synchronous replication, momentary write

overload or network load from other sources can increase response times unaccepta-

bly. Asynchronous replication removes network dependencies from the application

response path, and may make replication practical where it might otherwise not be.

Storage Replication Log Overflow

VERITAS Volume Replicator’s Storage Replication Log is stored on a dedicated

volume (which should be mirrored for failure tolerance). For maximum efficiency,

the SRL is structured as a circular buffer that cannot be expanded during replication.

If a lengthy network outage or long period of heavy update activity causes the SRL to

fill, VERITAS Volume Replicator switches to keeping track locally of which regions

of replicated volumes are modified. When bandwidth is again available, VERITAS

Volume Replicator sends the contents of modified regions to affected secondary sites.

When primary and secondary sites are again synchronized, replication resumes.

Other VERITAS Volume Replicator policies for dealing with SRL overflow include:


20

➨ abandoning replication: Since it requires full resynchronization before replica-tion can recommence, this policy is seldom used in practice.

➨ delaying I/O request completion: VERITAS Volume Replicator will delay completion of application I/O requests until logged updates have been transmit-ted and a predetermined amount of SRL space is freed. An override causes rep-lication to be abandoned if the SRL fills while a network link is down. This cir-cumvents application failures that might result from repeated replication-related I/O request timeouts.

Getting Replication Started

Replication requires a starting point at which the contents of primary and secondary

volumes are known to be identical. Thus far, the discussion has assumed that there is

a starting point at which the contents of corresponding primary and secondary volume

groups are identical, without discussing how they might have become identical.

A replication starting point can be established by making an image (block-by-block)

backup of primary volumes (e.g., using the UNIX dd utility for small volumes, or an

image backup utility such as VERITAS NetBackup’s for larger ones). When the im-

age has been restored at each secondary site, replication can begin. This procedure

may not be workable if it is impractical to freeze primary site data until all secondary

site restores are complete.

To minimize the impact of backup-based startup, VERITAS Volume Replicator in-

cludes a replication checkpoint facility that enables volumes to be used while prepar-

ing for replication. Immediately prior to starting a primary volume image backup, an

administrator declares a replication checkpoint, causing a marker to be placed in the

SRL. When the backup is complete, the administrator places a checkend marker in the

SRL. Any updates to primary volumes between the checkpoint and checkend markers

are reflected in the SRL, bracketed by the checkstart and checkend markers. The pri-

mary volumes may be used by applications during the backup.

The completed image backup is then restored at secondary sites, and logical links

with the primary site are established. VERITAS Volume Replicator sends all updates

starting at the checkpoint marker to the secondary sites. When the checkend marker is

reached, primary and secondary volume contents are identical, and synchronous or

asynchronous replication, including update ordering, begins.

VERITAS Volume Replicator auto-synchronization can simplify replication startup

even further in some cases. To auto-synchronize, a system administrator establishes a

replication link between primary and secondary volume groups, and VERITAS Vol-


21

ume Replicator copies the entire primary volume group contents to the secondary

volumes. To expedite the copy, write ordering is not preserved during auto-

synchronization.

Applications may update volumes during auto-synchronization. VERITAS Volume

Replicator tracks primary volume updates by block region. When the copy is com-

plete, updated regions are re-copied. When none of their block regions differ, primary

and secondary volume groups are synchronized, and normal replication begins.

VERITAS Volume Replicator In-Band Control

VERITAS Volume Replicator maintains application write order between primary and

secondary sites to ensure that newer application updates are not overwritten by older

updates of the same blocks that arrive out of order because of network delays. This is

necessary, but not sufficient to guarantee the consistency of volume contents from a

file system or database management system viewpoint. When a file system or data-

base management system has:

➨ no transactions outstanding, and,

➨ no updated data in cache that has not yet been written to disk,

its on-disk image is said to be internally consistent. Internally consistent file system

and database images greatly simplify backup and disaster recovery.

Because VERITAS Volume Replicator deals with volumes, it cannot determine when

a database or file system’s on-disk image is internally consistent. Special in-band

control application programming interfaces (APIs), however, allow registered appli-

cations at a primary site to send messages to registered applications at secondary sites

within a replicated data stream. An in-band control message suspends updates at the

receiving secondary site until a registered application has processed it and used a

companion API to resume replication. Figure 5 illustrates in-band replication control.

Process in band control message

Invoke “resume updates” API

Write data-2 to replication log


Receive in band control message

Time

Send data-1 to secondary site

Write data-1 to secondary disk

Primary site

Secondary site

Write data-1 to primary disk

Send in band control message





Write data-2 to secondary disk

etc.




Updates are suspended when in band control message is received

Secondary site application uses API

resume updating

Primary site continues to update replicated data

etc.

Figure 5: In-Band Control Message Time Line


22

A primary site application can send an in-band control message when some signifi-

cant event occurs (e.g., close of a business day). The receiving secondary site can re-

spond to the event (e.g., by creating a VERITAS Volume Manager mirror frozen im-

age of the day’s data), and then allow secondary site updates to resume.

Reciprocal Disaster Recovery

The primary and secondary site roles are specific to each replicated volume group. A

server may be the primary site for one volume group and a secondary site for another.

Figure 6 illustrates the use of this feature for reciprocal disaster recoverability.

Volume

File A2.ORA

Volume Primary Server for

Application A

Application A pdates Application A

Updates Application A Updates Application A

Updates

File A!.ORA

Application A Volumes

Primary Server for

Application B

Replication direction

Application B Replica

Volume

File B2.ORA

Volume

File B1.ORA

Application B Volumes

Application A Replica

Application B pdates Application B

Updates Application B BBUpdates Application B

Updates

Secondary Server for

Application B

Secondary Server for

Application A

Volume Group A

Volume Group B

Volume Group B’

Volume Group A’

Figure 6: Reciprocal Disaster Recovery

The system illustrated in Figure 6 has servers dedicated to applications A and B. Data

for application A are stored on Volume Group A, which is replicated to Volume

Group A’ on application B’s server, and conversely. If either site is incapacitated, its

application can be restarted on the server at the surviving site with current data.

For enterprises with large numbers of servers, reciprocal disaster recoverability based

on replication technology can dramatically improve overall IT resiliency. For a hard-

ware investment of:

➨ the storage devices required for replicated data,

➨ incremental server processing power and memory capacity to handle replication and provide adequate performance in the event of fail over, and,

➨ sufficient network bandwidth to accommodate replication traffic in addition to operational traffic,

an enterprise can position itself to rapidly resume operations when a disaster incapaci-

tates any of its critical applications.


23

III. Volume Replication And Oracle

Disaster protection for Oracle databases

Oracle Database Replication Techniques

By one analyst’s estimate, relational database management systems manage up to

90% of the online business information stored on servers, with Oracle holding the

largest market share. As enterprises become increasingly conscious of the importance

of their operational data and begin to plan for disaster recoverability, they naturally

consider replication of their Oracle databases.

Aware of this need, the Oracle Corporation has begun to offer a comprehensive test-

ing-based certification program by which vendors of other replication products can

verify that their products properly recover Oracle databases from site disasters, either

by themselves or in conjunction with Oracle’s own Replication Manager. This section

describes techniques for using VERITAS Volume Replicator to replicate Oracle data-

bases both with and without assistance from the Oracle Replication Manager.

Volume Replication of Oracle Databases

VERITAS Volume Replicator can replicate a complete Oracle database. Any of the

techniques described earlier can be used to establish a replication starting point with

identical database volume images at primary and secondary sites. Auto-

synchronization (page 21) uses the network to establish the starting point. For very

large databases, a full database backup coupled with the replication checkpoint tech-

nique described on page 21 may be more practical. In either case, the database may

be used during initial synchronization, but VERITAS Volume Replicator does not or-

der writes until initial synchronization is complete.

For disaster recovery, replicated database volumes must contain database control files

and redo logs as well as data files. Synchronous replication is clearly preferable to

preserve committed transactions, but performance considerations may require asyn-

chronous replication for long distances or low performance network links.

From a database standpoint, an unrecoverable disaster at the primary site is approxi-

mately equivalent to a local system crash at the secondary site. To activate the secon-

dary site database for application use:

➨ replication is stopped,


24

➨ secondary site replicated volumes are changed from secondary mode to primary mode and mounted, and,

➨ Oracle’s crash recovery procedures are used to restore the database image to the most current transactionally consistent point possible by applying all complete transactions from the database redo log and undoing any incomplete ones.

Oracle Standby Database Replication2

Another option for making Oracle databases recoverable is Oracle’s built-in replica-

tion facility based on database transactions rather than volume block updates. As

Oracle processes transactions, it captures them in an online redo log that can be used

to rebuild a current database from a backup if necessary due to hardware, software, or

operations failure. Periodically, the online redo log is emptied into an archive redo

log, and a new online redo log is started.

Oracle’s built-in replication facility uses these archive redo logs. Standby database

replication starts with the creation of a complete point-in-time copy of an inactive da-

tabase at a remote site. When the remote database, called a standby database, is iden-

tical to the primary database, replication begins. Each time an online redo log is

closed and copied to an archive redo log, Oracle’s Replication Manager transmits the

archive redo log to the remote site and applies the transactions in it to the standby da-

tabase. If a disaster incapacitates the primary site, the standby database is or can be

made as current as the time of the newest archive redo log transaction.

The advantage of standby database replication is that the database is always recovered

to transactional consistency. In other words, the recovered database may be out of

date (by the age of the newest archive redo log), but it does not contain partial trans-

actions (e.g., debits without matching credits), or metadata updates, and can therefore

be started and used immediately, without crash recovery.

Hybrid Replication of Oracle Databases

The disadvantage of standby database replication is that it does not restore a database

to the most current state reflected in the online redo log. Standby database replication

can be combined with volume replication of redo logs to improve the currency of a

recovered database.

2 Oracle database replication can also be used to distribute a database, allowing read/write access to data at multiple sites from

multiple sites, with user-defined policies for resolving conflicts. Since this usage is not comparable to volume replication, it is not discussed in this paper.


25

The most recent transactions against an Oracle database are recorded in the database’s

online redo log. The file containing this log must always be open, because the data-

base management system is continually adding to it. For performance reasons, the

must also be located near the database (i.e., at the primary site). More up to date

standby database recovery would be possible if the Oracle online redo log could

somehow be instantly transported from the disaster site to the recovery site.

Cooperation between Oracle Replication and VERITAS Volume Replicator

But transporting data instantly and transparently from a primary site to disaster recov-

ery sites is exactly what VERITAS Volume Replicator does. Allocating the Oracle

online redo log in a replicated volume makes the log available at the disaster recovery

site. In a disaster recovery scenario, after all archive redo logs have been applied to a

standby database, the replicated online redo log can be applied to roll the database

forward to the point of the last complete primary site transaction.

The Oracle Storage Compatibility Program

Up to date database recovery after a site disaster is obviously extraordinarily valuable

to enterprises. Recognizing this, Oracle Corporation has instituted an Oracle Storage

Compatibility Program (OSCP), an audited program of pre-defined tests that verify

the ability of a data replication solution to support up to date transactionally consis-

tent recovery of Oracle databases at remote sites.

Any storage configuration that supports replication of Oracle databases is eligible to

participate in OSCP, including both “hardware” solutions that replicate data between

disk controllers, and hybrid solutions such as VERITAS Volume Replicator, that

combine basic storage hardware with host-based software. VERITAS has submitted

the hybrid VERITAS Volume Replicator/Standby Database Replication technique de-

scribed above to the OSCP testing process, and is the first vendor to successfully pass

Oracle’s tests. VERITAS Volume Replicator is approved by Oracle Corporation for

up to date disaster recovery of Oracle databases.

Comparison of Oracle Replication Techniques

Whether standby database replication augmented by the volume replication of the

Oracle online redo log, or full replication of the volumes holding the database and

other application data is preferable depends on enterprise priorities. For example, if a

database must be replicated across thousands of miles, data propagation time might

preclude synchronous volume replication because of the adverse effect on application

responsiveness. Similarly, if database corruption by unstable applications is a con-


26

cern, the standby database technique with delayed application of archive redo logs

would probably be the preferred solution. On the other hand, if rapid recovery from a

primary site disaster is the main concern, volume replication is probably the best solu-

tion, because the recovery process is less time consuming and complex than with

standby database recovery. Table 2 compares seven key aspects of the hybrid and full

volume replication techniques.

Characteristic Standby Database Replication with Volume Replication for

Online Redo Log

Volume Replication of Entire Database

Cost Potentially lower communication cost because archive redo logs are batched (greater efficiency) and are not latency-critical

Potentially higher communication cost because synchronous replication I/O is in the application response path and therefore latency is critical.

Replication Granularity Individual tables Contents of a volume (usually more than one table)

Protection againstDatabase Corruption

Better, because application, operating system, or storage subsystem-caused corruption is not reflected in the standby database until archive redo logs are applied.

Any application, operating system, or database management system-caused corruption is reflected immediately in the secondary image.

Database OperationalPerformance

No impact on database I/O; minor impact possible when archive redo logs are copied and sent to secondary site; some impact if replication of the online redo log is synchronous.

Each write request incurs SRL write delay plus message round trip time (for synchronous replication only)

Failover Time(time until applications

can begin using databaseat secondary site)

Possibly longer. All archive redo logs not yet applied to the standby database must be applied, and the online redo log must be replayed.

Typically shorter. Treated as an Oracle crash recovery. Only the online redo log need be replayed.

Failback Time(move operational

database back to originalprimary site)

Typically much shorter if database image at primary site survived the disaster. Archive and redo logs recorded since failover must be sent to the primary site and replayed against the database image there.

Typically longer. Image of operational database at secondary site must be created at primary site. If operational database is used by applications during restore, a switch of primary sites is also required.

Management Simplicity Potentially more complex to configure; both VERITAS Volume Replicator and Standby Database Replication must be configured (Oracle8i simplifies setup).

Probably simpler. Worst case disaster recovery scenario is approximately equivalent to Oracle local crash recovery.

Table 2: Comparison of Standby Database Hybrid and Pure Volume Replication3

Another Hybrid Replication Alternative

A variation of the hybrid alternative described above is to store both online redo log

and archive redo logs on volumes replicated at the recovery site. With this solution,

3 Table 2 is based on the paper Guidelines for Using Remote Mirroring Storage Systems for Oracle Database by J. Bill Lee, pub-

lished by Oracle Corporation.


27

archive redo logs and the online redo log are both immediately available at the secon-

dary site, so there is no possibility of unsent archive logs being destroyed in a primary

site disaster. Archive logs can be applied at the secondary site by creating and mount-

ing VERITAS Volume Manager split mirrors of the volumes containing them.

This solution also allows the replaying of archive logs against the standby database to

be deferred. This provides the same protection against application, operating system,

or database management system-caused database corruption as conventional standby

database replication with the Oracle Replication Manager managing the transmission

of the archive replication logs over the network.


29

IV. Summing Up

Volume replication in context

VERITAS Total Data Storage Management

VERITAS Volume Replicator is just one component of VERITAS Total Data Stor-

age Management. Total Data Storage Management consists of:

➨ a solid foundation including a flexible, failure tolerant volume manager and an enterprise-class file system,

➨ automated backup and media management for distributed enterprise require-ments,

➨ clustering for automated continuous application availability.

Why the VERITAS Solution?

Five factors qualify VERITAS uniquely to provide total data storage management

solutions for the enterprise:

➨ technology leadership,

➨ product integration,

➨ multi-platform support,

➨ support, training, and service, and,

➨ experience and reputation.

Technology Leadership

Since the company’s founding in 1992, VERITAS has been an innovator in enterprise

data storage management software technology, with foundation technologies such as:

➨ high-performance, failure tolerant, logical volumes,

➨ extent-based file allocation,

➨ online volume and file system expansion and defragmentation,

➨ several frozen image technologies for volumes and file systems,

➨ direct I/O for database management systems (Quick I/O for Databases).

In backup, VERITAS Block Level Incremental Backup (BLIB) has enabled fast, non-

intrusive backups of Oracle databases. VERITAS clustering is also making highly


30

available enterprise computing a reality. Volume replication is only the latest tech-

nology in which VERITAS is innovating, to provide practical disaster tolerance.

As the data storage management technology leader, VERITAS is actively driving new

I/O architecture standards. Working through organizations like the Storage Network-

ing Industry Association, VERITAS is helping to create open technologies for SAN

management, for file system enhancements for effective SAN use, for increased

backup efficiency, for intelligent storage devices, and in a variety of other areas. Be-

cause VERITAS is co-developing these technologies, the company will be among the

first to bring their benefits to users in future data storage management products.

Product Integration

Product integration is the second factor that makes VERITAS the right choice for to-

tal data storage management. Examples of VERITAS product integration include:

➨ File system and volume manager cooperation that enables online expansion and defragmentation, snapshots, and Quick I/O for Databases.

➨ In addition to the database replication capabilities described in this paper, VERITAS Volume Replicator cooperates with the VERITAS Cluster Server and Global Cluster Manager to make cross-cluster disaster recoverability and global applications a practical reality.

➨ VERITAS Cluster Server agents make VERITAS File System, VERITAS Vol-ume Manager, VERITAS Volume Replicator, and VERITAS NetBackup inte-gral parts of a failure-tolerant data storage management environment.

VERITAS develops new products and enhances existing ones with an awareness that

integrated data storage management is much more compelling to users than a collec-

tion of unrelated or loosely related products. This thinking causes product decisions

to be made in the context of integrated solutions.

Multi-Platform Support

The third reason to choose VERITAS total data storage management is the com-

pany’s support of major UNIX and Windows server operating systems. With a

VERITAS Total Data Storage Management strategy, the most important IT asset,

skills and knowledge, is applicable across the entire enterprise. Adding or changing

computing platforms does not require new data storage management skills.

Support, Training and Service

The fourth reason to choose VERITAS is the company’s extensive network of cus-

tomer support, service, and training facilities. VERITAS products are applied to some

of today’s most difficult IT problems, where the answers aren’t always obvious.


31

VERITAS training gives users the knowledge to design optimal data storage man-

agement architectures. VERITAS consulting services can help plan, design and im-

plement turn-key information services. And VERITAS service is a phone call away if

something does go wrong.

Experience and Reputation

The final, and perhaps most compelling reason that makes VERITAS the right data

storage management choice is the company’s experience and reputation in large en-

terprises. Enterprise computing is a tough business. Only the best survive. VERITAS

has not only survived, but has become the acknowledged leader in enterprise UNIX

and Windows data management, with over 60% of the Fortune 1000 as its customers.

VERITAS’ reputation for solid, dependable, yet innovative data storage management

solutions gives customers the confidence to adopt the company’s solutions in new

technology areas such as volume replication to provide heretofore unavailable IT

quality of service. Being first to deploy these advanced capabilities gives VERITAS

customers a competitive edge in their fields.

A VERITAS data storage management solution isn’t just a set of products, it’s a

company with the technology, vision, products, experience, and resources to make IT

the robust, responsive service it needs to be for success in today’s competitive world.

Business without Interruption

veritas volume replication and oracle databases

Documents