oracle optimized solution for disaster recovery · oracle optimized solution for disaster recovery...

Oracle Optimized Solution for Disaster Recovery

Oracle E-Business Suite 12.1.3 Using Oracle Solaris Cluster Geo Edition – Implementation Guide

AU G U S T 2 0 1 8 | E 9 8 6 1 8 - 0 1

Table of Contents

1. Overview ......................................................................................................................................................... 1

2. Architecture ..................................................................................................................................................... 2

Sites Configuration with Oracle E-Business Suites 12.1.3 ........................................................................ 2

Hostnames and IP Address Information .................................................................................................... 3

2.2.1. The /etc/hosts File .................................................................................................................... 3

2.2.2. The FND_NODES Table .................................................................................................................... 4

Sites Configuration with Oracle Solaris Cluster Geo Edition ..................................................................... 6

3. Configure the Oracle E-Business Suite 12.1.3 DB Context Files ................................................................... 7

Edit the /etc/hosts Files ....................................................................................................................... 8

Set the s_virtual_hostname Variable ................................................................................................. 8

Run AutoConfig on the Primary Cluster ..................................................................................................... 9

Run AutoConfig on the Secondary Cluster .............................................................................................. 23

Verify the Contents of the FND_NODES Table .......................................................................................... 25

4. Configure Flashback on the Database .......................................................................................................... 26

5. Configure Oracle Solaris Cluster Geo Edition ............................................................................................... 30

Install Geographic Edition Software ........................................................................................................ 30

Prepare Each Zone Cluster for Partner Membership .............................................................................. 31

5.2.1. Set Up the Network Address for Each Zone Cluster ..................................................................... 31

5.2.2. Set Up Security on the Primary Cluster ........................................................................................ 33

5.2.3. Set Up Security on the Secondary Cluster ................................................................................... 34

Enable Geographic Edition Software ....................................................................................................... 36

Configure Partnerships ............................................................................................................................ 42

5.4.1. Configure Trust Between Partner Clusters ................................................................................... 42

5.4.2. Create Partnerships ...................................................................................................................... 52

5.4.3. Configure the Secondary Cluster to Join the Partnership ............................................................. 55

Configure Protection Groups ................................................................................................................... 56

5.5.1. Create the First Protection Group (opmn-ebs-pg) ........................................................................ 56

Overview ii

5.5.2. Create the Second Protection Group (cm-ebs-pg) ....................................................................... 60

5.5.3. Create the Third Protection Group (cm-odg-pg) ........................................................................... 64

5.5.4. Display the Oracle Solaris Cluster Listings ................................................................................... 67

Configure Sites and Multigroups ............................................................................................................. 74

5.6.1. Create and Configure the santaclara Site ..................................................................................... 75

5.6.2. Create and Configure the hillsboro Site ........................................................................................ 77

6. Create and Configure a DNS Server ............................................................................................................ 78

Configure the DNS Service ..................................................................................................................... 79

Configure the DNS Update Scripts .......................................................................................................... 82

6.2.1. Create the dns_update.sh Script .............................................................................................. 82

6.2.2. Verify the DNS Update Scripts on the DNS Server ...................................................................... 84

Configure the DNS Client ........................................................................................................................ 84

Configure the Protection Group Role Change Script ............................................................................... 86

7. Load Testing with Oracle Application Testing Suite (OATS) .......................................................................... 86

The Test Configuration ............................................................................................................................ 86

OLTP Load Generator: the OATS Machine ............................................................................................. 88

OATS Testing Information ........................................................................................................................ 89

Concurrent Manager Reports .................................................................................................................. 90

8. Switchover ebs-mg Multigroup to the Secondary Cluster ............................................................................. 90

Switchover Operation .............................................................................................................................. 91

Switchover Timing Information ................................................................................................................ 95

9. Takeover ebs-mg Multigroup from the Secondary Cluster ............................................................................ 96

Takeover Operation ................................................................................................................................. 96

Takeover Timing Information ................................................................................................................... 98

10. Restore the Primary Site ............................................................................................................................... 99

11. Apply SuperCluster Quarterly Maintenance Updates ................................................................................. 106

Overview ii

Overview 1

1. Overview

This document provides a step-by-step guide to set up disaster recovery for an Oracle E-Business Suite 12.1.3 installation on Oracle SuperCluster. The configuration uses Oracle Solaris Cluster Geo Edition to switch Oracle E-Business Suite 12.1.3 to a standby Oracle SuperCluster in the event of a failure. The primary and standby SuperClusters can be in different geographical locations.

The document assumes that you have installed Oracle E-Business Suite 12.1.3 and configured Oracle Data Guard and Oracle’s ZFS Storage Appliance by following the instructions provided in the following guides (available on My Oracle Support):

Oracle Optimized Solution for Oracle E-Business Suite on Oracle SuperCluster Example Implementation Guide (Doc ID 1557489.1)

Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Example Implementation Guide (Doc ID 1566368.1)

The guides contain procedures for duplicating database content and applications between primary and standby Oracle SuperClusters.

This document builds on the configurations from the implementation guides by deploying Oracle Cluster Geo Edition to automatically orchestrate the stopping and starting of Oracle E-Business Suite 12.1.3 services after a failover to the standby SuperCluster. Furthermore, in the configuration of the primary and standby SuperClusters, logical hostnames are used for the APP Tiers. Likewise, an alias hostname for each Oracle RAC VIP is used as a virtual hostname for each Oracle RAC database host. Thus, in the event of a switchover, you do not need to repeatedly run AutoConfig beyond initial configuration.

The topics for a disaster recovery solution are presented as follows:

1. Description of the reference architecture.

2. Configuration of Oracle E Business Suite 12.1.3 in both the primary and secondary SuperClusters.

3. Installation and configuration of Oracle Solaris Cluster Geo Edition.

4. Description of the switchover process between the primary and standby SuperClusters which ends with the standby SuperCluster taking subsequent control.

https://support.oracle.com/

Architecture 2

2. Architecture

This section describes the reference architecture for the implementation of the disaster recovery program.

Sites Configuration with Oracle E-Business Suites 12.1.3

The configuration of Oracle E-Business Suites 12.1.3 in the primary and standby sites is illustrated in the following figure:

Figure 1- Starting Configuration of Primary and Standby Sites

Note: In this document, the terms “standby” and “secondary” are used interchangeably.

As shown in Figure 1, each site consists of four Oracle VM SPARC Server domains.

Two clustered database domain containing Oracle RAC

Two clustered application domains containing Oracle Solaris Cluster, which is configured with the following zone clusters:

o Zone cluster containing a Web Entry Point (Web) and 2 Oracle Process Manager and Server (OPMN1 and OPMN2) services

o Zone cluster containing Concurrent Manager (CM1) services

Architecture 3

Additionally, redo log shipping and ZFSSA continuous replication are configured for both primary and standby sites.

Db_unique_names are assigned for the two sites:

VIS at the primary site

VISION at the standby site

In this configuration, Oracle Solaris Cluster Geo Edition is not yet installed on either site. Only the primary site is active.

Note: In the configuration on which this document is based, a symlink is set up to an ENV file.

This is in place for each database node. Below is an example.

root@etc18cn1dbiodom1:~# su – oracle

Oracle Corporation SunOS 5.11 11.3 December 2015

oracle@etc18cn1dbiodom1:~$

oracle@etc18cn1dbiodom1:~$ ls –l VIS.env

lrwxrwxrwx 1 oracle oinstall 67 Oct 2 2015 VIS.env ->

/u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc18cn1dbiodom1.env

The file enables you to access the SQL and the DGMGRL prompts. Accessing either prompt is part of the subsequent procedures in this document.

When accessing the DGMGRL prompt, make sure that you use the correct DB sys user password. For example, after sourcing the ENV file, type:

oracle@etc18cn1dbiodom1:~$ dgmgrl sys/<password>

Hostnames and IP Address Information

The Oracle SuperCluster provides an external 10 gigabit Ethernet (10GbE) network and internal Internet Protocol over InfiniBand (IPoIB) network. The latter is used as a private non-routable network (192.168/16 prefix) that is only accessible from within the Oracle SuperCluster.

One of the tasks in this document to implement disaster recovery is to map an alias hostname for each Oracle RAC VIP as a virtual hostname for each Oracle RAC database host.

After you complete the task, the mapping of hostnames, IP addresses, and the aliases appears in both the

/etc/hosts files of the DB and APP tier nodes as well as FND_NODES table for both sites.

2.2.1. The /etc/hosts File

For example, on the primary site, the /etc/hosts file on each DB and APP Tier contains the following entries:

192.168.28.101 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip idb1-vip.us.oracle.com idb1-vip

192.168.28.102 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip idb2-vip.us.oracle.com idb2-vip

Architecture 4

On the standby site, the /etc/hosts file on each DB and APP Tier contains the following entries:

192.168.28.101 etc18db01-ibvip.us.oracle.com etc18db01-ibvip idb1-vip.us.oracle.com idb1-vip

192.168.28.102 etc18db02-ibvip.us.oracle.com etc18db02-ibvip idb2-vip.us.oracle.com idb2-vip

In these sample /etc/hosts files, the alias hostnames idb1-vip and idb2-vip map to the same IP

addresses at the primary and standby sites, respectively, 192.168.28.101 and 192.168.28.102.

2.2.2. The FND_NODES Table

The mapping also appears in the FND_NODES table within the APPS schema as shown in this example.




oracle@etc18cn1dbiodom1:~$ . VIS.env

oracle@etc18cn1dbiodom1:~$ sqlplus “/as sysdba”

SQL *Plus: Release 11.2.0.3.0 Production

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage

Management, OLAP,

Data Mining and Real Application Testing options

SQL>

SQL> select node_name, server_address, virtual_ip from fnd_nodes;

NODE_NAME SERVER_ADDRESS VIRTUAL_IP

---------------- -------------- ----------

EWEB-LH m.m.m.68

IOPMN2-LH 192.168.28.34

ETC11-CN01 idb1-vip

ETC18CN1DBIODOM1 idb1-vip

ICM1-LH 192.168.28.43

ETC11-CN02 idb2-vip


AUTHENTICATION *

IOPMN1-LH 192.168.28.33

9 rows selected.

SQL>

Architecture 5

Note: In all the examples in this documentation, some IP addresses have been masked on the first 3 octets to indicate that these should reflect valid addresses that are actually used at your company’s configuration. For the primary cluster, the masked address is n.n.n.#, such as n.n.n.27. For the secondary cluster, the masked address is m.m.m.#, such as m.m.m.68.The distinction also indicates the typical configuration where primary and secondary clusters are on different subnets.

The FND_NODES table provides the following information:

Node names for the nodes in the Cluster configuration. Oracle Solaris Cluster manages the location of the logical hostname for the node names. Consequently, the logical hostname is hosted only on one node at any one time.

The hostnames are prefixed with either E or I. The E prefix designates that the logical hostname is an external address on the 10GbE network. The I prefix indicates that the hostname is in an internal address on the IPoIB network.

The same logical hostnames for the services are used at the primary and standby sites, specifically EWEB-LH, IOPMN1-LH, IOPMN2-LH and ICM1-LH.

Two Oracle SuperCluster DB hostname pairs exist:

o ETC11-CN01 and ETC11-CN02 on the primary site. o ETC18CN1DBIODOM1 and ETC18CN2DBIODOM1 on the standby site.

Each pair of Oracle SuperCluster DB hostnames is mapped to corresponding virtual IP names in the internal IPoIB network as follows:

o ETC11-CN01 and ETC18CN1DBIODOM1 are mapped to idb1-vip. o ETC11-CN02 and ETC18CN2DBIODOM1 are mapped to idb2-vip.

Architecture 6

Sites Configuration with Oracle Solaris Cluster Geo Edition

The following figure shows the primary and standby sites with Oracle Solaris Cluster Geo Edition installed and configured.

Figure 2- Intended Final Configuration of Primary and Standby Sites

Figure 2 still reflects the basic architecture as illustrated in Figure 1. However, this figure also shows how partnerships are configured through Oracle Solaris Cluster Geo Edition:

Partnership 1 consists of the OPMN zone clusters of both sites

Partnership 2 consists of the CM zone clusters of both sites.

Likewise, protection groups are also configured for the partnerships:

Protection Group 1 consists of the WEB and OPMN services.

Protection Group 2 consists of the CM services and ZFS Storage Appliance

Protection Group 3 consists of the Oracle Data Guard

Note that Protection Groups 2 and 3 are set for Partnership 2. The steps to configure partnerships and protection groups are provided later in the section Configure Oracle Solaris Cluster Geo Edition.

With this configuration, Oracle Solaris Cluster Geo Edition provides the capability that orchestrates the stopping and starting of Oracle E-Business Suite 12.1.3 services after a switchover or failover from the primary site to the

Configure the Oracle E-Business Suite 12.1.3 DB Context Files 7

standby site.

In summary, the architecture resembles the configurations described in the Oracle Optimized Solution for Oracle E-Business Suite on Oracle SuperCluster Example Implementation Guide (Doc ID 1557489.1) and Oracle Optimized Solution for Disaster Recovery Oracle E-Business Suite Example Implementation Guide (Doc ID 1566368.1).

3. Configure the Oracle E-Business Suite 12.1.3 DB Context

Files

This section provides the procedure to configure the Oracle E-Business Suite 12.1.3. DB context files to prepare for disaster recovery setup.

Based on the reference architecture described previously, the following DB and APP Tier nodes exist in the current configuration. Refer to this table while performing the configurations as instructed:

On the Primary Cluster

Tier Node RAC VIP

DB (RAC) etc11-cn01 etc11-01-ib-vip

DB (RAC) etc11-cn02 etc11-02-ib-vip

APP (opmn-zc) eopmn1

APP (opmn-zc) eopmn2

APP (cm-zc) ecm1

APP (cm-zc) ecm2

On the Secondary Cluster

Tier Node RAC VIP

DB (RAC) etc18cn1dbiodom1 etc18db01-ibvip

DB (RAC) etc18cn2dbiodom1 etc18db02-ibvip

APP (dr-opmn-zc) deopmn1

APP (dr-opmn-zc) deopmn2

APP (dr-cm-zc) decm1

APP (dr-cm-zc) decm2

In the procedure that follows, you would perform the steps on all 12 nodes as indicated.


Edit the /etc/hosts Files

This step maps the hostnames and their addresses to the virtual IP names idb1-vip and idb2-vip. The

revisions are in bold.

On the primary cluster, edit the /etc/hosts files to reference the virtual IP names.

192.168.28.101 etc11-01-ib-vip.us.oracle.com etc11-01-ib-vip idb1-

vip.us.oracle.com idb1-vip

192.168.28.102 etc11-02-ib-vip.us.oracle.com etc11-02-ib-vip idb2-


On the secondary cluster, edit the /etc/hosts files to reference the virtual IP names.

192.168.28.101 etc18db01-ibvip.us.oracle.com etc18db01-ibvip idb1-


192.168.28.102 etc18db02-ibvip.us.oracle.com etc18db02-ibvip idb2-


Set the s_virtual_hostname Variable

This step populates the FND_NODES table and subsequent TNS entries at each DB and APP Tier.

On the primary cluster, perform the following steps.

a. On etc11-cn01, set s_virtual_hostname to idb1-vip.

oracle@etc11-cn01:~$ . /u01/app/oracle/product/11.2.0.3/dbhome_1/VIS1_etc11-

cn01.env

oracle@etc11-cn01:~$ echo $CONTEXT_FILE

/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS1_etc11-cn01.xml

oracle@etc11-cn01:~$

oracle@etc11-cn01:~$ grep s_virtual_hostname $CONTEXT_FILE

<host oa_var="s_virtual_hostname">idb1-vip</host>


b. On etc11-cn02, set s_virtual_hostname to idb2-vip.


cn02.env


oracle@etc11-cn02:~$ echo $CONTEXT_FILE




oracle@etc11-cn02:~$ grep s_virtual_hostname $CONTEXT_FILE



On the secondary cluster, perform the following steps:

a. On etc18cn1dbiodom1, set s_virtual_hostname to idb1-vip.

oracle@etc18cn1dbiodom1:~$ .


oracle@etc18cn1dbiodom1:~$ echo $CONTEXT_FILE

/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS1_etc18cn1dbiodom1.xml


oracle@etc18cn1dbiodom1:~$ grep s_virtual_hostname $CONTEXT_FILE



b. On etc18cn2dbiodom1, set s_virtual_hostname to idb2-vip.

oracle@etc18cn2dbiodom1:~$ .


oracle@etc18cn2dbiodom1:~$ echo $CONTEXT_FILE

/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/VIS2_etc18cn2dbiodom1.xml


oracle@etc18cn2dbiodom1:~$ grep s_virtual_hostname $CONTEXT_FILE



Run AutoConfig on the Primary Cluster

On etc11-cn01, run the adautocfg.sh script.


cn01.env

oracle@etc11-cn01:~$ echo $ORACLE_HOME

/u01/app/oracle/product/11.2.0.3/dbhome_1


oracle@etc11-cn01:~$ cd $ORACLE_HOME/appsutil/scripts/VIS1_etc11-cn01

oracle@etc11-cn01:~$ ./adautocfg.sh

Enter the APPS user password:

The log file for this session is located at:

/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/log/

VIS1_etc11-cn01/10210119/adconfig.log

AutoConfig is configuring the Database environment...


AutoConfig will consider the custom templates if present.

Using ORACLE_HOME location : /u01/app/oracle/product/11.2.0.3/dbhome_1

Classpath :

:/u01/app/oracle/product/11.2.0.3/dbhome_1/

jdbc/lib/ojdbc5.jar:/u01/app/oracle/product/11.2.0.3/dbhome_1/

appsutil/java/xmlparserv2.jar:/u01/app/oracle/product/11.2.0.3/

dbhome_1/appsutil/java:/u01/app/oracle/product/11.2.0.3/dbhome_1/

jlib/netcfg.jar:/u01/app/oracle/product/11.2.0.3/dbhome_1/jlib/

ldapjclnt11.jar

Using Context file :


Context Value Management will now update the Context file

Updating Context file...COMPLETED

Attempting upload of Context file and templates to

database...COMPLETED

Updating rdbms version in Context file to db112

Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.




cn02.env







The log file for this session is located at: /u01/app/oracle/product/11.2.0.3/

dbhome_1/appsutil/log/VIS2_etc11-cn02/10210118/adconfig.log



Using ORACLE_HOME location : /u01/app/oracle/product/11.2.0.3/dbhome_1

Classpath :

:/u01/app/oracle/product/11.2.0.3/dbhome_1/jdbc/lib/ojdbc5.jar:/u01/app/

oracle/product/11.2.0.3/dbhome_1/appsutil/java/xmlparserv2.jar:/

u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/java:/u01/app/oracle/


product/11.2.0.3/dbhome_1/jlib/netcfg.jar:/u01/app/oracle/product/11.2.0.3/

dbhome_1/jlib/ldapjclnt11.jar

Using Context file :




Attempting upload of Context file and templates to

database...COMPLETED

Updating rdbms version in Context file to db112

Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...




This second run of AutoConfig on the node updates the $ORACLE_HOME/VIS1_etc11-

cn01/tnsnames.ora file to make its entries for the RAC instance on etc11-cn02 reflect the idb2-vip alias.

The file can only be updated after AutoConfig is run on etc11-cn02 in the preceding step.

In the following example, the output is intentionally truncated.


cn01.env








/u01/app/oracle/product/11.2.0.3/dbhome_1/appsutil/log/VIS1_etc11-

cn01/10210119/adconfig.log


On eopmn1 (eweb-lh), run the adautocfg.sh script.

bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_eweb-lh.env

-bash-4.1$

-bash-4.1$ echo $ADMIN_SCRIPTS_HOME


/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/admin/scripts

-bash-4.1$

-bash-4.1$ echo $CONTEXT_FILE

/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_eweb-lh.xml

-bash-4.1$

-bash-4.1$ $ADMIN_SCRIPTS_HOME/adautocfg.sh


The log file for this session is located at: /ebsfs/apps/VIS/inst/apps/

VIS_eweb-lh/admin/log/10070650/adconfig.log

AutoConfig is configuring the Applications environment...


Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_eweb-lh

Classpath :

/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/

apps/apps_st/comn/java/classes

Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_eweb-lh.xml



Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...

Configuring AD_TOP........COMPLETED

Configuring FND_TOP.......COMPLETED

Configuring ICX_TOP.......COMPLETED

Configuring MSC_TOP.......COMPLETED

Configuring IEO_TOP.......COMPLETED

Configuring BIS_TOP.......COMPLETED

Configuring AMS_TOP.......COMPLETED

Configuring CCT_TOP.......COMPLETED

Configuring WSH_TOP.......COMPLETED

Configuring CLN_TOP.......COMPLETED

Configuring OKE_TOP.......COMPLETED

Configuring OKL_TOP.......COMPLETED

Configuring OKS_TOP.......COMPLETED

Configuring CSF_TOP.......COMPLETED

Configuring IGS_TOP.......COMPLETED

Configuring IBY_TOP.......COMPLETED

Configuring JTF_TOP.......COMPLETED

Configuring MWA_TOP.......COMPLETED

Configuring CN_TOP........COMPLETED

Configuring CSI_TOP.......COMPLETED

Configuring WIP_TOP.......COMPLETED


Configuring CSE_TOP.......COMPLETED

Configuring EAM_TOP.......COMPLETED

Configuring FTE_TOP.......COMPLETED

Configuring ONT_TOP.......COMPLETED

Configuring AR_TOP........COMPLETED

Configuring AHL_TOP.......COMPLETED

Configuring OZF_TOP.......COMPLETED

Configuring IES_TOP.......COMPLETED

Configuring CSD_TOP.......COMPLETED

Configuring IGC_TOP.......COMPLETED


-bash-4.1$

On eopmn1 (iopmn1-lh), run the adautocfg.sh script.

-bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_iopmn1-lh.env

-bash-4.1$


/ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh/admin/scripts

-bash-4.1$


/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_iopmn1-lh.xml

-bash-4.1$




/ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh/admin/log/10070648/adconfig.log



Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh

Classpath :

/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/

apps/apps_st/comn/java/classes

Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_iopmn1-lh/

appl/admin/VIS_iopmn1-lh.xml






































-bash-4.1$

On eopmn2 (iopmn2-lh), run the adautocfg.sh script.

-bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_iopmn2-lh.env

-bash-4.1$


/ebsfs/apps/VIS/inst/apps/VIS_iopmn2-lh/admin/scripts

-bash-4.1$


/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_iopmn2-lh.xml

-bash-4.1$





/ebsfs/apps/VIS/inst/apps/VIS_iopmn2-

lh/admin/log/10070649/adconfig.log



Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_iopmn2-lh

Classpath :

/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/apps/

apps_st/comn/java/classes

Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_iopmn2-

lh/appl/admin/VIS_iopmn2-lh.xml






































-bash-4.1$

On ecm1 (icm1-lh), run the adautocfg.sh script.

-bash-4.1$ . /ebsfs/apps/VIS/apps/apps_st/appl/APPSVIS_icm1-lh.env

-bash-4.1$


/ebsfs/apps/VIS/inst/apps/VIS_icm1-lh/admin/scripts

-bash-4.1$


/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/appl/admin/VIS_icm1-lh.xml

-bash-4.1$



The log file for this session is located at: /ebsfs/apps/VIS/inst/apps/VIS_icm1-

lh/admin/log/10070641/adconfig.log



Using CONFIG_HOME location : /ebsfs/apps/VIS/inst/apps/VIS_icm1-lh

Classpath :

/ebsfs/apps/VIS/apps/apps_st/comn/java/lib/appsborg2.zip:/ebsfs/apps/VIS/apps/

apps_st/comn/java/classes

Using Context file : /ebsfs/apps/VIS/inst/apps/VIS_icm1/appl/

admin/VIS_icm1-lh.xml






































-bash-4.1$

Verify that the contents of the tnsnames.ora file are correct.

Ensure that the file’s entries correctly reference the idb1-vip and idb2-vip aliases. Confirm also that

FNDFS_APPLTOP_AT_eweblh references all the other APP Tiers.

If an address entry is missing on an APP Tier node entry, rerun AutoConfig on that node.

-bash-4.1$ cat $TNS_ADMIN/tnsnames.ora

###############################################################

#

# This file is automatically generated by AutoConfig. It will be read and

# overwritten. If you were instructed to edit this file, or if you are not

# able to use the settings created by AutoConfig, refer to Metalink Note

# 387859.1 for assistance.

#


#$Header: NetServiceHandler.java 120.19.12010000.6 2010/03/09 08:11:36 jmajumde ship $

#

###############################################################

VIS=

(DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=idb1-vip.us.oracle.com)(PORT=1521))

(CONNECT_DATA=

(SERVICE_NAME=VIS)

(INSTANCE_NAME=VIS1)

)

)

VIS1=

(DESCRIPTION=



(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

VIS1_FO=

(DESCRIPTION_LIST=

(LOAD_BALANCE=NO)

(FAILOVER=YES)

(DESCRIPTION=


(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

(DESCRIPTION=


(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

)

VIS2=

(DESCRIPTION=



(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

VIS2_FO=

(DESCRIPTION_LIST=


(LOAD_BALANCE=NO)

(FAILOVER=YES)

(DESCRIPTION=


(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

(DESCRIPTION=


(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

)

VIS_FO=

(DESCRIPTION=


(CONNECT_DATA=

(SERVICE_NAME=VIS)


)

)

VIS_BALANCE=

(DESCRIPTION=

(ADDRESS_LIST=

(LOAD_BALANCE=YES)

(FAILOVER=YES)





)

(CONNECT_DATA=

(SERVICE_NAME=VIS)

)

)

FNDFS_IOPMN1-LH=

(DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=IOPMN1-LH)(PORT=1626))

(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_IOPMN1-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)


)

)

FNDFS_VIS_IOPMN1-LH=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_VIS_IOPMN1-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_IOPMN2-LH=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_IOPMN2-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_VIS_IOPMN2-LH=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_VIS_IOPMN2-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_EWEB-LH=

(DESCRIPTION=


(ADDRESS=(PROTOCOL=tcp)(HOST=EWEB-LH.us.oracle.com)(PORT=1627))

(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_EWEB-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_VIS_EWEB-LH=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_VIS_EWEB-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_ICM1-LH=

(DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=ICM1-LH)(PORT=1627))

(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_ICM1-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDFS_VIS_ICM1-LH=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)


FNDFS_VIS_ICM1-LH.us.oracle.com=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDFS)

)

)

FNDSM_IOPMN1-LH_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDSM_IOPMN1-LH.us.oracle.com_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDSM_IOPMN2-LH_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDSM_IOPMN2-LH.us.oracle.com_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDSM_EWEB-LH_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDSM_EWEB-LH.us.oracle.com_VIS=

(DESCRIPTION=


(CONNECT_DATA=


(SID=FNDSM)

)

)

FNDSM_ICM1-LH_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDSM_ICM1-LH.us.oracle.com_VIS=

(DESCRIPTION=


(CONNECT_DATA=

(SID=FNDSM)

)

)

FNDFS_APPLTOP_AT_eweblh=

(DESCRIPTION=

(ADDRESS_LIST=





)

(CONNECT_DATA=

(SID=FNDFS)

)

)

IFILE=/ebsfs/apps/VIS/inst/apps/VIS_eweb-lh/ora/10.1.2/network/admin/VIS_eweb-lh_ifile.ora

-bash-4.1$

Run AutoConfig on the Secondary Cluster

Repeat the step to run AutoConfig on the secondary cluster to populate the FND_NODES table as well as update

the relevant tnsnames.ora entries for the DB and APP nodes.

Access Oracle Data Guard broker.







DGMGRL for Solaris Version 11.2.0.3.0 – 64bit Production

Copyright (c) 2000, 2009, Oracle. All Rights reserved.

Welcome to DGMGRL, type “help” for information.

Connected.

DGMGRL>

Switchover the database.

This step causes VISION to become the primary database.

DGMGRL> switchover to 'VISION'

Performing switchover NOW, please wait...

New primary database "VISION" is opening...

Operation requires shutdown of instance "VIS2" on database "VIS"

Shutting down instance "VIS2"...

ORACLE instance shut down.

Operation requires startup of instance "VIS2" on database "VIS"

Starting instance "VIS2"...

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Database mounted.

Switchover succeeded, new primary is "VISION"

Display database configuration.

DGMGRL> show configuration

Configuration - DR Orchestration for EBS

Protection Mode: MaxAvailability

Databases:

VISION - Primary database

VIS - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS

DGMGRL>


Run the adautocfg.sh script on the nodes.

a. etc18cn1dbiodom1

b. etc18cn2dbiodom1

c. etc18cn1dbiodom1

d. deopmn1 (eweb-lh)

e. deopmn1 (iopmn1-lh)

f. deopmn2 (iopmn2-lh)

g. deopmn2 (icm1-lh)

Verify that the contents of the tnsnames.ora file are correct.

Verify the Contents of the FND_NODES Table

The table within the APPS schema should contain relevant hostnames for the primary and secondary clusters as follows:








Connected to:



Management, OLAP,


SQL>

SQL> select node_name, server_address, virtual_ip from fnd_nodes;

NODE_NAME SERVER_ADDRESS VIRTUAL_IP

---------------- -------------- ----------

EWEB-LH m.m.m.68

IOPMN2-LH 192.168.28.34

ETC11-CN01 idb1-vip

Configure Flashback on the Database 26


ICM1-LH 192.168.28.43

ETC11-CN02 idb2-vip


AUTHENTICATION *

IOPMN1-LH 192.168.28.33

9 rows selected.

SQL>

4. Configure Flashback on the Database

You can configure the database with flashback retention. The flashback logs can be used to recover the failed primary database to a point in time before the switchover to the standby database occurred and convert that database to a physical standby database.

By default, flashback retention is set to 1440 minutes or 1 day. You can set it to a different value depending on how much flashback data is kept in the fast recovery area.

You define the flashback retention time through the db_flashback_retention_target parameter.

Do the following steps:

Check the status of the databases.

The output shows that as a consequence of Step 2 in Section 3.4, VISION is the primary database while VIS is the standby database.

oracle@etc18cn1dbiodom1:~$ srvctl status db –d vision –v

Instance VIS1 is running on node etc18cn1dbiodom1. Instance status: Open.

Instance VIS1 is running on node etc18cn2dbiodom1. Instance status: Open.

oracle@etc11-cn01:~$ srvctl status db –d vision –v

Instance VIS1 is running on node etc11-cn01. Instance status: Mounted

(Closed).

Instance VIS1 is running on node etc11-cn02. Instance status: Mounted

(Closed).

Access the primary database.









Connected to:



Management, OLAP,


SQL>

Configure the flashback retention as shown.

SQL> alter database flashback on;

SQL> select flashback_on from v$database;

FLASHBACK_ON

------------------

YES

SQL>

SQL> alter system set db_flashback_retention_target=240 scope=both;

System altered.

SQL> show parameters db_flashback_retention_target

NAME TYPE VALUE

------------------------------------ ----------- -----------------------------

-

db_flashback_retention_target integer 240

SQL>











Connected.

DGMGRL>

Display standby database information.

DGMGRL> show database ‘VIS’

Database – VIS

Role: PHYSICAL STAND BY

Intended State: APPLY-ON

Transport Lag: 0 seconds

Apply Lag: 0 seconds

Real Time Query: OFF

Instance(s):

VIS1 (apply instance)

VIS2

Database Status:

SUCCESS

Switchover the database.

This step causes VIS to become the primary database.

DGMGRL> switchover to 'VIS'

Performing switchover NOW, please wait...

New primary database "VIS" is opening...

Operation requires shutdown of instance "VIS2" on database "VISION"



Operation requires startup of instance "VIS2" on database "VISION"




Database mounted.

Switchover succeeded, new primary is "VIS"


Display database configuration.




Databases:

VIS - Primary database

VISION - Physical standby database



SUCCESS

DGMGRL>

Display both VIS and VISION database information.

DGMGRL> show database ‘VIS’

Database – VIS

Role: Primary

Intended State: TRANSPORT-ON

Instance(s):

VIS1

VIS2

Database Status:

SUCCESS

DGMGRL> show database ‘VISION’

Database – VISION

Role: PHYSICAL STANDBY

Intended State: APPLY-ON

Transport Lag: 0 seconds

Apply Lag: 0 seconds

Real Time Query: OFF

Instance(s)

VIS1 (apply instance)

VIS2

Database Status

SUCCESS

Configure Oracle Solaris Cluster Geo Edition 30

Check the status of the databases.

oracle@etc11-cn01:~$ srvctl status db –d vis –v

Instance VIS1 is running on node etc11-cn01. Instance status: Open

Instance VIS2 is running on node etc11-cn02. Instance status: Open

oracle@etc18cn1dbiodom1:~$ srvctl status db –d vision –v

Instance VIS1 is running on node etc18cn1dbiodom1. Instance status: Mounted

(Closed).

Instance VIS2 is running on node etc18cn2dbiodom1. Instance status: Mounted

(Closed).

On the new primary database, repeat Steps 2 and 3 to turn on flashback retention.

5. Configure Oracle Solaris Cluster Geo Edition

Oracle Solaris Cluster Geo Edition manages the disaster recovery orchestration by using a cluster partnership, protection groups, multigroups and sites.

Install Geographic Edition Software

Reference documentation:

How to Install Geographic Edition Software

Install the Oracle Solaris Cluster Geo Edition software on each Oracle Solaris Cluster node.

root@eopmn1:~# pkg install ha-cluster/group-package/ha-cluster-geo-full

Verify that the package installed successfully on each Oracle Solaris Cluster node.

root@eopmn1:~# pkg info ha-cluster/group-package/ha-cluster-geo-full

Name: ha-cluster/group-package/ha-cluster-geo-full

Summary: Oracle Solaris Cluster Geographic Edition full group package

Description: Oracle Solaris Cluster Geographic Edition full group package

Category: Meta Packages/Group Packages

State: Installed

Publisher: ha-cluster

Version: 4.3 (Oracle Solaris Cluster Geographic Edition 4.3.0.24.0)

Build Release: 5.11

Branch: 0.24.0

http://docs.oracle.com/cd/E56676_01/html/E56740/ewpqw.html#GEINSgbtbq


Packaging Date: August 26, 2015 11:33:37 PM

Size: 5.53 kB

FMRI: pkg://ha-cluster/ha-cluster/group-package/ha-cluster-geo-

[email protected],5.11-0.24.0:20150826T233337Z

root@eopmn1:~#

Prepare Each Zone Cluster for Partner Membership


How to Prepare a Zone Cluster for Partner Membership

5.2.1. Set Up the Network Address for Each Zone Cluster

Check first that the correct entries are in the /etc/hosts file before you set the address for the

zone clusters.

On the primary cluster, set the network address for opmn-zc and cm-zc.

a. Verify zone cluster entries on the /etc/hosts file.

root@eopmn1:~# grep opmn-zc /etc/hosts

n.n.n.115 opmn-zc.us.oracle.com opmn-zc

root@eopmn1:~#

root@eopmn2:~# grep opmn-zc /etc/hosts

n.n.n.115 opmn-zc.us.oracle.com opmn-zc

root@eopmn2:~#

root@ecm1:~# grep cm-zc /etc/hosts

n.n.n.116 cm-zc.us.oracle.com cm-zc

root@ecm1:~#

root@ecm2:~# grep cm-zc /etc/hosts

n.n.n.116 cm-zc.us.oracle.com cm-zc

root@ecm2:~#

b. Set the zone clusters’ network address.

root@etc11cn01-appiodom1:~# clzc configure opmn-zc

clzc:opmn-zc> add net

clzc:opmn-zc:net> set address=opmn-zc

clzc:opmn-zc:net> end

clzc:opmn-zc> verify

clzc:opmn-zc> commit

http://docs.oracle.com/cd/E56676_01/html/E56740/gkcda.html#GEINSgkcdy


clzc:opmn-zc> exit

root@etc11cn01-appiodom1:~#

root@etc11cn01-appiodom1:~# clzc configure cm-zc

clzc:cm-zc> add net

clzc:cm-zc:net> set address=cm-zc

clzc:cm-zc:net> end

clzc:cm-zc> verify

clzc:cm-zc> commit

clzc:cm-zc> exit

root@etc11cn01-appiodom1:~#

On the secondary cluster, set the network address for dr-opmn-zc and dr-cm-zc.

a. Verify zone cluster entries on the /etc/hosts file.

root@deopmn1:~# grep dr-opmn-zc /etc/hosts

m.m.m.81 dr-opmn-zc.us.oracle.com dr-opmn-zc

root@deopmn1:~#

root@deopmn2:~# grep dr-opmn-zc /etc/hosts

m.m.m.81 dr-opmn-zc.us.oracle.com dr-opmn-zc

root@deopmn2:~#

root@decm1:~# grep dr-cm-zc /etc/hosts

m.m.m.82 dr-cm-zc.us.oracle.com dr-cm-zc

root@decm1:~#

root@decm2:~# grep dr-cm-zc /etc/hosts

m.m.m.82 dr-cm-zc.us.oracle.com dr-cm-zc

root@decm2:~#

b. Set the zone clusters’ network address.

root@etc18app01:~# clzc configure dr-opmn-zc

clzc:dr-opmn-zc> add net

clzc:dr-opmn-zc:net> set address=dr-opmn-zc

clzc:dr-opmn-zc:net> end

clzc:dr-opmn-zc> verify

clzc:dr-opmn-zc> commit

clzc:dr-opmn-zc> exit

root@etc18app01:~# clzc configure dr-cm-zc

clzc:dr-cm-zc> add net

clzc:dr-cm-zc:net> set address=dr-cm-zc

clzc:dr-cm-zc:net> end

clzc:dr-cm-zc> verify

clzc:dr-cm-zc> commit


clzc:dr-cm-zc> exit

root@etc18app01:~#

5.2.2. Set Up Security on the Primary Cluster

In this step, you copy the security files for the common agent container to all zone cluster nodes on the primary cluster.

For the opmn-zc cluster nodes, perform the following steps:

a. On eopmn1, issue the commands as shown.

root@eopmn1:~# cacaoadm status

default instance is DISABLED at system startup.

default instance is not running.

root@eopmn1:~# cacaoadm stop

root@eopmn1:~# cacaoadm create-keys --force

root@eopmn1:~# cd /etc/cacao/instances/default

root@eopmn1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar ./security

root@eopmn1:/etc/cacao/instances/default# scp /tmp/SECURITY.tar iopmn2:/tmp

Password:

SECURITY.tar 100% |*****************************| 984 KB 00:00

root@eopmn1:/etc/cacao/instances/default# cd

root@eopmn1:~# cacaoadm set-param network-bind-address=0.0.0.0

root@eopmn1:~# cacaoadm enable

root@eopmn1:~# cacaoadm start

b. On eopmn2, issue the commands as shown.

root@eopmn2:~# cacaoadm status



root@eopmn2:~# cacaoadm stop

root@eopmn2:~# cd /etc/cacao/instances/default

root@eopmn2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar

root@eopmn2:/etc/cacao/instances/default# cd

root@eopmn2:~# cacaoadm set-param network-bind-address=0.0.0.0

root@eopmn2:~# cacaoadm enable

root@eopmn2:~# cacaoadm start


For the cm-zc nodes, perform the following steps:

a. On ecm1, issue the commands as shown.

root@ecm1:~# cacaoadm status



root@ecm1:~# cacaoadm stop

root@ecm1:~# cacaoadm create-keys --force

root@ecm1:~# cd /etc/cacao/instances/default

root@ecm1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar ./security

root@ecm1:/etc/cacao/instances/default# scp /tmp/SECURITY.tar icm2:/tmp

Password:

SECURITY.tar 100% |*****************************| 984 KB 00:00

root@ecm1:/etc/cacao/instances/default# cd

root@ecm1:~# cacaoadm set-param network-bind-address=0.0.0.0

root@ecm1:~# cacaoadm enable

root@ecm1:~# cacaoadm start

b. On ecm2, issue the commands as shown.

root@ecm2:~# cacaoadm status



root@ecm2:~# cacaoadm stop

root@ecm2:~# cd /etc/cacao/instances/default

root@ecm2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar

root@ecm2:/etc/cacao/instances/default# cd

root@ecm2:~# cacaoadm set-param network-bind-address=0.0.0.0

root@ecm2:~# cacaoadm enable

root@ecm2:~# cacaoadm start

5.2.3. Set Up Security on the Secondary Cluster

In this step, you copy the security files for the common agent container to all zone cluster nodes on the secondary cluster.

For the dr-opmn-zc cluster nodes, perform the following steps:

a. On deopmn1, issue the commands as shown.

root@deopmn1:~# cacaoadm status

default instance is ENABLED at system startup.

Smf monitoring process:

15282


15284

Uptime: 0 day(s), 22:56

root@deopmn1:~# cacaoadm stop

root@deopmn1:~# cacaoadm create-keys --force

root@deopmn1:~# cd /etc/cacao/instances/default

root@deopmn1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar

./security

root@deopmn1:/etc/cacao/instances/default# cd

root@deopmn1:~# scp /tmp/SECURITY.tar deopmn2:/tmp

Password:

SECURITY.tar 100% |*****************************| 985 KB 00:00


root@deopmn1:~# cacaoadm set-param network-bind-address=0.0.0.0

root@deopmn1:~# cacaoadm enable

root@deopmn1:~# cacaoadm start

b. On deopmn2, issue the commands as shown.

root@deopmn2:~# cacaoadm status



29308

29309


root@deopmn2:~# cacaoadm stop

root@deopmn2:~# cd /etc/cacao/instances/default

root@deopmn2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar


root@deopmn2:~# cacaoadm set-param network-bind-address=0.0.0.0

root@deopmn2:~# cacaoadm enable

root@deopmn2:~# cacaoadm start

For the dr-cm-zc cluster nodes, perform the following steps:

a. On decm1, issue the commands as shown.

root@decm1:~# cacaoadm status




18667

18668


root@decm1:~# cacaoadm stop

root@decm1:~# cacaoadm create-keys --force


root@decm1:~# cd /etc/cacao/instances/default

root@decm1:/etc/cacao/instances/default# tar cf /tmp/SECURITY.tar ./security

root@decm1:/etc/cacao/instances/default# scp /tmp/SECURITY.tar decm2:/tmp

Password:

SECURITY.tar 100% |*****************************| 984 KB 00:00

root@decm1:/etc/cacao/instances/default# cd

root@decm1:~# cacaoadm set-param network-bind-address=0.0.0.0

root@decm1:~# cacaoadm enable

root@decm1:~# cacaoadm start

b. On decm2, issue the commands as shown.

root@decm2:~# cacaoadm status



2668

2670


root@decm2:~# cacaoadm stop

root@decm2:~# cd /etc/cacao/instances/default

root@decm2:/etc/cacao/instances/default# tar xf /tmp/SECURITY.tar

root@decm2:/etc/cacao/instances/default# cd

root@decm2:~# cacaoadm set-param network-bind-address=0.0.0.0

root@decm2:~# cacaoadm enable

root@decm2:~# cacaoadm start

Enable Geographic Edition Software


How to Enable the Geographic Edition Framework

This step enables the Geographic Edition infrastructure. You must enable it on all the clusters of the geographically separated clusters.

Enable the Geographic Edition framework on the primary cluster.

a. For the zone cluster opmn-zc, issue the commands on eopmn1 as shown.

root@eopmn1:~# geoadm start

... checking for management agent ...

... management agent check done ....

... starting product infrastructure ... please wait ...

Registering resource type <SUNW.HBmonitor>...done.

Registering resource type <SUNW.SCGeoInitSvc>...done.

http://docs.oracle.com/cd/E56676_01/html/E56740/extal.html#GEINSegjzg


Resource type <SUNW.scmasa> has been registered already

Registering resource type <SUNW.SCGeoZC>...done.

Creating scalable resource group <geo-clusterstate>...done.

Creating service tag management resource <geo-servicetag>...

Service tag management resource created successfully ....

Creating failover resource group <geo-infrastructure>...done.

Creating logical host resource <geo-clustername>...

Logical host resource created successfully ....

Creating resource <geo-hbmonitor> ...done.

Creating resource <geo-failovercontrol> ...done.

Bringing RG <geo-clusterstate> to managed state ...done.

Bringing resource group <geo-infrastructure> to managed state ...done.

Enabling resource <geo-clustername> ...done.

Enabling resource <geo-hbmonitor> ...done.

Enabling resource <geo-failovercontrol> ...done.

Node eopmn1: Bringing resource group <geo-infrastructure> online ...done.

Oracle Solaris Cluster Geographic Edition infrastructure started successfully.

root@eopmn1:~#

root@eopmn1:~# geoadm show

--- CLUSTER LEVEL INFORMATION ---

Oracle Solaris Cluster Geographic Edition software is active on : node eopmn1, cluster

opmn-zc

Command ran successfully.

root@eopmn1:~#

root@eopmn1:~# clrg status geo-clusterstate geo-infrastructure

=== Cluster Resource Groups ===

Group Name Node Name Suspended Status

---------- --------- --------- ------

geo-clusterstate eopmn1 No Online

eopmn2 No Online

geo-infrastructure eopmn1 No Online

eopmn2 No Offline

root@eopmn1:~# clrs status -g geo-clusterstate, geo-infrastructure

=== Cluster Resources ===

Resource Name Node Name State Status Message

------------- --------- ----- --------------

geo-zc-sysevent eopmn1 Online_not_monitored Online_not_monitored

eopmn2 Online_not_monitored Online_not_monitored

geo_servicetag eopmn1 Online_not_monitored Online_not_monitored


geo-failovercontrol eopmn1 Online Online - Service is online.

eopmn2 Offline Offline


geo-hbmonitor eopmn1 Online Online - Daemon OK


geo-clustername eopmn1 Online Online - LogicalHostname

online.


root@eopmn1:~#

b. For the zone cluster cm-zc, issue the commands on ecm1 as shown.

root@ecm1:~# geoadm start





















Node ecm1: Bringing resource group <geo-infrastructure> online ...done.



7175

7180


root@ecm1:~#

root@ecm1:~# geoadm show


Oracle Solaris Cluster Geographic Edition software is active on : node ecm1, cluster dr-cm-

zc


root@ecm1:~#

root@ecm1:~# clrg status geo-clusterstate geo-infrastructure




---------- --------- --------- ------

geo-clusterstate ecm1 No Online

ecm2 No Online

geo-infrastructure ecm1 No Online

ecm2 No Offline

root@ecm1:~# clrs status -g geo-clusterstate, geo-infrastructure



------------- --------- ----- --------------

geo-zc-systevent ecm1 Online_not_monitored Online_not_monitored

ecm2 Online_not_monitored Online_not_monitored

geo-servicetag ecm1 Online_not_monitored Online_not_monitored

ecm2 Online_not_monitored Online_not_monitored

geo-failovercontrol ecm1 Online Online - Service is online.

Ecm2 Offline Offline

geo-hbmonitor ecm1 Online Online - Daemon OK

ecm2 Offline Offline

geo-clustername ecm1 Online Online - LogicalHostname

online.

Ecm2 Offline Offline

root@ecm1:~#

Enable the Geographic Edition framework on the secondary cluster.

a. For the zone cluster dr-opmn-zc, issue the commands on deopm1 as shown.

root@deopmn1:~# geoadm start






















Node deopmn1: Bringing resource group <geo-infrastructure> online ...done.


root@deopmn1:~#

root@deopmn1:~# geoadm show


Oracle Solaris Cluster Geographic Edition software is active on : node deopmn1, cluster dr-

opmn-zc


root@deopmn1:~#

root@deopmn1:~# clrg status geo-clusterstate geo-infrastructure



---------- --------- --------- ------

geo-clusterstate deopmn1 No Online

deopmn2 No Online

geo-infrastructure deopmn1 No Online

deopmn2 No Offline

root@deopmn1:~# clrs status -g geo-clusterstate, geo-infrastructure



------------- --------- ----- --------------

geo-zc-sysevent deopmn1 Online_not_monitored Online_not_monitored

deopmn2 Online_not_monitored Online_not_monitored

geo-servicetag deopmn1 Online_not_monitored Online_not_monitored


geo-failovercontrol deopmn1 Online Online - Service is online.

deopmn2 Offline Offline

geo-hbmonitor deopmn1 Online Online - Daemon OK


geo-clustername deopmn1 Online Online - LogicalHostname

online.



root@deopmn1:~#

b. For the zone cluster dr-cm-zc, issue the commands on decm1 as shown.

root@decm1:~# geoadm start





















Node decm1: Bringing resource group <geo-infrastructure> online ...done.


root@decm1:~#

root@decm1:~# geoadm show


Oracle Solaris Cluster Geographic Edition software is active on : node decm1, cluster dr-

cm-zc

--- LOCAL NODE INFORMATION ---

Node decm1 does not host active product module.


root@decm1:~#

root@decm1:~# clrg status geo-clusterstate geo-infrastructure



---------- --------- --------- ------

geo-clusterstate decm1 No Online

decm2 No Online


geo-infrastructure decm1 No Online

decm2 No Offline

root@decm1:~# clrs status -g geo-clusterstate, geo-infrastructure



------------- --------- ----- --------------

geo-zc-sysevent decm1 Online_not_monitored Online_not_monitored

decm2 Online_not_monitored Online_not_monitored

geo-servicetag decm1 Online_not_monitored Online_not_monitored

decm2 Online_not_monitored Online_not_monitored

geo-failovercontrol decm1 Online Online - Service is online.

decm2 Offline Offline

geo-hbmonitor decm1 Online Online - Daemon OK


geo-clustername decm1 Online Online - LogicalHostname

online.


root@decm1:~#

Configure Partnerships

The procedures in this section cover the following topics:

Configuration of partnership trusts

Creation of partnerships

Joining of partner clusters into the partnership

5.4.1. Configure Trust Between Partner Clusters


How to Configure Trust Between Two Clusters

Before creating a partnership between two clusters, you must first configure Geographic Edition software for secure communication between the two clusters. The configuration must be reciprocal.

On the primary cluster, configure partnership trusts for opmn-zc.

a. On eopmn1, create trust between opmn-zc and the remote dr-opmn-zc.

root@eopmn1:~# geops add-trust -c dr-opmn-zc

http://docs.oracle.com/cd/E56676_01/html/E56740/gbowv.html#GEINSgboxr


Local cluster : opmn-zc

Local node : eopmn1

Cleaning up certificate files in /etc/cacao/instances/default/security/jsse

on eopmn1

Retrieving certificates from dr-opmn-zc ... Done

New Certificate:

Owner: CN=deopmn1_agent

Issuer: CN=deopmn1_ca

Serial number: 701b5d1c

Valid from: Wed May 14 23:36:48 PDT 1969 until: Sun Oct 14 23:36:48 PDT 2035

Certificate fingerprints:

MD5: 0E:D6:7A:73:47:8D:7F:7F:7A:5A:38:AA:1B:5C:2D:62

SHA1: 57:62:C6:F2:AF:84:8A:AE:B0:18:2E:AE:B5:D8:AB:31:E3:87:7C:59

SHA256:

7C:16:92:01:94:0F:80:39:D3:56:30:86:5C:56:EF:69:97:70:EA:60:85:2F:FB:

6E:6C:EF:C0:72:28:0B:27:26

Signature algorithm name: SHA1withRSA

Version: 3

Extensions:

#1: ObjectId: 2.5.29.19 Criticality=false

BasicConstraints:[

CA:false

PathLen: undefined

]

Do you trust this certificate? [y/n] y

Adding certificate to truststore on eopmn1 ... Done


New Certificate:

Owner: CN=deopmn1_ca

Issuer: CN=deopmn1_ca

Serial number: 3b569267



MD5: 8C:3C:9E:A9:4B:7E:E6:40:96:B5:E0:4F:29:64:29:0F

SHA1: 8B:D1:8D:0C:01:9D:11:75:E9:65:CE:C9:BA:B1:68:55:F4:96:78:0B

SHA256:

ED:58:C4:42:C9:55:12:37:5C:64:C1:34:4F:D3:E9:FB:37:68:7E:6D:87:7F:C2:8F:96:


06:FF:D3:6B:15:41:1A


Version: 3

Extensions:

#1: ObjectId: 2.5.29.19 Criticality=true

BasicConstraints:[

CA:true

PathLen:1

]




Operation completed successfully. All certificates are added to truststore on

nodes of cluster opmn-zc

root@eopmn1:~#

b. On both eopmn1 and eopmn2, verify that the trust was created.

root@eopmn1:~# geops verify-trust -c dr-opmn-zc


Local node : eopmn1

Retrieving information from cluster dr-opmn-zc ... Done

Verifying connections to nodes of cluster dr-opmn-zc:

eopmn1 -> deopmn2 / m.m.m.64 ... OK


Operation completed successfully, able to establish secure connections from

eopmn1 to all nodes of cluster dr-opmn-zc

root@eopmn1:~#

root@eopmn2:~# geops verify-trust -c dr-opmn-zc


Local node : eopmn2

Retrieving information from cluster dr-opmn-zc ... Done

Verifying connections to nodes of cluster dr-opmn-zc:





eopmn2 to all nodes of cluster dr-opmn-zc.

root@eopmn2:~#

On the primary cluster, configure partnership trusts for cm-zc.

a. On ecm1, create trust between cm-zc and the remote dr-cm-zc.

root@ecm1:~# geops add-trust -c dr-cm-zc

Local cluster : cm-zc

Local node : ecm1

Cleaning up certificate files in /etc/cacao/instances/default/security/jsse on

ecm1

Retrieving certificates from dr-cm-zc ... Done

New Certificate:

Owner: CN=decm1_agent

Issuer: CN=decm1_ca

Serial number: 63ee75ca



MD5: BD:4C:9A:55:BB:68:FB:61:95:A9:3F:8E:80:6A:B9:DE

SHA1: 71:62:6E:DF:E0:CA:A2:52:D8:D2:CC:63:5E:D3:2F:1B:6C:A1:BF:AB

SHA256: 19:87:53:BE:E8:93:F7:2D:36:D5:C7:04:17:45:70:0F:1F:A4:32:

47:39:F6:80:1E:52:0A:95:35:B0:96:78:1E


Version: 3

Extensions:


BasicConstraints:[

CA:false

PathLen: undefined

]


Adding certificate to truststore on ecm1 ... Done



New Certificate:

Owner: CN=decm1_ca

Issuer: CN=decm1_ca

Serial number: 1249454



MD5: 32:8D:3B:3F:83:8E:20:B1:55:95:5F:FC:87:76:57:55

SHA1: CE:18:C4:0D:AD:39:E4:04:A0:9D:1D:7B:03:E4:2B:C4:8A:9E:9D:89

SHA256: 52:0D:B8:C4:0E:BD:3E:D5:E6:8B:49:DB:66:9D:38:22:11:

0F:35:4D:B9:8B:BA:78:B1:26:C8:4F:21:0B:AD:67


Version: 3

Extensions:


BasicConstraints:[

CA:true

PathLen:1

]




Operation completed successfully. All certificates are added

to truststore on nodes of cluster cm-zc

root@ecm1:~#

b. On both ecm1 and ecm2, verify that the trust was created.

root@ecm1:~# geops verify-trust -c dr-cm-zc


Local node : ecm1

Retrieving information from cluster dr-cm-zc ... Done

Verifying connections to nodes of cluster dr-cm-zc:

ecm1 -> decm1 / m.m.m 65 ... OK

ecm1 -> decm2 / m.m.m.66 ... OK



ecm1 to all nodes of cluster dr-cm-zc

root@ecm1:~#

root@ecm2:~# geops verify-trust -c dr-cm-zc


Local node : ecm2

Retrieving information from cluster dr-cm-zc ... Done

Verifying connections to nodes of cluster dr-cm-zc:

ecm2 -> decm1 / m.m.m 65 ... OK

ecm2 -> decm2 / m.m.m 66 ... OK


ecm2 to all nodes of cluster dr-cm-zc

root@ecm2:~#

On the secondary cluster, configure partnership trusts for dr-opmn-zc.

a. On deopmn1, create trust between dr-opmn-zc and the remote opmn-zc.

root@deopmn1:~# geops add-trust -c opmn-zc

Local cluster : dr-opmn-zc

Local node : deopmn1

Cleaning up certificate files in /etc/cacao/instances/default/security/jsse

on deopmn1

Retrieving certificates from opmn-zc ... Done

New Certificate:

Owner: CN=eopmn1_agent

Issuer: CN=eopmn1_ca

Serial number: 4855faab

Valid from: Wed May 14 23:32:46 MeDT 1969 until: Sun Oct 14 22:32:46 MeST

2035


MD5: 90:02:22:E5:06:FB:9C:6A:6C:BA:1C:EE:1F:55:14:6A

SHA1: 4A:8D:FD:7B:7D:29:54:06:70:A1:52:27:63:5A:13:C7:56:EA:AA:62

SHA256:

24:6D:88:5F:E6:AD:B5:06:00:84:96:FE:B9:C0:2E:B2:C3:B0:26:23:66:

D6:7A:33:13:F5:06:0C:21:EE:FE:2C


Version: 3


Extensions:


BasicConstraints:[

CA:false

PathLen: undefined

]


Adding certificate to truststore on deopmn1 ... Done


New Certificate:

Owner: CN=eopmn1_ca

Issuer: CN=eopmn1_ca

Serial number: 5adf1784

Valid from: Wed May 14 23:32:45 MeDT 1969 until: Sun Oct 14 22:32:45 MeST

2035


MD5: AB:9F:B8:79:A0:4B:66:48:26:64:62:43:1D:AD:00:47

SHA1: 18:5E:E8:C8:A2:FC:3C:B1:D2:AD:92:CF:1B:2F:AB:D4:12:44:04:5E

SHA256: 52:E6:97:A9:22:16:98:71:43:21:10:43:97:39:45:05:05:36:77:

99:AB:0D:65:0E:C6:55:7D:19:AB:26:9A:4A


Version: 3

Extensions:


BasicConstraints:[

CA:true

PathLen:1

]




Operation completed successfully. All certificates are added to truststore on

nodes of cluster dr-opmn-zc

root@deopmn1:~#


b. On both deopmn1 and deopmn2, verify that the trust was created.

root@deopmn1:~# geops verify-trust -c opmn-zc



Retrieving information from cluster opmn-zc ... Done

Verifying connections to nodes of cluster opmn-zc:

deopmn1 -> eopmn1 / n.n.n.22 ... OK



deopmn1 to all nodes of cluster opmn-zc

root@deopmn1:~#

root@deopmn2:~# geops verify-trust -c opmn-zc



Retrieving information from cluster opmn-zc ... Done

Verifying connections to nodes of cluster opmn-zc:




deopmn2 to all nodes of cluster opmn-zc

root@deopmn2:~#

On the secondary cluster, configure partnership trusts for dr-cm-zc.

a. On decm1, create trust between dr-cm-zc and the remote cm-zc.

root@decm1:~# geops add-trust -c cm-zc

Local cluster : dr-cm-zc

Local node : decm1

Cleaning up certificate files in /etc/cacao/instances/default/security

/jsse on decm1

Retrieving certificates from cm-zc ... Done

New Certificate:


Owner: CN=ecm1_agent

Issuer: CN=ecm1_ca

Serial number: 3e192ee1

Valid from: Wed May 14 23:13:50 MeDT 1969 until: Sun Oct 14 22:13:50 MeST 2035


MD5: AB:25:0D:95:56:6B:14:DB:49:98:D2:D0:E1:30:71:E2

SHA1: DF:44:CC:6A:C5:3C:E7:D6:3B:DC:16:BB:BE:47:1C:F5:61:79:AB:00

SHA256: B8:87:52:7F:B4:32:2E:5E:9D:35:D1:22:75:44:FC:08:35:19:05:

E6:6A:D5:97:69:22:86:56:21:DA:1E:F7:A9


Version: 3

Extensions:


BasicConstraints:[

CA:false

PathLen: undefined

]


Adding certificate to truststore on decm1 ... Done


New Certificate:

Owner: CN=ecm1_ca

Issuer: CN=ecm1_ca

Serial number: 19e8a71e

Valid from: Wed May 14 23:13:48 MeDT 1969 until: Sun Oct 14 22:13:48 MeST 2035


MD5: B7:31:32:F3:7B:D3:6F:73:41:46:FE:88:B1:37:69:AB

SHA1: 1B:14:98:BC:C0:62:EE:21:31:2D:34:02:68:FC:A7:6F:68:D0:AC:06

SHA256: 3B:52:5E:47:10:FE:DB:E0:F1:B7:BA:42:A4:FB:AB:48:0F:43:

4A:A4:D7:E5:9C:6B:11:06:CF:0F:71:18:CE:E4


Version: 3

Extensions:


BasicConstraints:[

CA:true

PathLen:1

]





Operation completed successfully. All certificates are added to

truststore on nodes of cluster dr-cm-zc

root@decm1:~#

b. On both decm1 and decm2, verify that the trust was created.

root@decm1:~# geops verify-trust -c cm-zc


Local node : decm1

Retrieving information from cluster cm-zc ... Done

Verifying connections to nodes of cluster cm-zc:

decm1 -> ecm1 / n.n.n.24 ... OK

decm1 -> ecm2 / n.n.n.25 ... OK


decm1 to all nodes of cluster cm-zc

root@decm1:~#

root@decm2:~# geops verify-trust -c cm-zc


Local node : decm2

Retrieving information from cluster cm-zc ... Done

Verifying connections to nodes of cluster cm-zc:

decm2 -> ecm1 / n.n.n.24 ... OK

decm2 -> ecm2 / n.n.n.25 ... OK


decm2 to all nodes of cluster cm-zc

:~#

mailto:root@decm2


5.4.2. Create Partnerships


How to Create a Partnership

A partnership establishes heartbeat monitoring between two clusters that are running the Geographic Edition framework. Partnered clusters exchange heartbeats to monitor each other's presence and health. A partnership can involve only two clusters, and only one partnership can be defined between the clusters.

The steps that follow create partnerships between the two clusters as reflected in Figure 2:

“Partnership 1” involves the primary zone cluster opmn-zc and the standby zone cluster dr-opmn-zc. This partnership is created as opmn-ps.

“Partnership 2” involves the primary zone cluster cm-zc and the standby zone cluster dr-cm-zc. This partnership is created as cm-ps.

The procedure includes verification that the partnerships are reflected in both the primary and secondary clusters.

On the primary cluster, configure the partnerships.

a. On eopmn1, create a partnership between zone clusters opmn-zc and dr-opmn-zc.

root@eopmn1:~# geops create -c dr-opmn-zc opmn-ps

Partnership between local cluster "opmn-zc" and remote cluster "dr-opmn-zc"

successfully created.

root@eopmn1:~# geops status

Partnership "opmn-ps" : OK

Partner clusters : dr-opmn-zc

Synchronization : OK

ICRM Connection : OK

Heartbeat "hb_opmn-zc~dr-opmn-zc" monitoring "dr-opmn-zc": OK

Plug-in "ping_plugin" : Inactive

Plug-in "tcp_udp_plugin" : OK

root@eopmn1:~#

b. On eopmn2, verify the created partnership.

root@eopmn2:~# geops status


Partner clusters : dr-opmn-zc



Heartbeat "hb_opmn-zc~dr-opmn-zc" monitoring "dr-opmn-zc": OK

http://docs.oracle.com/cd/E56676_01/html/E56740/gnojr.html#scrolltoc




root@eopmn2:~#

c. On ecm1, create a partnership between zone clusters cm-zc and dr-cm-zc.

root@ecm1:~# geops create -c dr-cm-zc cm-ps

Partnership between local cluster "cm-zc" and remote cluster "dr-cm-zc"

successfully created.

root@ecm1:~# geops status

Partnership "cm-ps" : OK

Partner clusters : dr-cm-zc



Heartbeat "hb_cm-zc~dr-cm-zc" monitoring "dr-cm-zc": OK



root@ecm1:~#

d. On ecm2, verify the created partnership.

root@ecm2:~# geops status


Partner clusters : dr-cm-zc



Heartbeat "hb_cm-zc~dr-cm-zc" monitoring "dr-cm-zc": OK



root@ecm2:~#

On the secondary cluster, verify the partnerships.

a. On deopmn1 and deopmn2, verify the partnership for the zone cluster dr-opmn-zc.

root@deopmn1:~# geops status



Partner clusters : opmn-zc



Heartbeat "hb_dr-opmn-zc~opmn-zc" monitoring "opmn-zc": OK



root@deopmn1:~#

root@deopmn2:~# geops status


Partner clusters : opmn-zc



Heartbeat "hb_dr-opmn-zc~opmn-zc" monitoring "opmn-zc": OK



root@deopmn2:~#

b. On decm1 and decm2, verify the partnership for dr-cm-zc.

root@decm1:~# geops status


Partner clusters : cm-zc



Heartbeat "hb_dr-cm-zc~cm-zc" monitoring "cm-zc": OK



root@decm1:~#

root@decm2:~# geops status


Partner clusters : cm-zc




Heartbeat "hb_dr-cm-zc~cm-zc" monitoring "cm-zc": OK



root@decm2:~#

5.4.3. Configure the Secondary Cluster to Join the Partnership


How to Join a Partnership

In this step, you complete the partnership configuration by defining the secondary cluster to become the second member of the partnership. You perform this step on the secondary cluster.

On deopmn1, configure the local cluster to join the cluster opmn-zc in the partnership opmn-ps.

root@deopmn1:~# geops join-partnership opmn-zc opmn-ps

Local cluster "dr-opmn-zc" is now partner of cluster "opmn-zc".

root@deopmn1:~#

Verify the partnership’s status.

root@deopmn1:~# geops show opmn-ps

Partnership: opmn-ps

Description :

Local heartbeat : hb_dr-opmn-zc~opmn-zc

Remote heartbeat : hb_opmn-zc~dr-opmn-zc

Notification_ActionCmd :

Notification_EmailAddrs :

root@deopmn1:~#

On decm1, configure the local cluster to join the cluster cm-zc in the partnership cm-ps.

root@decm1:~# geops join-partnership cm-zc cm-ps

Local cluster "dr-cm-zc" is now partner of cluster "cm-zc".

root@decm1:~#

Verify the partnership’s status.

root@decm1:~# geops show cm-ps

http://docs.oracle.com/cd/E56676_01/html/E56740/eyprd.html#GEINSeyppk


Partnership: cm-ps

Description :

Local heartbeat : hb_dr-cm-zc~cm-zc

Remote heartbeat : hb_cm-zc~dr-cm-zc

Notification_ActionCmd :

Notification_EmailAddrs :

root@decm1:~#

Configure Protection Groups

Protection groups enable a set of clusters to tolerate and recover from disaster by managing the resource groups for services. Protection groups can exist only in a partnership. Thus, you must first create partnerships before you can define protection groups.

In a protection group for a specific partnership, one partner cluster becomes the primary cluster while the other partner becomes the secondary cluster.

A protection group contains application resource groups and properties for managing data replication for those application resource groups.

The configuration for a protection group is identical on partner clusters. Therefore, partner clusters must have the application resource groups of the protection group defined in their configuration. The Geographic Edition framework propagates protection group configurations between partners.

5.5.1. Create the First Protection Group (opmn-ebs-pg)


Creating a Protection Group That Does Not Require Data Replication

Based on Figure 2, Protection Group 1 manages Web Entry Point (WEB) and Oracle Process Manager and Server (OPMN1 & OPMN2) services. Thus, the group is called opmn-ebs-pg. This group is set up for Partnership 1 (opmn-ps).

Ensure that the required resource groups are online.

root@eopmn1:~# clrg online -eM opmn1-lh-rg opmn2-lh-rg

root@eopmn1:~# clrg online -eM web-rg opmn1-rg opmn2-rg

root@eopmn1:~# clrg status



---------- --------- --------- ------

web-rg eopmn1 No Online

eopmn2 No Offline

http://docs.oracle.com/cd/E56676_01/html/E56740/gnoly.html#scrolltoc


opmn1-lh-rg eopmn1 No Online

eopmn2 No Offline


eopmn1 No Offline

opmn1-rg eopmn1 No Online


scalmnt-rg eopmn1 No Online

eopmn2 No Online


eopmn2 No Online


eopmn2 No Offline

On the primary cluster, perform the following steps:

a. Create the group.

root@eopmn1:~# geopg list

*** No protection group configured ***

root@eopmn1:~#

root@eopmn1:~# geopg create -s opmn-ps -o primary opmn-ebs-pg

Protection group "opmn-ebs-pg" successfully created.

root@eopmn1:~# geopg validate opmn-ebs-pg

Protection group "opmn-ebs-pg" successfully validated.

b. Configure the group.

root@eopmn1:~# clrg set -p Auto_start_on_new_cluster=false web-rg opmn1-rg

opmn2-rg

root@eopmn1:~# geopg set-prop -p External_Dependency_Allowed=true opmn-ebs-pg

Protection group "opmn-ebs-pg" successfully modified.

root@eopmn1:~#

root@eopmn1:~# geopg status opmn-ebs-pg


Protection group "opmn-ebs-pg" : OK

Partnership : opmn-ps


Cluster opmn-zc : OK

Role : Primary

Activation state : Activated

Configuration : OK

Data replication : None

Resource groups : None

Cluster dr-opmn-zc : OK

Role : Secondary


Configuration : OK


Resource groups : None

c. Add resource groups to the protection group.

root@eopmn1:~# geopg add-resource-group web-rg,opmn1-rg,opmn2-rg opmn-ebs-pg

Following resource groups successfully added:

"web-rg,opmn1-rg,opmn2-rg".



root@eopmn1:~#

root@eopmn1:~# geopg show opmn-ebs-pg

Protection Group: opmn-ebs-pg

Partnership name : opmn-ps

Local Role : Primary

Deployed on clusters : opmn-zc, dr-opmn-zc

Data Replication type : NONE

Configuration status : OK

Synchronization status : OK

Creation signature : opmn-zc Oct 16, 2015 5:42:31 PM PDT

Last update : Oct 16, 2015 4:54:19 PM MeST

Local properties : None

Global properties :


Description :

Timeout : 3600 seconds

RoleChange_ActionCmd :

External_Dependency_Allowed : true

RoleChange_ActionArgs :

Protected resource groups:

opmn2-rg

web-rg

opmn1-rg

*** No device groups defined in protection group "opmn-ebs-pg" ***

On the secondary cluster, create the same protection group by retrieving the configuration from the partner cluster.

root@deopmn1:~# geopg get -s opmn-ps opmn-ebs-pg

Protection group "opmn-ebs-pg" successfully created.

root@deopmn1:~# geopg validate opmn-ebs-pg


On the primary cluster, start the protection group.

root@eopmn1:~# geopg start -e global opmn-ebs-pg

Processing operation... The timeout period for this operation on each cluster

is 3600 seconds...

Protection group "opmn-ebs-pg" successfully started.

root@eopmn1:~#



root@eopmn1:~#







Role : Primary


Configuration : OK


Resource groups : OK


Role : Secondary


Configuration : OK



5.5.2. Create the Second Protection Group (cm-ebs-pg)


How to Create and Configure an Oracle ZFS Storage Appliance Protection Group

Based on Figure 2, Protection Group 2 manages Concurrent Manager (CM1) services as well as providing data replication for the Oracle ZFS Storage Appliance. Thus, the group is called cm-ebs-pg. This group is set up for Partnership 2 (cm-ps).

You create this protection group by creating and configuring the Oracle ZFS Storage Appliance replication.


a. Create a resource of SUNW.ScalMountPoint for the Concurrent Manager Logs.

root@ecm1:~# grep cmlogs /etc/vfstab

etc11-storIB:/export/ebslogs/ebsfs_cmlogs - /ebsfs_cmlogs nfs - no

rw,bg,hard,nointr,nosuid,rsize=131072,wsize=131072,proto=tcp,vers=3

root@ecm2:~# grep cmlogs /etc/vfstab

etc11-storIB:/export/ebslogs/ebsfs_cmlogs - /ebsfs_cmlogs nfs - no

rw,bg,hard,nointr,nosuid,rsize=131072,wsize=131072,proto=tcp,vers=3

root@ecm1:~# mount /ebsfs_cmlogs

root@ecm1:~# df -h /ebsfs_cmlogs

Filesystem Size Used Available Capacity Mounted on

etc11-storIB:/export/ebslogs/ebsfs_cmlogs

13T 31K 13T 1% /ebsfs_cmlogs

http://docs.oracle.com/cd/E56676_01/html/E39675/fncnr.html#scrolltoc


root@ecm2:~# mount /ebsfs_cmlogs

root@ecm2:~# df -h /ebsfs_cmlogs

Filesystem Size Used Available Capacity Mounted on

etc11-storIB:/export/ebslogs/ebsfs_cmlogs

13T 31K 13T 1% /ebsfs_cmlogs

root@ecm1:~# clrg create -S scalmnt2-rg

root@ecm1:~# clrs create -d -g scalmnt2-rg -t ScalMountPoint -x

MountPointDir=/ebsfs_cmlogs -x FileSystemType=nas -x TargetFileSystem=etc11-

storIB:/export/ebslogs/ebsfs_cmlogs cm-logs

root@ecm1:~#

root@ecm1:~# mkdir -p /var/geo/config/

root@ecm1:~# echo "ebs-apps-cmlogs|any|ecm2,ecm1" >

/var/geo/config/geozfssaconfig

root@ecm1:~# grep vnic1 /etc/hosts

n.n.n.90 etc11-vnic1.us.oracle.com etc11-vnic1

m.m.m.35 etc18-vnic1.us.oracle.com etc18-vnic1

root@ecm1:~#

root@ecm1:~# mkdir -p /var/tmp/geo

root@ecm1:~# cp /opt/ORCLscgrepzfssa/etc/zfssa_geo_config /var/tmp/geo/

root@ecm1:~# grep -v ^# /var/tmp/geo/zfssa_geo_config | grep -v ^$

PS=cm-ps

PG=cm-ebs-pg

REPCOMP=ebs-apps-cmlogs

REPRS=zfssa-rep-rs

REPRG=zfssa-rep-rg

DESC="Geo ZFSSA"

APPRG=scalmnt2-rg,cm-rg

CONFIGFILE=/var/geo/config/geozfssaconfig

LOCAL_CONNECT_STRING=geouser@etc11-vnic1

REMOTE_CONNECT_STRING=geouser@etc18-vnic1

CLUSTER_DGS=

root@ecm1:~#

root@ecm1:~# clrg offline cm-rg

root@ecm1:~# clrg offline scalmnt2-rg

root@ecm1:~# /opt/ORCLscgrepzfssa/util/zfssa_geo_register -f

/var/tmp/geo/zfssa_geo_config

sourcing /var/tmp/geo/zfssa_geo_config

Creating protection group "cm-ebs-pg"

This operation might take a while...

Protection group "cm-ebs-pg" successfully created.


Adding the replication component "ebs-apps-cmlogs" to the protection group

"cm-ebs-pg"

Password for property local_service_password :

Password for property remote_service_password :

Script-based plug-in configuration "ebs-apps-cmlogs" successfully added to

the protection group "cm-ebs-pg".

Run "geopg get --partnership cm-ps cm-ebs-pg" on the partner cluster if this

has not been done.

root@ecm1:~#

b. Add resource groups to the protection group.

root@ecm1:~# geopg add-resource-group cm-rg cm-ebs-pg


"cm-rg".

root@ecm1:~#

root@ecm1:~# geopg show cm-ebs-pg

Protection Group: cm-ebs-pg

Partnership name : cm-ps


Deployed on clusters : cm-zc, dr-cm-zc

Data Replication type : SBP



Creation signature : cm-zc Oct 16, 2015 9:14:17 AM PDT

Last update : Oct 16, 2015 9:31:10 AM PDT


Global properties :

create_config_script :

/opt/ORCLscgrepzfssa/bin/zfssa_geo_control


Description : Geo ZFSSA

add_app_rg_args :

remove_app_rg_args :


remove_config_script :



switchover_script :



takeover_script :


start_replication_script :


remove_app_rg_script : /bin/true

stop_replication_script :


add_app_rg_script :



configuration_file : /var/geo/config/geozfssaconfig


cm-rg

scalmnt2-rg

SBP Script-based plug-in configurations:

ebs-apps-cmlogs

ebs-apps-cmlogs_create_config_args : -O ebs-apps-cmlogs -r zfssa-rep-

rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1

ebs-apps-cmlogs_start_replication_args : -O ebs-apps-cmlogs -r zfssa-

rep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-

vnic1

ebs-apps-cmlogs_switchover_args : -O ebs-apps-cmlogs -r zfssa-rep-rs

-g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1

ebs-apps-cmlogs_local_service_password : ********

ebs-apps-cmlogs_stop_replication_args : -O ebs-apps-cmlogs -r zfssa-

rep-rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-

vnic1

ebs-apps-cmlogs_remote_service_password : ********

ebs-apps-cmlogs_remove_config_args : -O ebs-apps-cmlogs -r zfssa-rep-

rs -g zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1

ebs-apps-cmlogs_takeover_args : -O ebs-apps-cmlogs -r zfssa-rep-rs -g

zfssa-rep-rg -S cm-zc:geouser@etc11-vnic1,dr-cm-zc:geouser@etc18-vnic1

root@ecm1:~#

root@ecm1:~# geopg validate cm-ebs-pg

Protection group "cm-ebs-pg" successfully validated.


root@decm1:~# geopg get -s cm-ps cm-ebs-pg

Protection group "cm-ebs-pg" successfully created.


root@decm1:~#

root@decm1:~# geopg validate cm-ebs-pg



root@ecm1:~# geopg start -e global cm-ebs-pg


is 3600 seconds...

Protection group "cm-ebs-pg" successfully started.

root@ecm1:~#

root@ecm1:~# geopg validate cm-ebs-pg


root@ecm1:~#

root@ecm1:~# geopg status cm-ebs-pg

Protection group "cm-ebs-pg" : OK

Partnership : cm-ps


Cluster cm-zc : OK

Role : Primary


Configuration : OK

Data replication : OK


Cluster dr-cm-zc : OK

Role : Secondary


Configuration : OK



5.5.3. Create the Third Protection Group (cm-odg-pg)


How to Create and Configure an Oracle Data Guard Protection Group

http://docs.oracle.com/cd/E56676_01/html/E56742/fnrgf.html#scrolltoc


Based on Figure 2, Protection Group 3 provides data replication for Oracle Data Guard. Thus, the group is called cm-odg-pg. This group is also set up for Partnership 2 (cm-ps).


a. Create the group.

root@ecm1:~# geopg create -s cm-ps -o primary -d odg cm-odg-pg

Protection group "cm-odg-pg" successfully created.

root@ecm1:~# geopg validate cm-odg-pg

Protection group "cm-odg-pg" successfully validated.

root@ecm1:~#

b. Configure the protection group.

This step creates the following for Oracle Solaris Cluster:

EBS-oracle-external-proxy-shadow-rg resource group

EBS-oracle-external-proxy-shadow-rs resource

root@ecm1:~# cat /var/cluster/geo/odg/EBS_config ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1

root@ecm1:~#

root@ecm1:~# geopg add-replication-component \

-p local_database_name=VIS \

-p local_db_service_name=VIS_DGMGRL \

-p remote_database_name=VISION \

-p remote_db_service_name=VISION_DGMGRL \

-p standby_type=physical \

-p local_oracle_svr_rg_name=oep-rg \

-p remote_oracle_svr_rg_name=oep-rg \

-p sysdba_username=sys \

-p sysdba_password= \

-p replication_mode=maxavailability EBS cm-odg-pg

Password for property sysdba_password :

Oracle Data Guard configuration "EBS" successfully added to the protection group "cm-odg-pg".

c. Add resource groups to the protection group.

root@ecm1:# geopg add-resource-group EBS-oracle-external-proxy-shadow-rg cm-odg-pg


"EBS-oracle-external-proxy-shadow-rg".

root@ecm1:#




root@ecm1:~#

root@ecm1:~# geopg show cm-odg-pg

Protection Group: cm-odg-pg

Partnership name : cm-ps


Deployed on clusters : cm-zc, dr-cm-zc

Data Replication type : ODG



Creation signature : cm-zc Oct 16, 2015 7:31:09 AM PDT

Last update : Oct 20, 2015 4:42:00 AM MeST


Global properties :

Description :






EBS-oracle-external-proxy-shadow-rg

ODG Oracle Data Guard configurations:

EBS

EBS_remote_db_service_name : VIS_DGMGRL

EBS_remote_rac_proxy_svr_rg_name : oep-rg

EBS_replication_mode : maxavailability

EBS_sysdba_username : sys

EBS_local_db_service_name : VISION_DGMGRL

EBS_local_rac_proxy_svr_rg_name : oep-rg

EBS_sysdba_password : ********

EBS_local_database_name : VISION

EBS_standby_type : physical

EBS_remote_database_name : VIS



root@decm1:~# geopg get -s cm-ps cm-odg-pg

Protection group "cm-odg-pg" successfully created.

root@decm1:~#

root@decm1:~# geopg validate cm-odg-pg



root@ecm1:~# geopg start -e global cm-odg-pg


is 3600 seconds...

Protection group "cm-odg-pg" successfully started.

root@ecm1:~#



root@ecm1:~#

root@ecm1:~# geopg status cm-odg-pg

Protection group "cm-odg-pg" : OK

Partnership : cm-ps


Cluster cm-zc : OK

Role : Primary


Configuration : OK




Role : Secondary


Configuration : OK



5.5.4. Display the Oracle Solaris Cluster Listings

After you have completed the configuration of Oracle Solaris Cluster Geo Edition, display the

listings.


Display the listings on the primary cluster.

a. Check the status of the opmn-zc zone cluster.

root@eopmn1:~# clrg status



---------- --------- --------- -------


eopmn2 No Offline


eopmn1 No Offline

scalmnt-rg eopmn1 No Online

eopmn2 No Online


eopmn2 No Online


eopmn2 No Offline

web-rg eopmn1 No Online

eopmn2 No Offline



root@eopmn1:~#

root@eopmn1:~# clrs status


Resource Name Node State Status Message

------------- ----- ----- --------------

ebs-web-lsr eopmn1 Online Online - Service is online.


ebs-web eopmn1 Online Online - Service is online.


ebs-web-lh eopmn1 Online Online - LogicalHostname online.

eopmn2 Offline Offline - LogicalHostname offline.

ebs-opmn1-lh eopmn1 Online Online - LogicalHostname online.

eopmn2 Offline Offline - LogicalHostname offline.

ebs-opmn2-lh eopmn2 Online Online - LogicalHostname online.


ebs-opmn1 eopmn1 Online Online


ebs-opmn1-lsr eopmn1 Online Online - Service is online.

ebs-opmn2 eopmn2 Online Online

ebs-opmn2-lsr eopmn2 Online Online - Service is online.

ebs-storage eopmn1 Online Online

eopmn2 Online Online

geo-zc-sysevent eopmn1 Online_not_monitored Online_not_monitored


geo-servicetag eopmn1 Online_not_monitored Online_not_monitored


geo-failovercontrol eopmn1 Online Online - Service is online.


geo-hbmonitor eopmn1 Online Online - Daemon OK


geo-clustername eopmn1 Online Online - LogicalHostname online.


b. Check the status of the cm-zc zone cluster.

root@ecm1:~# clrg status



--------------- --------- --------- ------

oep-rg ecm1 No Online

ecm2 No Online

scalmnt-rg ecm1 No Online

ecm2 No Online

cm-rg ecm1 No Online

ecm2 No Offline

scalmnt2-rg ecm1 No Online

ecm2 No Online

geo-clusterstate ecm1 No Online

ecm2 No Online

geo-infrastructure ecm1 No Online

ecm2 No Offline


cm-odg-pg-odg-rep-rg ecm No Online

ecm2 No Offline

EBS-oracle-external-proxy-shadow-rg ecm1No Online

ecm2 No Online

zfssa-rep-rg ecm1 No Online

ecm2 No Offline

root@ecm1:~#

root@ecm1:~# clrs status



------------------- -------------- ------ ------------------

oep-rs ecm Online Online - Service VIS is UP [Instance is

OPEN]

ecm2 Online Online - Service VIS is UP [Instance is

OPEN]

ebs-storage ecm1 Online Online

ecm2 Online Online

ebs-cm ecm1 Online Online - Service is online.


ebs-cm-lsr ecm1 Online Online - Service is online.


ebs-cm-lh ecm1 Online Online - LogicalHostname online.

ecm2 Offline Offline - LogicalHostname offline.

cm-logs ecm1 Online Online

ecm2 Online Online

geo-zc-sysevent ecm1 Online_not_monitored

Online_not_monitored

ecm2 Online_not_monitored


geo-servicetag ecm1 Online_not_monitored


ecm2 Online_not_monitored


geo-failovercontrol ecm1 Online Online - Service is online.


geo-hbmonitor ecm1 Online Online - Daemon OK


geo-clustername ecm1 Online Online - LogicalHostname online.



EBS-odg-rep-rs ecm1 Online Online - Data protection mode is

MaxAvailability


EBS-oracle-external-proxy-shadow-rs ecm1 Online Online - Service VIS is UP

[Instance is OPEN] (Data Guard Primary)

ecm2 Online Online - Service VIS is UP [Instance is

OPEN] (Data Guard Primary)

zfssa-rep-rs ecm1 Online Online - Sending update


Display the listings on the secondary cluster.

a. Check the status of the dr-opmn-zc zone cluster.

root@deopmn1:~# clrg status



---------- --------- --------- ------

web-rg deopmn1 No Unmanaged

deopmn2 No Unmanaged

opmn1-lh-rg deopmn1 No Online

deopmn2 No Offline

opmn2-lh-rg deopmn2 No Online

deopmn1 No Offline

scalmnt-rg deopmn1 No Online

deopmn2 No Online

opmn1-rg deopmn1 No Unmanaged

opmn2-rg deopmn2 No Unmanaged

geo-clusterstate deopmn1 No Online

deopmn2 No Online

geo-infrastructure deopmn1 No Online

deopmn2 No Offline

root@deopmn1:~#

root@deopmn1:~# clrs status




------------- --------- ----- --------------

ebs-web deopmn1 Offline Offline


ebs-web-lsr deopmn1 Offline Offline


ebs-web-lh deopmn1 Offline Offline


ebs-opmn1-lh deopmn1 Online Online - LogicalHostname online.


ebs-opmn2-lh deopmn2 Online Online - LogicalHostname online.


ebs-storage deopmn1 Offline Online

deopmn2 Offline Online

ebs-opmn1 deopmn1 Offline Offline

ebs-opmn1-lsr deopmn1 Offline Offline

ebs-opmn2 deopmn2 Offline Offline

ebs-opmn2-lsr deopmn2 Offline Offline

geo-zc-sysevent deopmn1 Online_not_monitored Online_not_monitored


geo-servicetag deopmn1 Online_not_monitored Online_not_monitored


geo-failovercontrol deopmn1 Online Online - Service is online.


geo-hbmonitor deopmn1 Online Online - Daemon OK


geo-clustername deopmn1 Online Online - LogicalHostname online.


b. Check the status of the dr-cm-zc zone cluster.

root@decm1:~# clrg status



---------- --------- --------- ------


scalmnt-rg decm1 No Online

decm2 No Online

oep-rg decm1 No Online

decm2 No Online

cm-rg decm1 No Unmanaged

decm2 No Unmanaged

geo-clusterstate decm1 No Online

decm2 No Online

geo-infrastructure decm1 No Online

decm2 No Offline

cm-odg-pg-odg-rep-rg decm1 No Online

decm2 No Offline

EBS-oracle-external-proxy-shadow-rg decm1 No Unmanaged

decm2 No Unmanaged

zfssa-rep-rg decm1 No Online

decm2 No Offline

scalmnt2-rg decm1 No Unmanaged

decm2 No Unmanaged

root@decm1:~#

root@decm1:~# clrs status



------------- --------- ----- --------------

ebs-storage decm1 Online Online

decm2 Online Online

oep-rs decm1 Online Online - Service VIS

is UP [Instance is MOUNTED]

decm2 Online Online - Service VIS

is UP [Instance is MOUNTED]

geo-zc-sysevent decm1 Online_not_monitored


decm2 Online_not_monitored


geo-servicetag decm1 Online_not_monitored



decm2 Online_not_monitored


geo-failovercontrol decm1 Online Online - Service is

online.


geo-hbmonitor decm1 Online Online - Daemon OK


geo-clustername decm1 Online Online -

LogicalHostname online.


EBS-odg-rep-rs decm1 Online Online - Data

protection mode is MaxAvailability


zfssa-rep-rs decm1 Online Online - Receiving

update


ebs-cm decm1 Offline Offline


ebs-cm-lsr decm1 Offline Offline


ebs-cm-lh decm1 Offline Offline -

LogicalHostname offline.

decm2 Offline Offline -

LogicalHostname offline.

EBS-oracle-external-proxy-shadow-

rs decm1 Offline Offline


cm-logs decm1 Offline Offline


Configure Sites and Multigroups

A multigroup is a set of protection groups that is preconfigured for group migration between predetermined sets of clusters called sites. A switchover or takeover of all protection groups in a multigroup is performed in a single operation.


A protection group in a multigroup can be configured in a dependency chain, which defines which protection groups have a dependency on another protection group.

Likewise, a multigroup can contain multiple dependency chains. The dependency chain configuration determines the sequence protection groups must be taken offline and brought online. This sequence is based on the groups' roles in the dependency chain.

For example, an application that depends on a database needs to be stopped first before the database is shut down. In contrast, during restart operations after a switchover, the database takes precedence over the application.

Based on the Figure 1 and Figure 2, the two sites to be created are santaclara and hillsboro.


How to Create a Site

How to Create a Multigroup

5.6.1. Create and Configure the santaclara Site

Perform these steps on the primary cluster.

Create the site.

On this site, the zone cluster opmn-zc is the controller while cm-zc is a member.

root@eopmn1:~# geosite create -m cm-zc santaclara

Processing operation...might take several minutes to complete.

Site "santaclara" successfully created.

root@eopmn1:~#

root@ecm1:~# geosite join opmn-zc santaclara


Local cluster "cm-zc" successfully joined site "santaclara" on cluster "opmn-

zc".

root@ecm1:~#

Create the multigroup at the site.

root@eopmn1:~# geomg create -s santaclara ebs-mg


Multigroup "ebs-mg" successfully created.

root@eopmn1:~#

Add the protection groups to the multigroup.

This syntax means that later on, during multigroup switchover or takeover, opmn-ebs-pg and cm-ebs-pg can execute in parallel, but after the execution of cm-odg-pg. This arrangement makes sure that the database

http://docs.oracle.com/cd/E56676_01/html/E56740/create-site.html#scrolltoc

http://docs.oracle.com/cd/E56676_01/html/E56740/create-mg.html#scrolltoc


services are recovered before the application services.

root@eopmn1:~# geomg add-protection-group "(opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-

pg)/cm-zc:cm-odg-pg" ebs-mg

The following protection group list was successfully added to multigroup "ebs-

mg":

"(opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-zc:cm-odg-pg".

root@eopmn1:~#

Validate the multigroup.

root@eopmn1:~# geomg validate ebs-mg

Multigroup "ebs-mg" successfully validated.

root@eopmn1:~#

Display the multigroup’s status.

root@eopmn1:~# geomg status ebs-mg

Multigroup: ebs-mg

Description :

Site name : santaclara


Last update : Oct 16, 2015 6:15:50 PM PDT


Synchronization status with :

cm-zc : OK


Protection group list : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-

zc:cm-odg-pg

Status of protection group "opmn-zc:opmn-ebs-pg" :

Global status : OK

Role : Primary


Status of protection group "cm-zc:cm-ebs-pg" :

Global status : OK

Role : Primary


Status of protection group "cm-zc:cm-odg-pg" :

Global status : OK

Role : Primary



root@eopmn1:~#

5.6.2. Create and Configure the hillsboro Site

Perform these steps on the secondary cluster.

Create the site.

On this site, the zone cluster dr-opm-zc is the controller while the dr-cm-zc is a member.

root@deopmn1:~# geosite create -m dr-cm-zc hillsboro


Site "hillsboro" successfully created.

root@deopmn1:~#

root@decm1:~# geosite join dr-opmn-zc hillsboro


Local cluster "dr-cm-zc" successfully joined site "hillsboro" on cluster "dr-

opmn-zc".

root@decm1:~#

Create the multigroup at the site.

root@deopmn1:~# geomg create -s hillsboro ebs-mg


Multigroup "ebs-mg" successfully created.

root@deopmn1:~#

Add the protection groups to the multigroup.

root@deopmn1:~# geomg add-protection-group "(dr-opmn-zc:opmn-ebs-pg,dr-cm-

zc:cm-ebs-pg)/dr-cm-zc:cm-odg-pg" ebs-mg

The following protection group list was successfully added to multigroup

"ebs-mg":

"(dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-pg)/dr-cm-zc:cm-odg-pg".

root@deopmn1:~#

Create and Configure a DNS Server 78

Validate the multigroup.

root@deopmn1:~# geomg validate ebs-mg

Multigroup "ebs-mg" successfully validated.

root@deopmn1:~#

Display the multigroup’s status.

root@deopmn1:~# geomg status ebs-mg

Multigroup: ebs-mg

Description :

Site name : hillsboro

Creation signature : dr-opmn-zc Oct 16, 2015 5:19:33 PM MeST




dr-cm-zc : OK


Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-

pg)/dr-cm-zc:cm-odg-pg

Status of protection group "dr-cm-zc:cm-odg-pg" :

Global status : OK

Role : Secondary


Status of protection group "dr-cm-zc:cm-ebs-pg" :

Global status : OK

Role : Secondary


Status of protection group "dr-opmn-zc:opmn-ebs-pg" :

Global status : OK

Role : Secondary


root@deopmn1:~#

6. Create and Configure a DNS Server

This section provides steps in preparation for a switchover and takeover from the primary cluster to the secondary


cluster.

Note: This section is a simplified illustration of how you can remap network host names by using a dedicated DNS server that can be remotely updated. However, you must choose a method appropriate to the network administration of your own data center.

Before a switchover and takeover, the client address eweb-lh needs to be changed from the IP address that is used by the primary cluster to the IP address that is used by the secondary cluster. A script is provided that modifies the DNS server so that the client address eweb-lh refers to the correct address.

Whenever a role change occurs, modifying the DNS server is required to ensure that the primary cluster always refers to the correct IP address for eweb-lh. For example, if the primary cluster is running at the Santa Clara site, the DNS server resolves the client address eweb-lh to the IP address n.n.n.27. If the primary cluster is running at the Hillsboro site, the DNS server resolves eweb-lh to the IP address m.m.m.68.

In the procedures that follow, the examples use ebsdns as the name of the DNS server.

Configure the DNS Service

Install the DNS package.

root@ebsdns:~# pkg install service/network/dns/bind

Creating Plan (Solver setup): |

Creating Plan (Evaluating mediators): \

Verify the installed packages.

root@ebsdns:~# pkg list |grep -i bind

network/dns/bind 9.6.3.11.1-0.175.2.6.0.3.0 i--

service/network/dns/bind 9.6.3.11.1-0.175.2.6.0.3.0 i--

Create the /etc/named.conf file.

root@ebsdns:~# cat /etc/named.conf

options {

directory "/var/named";

dump-file "/var/named/dump/named_dump.db";

statistics-file "/var/named/named.stats";

};

zone "." in {

type hint ;

file "named.root" ;

};

zone "us.oracle.com" in {


type master ;

file "domain.us.oracle.com";

};

zone "in-addr.arpa" in {

type master ;

file "rdomain.us.oracle.com";

};

zone "0.0.127.in-addr.arpa" in {

type master ;

file "loopback";

};

logging {

category "default" { "debug"; };

category "general" { "debug"; };

category "database" { "debug"; };

category "security" { "debug"; };

category "config" { "debug"; };

category "resolver" { "debug"; };

category "xfer-in" { "debug"; };

category "xfer-out" { "debug"; };

category "notify" { "debug"; };

category "client" { "debug"; };

category "unmatched" { "debug"; };

category "network" { "debug"; };

category "update" { "debug"; };

category "queries" { "debug"; };

category "dispatch" { "debug"; };

category "dnssec" { "debug"; };

category "lame-servers" { "debug"; };

channel "debug" {

file "/tmp/nameddbg" versions 2 size 50m;

print-time yes;

print-category yes;

};

};

Create a forward lookup file for the DNS server.

In this file, the client address eweb-lh resolves to n.n.n.27.

root@ebsdns:~# cat /var/named/domain.us.oracle.com

; Forward map for us.oracle.com

$TTL 3h


@ in soa ebsdns.us.oracle.com. root.ebsdns.us.oracle.com. (

20151019

43200

3600

604800

86400 )

in ns ebsdns.oracle.com.

in mx 10 mail.us.oracle.com.

ebsdns in A n.n.n.20

localhost in A 127.0.0.1

mail in CNAME ebsdns

www in CNAME ebsdns

ftp in CNAME ebsdns

eweb-lh in A n.n.n.27

root@ebsdns:~#

Create a reverse lookup file for the DNS server.

root@ebsdns:~# cat /var/named/loopback

; Reverse map for 0.0.127.in-addr.arpa.

$TTL 1h

@ in soa ebsdns.oracle.com. root.ebsdns.oracle.com. (

20151020

43200

3600

604800

86400 )

in ns ebsdns.oracle.com.

1 in ptr localhost.oracle.com.

root@ebsdns:~#

Enable the DNS server.

root@ebsdns:~# svcadm enable dns/server

root@ebsdns:~# svcs dns/server

STATE STIME FMRI

online 1:09:58 svc:/network/dns/server:default

root@ebsdns:~#


Configure the DNS Update Scripts

Perform this configuration on each node in the zone clusters opmn-zc and dr-opmn-zc.

6.2.1. Create the dns_update.sh Script

On the primary cluster, the script remotely runs the dns_change_to_primaryip.sh script in the DNS server.

On the secondary cluster, the same script runs the dns_change_to_drip.sh script. Passwordless SSH has

already been configured for nodes of the opmn-zc and dr-opmn-zc clusters to access the DNS server.

On the primary cluster, create the scripts with execute permissions on the eopmn1 and eopmn2 nodes.

root@eopmn1:~# cat /var/geo/scripts/dns_update.sh

#!/bin/ksh

ssh ebsdns /var/geo/scripts/dns_change_to_primaryip.sh

root@eopmn1:~#

root@eopmn2:~# cat /var/geo/scripts/dns_update.sh

#!/bin/ksh

ssh ebsdns /var/geo/scripts/dns_change_to_primaryip.sh

root@eopmn2:~#

On the secondary cluster, create the scripts with execute permissions on the deopmn1 and deopmn2 nodes.

root@deopmn1:~# cat /var/geo/scripts/dns_update.sh

#!/bin/ksh

ssh ebsdns /var/geo/scripts/dns_change_to_drip.sh

root@deopmn1:~#

root@deopmn2:~# cat /var/geo/scripts/dns_update.sh

#!/bin/ksh

ssh ebsdns /var/geo/scripts/dns_change_to_drip.sh

root@deopmn2:~#

On the DNS server ebsdns, create two scripts with execute permissions as indicated.

a. Create the dns_change_to_drip.sh script.

root@ebsdns:~# cat /var/geo/scripts/dns_change_to_drip.sh

#!/bin/ksh

#

# This script sets eweb-lh hostname to m.m.m.68

#


DB_hosts=/var/named/domain.us.oracle.com

DB_ip2host=/var/named/rdomain.us.oracle.com

/usr/bin/ed ${DB_ip2host} << EOF

/^27.n.n.n/s/27.n.n.n/68.m.m.m/

w

q

EOF

/usr/bin/ed ${DB_hosts} << EOF

/^eweb-lh/s/n.n.n.27/m.m.m.68/

w

q

EOF

svcadm restart dns/server

root@ebsdns:~#

b. Create the dns_change_to_primaryip.sh script.

root@ebsdns:~# cat /var/geo/scripts/dns_change_to_primaryip.sh

#!/bin/ksh

#

# This script sets eweb-lh hostname to n.n.n.27

#

DB_hosts=/var/named/domain.us.oracle.com

DB_ip2host=/var/named/rdomain.us.oracle.com

/usr/bin/ed ${DB_ip2host} << EOF

/^68.m.m.m/s/68.m.m.m/27.n.n.n/

w

q

EOF

/usr/bin/ed ${DB_hosts} << EOF

/^eweb-lh/s/m.m.m.68/n.n.n.27/

w

q

EOF

svcadm restart dns/server

root@ebsdns:~#


6.2.2. Verify the DNS Update Scripts on the DNS Server

Run these commands.

root@ebsdns:~# cd /var/geo/scripts

root@ebsdns:/var/geo/scripts# ls -l

total 4

-rwxr-xr-x 1 root root 497 Oct 20 2015 dns_change_to_drip.sh

-rwxr-xr-x 1 root root 496 Oct 20 2015

dns_change_to_primaryip.sh

root@ebsdns:/var/geo/scripts#

root@ebsdns:/var/geo/scripts# ./dns_change_to_drip.sh

270

270

369

369


root@ebsdns:/var/geo/scripts# nslookup eweb-lh

Server: n.n.n.20

Address: n.n.n.20#53

Name: eweb-lh.us.oracle.com

Address: m.m.m.68

root@ebsdns:/var/geo/scripts# ./dns_change_to_primaryip.sh

270

270

369

369


root@ebsdns:/var/geo/scripts# nslookup eweb-lh

Server: n.n.n.20



Address: n.n.n.27


Configure the DNS Client

In the two zone clusters opmn-zc and dr-opmn-zc, their respective nodes have a constant mapping of eweb-lh to

the IP address in their /etc/hosts file. This mapping ensures that the eweb-lh within the cluster resolves to the

correct IP address for that cluster.


On each zone cluster node, add the DNS server to /etc/hosts.

The nodes are eopmn1, eopmn2, deopmn1, and deopmn2.

root@eopmn1:~# grep ebsdns /etc/hosts

n.n.n.20 ebsdns

root@eopmn1:~#

root@eopmn2:~# grep ebsdns /etc/hosts

n.n.n.20 ebsdns

root@eopmn2:~#

root@deopmn1:~# grep ebsdns /etc/hosts

n.n.n.20 ebsdns

root@deopmn1:~#

root@deopmn2:~# grep ebsdns /etc/hosts

n.n.n.20 ebsdns

root@deopmn1:~#

On each DNS client machine, configure the name server.

Note: In your own company configuration, multiple name servers might exist for redundancy.

root# svccfg -s network/dns/client setprop config/nameserver = net_address: n.n.n.20

root# svccfg -s network/dns/client setprop config/domain = astring: us.oracle.com

root# svccfg -s network/dns/client setprop config/search = astring: us.oracle.com

root# svccfg -s name-service/switch setprop config/ipnodes = astring: '"files dns"'

root# svccfg -s name-service/switch setprop config/host = astring: '"files dns"'

root# svccfg -s network/dns/client listprop config

config application

config/value_authorization astring solaris.smf.value.name-service.dns.client

config/nameserver net_address n.n.n.20

config/domain astring us.oracle.com

config/search astring us.oracle.com

root#

Enable the DNS client on each client machine.

root# svcadm enable dns/client

root# svcs dns/client

STATE STIME FMRI

online Nov_08 svc:/network/dns/client:default

root#

Load Testing with Oracle Application Testing Suite (OATS) 86

On each DNS client machine, verify a name server lookup for eweb-lh.

root# nslookup eweb-lh

Server: n.n.n.20



Address: n.n.n.27

root# nslookup n.n.n.27

Server: n.n.n.20


27.n.n.n.in-addr.arpa name = eweb-lh.us.oracle.com.

root#

Configure the Protection Group Role Change Script

Add the dns_update.sh script to the RoleChange_ActionCmd property for the opmn-ebs-pg

protection group.

The system sets this property on the local cluster, and then propagates the value to the partner cluster.

root@eopmn1:~# geopg set-prop -p RoleChange_ActionCmd=/var/geo/scripts/dns_update.sh opmn-ebs-pg

Protection group "opmn-ebs-pg" successfully modified.

root@eopmn1:~#

7. Load Testing with Oracle Application Testing Suite (OATS)

This section describes a test performed using the disaster recovery configuration implemented in this documentation.

In this testing scenario, 600 virtual users perform online transactions using Oracle E-Business Suite 12.1.3. The test and subsequent graphs show the time disruption users experience during an outage and through the disaster recovery implementation, the recovery speed for all 600 virtual users to resume normal operations. There is a separate SQL script to submit batch jobs to the concurrent managers.

The Test Configuration

The test configuration is shown in the following figure:


Figure 3 - Primary and Standby Sites Configuration with Oracle Application Testing Suites

The test configuration consists of a Windows server that is using the Oracle Application Testing Suite (OATS) as a load generator. In its DNS configuration, the Windows server is set up to use the DNS Server ebsdns.

In a disaster recovery operation, the Oracle Solaris Cluster Geo Edition software modifies the eweb-lh entry on the DNS Server ebsdns. After that modification, the Oracle E-Business Suite 12.1.3 virtual users are directed to the correct SuperCluster where Oracle E-Business Suite 12.1.3 services are running.


OLTP Load Generator: the OATS Machine

Figure 4 shows that OATS is configured with 4 OATS scripts. Each script simulates 150 users across 4 different types of transactions for a total of 600 virtual users.

Figure 4 - Load Testing Configuration

The following list shows the parameters of the scripts:

Create_Query_Cash_Expenses 150 Virtual Users and OLT Server

Create_Query_Credit_Expenses 150 Virtual Users and OLT Server

TC_Entry 150 Virtual Users and OLT Server

View_Payslip 150 Virtual Users and OLT Server


OATS Testing Information

The test was run for approximately 15 minutes to measure the performance of Oracle E-Business Suite 12.1.3 in handling the workload of sustained usage by 600 virtual users.

Figure 5 shows the efficiency with which Oracle E-Business Suite 12.1.3 performed in supporting the transactions of 600 virtual users before and after disaster recovery.

Figure 5- OATS Test Performance Graph

Switchover ebs-mg Multigroup to the Secondary Cluster 90

Concurrent Manager Reports

In the test, an additional task was added to run concurrent to the constant OLTP transactions of the 600. The task simulated a DBA user logged to the SuperCluster DB node and repeatedly running a script that submits an Order-to-Cash job.

Figure 6 - Concurrent Manager Reports

Figure 6 shows the Oracle E-Business Suite System Administration screen for completed jobs. The screen displays the report name, status and links to the report details and output.

The Output column links to access the output files. In the test, part of the recovery verification was actual accessing of the output links. Overall, the entire test demonstrates the continuous replication feature of the ZFS SA for the Concurrent Manager output files based on the configuration prescribed in the architecture.

8. Switchover ebs-mg Multigroup to the Secondary Cluster

This section describes procedures for performing a switchover of the ebs-mg multigroup to the standby. The section first describes the steps for the switchover. Then timing information regarding the switchover operation is provided.


Switchover Operation

Check the status first before performing the switchover.


a. Check the status of the ebs-mg multigroup.

The command output shows that the santaclara site is activated and has the primary role for the three protection groups. Consequently, the hillsboro site has the secondary role for the protection groups.


Multigroup: ebs-mg

Description :






cm-zc : OK



zc:cm-odg-pg


Global status : OK

Role : Primary



Global status : OK

Role : Primary



Global status : OK

Role : Primary


root@eopmn1:~#


b. Check the status of the standby site.

root@deopmn1:~# geosite show -v hillsboro

Site: hillsboro

Description :




Query_interval : 120 seconds

Controller "dr-opmn-zc"

Configuration : OK

Member "dr-cm-zc"

Configuration : OK

Local heartbeat : hb_dr-opmn-zc~hillsboro~dr-cm-zc

Multigroup: ebs-mg

Protection group list : (dr-opmn-zc:opmn-ebs-pg,dr-cm-zc:cm-ebs-pg)/dr-cm-

zc:cm-odg-pg

The command output shows that for this site, the dr-opmn-zc is the

controller.

On the secondary cluster, perform the following steps:

a. Check the status of the ebs-mg multigroup.

This step shows that prior to the switchover, the ebs-mg multigroup on the standby site has the secondary role.


Multigroup: ebs-mg

Description :






dr-cm-zc : OK






Global status : OK

Role : Secondary



Global status : OK

Role : Secondary



Global status : OK

Role : Secondary


root@deopmn1:~#

b. Switch the ebs-mg multigroup over to the standby site.

root@deopmn1:~# time geomg switchover -s hillsboro ebs-mg

Are you sure you want to switch over multigroup ‘ebs-mg’ to site ‘hillsboro’?

(yes|no) > yes

Processing operation... The timeout period for this multigroup operation is

10920 seconds...

Beginning "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg"

Beginning "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg"

Beginning "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg"

"Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" completed

successfully

"Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed

successfully

"Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed

successfully

"Switchover" operation succeeded for the multigroup "ebs-mg".

real 8m12.937s

user 0m3.078s

sys 0m2.151s

root@deopmn1:~#


c. Check the status of the multigroup.

This step shows that after the switchover, the protection groups in the standby site assume the primary role.


Multigroup: ebs-mg

Description :






dr-cm-zc : OK





Global status : OK

Role : Primary



Global status : OK

Role : Primary



Global status : OK

Role : Primary


root@deopmn1:~#


Switchover Timing Information

The following figure shows timing information about the switchover operation.

Figure 7 - Oracle SuperCluster - E-Business Suite Disaster Recovery Orchestration demo – Services Availability Timing

Figure 7 provides the following data:

Site switchover time <10 minutes including validations at the start and end of the operation.

Actual service interruption ~6 minutes.

Takeover ebs-mg Multigroup from the Secondary Cluster 96

9. Takeover ebs-mg Multigroup from the Secondary Cluster

This section describes procedures for performing a takeover of the ebs-mg multigroup by the standby site. The section first describes the steps for the takeover. Then timing information regarding the takeover operation is provided.

Takeover Operation

On the primary (santaclara) site, issue a stop command at each SuperCluster compute node’s service processor.

The command causes an outage by powering off all the site’s nodes.

This step simulates an unplanned outage at the site.

Stop /SYS –f -script

On the standby site, issue the command to execute a takeover.

root@deopmn1:~# time geomg takeover -f -s hillsboro ebs-mg


10920 seconds...

Beginning "Takeover" operation on protection group "dr-cm-zc:cm-odg-pg"

Beginning "Takeover" operation on protection group "dr-opmn-zc:opmn-ebs-pg"

Beginning "Takeover" operation on protection group "dr-cm-zc:cm-ebs-pg"

"Takeover" operation on protection group "dr-cm-zc:cm-odg-pg" completed

successfully

"Takeover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed

successfully

"Takeover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed

successfully

"Takeover" operation succeeded for the multigroup "ebs-mg".

real 10m19.878s

user 0m3.759s

sys 0m2.756s

root@deopmn1:~#











Connected.

DGMGRL>

Check and verify that the standby database has become primary.




Databases:


Warning: ORA-16629: database reports a different protection level from

the protection mode

VIS - Physical standby database (disabled)

ORA-16661: the standby database needs to be reinstated



WARNING

DGMGRL>

Check the status of the multigroup.

geomg status ebs-mg


Takeover Timing Information

The following figure shows timing information about the takeover operation.

Figure 8 - Oracle SuperCluster – E-Business Suite Disaster Recovery Orchestration demo – Services Availability Timing

Figure 8 provides the following data:

Site takeover time ~10 minutes including validations at the start and end of the operation.

Actual service interruption ~15 minutes, includes ~5 minutes of decision making time.

Restore the Primary Site 99

10. Restore the Primary Site

This section provides instructions for the restoration of the failed primary site that has been fixed.

Issue the following command.

The command powers up the server, and as a consequence, also boots up all the domains. However the software detects that the site has been taken over by the standby site. Thus, application and database services will remain offline.

Make sure to issue the command at each of the SuperCluster compute node’s service processor.

Start /SYS –f -script

When the primary site is ready for use again, reinstate its database as an active standby database.

a. Access Oracle Data Guard broker.

root@etc11-cn01:~# su – oracle



oracle@etc11-cn01:~$ . VIS.env

oracle@etc11-cn01:~$ dgmgrl sys/<password>




Connected.

DGMGRL>

b. Reinstate the database.

DGMGRL> reinstate database 'VIS'

Reinstating database "VIS", please wait...

Operation requires shutdown of instance "VIS1" on database "VIS"


ORA-01109: database not open

Database dismounted.


Operation requires startup of instance "VIS1" on database "VIS"





Database mounted.

Continuing to reinstate database "VIS" ...

Reinstatement of database "VIS" succeeded




Databases:


VIS - Physical standby database



SUCCESS

DGMGRL>

Display the status of the multigroup at santaclara.

In the following example, the global status shows that the ebs-mg multigroup is in an error state. This state is expected after the former primary cluster of the protection groups is up and running after a takeover by a secondary cluster. The next three steps in this procedure remedy this error state.


Multigroup: ebs-mg

Description :






cm-zc : OK


Protection group list : (opmn-zc:opmn-ebs-pg,cm-zc:cm-ebs-pg)/cm-zc:cm-odg-pg


Global status : Error

Role : Primary

Activation state : Deactivated



Role : Primary





Role : Primary


Update and start the protection group opmn-ebs-pg.

root@eopmn1:~# geopg update opmn-ebs-pg

WARNING:This operation overwrites the existing local configuration

for protection group "opmn-ebs-pg" with the configuration of the

remote cluster.

Check the configuration for "opmn-ebs-pg" on the

remote cluster before proceeding.

Do you want to continue (yes|no) > yes

Protection group "opmn-ebs-pg" successfully updated.

root@eopmn1:~#

root@eopmn1:~# geopg start -e global opmn-ebs-pg


is 3600 seconds...

Protection group "opmn-ebs-pg" successfully started.






Role : Secondary


Configuration : OK




Role : Primary


Configuration : OK



root@eopmn1:~#

Update and start the protection group cm-odg-pg.

root@ecm2:~# geopg update cm-odg-pg



for protection group "cm-odg-pg" with the configuration of the remote

cluster.

Check the configuration for "cm-odg-pg" on the



Protection group "cm-odg-pg" successfully updated.

root@ecm2:~#

root@ecm2:~# geopg start -e global cm-odg-pg


is 3600 seconds...

Protection group "cm-odg-pg" successfully started.

root@ecm2:~# geopg status cm-odg-pg

Protection group "cm-odg-pg" : OK

Partnership : cm-ps


Cluster cm-zc : OK

Role : Secondary


Configuration : OK




Role : Primary


Configuration : OK



root@ecm2:~#

Update and start the protection group cm-ebs-pg.

The cm-ebs-pg group involves the ZFS SA replication. This step performs a cleanup on the ZFS SA to destroy the ebs-app-cmlogs share.

a. On the standby site’s ZFS SA, issue the commands as shown.

e tc18sn1:shares ebs-apps-cmlogs replication> ls

Actions:

TARGET STATUS NEXT

action-000 etc11-vnic1 idle Sync now

etc18sn1:shares ebs-apps-cmlogs replication> select action-000


etc18sn1:shares ebs-apps-cmlogs action-000> sendupdate

etc18sn1:shares ebs-apps-cmlogs action-000> set continuous=true

continuous = true (uncommitted)

etc18sn1:shares ebs-apps-cmlogs action-000> set enabled=true

enabled = true (uncommitted)

etc18sn1:shares ebs-apps-cmlogs action-000> commit

etc18sn1:shares ebs-apps-cmlogs action-000> done

etc18sn1:shares ebs-apps-cmlogs replication> ls

Actions:

TARGET STATUS NEXT

action-000 etc11-vnic1 sending Continuous

b. On the primary site’s ZFS SA, issue the commands as shown.

etc11-sn1:shares> ls

Properties:

pool = ssc

Projects:

QuorumProject

default

ebs-apps

ebs-apps-cmlogs

ebs-appzones

ebs-backup

ebs-db

iodomrpool

osc-backup

ssc

Children:

replication => Manage remote replication

schema => Define custom property schema

etc11-sn1:shares> destroy ebs-apps-cmlogs

This will destroy all data in "ebs-apps-cmlogs"! Are you sure? (Y/N) Y

etc11-sn1:shares> ls

Properties:

pool = ssc

Projects:

QuorumProject

default

ebs-apps

ebs-appzones

ebs-backup


ebs-db

iodomrpool

osc-backup

ssc

Children:

replication => Manage remote replication

schema => Define custom property schema

etc11-sn1:shares>

c. Update and start the cm-ebs-pg group.

root@ecm2:~# geopg update cm-ebs-pg


for protection group "cm-ebs-pg" with the configuration of the remote

cluster.

Check the configuration for "cm-ebs-pg" on the



Protection group "cm-ebs-pg" successfully updated.

root@ecm2:~#

root@ecm2:~# geopg start -e global cm-ebs-pg


is 3600 seconds...

Protection group "cm-ebs-pg" successfully started.

root@ecm2:~# geopg status cm-ebs-pg

Protection group "cm-ebs-pg" : OK

Partnership : cm-ps


Cluster cm-zc : OK

Role : Secondary


Configuration : OK




Role : Primary


Configuration : OK



root@ecm2:~#


Verify the status on both clusters to ensure that normal operations have resumed.

a. Check the status of the primary site.

root@eopmn1:~# geomg status

Multigroup: ebs-mg

Description :






cm-zc : OK



zc:cm-odg-pg


Global status : OK

Role : Secondary



Global status : OK

Role : Secondary



Global status : OK

Role : Secondary


root@eopmn1:~#

b. Check the status of the standby site.

root@deopmn1:~# geomg status

Multigroup: ebs-mg

Description :



Apply SuperCluster Quarterly Maintenance Updates 106




dr-cm-zc : OK





Global status : OK

Role : Primary



Global status : OK

Role : Primary



Global status : OK

Role : Primary


root@deopmn1:~#

(Optional): If the status of both sites indicates OK, switch over the ebs-mg multigroup to the original primary site.

The sample output is intentionally truncated.

root@deopmn1:~# time geomg switchover -s santaclara ebs-mg

Are you sure you want to switch over multigroup ‘ebs-mg’ to site

‘santaclara’? (yes|no) > yes


10920 seconds...

11. Apply SuperCluster Quarterly Maintenance Updates

Periodically the SuperCluster needs to be patched with a Quarterly Maintenance Update (QMU). Every quarter, Oracle provides a SuperCluster Quarterly Full Stack Download Patches (QFSDP).

QFSDP is the only mechanism that delivers the proactive patching policy for all SuperClusters. Systems should not be kept out of date for more than 6 months (2 QFSDPs). Oracle makes no guarantees that SuperCluster patches older than 6 months will be maintained.


The QFSDP should be applied as a whole unit in a single operation. Partial or selective patching is not

recommended. Reactive patching policy does allow individual patches to be applied to individual components to address critical issues with no reasonable workaround. However, before applying patches selectively, seek advice from Oracle Support first. Refer to the README files within the QFSDP for further details.

For example, for the project that is the basis of this document’s implementation, the Jan 2016 QFSDP was applied to upgrade the OS release from Solaris 11.2 to Solaris 11.3 and Oracle Solaris Cluster from 4.2 to 4.3.

Refer to the following MOS notes:

Contents of each SuperCluster Quarterly Full Stack Download Patch (QFSDP) (Doc ID 2056975.1)

SuperCluster QFSDP Installation Checklists (Doc ID 2090692.1)

The following is an extract from the QFSDP Installation checklist for T4-4 SuperCluster, January 2016 that is referenced from Doc ID 2090692.1.

The intent of this document is to provide a high-level overview of the Quarterly Full Stack Download Patch (QFSDP) installation. This document does not replace the requirement to read & follow the instructions in the QFSDP README files and the QFSDP SuperCluster Patching Best Practices document (Oracle Support Document 1569461.1), this document is meant as a navigation help and for tracking progress during the update process.

When planning a QFSDP, follow this recommended sequence.

Update the standby site first.

For example, in this document, the patch update is applied to the hillsboro site first.

After the update of the standby site, perform a planned switchover of the multigroup to the standby site.

For example, in this project, the ebs-mg multigroup is switched over to the hillsboro site as follows:

root@deopmn1:~# geomg switchover -s hillsboro ebs-mg

Are you sure you want to switch over multigroup ‘ebs-mg’ to site ‘hillsboro’?

(yes|no) > yes


10920 seconds...

Beginning "Switchover" operation on protection group "dr-cm-zc:cm-odg-pg"

Beginning "Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg"

Beginning "Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg"

"Switchover" operation on protection group "dr-cm-zc:cm-odg-pg" completed

successfully

"Switchover" operation on protection group "dr-opmn-zc:opmn-ebs-pg" completed

successfully

"Switchover" operation on protection group "dr-cm-zc:cm-ebs-pg" completed

successfully

"Switchover" operation succeeded for the multigroup "ebs-mg".

root@deopmn1:~#


After the standby site takes over the services, apply the same patch update to the primary site.

As an option, you can perform a load test during the switchover between the sites to monitor any performance changes after the QFSDP application.

The following figure shows the load testing graph when OATS was used to perform load testing after the standby site has been updated. No performance degradation observed.

Figure 9 – E-Business Suite Disaster Recovery Orchestration demo – Services Availability Timing

ORACLE CORPORATION

Worldwide Headquarters

500 Oracle Parkway, Redwood Shores, CA 94065 USA

Worldwide Inquiries

TELE + 1.650.506.7000 + 1.800.ORACLE1 FAX + 1.650.506.7200 oracle.com

CONNECT WITH US

Call +1.800.ORACLE1 or visit oracle.com. Outside North America, find your local office at oracle.com/contact.

blogs.oracle.com/oracle facebook.com/oracle twitter.com/oracle

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other

warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability

or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual

obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any

form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are

registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license

and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron

logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

0818

Oracle E-Business Suite 12.1.3 Using Oracle Solaris Cluster Geo Edition – Implementation Guide

August 2018

Author: Neil Garthwaite, Gia-Khanh Nguyen

https://www.oracle.com/

http://www.oracle.com/contact

oracle optimized solution for disaster recovery · oracle optimized solution for disaster recovery...

Documents