1 copyright © 2013, oracle and/or its affiliates. all rights reserved. … · red hat enterprise...

Cloning of Oracle Homes on Oracle RAC 11.2 Cluster Nodes after Linux Upgrade

Andreas Becker,

Principal Member Technical Staff

Oracle Server Technologies, SAP Dev.


Program Agenda

The Project

The Plan

The Big Picture

DelNode/AddNode vs. Oracle Home Cloning

Summary


The following is intended to outline our general product direction. It is intended

for information purposes only, and may not be incorporated into any contract.

It is not a commitment to deliver any material, code, or functionality, and should

not be relied upon in making purchasing decisions. The development, release,

and timing of any features or functionality described for Oracle’s products

remains at the sole discretion of Oracle.


The Project


Project Description

SAP System on Oracle RAC

Oracle RAC Cluster with 4 nodes

Oracle Release 11.2.0.3

Oracle Linux Release 6.1 on SAP Application Server

Oracle Linux Release 5.8 on Oracle RAC nodes

ASM + ACFS

Target: Oracle Linux Release 6.1 on all Oracle RAC nodes

Current Situation and Project Target


Project Description SAP System + Oracle RAC System Overview

SAP

Oracle

RAC

Database

RAC Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

RAC Instances Services

SAP User

OL6.1 Node #1 Node #2 Node #3 Node #4

SAP Instances

Database

SAP Instance 2

OL6.1

SAP Instance 1

OL6.1

SAP Instance 3

OL6.1

SAP Instance 4

OL6.1

OL5.8

ASM / ACFS


Linux OS Upgrade Oracle Linux

In-place upgrading from a major version of Oracle

Linux 5 or earlier is not supported. Although

Anaconda provides an option to perform an upgrade,

fresh installation is strongly recommended.

Oracle® Linux

Release Notes for Release 6.4

http://docs.oracle.com/cd/E37670_01/E39522/html/ol

_sup_64rn.html

http://docs.oracle.com/cd/E37670_01/E39522/html/ol_sup_64rn.html




Linux OS Upgrade Red Hat Linux

Red Hat does not support in-place upgrades between

any major versions of Red Hat Enterprise Linux.

Red Hat Enterprise Linux 6 Installation Guide

https://access.redhat.com/site/documentation/en-

US/Red_Hat_Enterprise_Linux/6/html/Installation_Guid

e/ch-upgrade-x86.html

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/ch-upgrade-x86.html












Project Description

Technical restrictions

– Linux upgrade requires fresh installation of all RAC nodes

SAP System HA requirements

– SAP application must remain available, no longer downtimes

– Downtimes due to SAP application server restarts are acceptable

– 2 of the 4 RAC nodes are sufficient run the SAP system

Restrictions and Requirements


Project Description

Project goal: Linux OS Upgrade

– No other requirements or changes

– No hardware replacement

– No host name change

– No network configuration change, no IP change

Restrictions and Requirements


The Plan


The Plan

2 RAC nodes are sufficient to run the SAP load (for a while)

2-phases

– Phase 1: „OS upgrade“ of RAC nodes 1 and 2

– Phase 2: „OS upgrade“ of RAC nodes 3 and 4

2-Phase-Approach


The Plan

Between Phase 1 and Phase 2 the system would run with different OS

versions (for a while)

MOS 220970.1 For Oracle Database 11g Release 2, Oracle Clusterware and ASM binaries are combined into a single

ORACLE_HOME called the grid infrastructure home. This home fully supports rolling upgrades for patches, bundles,

patchsets and releases. (If you are upgrading ASM from Oracle Database 10g to 11g Release 2, you will not be able

to upgrade ASM in a rolling fashion.)

The Oracle Clusterware and Oracle Real Application Clusters both support rolling upgrades of the OS software

when the version of the Oracle Database is certified on both releases of the OS (and the OS is the same, no Linux

and Windows or AIX and Solaris, or 32 and 64 bit etc.). This can apply a patch to the operating system, a patchset

(such as EL4u4 to EL4u6) or a release (EL4 to EL5).

2-Phase-Approach

https://support.oracle.com/epmos/faces/DocumentDisplay?id=220970.1


The Plan

Oracle® Database SAP® Bundle Patch 11.2.0.3.7 - 201309 for Grid Infrastructure

If you restrict installation of GI-PSU 11.2.0.3.7 and this SBP to the grid homes and the non-SAP homes as described above, you can also keep

the SAP databases on the cluster and the SAP applications running on these available. However, due to limitations in Oracle's connection

handling such a RAC-rolling grid home SBP installation is not transparent to the SAP applications connected against the databases of the

cluster. To avoid transaction or other errors in the SAP applications when installing this SBP in a rolling manner, follow the additional

instructions in section 2.4, "Maintaining SAP Applications during a RAC-Rolling Patch Installation".

…To avoid transaction or other errors in the SAP applications when installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this

SBP in a rolling manner, you must follow the installation instructions below on each of the cluster nodes in turn:

All connections from SAP applications to SAP database instances running on the cluster node must be terminated by the SAP application

before installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP on the cluster node.

Any SAP application component (for example, the Enqueue Server) which is dependent on the Oracle Clusterware on a cluster node must be

stopped by the SAP application before installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP on the cluster node.

…

RAC-Rolling Grid Home Patching


The Plan

…To avoid transaction or other errors in the SAP applications…

1. All connections from SAP applications to SAP database instances running on

the cluster node must be terminated by the SAP application. …

2. Any SAP application component (for example, the Enqueue Server) which is

dependent on the Oracle Clusterware on a cluster node must be stopped by

the SAP application…

RAC-Rolling Grid Home Patching


The Big Picture


The Big Picture Target

SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

SAP User

OL6.1 OL6.1 OL6.1 OL6.1 OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

SAP Instance 2

OL6.1

SAP Instance 1

OL6.1

SAP Instance 3

OL6.1

SAP Instance 4

OL6.1


The Big Picture Starting Point

SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

SAP User

OL6.1

OCR

Voting


SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4


The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

1. Block SAP instances 1 and 2 for logon and

disable batch jobs

2. Inform SAP user to reconnect

3. Inform SAP user about SAP instance restart

4. Relocate database services

5. Restart SAP instance(s) 1 and 2

6. Allow logon on SAP instances 1 and 2

and enable batch jobs again

OL6.1

OCR

Voting





SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services


disable batch jobs







OL6.1

OCR

Voting





SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services


disable batch jobs







OL6.1

OCR

Voting


S1 S2

$ srvctl relocate service… $ srvctl relocate service…




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services


disable batch jobs







OL6.1

OCR

Voting


S1 S2 S3 S4




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services


disable batch jobs







OL6.1

OCR

Voting





SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ERS

Services

1. Relocate SAP ERS instance (sapctl)

2. Relocate SAP ASCS instance (sapctl)

OL6.1

OCR

Voting


ASCS ERS




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services



OL6.1

OCR

Voting


ASCS ASCS ERS




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services



OL6.1

OCR

Voting


ASCS ERS ERS




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

SAP preparations are completed.

OL6.1

OCR

Voting


ERS ASCS




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Stop RAC instances on nodes 1 and 2

$ srvctl stop instance –i <instance> -d

<db>

2. Stop CRS on nodes 1 and 2 (root)

# crsctl stop crs

OL6.1

OCR

Voting


ERS ASCS



The Big Picture Preparations completed

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

Preparations for phase 1 complete.

OL6.1

OCR

Voting


ERS ASCS



Program Agenda

The Project

The Plan

The Big Picture


Summary


The Oracle Home Cloning Approach


The Alternatives

1. Backup Grid home, database home(s), oratab, host configuration files,

user home directories, user configuration, /etc/hosts, /etc/passwd, …

2. Stop database instances and CRS

3. Install the new Linux version and configure the node exactly as before

hostname, IP, accounts, user homes, …).

4. Ensure that installation prerequisites are met

5. Restore backup of Grid home and database home(s)

6. Clone Grid home and database home(s)

7. Start CRS and database instances

Oracle home Cloning Approach


The Big Picture Preparations completed

SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

Preparations for phase 1 complete.

OL6.1

OCR

Voting


ERS ASCS

Node 1

Node 2

Node 3

Node 4



The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Backup Grid home and database home on

each node.

2. Backup /etc/oraInst.loc, /etc/oratab

3. Backup OS configuration files like

/etc/hosts, /etc/groups, /etc/passwd

4. ASM: mapping of ASM disks

OL6.1

OCR

Voting


ERS ASCS

Node 1

Node 2

Node 3

Node 4

gi_home_node1.tar gi_home_node2.tar

db_home_node1.tar db_home_node2.tar

$ tar cvf … $ tar cvf …




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Install new OS version

OL6.1

OCR

Voting


ERS ASCS

Node 1

Node 2

Node 3

Node 4

OL6.1 OL6.1

Install Linux OS




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Restore & Clone Grid Home

2. Restore & Clone Database Home

OL6.1

OCR

Voting


ERS ASCS

Node 1

Node 2

Node 3

Node 4

OL6.1 OL6.1

gi_home_node1.tar gi_home_node2.tar

db_home_node1.tar db_home_node2.tar

$ tar xvf …

$ clone.pl $ tar xvf …

$ clone.pl



The Alternatives

Cloning the Grid home $ su root -c "$OHGRID/crs/install/rootcrs.pl -unlock“

$ /usr/bin/perl $OHGRID/clone/bin/clone.pl ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$OHGRID -defaultHomeName

INVENTORY_LOCATION=/oracle/oraInventory -O'"CLUSTER_NODES={<node list>}"' -O'"LOCAL_NODE=<node>"' CRS=false -

O"SHOW_ROOTSH_CONFIRMATION=false“

su root -c "$OHGRID/rdbms/install/rootadd_rdbms.sh"

su root -c "$OHGRID/crs/install/rootcrs.pl -patch"

su root -c "chmod 755 /opt/oracle“

Cloning the Database home $ $ORACLE_HOME/clone/bin/clone.pl ORACLE_HOME=$ORACLE_HOME ORACLE_BASE=$ORACLE_BASE

-defaultHomeName

$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={<comma

separated list of cluster nodes>}"

# $ORACLE_HOME/root.sh

For details, see My Oracle Support note MOS 1136544.1.

Oracle Home Cloning Approach



SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 OL5.8 Node #1 Node #2

OL6.1 OL6.1

1. Start CRS on new nodes

2. Start RAC instances on new nodes

Next: Rebalance


Node 1

Node 2

Node 3

Node 4




SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1

Node #2 OL6.1 OL6.1

1. Rebalance to complete phase 1

Phase 1 complete

Next steps:

1. Start phase 2

2. Rebalance to complete phase 2


Node 1

Node 2

Node 3

Node 4

S1 S2

ASCS ERS

Zero OCR

updates




SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1

Node #2 OL6.1 OL6.1


Node 1

Node 2

Node 3

Node 4

S1 S2

ASCS ERS

Zero OCR

updates

Phase 1 completed.

Phase 2 completed.

Oracle RAC is running OL 6.1.

OL6.1 OL6.1



The DelNode/AddNode Approach


The Alternatives

1. Remove a RAC node from the cluster

2. Install the new Linux version

3. Add the node to the cluster again.

Repeat steps 1-3 for all nodes that need to be installed. When finished,

backup OCR.

DelNode: <GRIDHOME>/bin/crsctl delete node -n <nodename>

AddNode: <GRIDHOME>/oui/bin/addNode.sh

DelNode/AddNode Approach


The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Remove nodes


3. Add nodes

OL6.1

OCR

Voting


ERS ASCS

Node 1

Node 2

Node 3

Node 4




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8 OL5.8

Services

1. Remove nodes


3. Add nodes

OL6.1

OCR

Voting

OL5.8 Node #1

Node #2 Node #3 Node #4

ERS ASCS

Node 2

Node 3

Node 4

$ crsctl delete node node 1




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes


3. Add nodes

OL6.1

OCR

Voting

OL5.8 Node #1

OL5.8 Node #2

Node #3 Node #4

ERS ASCS

Node 3

Node 4

$ crsctl delete node node 2

2 OCR

updates




SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes


3. Add nodes

OL6.1

OCR

Voting

OL5.8 Node #1n

OL5.8 Node #2n

Node #3 Node #4

ERS ASCS

Node 3

Node 4

OL6.1

OL6.1


2 OCR

updates



SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes from RAC cluster


3. Add nodes to RAC cluster again

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 Node #1n OL6.1

Node 1n

Node 3

Node 4

OL5.8 Node #2n OL6.1

$ addNode.sh


3 OCR

updates



SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes from RAC cluster


3. Add nodes to RAC cluster again

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 OL5.8 Node #1n Node #2n

OL6.1 OL6.1

Node 1n

Node 2n

Node 3

Node 4

$ addNode.sh


4 OCR

updates



SAP

Oracle

RAC

Database

RAC

Nodes


S1 S2 S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 OL5.8 Node #1n Node #2n

OL6.1 OL6.1

1. Start CRS on new nodes

2. Start RAC instances on new nodes


Node 1n

Node 2n

Node 3

Node 4




SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1n

Node #2n OL6.1 OL6.1

1. Rebalance again: move SAP users, SAP

Database Services, and SAP instances

back to RAC instances on nodes 1 and 2


Node 1n

Node 2n

Node 3

Node 4

S1 S2

ASCS ERS




SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1n

Node #2n OL6.1 OL6.1


Node 1n

Node 2n

Node 3

Node 4

S1 S2

ASCS ERS

Phase 1 completed.

Phase 2: next to start

Oracle RAC is running OL 6.1 and OL 5.8

(mix).

4 OCR

updates




SAP

Oracle

RAC

Database

RAC

Nodes


OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

OL6.1 OL6.1 OL6.1 OL6.1 OL6.1 Node #1n

Node #2n Node #3n Node #4n

Phase 1 completed.

Phase 2 completed.

Oracle RAC is running OL 6.1.

OCR

Voting

Node 1n

Node 2n

Node 3n

Node 4n

8 OCR

updates



Program Agenda

The Project

The Plan

The Big Picture


Summary


Summary


Summary

DelNode/AddNode Oracle Home Cloning

• Nodes are removed from cluster and added to

cluster

• Every ‚node add‘/‘node remove‘

operation changes the OCR and affects

the remaining cluster

• N-node-RAC 2xN OCR configuration

changes

• Zero Addnode/DelNode operations

• OCR configuration before/after is not identical

• OCR backup required when finished

• OCR/Cluster configuration remains

unchanged

Comparison DelNode/AddNode vs. Oracle Home Cloning Approach


Summary

DelNode/AddNode Oracle Home Cloning

• General approach • More specific approach

Suitable for

• Cluster extensions

• Cluster configuration changes (host names,

hardware, IP address)

• Preparation of new nodes while system is

online

• Hardware exchange

Suitable for

• OS upgrade / OS fresh installations

• Rolling GI Home Patching

Comparison DelNode/AddNode vs. Cloning Approach


1696869 - Patching of Oracle Homes with Minimal Downtime

http://service.sap.com/sap/support/notes/1696869

Rolling Patching

Oracle® Database SAP® Bundle Patch

11.2.0.3.7 - 201309 for Grid Infrastructure

README

References

SAP Notes

http://service.sap.com/sap/support/notes/1696869


Oracle Support Document 1136544.1 (Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1136544.1

Oracle Support Document 1262925.1 (How to remove/delete a node from Grid Infrastructure Clusterware when the node has failed) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1262925.1

Oracle Support Document 1332451.1 (How to Add Node/Instance or Remove Node/Instance in 10gR2, 11gR1 and 11gR2 Oracle Clusterware and RAC) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1332451.1

Oracle Support Document 220970.1 (RAC: Frequently Asked Questions) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=220970.1

References

MOS Notes







Red Hat Enterprise Linux 6 Product Documentation

https://access.redhat.com/site/documentation/en-

US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/ch-upgrade-x86.html

Oracle Linux Release Notes for Release 6.4

Supported Upgrade Path


References

Other














Questions?

1 copyright © 2013, oracle and/or its affiliates. all rights reserved. … · red hat enterprise...

Documents