PRE-CHECKSSr. No. Description User

1 Check all the Cluster Services daemon oracle

2 root

3 root

4 Run dbupgdiag.sql and save the output for reference oracle5 Shutdown the database oracle

6 Shutdown the cluster root

7 Take backup of Grid and Oracle homeoracle

root

8 Take backup of oraInventory oracle

9 Start crs root10 Start cluster root

11 oracle

12 oracle

13 Unzip 11.2.0.4 patchset - 3rd zip file is for grid. oracle

14root

oracle

15

16 Verify cluster components oracle

17 Remove timeouts configured for Putty sessions

18 Cluster freeze and verification steps

Check configuration of Voting Disk and Oracle Cluster Registry

Take backup of OCR and voting disks (no need to take backup of voting disk manually with dd command as oracle automatically takes every 4 hours)

Check all cluster resources , it should show DB as up and running

Configure SSH passwordless authenticaltion between each RAC node

Create 4 empty folders i.e. grid_2 and db_2 on both the nodes, give permission and change ownership

Make sure we have sufficient space available in /tmp location

oracle / root

19 Unset Oracle environment variables oracle

20 oracleCollect orachk output to check the system readiness and save it for reference

PRE-CHECKSCommand

ps -ef|grep d.binps -ef|grep d.bin| wc -l cd /opt/app/grid/oracle/product/11.2.0/grid_1/bin./ocrcheck./crsctl query css votedisk./ocrconfig -showbackup

./ocrconfig -backuploc /opt/app/oracle

./ocrconfig -manualbackupSQL> @?/rdbms/admin/dbupgdiag.sql./srvctl stop database -d uc1asvt./crsctl stop cluster -all ./crsctl stop crs (on both the nodes)cd /opt/app/oracle/product/11.2.0cp -r db_1 db_11.2.0.3cd /opt/app/grid/oracle/product/11.2.0cp -r grid_1 grid_11.2.0.3

cat /etc/oraInst.loc

./crsctl start crs

./crsctl start cluster -all

./crsctl stat res -t

unzip p13390677_112040_Linux-x86-64_3of7.zipmkdir grid_2chmod 775 grid_2chown root:dba grid_2mkdir db_2

cp -r /opt/app/oracle/product/oraInventory /opt/app/oracle/product/oraInventory_bkp

./cluvfy comp software -n all

./cluvfy comp software -n all -verbose

./cluvfy stage -pre crsinst -n [nodelist] -verbose

With root access, we set the below options in the global SSHd configuration file (sshd_config) and restarted the SSHd service:

ClientAliveInterval 30 TCPKeepAlive yes ClientAliveCountMax 99999

/etc/init.d/sshd restart

# haconf -makerw# hagrp -freeze cvm -persistent# hastatus -sum# haconf -dump -makero# haclus -display |grep -i Read

./orachk -u -o pre

unset ORACLE_BASEunset ORACLE_HOMEunset ORACLE_SIDunset ORA_CRS_HOMEunset ORA_NLS10unset TNS_ADMIN

PRE-CHECKSExpected Output / Key Notes

It would list all the daemons - main 4 are ohasd,crsd,evmd and cssd10

All files currently storing OCRAll files currently storing Voting DisksIt will show all auto backups

Take a manual backup

It will show the recently taken manual backup of OCR and Vote diskRefer Metalink Doc ID 556610.1 for more detailsIt will do a graceful shutdown of database.

It will shutdown the crs daemon and all cluster services across nodes

Take backup of Oracle Home

Take backup of Grid Home

Take backup of Inventory

It will start ohasd and other daemonsIt will start remaining daemons in both the nodes

Database should be open and stable

Refer Metalink Doc ID 300548.1 for steps to configure SSH

On Node1 only.

On node 1 and node 2 both

On node 1 and node 2 both

Involve Infra team if required

inst_group=dbainventory_loc=/opt/app/oracle/product/oraInventory

Run these commands from Grid Home. Make sure all the failures are resolved. Warnings can be ignored selectively.

Test this throughly to avoid any issues with ssh client terminating in the middle of the upgrade.

The steps mentioned are applicable for Veritas cluster. These are mandatory to avoid known issues while performing upgrade. Refer Metalink Doc ID 1360034.1 for more details.

Make sure all the failures are resolved. Warnings can be ignored selectively. Refer Metalink Doc ID 1457357.1 for details.

IMPLEMENTATIONSr. No. Description User

1 oracle

2 Apply Grid PSU locally on both nodes oracle

3 root

4 oracle

5 Run root.sh on both the nodes root

6 Apply Database PSU locally on both nodes oracle

7 root

8 Perform Database upgrade Pre-Checks oracle

9 Unset & set variables as mentioned oracle

10 oracle

Invoke ./runInstaller from "grid" folder (created in the location where 11.2.0.4 binaries were unzipped) to install and upgrade current Grid Home on 11.2.0.3 version

Run rootupgrade.sh on both the nodes after Grid PSU is applied on all the nodes successfully

Invoke ./runInstaller from "database" folder (created in the location where 11.2.0.4 binaries were unzipped) to install and upgrade current Database Home on 11.2.0.3 version

Run root.sh on both the nodes after Databse PSU is applied on all the nodes successfully

Invoke dbua to upgrade the database from old Oracle Home to the new one installed

IMPLEMENTATIONCommand

sh /opt/app/grid/product/11.2.0/grid_2/rootupgrade.sh

sh /opt/app/oracle/product/11.2.0/db_2/root.sh

sh /opt/app/grid/product/11.2.0/grid_2/rootupgrade.sh

Sample command:/opt/app/grid/oracle/product/11.2.0/grid_2/OPatch/opatch apply /opt/app/oracle/product/11.2.0.4_Patches/19955028/19769489 -oh /opt/app/grid/oracle/product/11.2.0/grid_2 -local

Sample command:/opt/app/oracle/product/11.2.0/db_2/OPatch/opatch apply /opt/app/oracle/product/11.2.0.4_Patches/19955028/19769489 -oh /opt/app/oracle/product/11.2.0/db_2 -local

Unset below environment variables:

unset ORACLE_BASEunset ORACLE_HOMEunset ORACLE_SIDunset ORA_CRS_HOMEunset ORA_NLS10unset TNS_ADMIN

Set environment variables as below:

export ORACLE_SID=INSTANCE_NAMEexport ORACLE_HOME=/opt/app/oracle/product/11.2.0/db_2export PATH=$ORACLE_HOME/bin:$PATHexport LD_LIBRARY_PATH=$ORACLE_HOME/lib

GRID_Upgrade.docx

DB_Software_Install.docx

DB_Upgrade_Pre-Checks.docx

DB_Upgrade.docx

IMPLEMENTATIONExpected Output / Key Notes

Do NOT run rootupgrade.sh script when you get the pop up to do so.

Refer Metalink Doc ID 1410202.1 for reference

Execute rootupgrade.sh as real ROOT user and not using "sudo su - root"

On node 1 only

Execute root.sh as real ROOT user and not using "sudo su - root"

Refer Metalink Doc ID 1410202.1 for reference

Do NOT run root.sh script when you get the pop up to do so.

Disable FRA before upgrading the database. You can enable it back post upgrade is completed successfully. We did this as we were facing some errors during upgrade.

POST-CHECKSSr. No. Description User

1 oracle

2 root

3 Shutdown the database oracle

4 Shutdown the cluster root

5 Take backup of new Grid and Oracle Homeoracle

root

6 Take backup of oraInventory oracle

7 Start crs root8 Start cluster root

9 oracle

10 Run catbundle.sql psu apply sys

11 oracle

12 Verify the applied patch from dba_registry_history sys

13 Registry components validity check sys

14 Gather dictionary stats, using any one of the command sys

15 Gather the fixed objects stats sys

Run the orachk and dbupgdiag.sql again and save the output

Take backup of OCR and voting disks (no need to take backup of voting disk manually with dd command as oracle automatically take every 4 hours)

Check all cluster resources , it should show DB as up and running

Check opatch lsinventory on both the servers for all 4 homes

POST-CHECKSCommand

./orachk -u -o post

./ocrconfig -showbackup

./ocrconfig -backuploc /opt/app/oracle

./ocrconfig -manualbackup

./srvctl stop database -d uc1asvt

./crsctl stop cluster -all

./crsctl stop crs (on both the nodes)cd /opt/app/oracle/product/11.2.0cp -r db_2 db_11.2.0.4cd /opt/app/grid/oracle/product/11.2.0cp -r grid_2 grid_11.2.0.4

cat /etc/oraInst.loc

./crsctl start crs

./crsctl start cluster -all

./crsctl stat res -t

SQL> @?/rdbms/admin/catbundle.sql psu apply

$ORACLE_HOME/Opatch/opatch lsinventory

EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

cp -r /opt/app/oracle/product/oraInventory /opt/app/oracle/product/oraInventory_bkp

select action_time,action,version,comments from dba_registry_history;

col COMP_NAME format a50set lines 500select COMP_ID, COMP_NAME, VERSION, STATUS from dba_registry;

SQL> EXEC DBMS_STATS.GATHER_SCHEMA_STATS ('SYS');SQL> exec DBMS_STATS.GATHER_DATABASE_STATS (gather_sys=>TRUE);SQL> EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

POST-CHECKSExpected Output / Key Notes

It will show all auto backupsTake a manual backupIt will show the recently taken manual backup of OCR and Vote diskIt will do a graceful shutdown of database.

It will shutdown the crs daemon and all cluster services across nodes

Take backup of oracle home

Take backup of grid home

It will start ohasd and other daemonsIt will start remaining daemons in both the nodes

Database should be open and stable

On any one node from each environment.

On all the nodes (both Grid and Oracle)

It would show the time when patch was applied with the comments

All the components should be valid. Any invalid components should be fixed.

Refer Metalink Doc ID 457926.1 for details.

Check the score and see if it can be make better by using the recommendations. Verify all the db components are valid.

inst_group=dbainventory_loc=/opt/app/oracle/product/oraInventory

Gather fixed objects stats if the load is heavy and if the system is busy. For more details when to gather fixed object stats, read following note:

Document:798257.1GATHER_FIXED_OBJECTS_STATS Considerations