solution delivery guide - netapp.com · • move logical interfaces (lifs) along with volumes for...

Technical Report

NetApp for Oracle Database 18c Solution Delivery Guide

Jimmie Cox, NetApp

July 2019 | TR-4760

2 NetApp for Oracle Database 18c Solution Delivery Guide © 2019 NetApp, Inc. All rights reserved.

TABLE OF CONTENTS

1 Solution Overview ................................................................................................................................ 4

1.1 NetApp Solution for Oracle Database .............................................................................................................4

2 Technology Requirements .................................................................................................................. 5

2.1 Hardware Requirements .................................................................................................................................5

2.2 License Requirements ....................................................................................................................................6

2.3 Storage ...........................................................................................................................................................7

2.4 Networking Requirements ...............................................................................................................................8

3 Solution Architecture ......................................................................................................................... 10

4 Install, Deploy, and Configure Oracle 18c Application ................................................................... 11

4.1 Install the Oracle 18c Application .................................................................................................................. 11

4.2 Install RAC on Two Linux Nodes .................................................................................................................. 21

4.3 Provisioning .................................................................................................................................................. 70

4.4 Use Space-Reserved Files and LUNs .......................................................................................................... 71

4.5 Support for SCSI Thin-Provisioned LUNs ..................................................................................................... 71

4.6 Configure Volume Provisioning Options ....................................................................................................... 71

5 Extensibility (Cloud) ........................................................................................................................... 72

5.1 What is Scalability in Cloud? ......................................................................................................................... 72

5.2 NetApp Cloud Volumes ONTAP ................................................................................................................... 73

5.3 Oracle RAC ................................................................................................................................................... 74

6 Oracle ASM Best Practices ................................................................................................................ 75

6.1 Use Disk Multipathing Software to Protect from Path Failure........................................................................ 75

6.2 Set the Processes Initialization Parameter.................................................................................................... 76

6.3 Use Disk Labels ............................................................................................................................................ 76

6.4 Set the FAILGROUP_REPAIR_TIME or DISK_REPAIR_TIME Disk Group Attribute Appropriately ............. 76

6.5 Use ASMLib On Supported Systems ............................................................................................................ 76

Where to Find Additional Information .................................................................................................... 80

Version History ......................................................................................................................................... 80

LIST OF TABLES

Table 1) NetApp ONTAP 9.x hardware requirements. ...................................................................................................5

Table 2) Oracle operating system requirements. ............................................................................................................5

Table 3) Differences in the three-volume provisioning option. ...................................................................................... 71

Table 4) Oracle formula. ............................................................................................................................................... 76


LIST OF FIGURES

Figure 1) ONTAP network architecture. .........................................................................................................................8

Figure 2) Physical architecture of the NetApp for Oracle Database solution (with repetitive connectivity lines added for readability). ................................................................................................................................................................... 10

Figure 3) Oracle Database with RAC and Data Guard (graphic supplied by Oracle). .................................................. 75

Figure 4) Two-volume layout. ....................................................................................................................................... 80


1 Solution Overview

Oracle databases drive business-critical applications and are key to the success of countless data-driven

enterprises like yours. Your service goals include a responsive, uninterrupted experience for employees,

customers, and partners accessing Oracle data. You need an infrastructure for Oracle that can deliver

consistently high levels of performance, data availability, and data protection while meeting your

operational cost guidelines.

Your database team needs a storage infrastructure that’s responsive enough to facilitate, not inhibit

database performance. In addition, the explosive growth of database instances and the critical data they

house creates various management challenges, from provisioning and cloning to backup and disaster

recovery. Without automated management tools that integrate databases with underlying storage, DBAs

can’t respond to rapidly changing business needs.

Your storage teams are under pressure to provide an infrastructure that exceeds user expectations while

driving out complexity and capital and operating inefficiencies. The infrastructure must remain up and

running, meet performance and availability SLAs, and provide ample resources to accommodate rapidly

expanding database environments without wasted capacity. Together, your IT teams need a highly

efficient, integrated solution to eliminate these issues, and deliver superior IT services for Oracle

databases.

1.1 NetApp Solution for Oracle Database

Address your biggest database challenges with an agile data infrastructure. The NetApp® solution for

Oracle Database delivers industry-leading storage, unprecedented scalability, continuous data access,

and automated data management for immediate response to business opportunities.

The following use cases are integrated into the NetApp and Oracle Database solution.

Use case 1: Data replication in database environments

• Use NetApp volume snap mirror to replicate production database volumes

Use case 2: Storage migration and database consolidation

• Use Oracle Data Guard to migrate Oracle Database storage from NetApp Data ONTAP® data management software.

• Use Data ONTAP to consolidate databases with Oracle RAC and/or Oracle RAC one node.

Use case 3: Dynamic scalability with online scale-up and scale-out

• Add more storage disk shelves (scale up) or add extra storage controllers (scale out).

• Achieve transparent controller clustering and failover capability in order to provide continuous operations.

Use case 4: Live migration of data volumes

• Use NetApp DataMotion for volumes to move database volumes across storage controllers

• Move logical interfaces (LIFs) along with volumes for continued access to data, load balancing, and workload balancing.

Use case 5: Operational flexibility

• Use data motion for volume to move data between different storage aggregates and controllers within the cluster.

• Use data motion for volumes to move the data between aggregates made up of a higher number of disk or faster drive types


2 Technology Requirements

2.1 Hardware Requirements

Table 1 lists hardware requirements for NetApp ONTAP 9.x.

Table 1) NetApp ONTAP 9.x hardware requirements.

Storage Controller Model Minimum ONTAP Version Maximum ONTAP Version

NetApp FAS2600 Series 9.1RC1 9.4.x

NetApp FAS2700 Series 9.4RC1 9.4.x

NetApp FAS8200 9.1RC1 9.4.x

NetApp FAS9000 9.1RC2 9.4.x

NetApp AFF A200 9.1RC2 9.4.x



NetApp AFF A700 all-flash storage system

9.1RC2 9.4.x

NetApp AFF A700s 9.1 9.4.x

NetApp® AFF A800 all-flash storage system

9.4RC1 9.4.x

Table 2 lists the operating system requirements for Oracle.

Table 2) Oracle operating system requirements.

Operating System Supported Operating Systems

Windows • Windows 7 x64 - Professional, Enterprise, and Ultimate editions

• Windows 8.1 x64 - Pro and Enterprise editions

• Windows 10 x64 - Pro, Enterprise, and Education editions

• Windows Server 2012 x64 - Standard, Datacenter, Essentials, and Foundation editions

• Windows Server 2012 R2 x64 - Standard, Datacenter, Essentials, and Foundation editions

• Windows Server 2016 x64 - Standard, Datacenter, and Essentials editions

Linux • Oracle Linux 7.5 with the Unbreakable Enterprise Kernel 5: 4.14.35-1818.5.3.el7uek.x86_64 or later

• Oracle Linux 7.2 with the Unbreakable Enterprise Kernel 4: 4.1.12-32.2.3.el7uek.x86_64 or later

• Oracle Linux 7 with the Unbreakable Enterprise Kernel 3: 3.8.13-35.3.1.el7uek.x86_64 or later

• Oracle Linux 7 with the Red Hat Compatible kernel: 3.10.0-123.el7.x86_64 or later

Note: When new Linux kernels and distributions are released, Oracle modifies and tests its products for stability and reliability on these systems. Oracle makes every effort to add support for new


kernels and distributions in a timely manner. However, until a kernel or distribution is added to this list, its use with Oracle products is not supported. If you are running any popular Linux distribution with a 2.6 kernel and libc version 6, Oracle endeavors to support you if issues arise.

Policy Center

The default port used by the Policy Center for traffic and management is 8060.

Service Monitor

The default port used by the Service Monitor for viewing reports and management is 8040.

Policy Studio

The default URL addresses used by the Policy Studio to connect to other components are as follows:

• Software versions and configurations. ONTAP 9.1, 9.2, 9.3, and 9.5 are general availability (GA) in the ONTAP 9 release family.

Note: You can run ONTAP 9.1, 9.2, 9.3, and 9.5 in clustered configurations on all NetApp FAS systems, NetApp AFF systems, and NetApp FlexArray® Virtualization systems that are supported with this release.

• Oracle Database 18c software supports the following operating systems:

− Microsoft Windows x64 (64 bits)

− Linux x86-64

2.2 License Requirements

The access the ONTAP features, download the following licenses:

• NetApp SnapMirror®. This license is for mirroring backup sets to a destination storage system. Where this would be required would be on primary storage systems only. Typically for FAS-to-FAS replication, SnapMirror is required on both primary and destination systems. With your data fabric powered by NetApp solution for cloud backup, a SnapMirror license is required only on the primary storage.

• NetApp SnapRestore®. This license enables SnapCenter to restore and verify backup sets. Where this would be required would be on primary storage systems. Also required on SnapVault destination systems to perform remote verification and to restore from a backup. Also required on SnapMirror destination systems to perform remote verification.

• FlexClone. This license is required for your data fabric powered by NetApp solution for cloud backup. Where this would be required would be on primary storage systems.

• Standard NetApp SnapCenter® license (optional). This license enables you to add a storage virtual machine (SVM) to a SnapCenter instance to support backup and recovery of ONTAP storage, Clone lifecycle management, basic reporting, task automation, host file systems (Windows, Linux, UNIX), support for custom applications or databases, update of NetApp Snapshot™ copies to NetApp SnapMirror and SnapVault® secondary destinations, virtualization with VMware, and support for enterprise applications (Microsoft SQL Server and Oracle).

Note: The Standard SnapCenter license is a prerequisite for the Advanced license.

• FAS, AFF, ONTAP Select, ONTAP Cloud licenses. These individual licenses are charged on the used storage capacity that is managed by SnapCenter.

Note: ONTAP Select is not supported in the data fabric powered by NetApp solution for cloud backup.

Note: A 90-day trial license is available for SnapCenter Standard and Advanced licenses.


2.3 Storage

With NetApp AFF storage on Oracle, you can provision storage in minutes with the deep integration with

business critical Oracle applications. Within Cloud Volumes ONTAP you can create several volumes to

spread out the workload in the environments.

Cloud Volumes ONTAP is similar to ONTAP Select, except that it runs in a hyperscaler cloud

environment, bringing intelligence and data fabric connectivity to hyperscaler storage volumes. The best

practices for running Oracle on ONTAP are not affected. The primary considerations are performance

and to a lesser extent cost.

Cloud Volumes ONTAP is partially limited by the performance of the underlying volumes managed by the

cloud provider. The result is more manageable storage, and, in some cases, the caching capability of

offers a performance improvement. However, there are always some limitations in terms of IOPS and

latency due to the reliance on the public cloud provider. This does not mean that database performance is

unacceptable. It simply means that the performance ceiling is lower than options such as an actual

physical AFF system. Furthermore, the performance of storage volumes offered by the various cloud

providers that are used by Cloud Volumes ONTAP are continuously improving.

The prime use case for Cloud Volumes ONTAP is currently for development and testing work, but some

customers have used it for production activity as well. One notable report was the use of Oracle’s In-

Memory feature to mitigate storage performance limitations. This allows more data to be stored in RAM

on the virtual machine hosting the database server, thus reducing performance demands on storage.

Capacity Limits

In order to provide high and predictable performance on a storage array, some free space is required for

metadata and data organizational tasks. Free space is defined as any space that is not used for actual

data and includes unallocated space on the aggregate itself and unused space within the constituent

volumes. Thin provisioning must also be considered. For example, a volume might contain a 1TB LUN of

which only 50% is used by actual data. In a thin provisioned environment, this would correctly appear to

be consuming 500GB of space. However, in a fully provisioned environment, the full capacity of 1TB will

appear to be in use. The 500GB of unallocated space will be hidden. This space is unused by actual data

and should therefore be included in the calculation of total free space.

SSD Aggregates, Including AFF Systems

NetApp recommends a minimum of 10% free space. This includes all unused space, including free space

within the aggregate or a volume and any free space that is allocated due to the use of full provisioning

but is not used by actual data.

The recommendation of 10% free space is conservative. SSD aggregates can support database

workloads at even higher levels of utilization without any effect on performance. However, as the

utilization of the aggregate increases the risk of running out of space also increases if utilization is not

monitored carefully.

HDD Aggregates, Including Flash Pool Aggregates

NetApp recommends a minimum of 15% free space. This includes all unused space, including free space

within the aggregate or a volume and any free space that is allocated due to the use of full provisioning

but is not used by actual data.

There should be no measurable performance effect when utilization is less than 85%. As utilization

approaches 90%, some reduction in performance might become noticeable for certain workloads. As

utilization reaches 95%, most database workloads experience a degradation in performance.


2.4 Networking Requirements

Network Architecture

The network architecture for an ONTAP data center implementation typically consists of a cluster

interconnect, a management network for cluster administration, and a data network. Network interface

cards (NICs) provide physical ports for Ethernet connections and host bus adapters (HBAs) provide

physical ports for Fibre Channel (FC) connections.

Figure 1) ONTAP network architecture.

Logical and physical ports are provided on each node, you can use logical ports to manage network

traffic. Logical ports are interface groups or virtual LANs (VLANs).

Interface Groups

Interface groups combine multiple physical ports into a single logical trunk port. You might want to create

an interface group consisting of ports from NICs in different Peripheral Component Interconnect (PCI)

slots to ensure against a slot failure bringing down business-critical traffic. An interface group can be

single mode, multimode, or dynamic multimode. Each mode offers differing levels of fault tolerance. You

can use either type of multimode interface group to load balance network traffic.

VLANs

VLANs separate traffic from a network port (which could be an interface group) into logical segments

defined on a switch port basis, rather than on physical boundaries. The end-stations belonging to a VLAN

are related by function or application. You might group end-stations by department, such as engineering

and marketing, or by project, such as release1 and release2. Because physical proximity of the end-

stations is irrelevant in a VLAN, the end-stations can be geographically remote.

Support for industry-standard network technologies ONTAP supports all major industry-standard network

technologies. Key technologies include IPspaces, DNS load balancing, and SNMP traps. Broadcast

domains fail over groups, and subnets are described in NAS path failover.


IPspaces

You can use an IPspace to create a distinct IP address space for each virtual data server in a cluster.

Doing so enables clients in administratively separate network domains to access cluster data while using

overlapping IP addresses from the same IP address subnet range. A service provider, for example, could

configure different IPspaces for tenants using the same IP addresses to access a cluster.

You can use DNS load balancing to distribute user network traffic across available ports. A DNS server

dynamically selects a network interface for traffic based on the number of clients that are mounted on the

interface.

SNMP Traps

You can use SNMP traps to check periodically for operational thresholds or failures. SNMP traps capture

system monitoring information sent asynchronously from an SNMP agent to an SNMP manager.

Federal Information Processing Standards (FIPS) compliance ONTAP is compliant with the FIPS 140-2

for all Secure Sockets Layer (SSL) connections. You can turn on and off Secure Sockets Layer (SSL)

FIPS mode, set SSL protocols globally, and turn off any weak ciphers such as RC4.

PFILE (Per Oracle)

A PFILE is a traditional text-based init.ora parameter file. Typically, this resides on the server in the

$ORACLE_BASE/admin/SID/pfile directory, with a symbolic link pointing to it from the

$ORACLE_HOME/dbs directory. In addition, you can keep copies of this file on your local PC to allow

remote startup.

SQL> CONNECT sys/password AS SYSDBA

SQL> STARTUP PFILE=C:LocalInit.ora

A PFILE is necessary in order to create a server parameter file (SPFILE) to enable persistent initialization

parameters.

If you already have a SPFILE, you can generate a PFILE from it by running one of the following

commands:

CREATE PFILE FROM SPFILE;

CREATE PFILE FROM SPFILE = 'production.ora';

CREATE PFILE = '$ORACLE_HOME/dbs/my_pfile.ora' FROM SPFILE;

CREATE PFILE = '$ORACLE_HOME/dbs/my_pfile.ora' FROM SPFILE = '$ORACLE_HOME/dbs/my_spfile.ora';

If the SPFILE name is not specified, Oracle will look for the default SPFILE. If this is not present an error

will be returned. If the PFILE name is not specified Oracle will use a platform specific default name.

After the PFILE is produced you can edit it and use it to create a new or modified SPFILE.

SPFILE

An SPFILE is a server-managed binary file that Oracle uses to hold persistent initialization parameters. If

a parameter is changed using the ALTER SYSTEM SET command, Oracle applies this parameter change

to the current SPFILE. Because the database uses this file during startup, all parameter changes persist

between shutdowns.


You can create an SPFILE from a PFILE by running one of the following commands:

SQL> CREATE SPFILE FROM PFILE = '$ORACLE_HOME/dbs/my_pfile.ora';

SQL> CREATE SPFILE = '$ORACLE_HOME/dbs/my_spfile.ora' FROM PFILE =

'$ORACLE_HOME/dbs/my_pfile.ora';

If the SPFILE is not specified, Oracle assumes that you are creating a default SPFILE. If a default

SPFILE already exists, Oracle overwrites it. If the SPFILE is specified, a nondefault SPFILE is created.

The named SPFILE must not be the same as a SPFILE that was used to start the current instance.

Oracle recommends that you leave it to decide on the name and location of the SPFILE.

3 Solution Architecture

Figure 2 illustrates the physical architecture of the NetApp for Oracle Database solution.

Figure 2) Physical architecture of the NetApp for Oracle Database solution (with repetitive connectivity lines added for readability).

In summary, the components of the NetApp for Oracle Database solution combine to accomplish the

following tasks:

• Replication of production databases running on NetApp or third-party storage to dev/test running on NetApp storage

• Cloning of databases to and from varying environments:

− Physical to physical

− Physical to virtual

− Virtual to virtual

• Conversion of physical server images to virtual machine (VM) images

• Rapid cloning and provisioning of virtual machines (VMs)

• Application of thin provisioning, space-efficient cloning, deduplication, and compression


4 Install, Deploy, and Configure Oracle 18c Application

4.1 Install the Oracle 18c Application

To install the Oracle 18c application on Linux 7 in a single instance, complete the following steps:

1. Download the Oracle Database file and RPM from the Oracle site to your Linux VM.

Note: You can access the downloadable files on the Oracle Database Software Downloads page.

2. Create the groups.

groupadd -g 1100 oinstall

groupadd -g 1131 dba

groupadd -g 1132 oper

useradd -u 1101 -g oinstall -G dba,oper oracle

passwd oracle

3. Create the following directory and change the ownership and permissions:

mkdir -p /u01/app/oracle/product/18.3.0/dbhome_1

chown -R oracle:oinstall /u01

chmod -R 775 /u01

4. Run the preinstall for Oracle by running the following command as root user from root directory:

yum install oracle-database-preinstall-18c.x86_64

5. When the system asks you if this is OK, enter "y" for yes.

https://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html


6. When the system asks you whether it is OK to install the next package, enter "y" for yes.

7. After the preinstall package is complete, proceed to the next step.

8. Switch the user to Oracle and use vi editor to the .bashprofile and enter the following parameters:


TMP=/tmp; export TMP

TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=testlab.com.bd; export ORACLE_HOSTNAME

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE ORACLE_TERM=xterm; export ORACLE_TERM

BASE_PATH=/usr/sbin:$PATH; export BASE_PATH PATH=$ORACLE_HOME/bin:$GRID_HOME/bin:$BASE_PATH;

export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

9. Copy the zip folder to the database home directory.

10. Change the directory to Oracle home and then unzip the db_home.zip folder.

11. Run the ls command to view the files that copied over.


12. Run the ./runInstaller command to initialize the GUI and start the installation process.

[oracle@localhost dbhome_1]$ ./runInstaller

13. Select the Set Up Software Only option and click Next.

14. Select the Single Instance option and click Next.

15. Select the Enterprise Edition option and click Next.


16. Specify the Oracle base and click Next.

17. Create an inventory directory and click Next.


18. Select the system privileges and click Next.

19. Go through the prerequisite checks. When the check are complete, click Next.


20. On the Summary page, click Install to proceed.

21. When the install begins, click OK to run the following scripts.

22. As root user, run the following script:

[root@localhost ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.


23. Run the second script (as instructed by Oracle):

[root@localhost ~]# /u01/app/oracle/product/18.3.0/dbhome_1/root.sh

24. After the scripts are complete, execute the configuration scripts and click OK.

25. When this part of the installation is complete, click Close,

26. To create a database, open the terminal and run the dbca command.


[oracle@localhost dbhome_1]$ dbca

27. Select the Create a Database option and click Next.

28. Add an administrator password and create a pluggable database. Click Next.

29. On the Summary page, click Finish.


30. The database creation is now in progress.

31. A database creation completion pages is displayed.


32. Return to the terminal and run the following command. This command will enable you to connect to the database.

[oracle@localhost ~]$ sqlplus / as sysdba

Version 18.3.0.0.0

Copyright © 1982, 2018, Oracle. All rights reserved.

Connected to:

Oracle Database 18c Enterprise Edition Release 18.0.0.0.0

Version 18.3.0.0.0

This concludes the process for creating a single instance for Oracle.

4.2 Install RAC on Two Linux Nodes

To install RAC on two Linux nodes, complete the following steps:

1. Download the Oracle Database and Grid installers.

Note: You can access the downloadable files on the Oracle Database Software Downloads page.

2. The two nodes in this example are war1 and war2. For the host file on war1, run the follow

commands to edit the host file:

Note: Make sure that you save the updated changes.

vi /ect/host

127.0.0.1 localhost localhost.localdomain localhost 4 localhost 4.localdomain4

:: 1 localhost localhost.localdomain localhost 6 localhost 6.localdomain6

https://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html


# public IPs

192.168.1.175 storage storage.localdomain

192.168.1.181 godofwar1 godofwar1.localdomain

192.168.1.182 godofwar2 godofwar2.localdomain

# VIP IPs

192.168.1.186 godofwar1-vip godofwar1-vip.localdomain

192.168.1.187 godofwar2-vip godofwar2-vip.localdomain

# Private IPs

192.168.2.175 storage-priv storage-priv.localdomain

192.168.2.181 godofwar1-priv godofwar1-priv.localdomain

192.168.2.182 godofwar2-priv godofwar2-priv.localdomain

# Scan IPs

192.168.1.191 godofwar-scan godofwar-scan.localdomain



In the private IPs, use the subnet for communication between the storage server and RAC notes. This subnet is also used for Oracle Automatic Storage Management (ASM) communication between the RAC nodes. In the following image, you can also see the scan IPs. These entries were added to both RAC nodes host file. Review and install the prerequisites packages. These packages were downloaded to both servers; therefore, you must install them on both RAC nodes. To view the downloads, run the following command:

ls -lh /software/downloads/preinstaller

3. Install the following packages:

cd /software/downloads/preinstall/

rpm -ivh oracle-database-preinstall-18c-1.0-1.e16.x86_64.rpm

rpm -ivh oracleasmlib-2.0.12-1.el6.x86_64.rpm

rpm -ivh oracleasm-support-2.1.11.-2.el6.x86_64.rpm


4. Run the following command in order to verify whether the packages are installed correctly. In this example, they were installed correctly.

Before you proceed, follow the same steps on the other node (war2).

5. Create the user and groups that are required by Oracle. First, create the following groups. If you want to manually add a group, run the groupadd groupname command from the terminal.

6. Create the users.

Note: To manually add a user, run the useradd username or adduser username command from the terminal.


7. Verify that grid and Oracle users are created and assigned to the proper OS group by running the id

grid and id oracle commands. Follow the same steps for the other node; in this example, that is

war2 server.

8. Create the required folders for Oracle. After you create the required folders, move over to war 2

server and follow the same steps before you proceed to the next step. You can also create these folders by running the mkdir command.

9. Configure Oracle ASMLib on the first node by running the oracleasm configure -I command.

Note: ASMLib is largely being replaced by Oracle ASM Filter Driver (ASMFD). Oracle ASMFD simplifies the configuration and management of disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.

Per Oracle:


− If ASMLib is installed and configured for an existing Oracle ASM installation, then you must explicitly deinstall the existing ASMLib configuration before installing and configuring Oracle ASMFD. Refer to Oracle Grid Infrastructure Installation and Upgrade Guide for your operating system for information about deinstalling Oracle ASMLib and the installation of Oracle Grid Infrastructure.

− If ASMLib has been previously installed on the system, then you must set the Oracle ASM disk string to '' (with ASMCMD dsset '') before running ASMCMD afd_configure to configure

Oracle ASMFD.

− For additional information about configuring Oracle ASM filter driver (ASMFD), see the Administering Oracle ASM Filter Driver section of the Administrators' Guide.

10. Run the oracleasm configure command to verify the ASM configuration.

11. Start the Oracle ASMLib. As you can see in the following screenshot, Oracle ASM is configured on the first node. To replicate this configuration, run the oracleasm init command and then check

the status by running the oracleasm status command. Repeat this step on the other node before

you proceed.

12. In this example, the logical unit number (LUN) from the storage server were discovered on both RAC nodes. In this example, these storage LUNS are used as an ASM disk. To see the disks, run the ls

-lh /dev/sd*1 command.

https://docs.oracle.com/en/database/oracle/oracle-database/18/ostmg/administer-filter-driver.html#GUID-2F5E344F-AFC2-4768-8C00-6F3C56302123


13. Create the ASM disk. You can do this manually by running the createdisk command from the

terminal.

Notes:

− The disk names can contain uppercase letters, numbers, and the underscore character; however, they must start with an uppercase letter.

− To create a database during the installation by using the Oracle ASM library driver, you must change the disk discovery string to ORCL:*.

− If you are using a multipathing disk driver with Oracle ASM, then make sure that you specify the correct logical device name for the disk.

14. After the ASM disk is created, run the oracleasm listdisks command to display the disks.

Repeat the same steps on your second node before you proceed.


15. Add the following entries to the bash profile for the grid user. Add these entries on the second node before you proceed. Use VI editor to add the following variables to the /.bash_profile:.

# Oracle ASM instance related environment variables

export CVUQDISK_GRP=oinstall

export ORACLE_SID=+ASM1

export ORACLE_UNQNAME=+ASM

export ORACLE_BASE=/u01/app/grid

export ORACLE_HOME=/u01/app/18.3.0/gridHome_1

export PATH=$PATH:.:$ORACLE_HOME/bin

Note: You are now logged in as grid user on war 1, which is the location where the zip file is located.

16. Run the following command:

unzip /software/downloads/Oracle18cR3_Grid.zip -d to unzip the file to /u01/app/18.3.0/gridHome_1

on our first RAC node war1

17. Log back into the war1 mode +as root user. Before you install the Oracle grid software, install one

more prerequisite RPM package by running the following commands:

cd /u01/app/18.3.0/gridHome_1/

cd ./cv/rpm

pwd to verify you’re in the correct directory

ls -lh cvuqdisk-1.0.10-1.rpm


This is the cluster verification utility package that is provided with the Oracle grid software. To continue

with installation process, complete the following steps:

1. Run the following command:

rpm -ivh cvuqdisk-1.0.10-1.rpm. To verify that it took you can run the following command: rpm -qa

| egrip -I “cvuqdisk”

2. Copy this utility package to the second node war 2 by running the following command:

scp cvuqdisk-1.0.10.1.rpm root@godofwar:/tmp/

3. You are now logged into the war 2 node. To set this utility package up on the war 2 node, complete

the following steps:

a. Run the following command:

ls -lh /tmp/cvuqdisk-1.0.10.1.rpm

b. Run the next command:

rpm -ivh /tmp/cvuqdisk-1.0.10.1.rpm

c. Run this last command:

rpm -qa | egrep -i “cvuqdisk”

4. Install the grid infrastructure software on the two-node cluster.

Note: You are now logged back into the war 1 node as grid user. Run this grid setup script in order to install the grid infrastructure software. Run the following command in order to locate the grid setup:


cd $ORACLE_HOME

pwd

ls

gridSetup.sh

5. Run the following command in order to open the grid GUI interface:

./gridSetup.sh &

6. Select the Configure Oracle Grid Infrastructure for a New Cluster option to set up a new cluster.

7. For this example, select the Configure an Oracle Standalone Cluster option.


8. Select the Create Local Scan option. Customize the names and their ports.

9. Identify the role (HUB or LEAF) of the server. In this example, we chose HUB. As you can see, the war1 node is already there. Click Add to add the war2 node .


10. Add war2 node. Enter the public and virtual host names and click OK. In this example, we left the

Node Role set to HUB.

You can see that node war 2 was successfully added.


11. On the Specify Network Interface Usage page, set ETH0 for public usage and ETH1 for ASM and private usage. ETH2 will not be used in this example.

12. Select the Configure ACM Using Block Storage option.


13. On the Grid Infrastructure Management Repository Option page, select Yes.

In this example, Oracle has not noticed the disk. Click Change Discovery Path.


14. For the disk discovery path, enter ORCL*. Click OK.

The ASM disk is now discovered.


15. On the Create ASM Disk Group page, create the ORCL disk group for storing Oracle cluster registry and noting files. Allocate three 10GB disks for this disk group.

16. Create a data disk group with external redundancy for storing grid infrastructure management repository data and the database file. Allocate three 20GB disks for this disk group.

17. Specify a password for the Oracle ASM accounts. In this example, we used a simple password.

Note: For security reasons, simple passwords are not recommended for production servers.


18. Select the desired Platform Failure Isolation (PMI) option. In this example, PMI is not configured for the cluster setup.

19. Do not select the Register with Enterprise Manage (EMI) Cloud Control option.


20. Select the privileged operating system groups.

21. The Specify Installation Location page displays location of the Oracle base and the Oracle Grid Infrastructure.


22. The Create Inventory page displays the Oracle inventory folder that is owned by the oinstall

group.

23. Click the Automatically Run Configuration Scripts option. Select Use Root User and supply a password.


During the installation process, the following error was displayed. To resolve this same error, complete

the following steps:

1. Click Fix and select Again to try to resolve the issue.

2. The system advises that you run a script as root user on both RAC nodes.


a. Cd /tmp/GridSetupActions2018-09-22_08-34-24PM/CVU_18.0.0.0.0_grid and then

run the sh runfixup.sh shell command.

Note: Perform this step on both nodes as root as requested.

b. After running the shell script, a message is displayed instructing you to restart the installer from a new session in order for changes to take effect. For now, this message can be ignored. Click OK.

c. The Fixup Result tab shows that the fixup results are successful.


d. Click Ignore All and proceed to the next step.

3. A configuration summary is displayed. Click Save Response File and save it to a location of your choosing. Click Save and then click Install to begin the grid installation.

4. The setup wizard wants to run the scripts as root user. Click Yes.


5. An error message is displayed instructing you to review the log file. In this example, all the configurations succeeded except for two.

6. Click Skip and then click Yes.


7. To view the active version of cluster, run the following command:

which crsctl

crsctl query crs activeversion -f

8. To show the status for all Oracle cluster resources, run the following command.

At this point, everything in the grid infrastructure is stable and looks good.

crsctl status resource -t | more


9. Connect to Oracle ASM instance using SQL Server plus and verify the software version by running the following commands:

sqlplus / as sysasm

set pagesize 1000

SELECT * FROM v$version ;


To launch the ASM configuration assistant, complete the following steps:

1. Run the asmca command.

2. Run the asmca & command.

3. Under ASM Instances, you can see that both ASM instances are up and running on the two RAC nodes.

4. Under Disk Groups, you can see that the data and OCR disk groups are mounted on both nodes.


5. Under Volumes, by default, the ASM volume is created and in a disabled state.

6. Under ACFS File Systems, by default, the ACFS mount point is created, but it is in a dismounted state.


7. Under Settings, set the root password. In this example, root is used for the user credential, but there is also a Use Sudo option. Click Save.

8. Go back to Disk Groups. Notice how the data disk group is running low on disk space. Add an ASM disk to the data disk group.

9. Right-click the data disk group and select Add Disks.


10. Select the disk that you want to add and click OK.

11. Create a fast recovery disk group. Click Create.


12. Enter FRA for the disk group name. Select external redundancy and then select the three disks. In this example, we selected three 20GB disks. Click OK.

To install Oracle18C database software on this cluster, complete the following steps:

Note: You are logged in as Oracle user on node one.

1. Extract the zipped folder from unzip /software/downloads/Oracle18cR3_Database.zip -

d and place it in /u01/app/oracle/product/18.3.0/dbHome_1 on the first RAC node war1.


2. Set the variable before you install the database software.

export ORACLE_HOSTNAME=godofwar1.local.domain

3. Run the following commands:

cd $ORACLE_HOME

pwd follow by ls command

4. Enter ./runInstaller to start the installation.

5. Select Set up Software only for RAC installations. You can use DBCA to create the database later.


6. Select the Oracle Real Application Clusters Database Installation option.

7. Make sure that both RAC nodes are listed and selected.


8. Select Enterprise Edition.

9. Enter the Oracle base location. In this example, dbHome_1 is the software location.


10. Enter the proper privileged groups. In this example, the proper groups are selected.

11. After running the prerequisite checks, there is an indication that some failed. For now, click Ignore All and proceed to the next step.


12. This Summary page shows the options that you chose during the installation. Save to /u01/app/oracle/dbinstall.rsp and click Save. Then click Install.

13. The installer is requesting to run the

/u01/app/oracle/product/18.3.0/dbHome.1/root.sh script on both RAC nodes.


14. Run the following commands:

cd /u01/app/oracle/product/18.3.0/dbHome_1

sh root.sh

15. Run the commands from step 14 on the second node. After it is complete, click Finish.

The database software was successfully installed on the cluster.


Oracle recommends that you take a backup of the root script located in the Oracle home folder by

completing the following steps:

1. Take a backup of root.sh file on each of our RAC nodes. Run the follow command on both nodes:

cp $ORACLE_HOME/root.sh $ORACLE_HOME/root.sh_20180923_AfterDBInstall

2. Modify the .bash_profile for the Oracle user on both RAC nodes. To start, run the vi

~/.bash_profile command.

3. Include the Oracle home bin folder in the path variable and then save the changes. Make the same changes on node 2. The bash profile now looks good with both changes.


To create the database for this cluster, complete the following steps:

1. Start the configuration by running the dbca command.

2. Select the Create a Database option.

3. Select the Advanced Configuration option.


4. Select Oracle Real Application Cluster (RAC) Database as the database type. Select Admin Managed as the configuration type. In this example, the General Purpose template is selected.

5. Make sure that the RAC nodes are listed and selected for the setup.


6. Change the global database name to o18C. Make sure that the Create as Container Database option is not selected.

7. In this example, ASM is selected as the default storage type for storing the database data files. The Use Oracle Managed Files (OMF) option is selected by default.


8. Oracle recommends that you select the Multiplex Redo Logs and Control File option.

9. Add the two entries DATA and FRA. Click OK.


10. Select the Specify Fast Recovery Area and Enable Archiving options. Enter FRA in the Fast Recovery field. In this example, the Fast Recovery Area size is set to 12GB.

11. Do not select either option on the Select Oracle Data Vault Config Option page.


12. Review the default configuration settings, but don't change them.


13. On the Specify Management Options page, do not change any of the default settings.


14. On the Specify Database User Credentials page, use a simple password for all administrative accounts. Click Yes.

Note: For security reasons, simple passwords are not recommended on production servers.

15. Select the Generate Database Creation Scripts option.


16. Proceed to the Perform Prerequisite Checks page.

17. On this page, all the options are selected for this database creation. At the bottom of the page, select Response File and save it to a location of your choice. Click Finish.


18. Database Configuration Assistant creates the database.

19. After Database Configuration Assistant successfully creates the database, the following notification is displayed. You can use the password management link to review or reset the password for database user accounts.


20. Check the status of cluster by running the following command:

Command: crsctl status resource -t


As you can see in the above screenshots, the two new databases are stable and all the other resources on the RAC are healthy and in good shape. This concludes the Oracle RAC installation.


4.3 Provisioning

ONTAP provides thin provisioning. This feature allows you to manage the space and the space

requirements for ONTAP. Understanding how the options work enables you to choose the best option for

your environment.

Thin Provisioning for Volumes

When a thinly provisioned is created, ONTAP does not reserve any extra space when the volume is

created. As data is written to the volume, the volume requests the storage it needs from the aggregate to

accommodate the write operation. Using thin-provisioned volumes enables you to over commit your

aggregate, which introduces the possibility of the volume not being able to secure the space it needs

when the aggregate runs out of free space. You create a thin-provisioned NetApp FlexVol® by setting its -

space-guarantee option to none.

LUN Thin Provisioning

Thin provisioning of active LUNs has limited use in an Oracle environment because Oracle initializes data

files to their full size at the time of creation. The efficiency of thin provisioning of active LUNs in a file

system environment can be lost over time as deleted and erased data occupy more unallocated white

space in the file system.

There is one exception when logical volume managers (LVM) are used. When an LVM such as Veritas

VxVM or Oracle ASM is used, the underlying LUNs are divided into extents that are only used when

needed. For example, if a database begins at 2TB in size but could grow to 10TB over time, this database

can be placed on 10TB of thin-provisioned LUNs organized in an LVM disk group. It would occupy only

2TB of disk space at the time of creation and would only claim additional space as extents are allocated

to accommodate database growth. This process is safe as long as space is monitored.

LUN Sizing

A LUN is a virtualized object on ONTAP that exists across all of the spindles in the hosting aggregate. As

a result, the performance of the LUN is unaffected by its size because the LUN draws on the full potential

of the aggregate no matter which size is chosen.

As a matter of convenience, customers might want to use a LUN of a particular size. For example, if a

database is built on an ASM disk group composed of two LUNS of 1TB each, then that ASM disk group

must be grown in increments of 1TB. It might be preferable to build the ASM disk group from eight LUNs

of 500GB each so that the disk group can be increased in smaller increments.

The practice of establishing a universal standard LUN size is discouraged, because doing so can

complicate manageability. For example, a standard LUN size of 100GB might work well when a database

is in the range of 1TB to 2TB, but a database 20TB in size would require 200 LUNs. This means that

server reboot times are longer, there are more objects to manage in the various UIs, and products such

as SMO must perform discovery on many objects. Using fewer, larger LUNs avoids such problems.

Notes:

• The LUN count is more important than the LUN size.

• LUN size is mostly controlled by LUN count requirements.

• Avoid creating more LUNs than required.


4.4 Use Space-Reserved Files and LUNs

A space-reserved file or LUN is one for which storage is allocated when it is created. Historically, NetApp

has used the term thin-provisioned LUN to mean a LUN for which space reservation is disabled (a non-

space-reserved LUN).

Note: Non-space-reserved files are not referred to as thin-provisioned files.

Table 3 summarizes the major differences in how the three-volume provisioning option can be used with

space-reserved files and LUNs.

Table 3) Differences in the three-volume provisioning option.

Volume Provisioning LUN or File Space Reservation

Overwrites Protections Data2 Storage Efficiency3

Thick Supported Guaranteed1 Guaranteed Supported

Thin No affect None Guaranteed Supported

Semi-thick Supported Best effort1 Best effort Not supported

1The ability to guarantee overwrites or provide a best-effort overwrite assurance requires that space reservation is enabled on the LUN or file. 2Protection data includes Snapshot copies, FlexClone files, and LUNs marked for automatic deletion (backup clones). 3Storage efficiency includes deduplication, compression, any FlexClone files, and LUNs not marked for automatic deletion (active clones), and FlexClone subfiles (used for copy offload).

4.5 Support for SCSI Thin-Provisioned LUNs

ONTAP supports T10 SCSI thin-provisioned LUNs and NetApp thin-provisioned LUNs. T10SCSI thin

provisioning enables host applications to support SCSI features including LUN space reclamation and

LUN space monitoring capabilities for blocks environments. T10 SCSI thin provisioning must be

supported by your SCSI host software.

You use the ONTAP space-allocation setting to enable/disable support for the T10 thin provisioning on a

LUN. You use the ONTAP space-allocation enable setting to enable T10SCSI thin provisioning on a LUN.

4.6 Configure Volume Provisioning Options

You can configure a volume for thin provisioning, thick provisioning, or semi-thick provisioning.

Setting the -space-slo option to thick ensures the following:

• The entire volume is preallocated in the aggregate. You cannot run the volume create or volume

modify command to configure the volume's -space-guarantee option.

• 100% of the space required for overwrites is reserved. You cannot run the volume modify

command to configure the volume's -fractional-reserve option.

• Setting the -space-slo option to semi-thick ensures the following:

− The entire volume is preallocated in the aggregate. You cannot run the volume create or

volume modify command to configure the volume's -space-guarantee option.

− No space is reserved for overwrites. You can run the volume modify command to configure

the volume's -fractional-reserve option.

− Automatic deletion of Snapshot copies is enabled.

To configure the volume provisioning options, complete the following steps:


1. Configure volume provisioning options:

volume create -vserver vserver_name -volume volume_name -aggregateaggregate_name -space-slo

none|thick|semi-thick -space-guarantee none|volume

The -space-guarantee option defaults to none for AFF systems. Otherwise, it defaults to volume. For

existing FlexVol volumes, run the volume modify command to configure provisioning options.

Example 1:

The following command configures vol1 on SVM vs1 for thin provisioning:

cluster1::> volume create -vserver vs1 -volume vol1 -space-guarantee none

Example 2:

The following command configures vol1 on SVM vs1 for thick provisioning:

cluster1::> volume create -vserver vs1 -volume vol1 -space-slo thick

Example 3:

The following command configures vol1 on SVM vs1 for semi-thick provisioning:

cluster::> volume create -vserver vs1 -volume vol1 -space-slo semi-thick

5 Extensibility (Cloud)

5.1 What is Scalability in Cloud?

Scalability is a desired attribute for storage, backup, and disaster recovery of appliances, and for cloud-

based storage, backup and disaster recovery. This is one of the most valuable and predominant features

of cloud computing. Through scalability, you can scale up your data storage capacity or scale it down to

meet the demands of your growing business.

Let’s take a step back and think about database scaling.

Database cloud scaling essentially means that allowing the database to do more work than it was

originally designed to without taking a huge performance hit.

This process can generally be done through optimization or hardware expansion.

For Oracle Database, IOPS usually means a random database I/O operation, which usually is at a 70/30

read/write ratio. For example, two NetApp AFF A700 nodes are good for around 700K database IHOPS.

Let’s use the following size estimates as guidelines.

Note: As an Oracle best practice, conduct benchmarks in your environment to get accurate sizing based on your workload mix, data volumes, and date composition.

Let’s use the following database tiers as an example:

• Small: 2x cores; 4GB of RAM

• Medium: 8x cores; 32GB of RAM

• Large: 16x cores; 128GB of RAM

This guideline leaves many options to be able to scale the database tiers in many ways. As the database

is growing, the needs will change and the request to scale out for more performance will be there.

Note: These are only loose guidelines, but they're in the ballpark.

When running database servers, you want higher performance from your storage tier, you can ask

yourself: What is needed to have scalable storage tiers? Can you can mix and match with the database


tiers? With the options below you are able to add storage tiers in the cloud as mix and max storage as

needed. Let’s look below and think about our options for the storage tier:

• Small: 5K IOPS

• Medium: 15K IOPS

• Large: 50K IOPS

These options give you the ability to scale. End users can get better performance to support the high data

change rates within database applications. With careful consideration and planning, you can then know

how much I/O you can fit on a given storage system before you need another. You can count a database

as scalable as long as it could perform at a higher level (say something like 5x or 10x original design

specs) without having to redesign the database (or more directly, the table structure) itself. Porting the

database to bigger hardware, a cloud, optimizing indexes, are all viewed as good practices as long as

you don't have to mess with the original table structure.

5.2 NetApp Cloud Volumes ONTAP

Volumes ONTAP offers you the power of ONTAP software with flexible performance for EBS (I01, GP2,

ST1, and SC1) and S3 capacity options: Explore (2TB), Standard (10TB), and Premium (368TB) based

on the EC2 instance type chosen.

EBS Volumes

The most common volume type used in Amazon Web Services (AWS) environments is the general-

purpose SSD type, called gp2. The performance capability of these volumes is based mostly on the size

of the volume and is set at three IOPS per gigabyte.

The optimal EBS volume size is 3.33TB. This results in a volume with 10K IOPS, which is the maximum

available in a gp2 volume. Using larger volumes does not yield better performance and using smaller

volumes results in proportionally lower performance volumes. Going directly to the 3.33TB volume size

helps make sure that, as the environment grows, more EBS volumes are not required to improve

performance. If an ONTAP Cloud aggregate is built with multiple smaller EBS volumes, there is a risk that

the maximum of six is reached, leaving no options for adding IOPS capability to the aggregate.

As an example, consider a 1TB database with an IOPS requirement of 10K IOPS.

This database could require up to 10K extra checksum IOPS for a total of 20K IOPS. You can determine

sizing based on performance or capacity.

• 1TB ×3 IOPS per gigabyte = 3,000 IOPS

• 3.3TB ×3 IOPS per gigabyte = 10,000 IOPS

A single terabyte of storage has a baseline I/O capability of 3K IOPS, which is insufficient for the 20K

IOPS requirement. Provisioning a pair of 3.3TB volumes of storage meets the 2 0K IOPS performance

requirement but results in three-fold overprovisioning.

If the performance requirements are met by overprovisioning ordinary gp2 volumes, the cost is $300 per

month:

• 6.6TB × $0.10 per gigabyte per month = $660/month

AWS also offers an option to provision bytes and IOPS independently using a volume type called io1. The

same 1TB database could be based on four 250GB io1 volumes that deliver 5K IOPS each. This

approach precisely meets both the capacity and performance requirements, but the cost is higher than for

overprovisioning gp2 volumes.

There is also a bursting capability that is applicable to volumes smaller than 1TB.


The purpose of the burst capability is to support initial startup I/O such as booting a VM or starting a

database. The bursting limit is 3,000 IOPS, and the duration of the burst period varies based on the size

of the volume.

If the example 1TB database is based on six 250GB gp2 volumes, it would have a burst capability of 18K

IOPS. However, after approximately 40 minutes, the burst credits would be exhausted, and performance

is limited to the baseline 3K IOPS.

If I/O remains below 3K IOPS, the burst credits are gradually replenished. If the performance target of

20K IOPS for the example 1TB database does not apply 100% of the time, then relying on burst credits

might be a more affordable option.

Caution: Bursting behavior complicates performance testing with any AWS VM, including ONTAP Cloud,

with volumes less than 1TB in size because performance varies significantly depending on the available

burst credits.

5.3 Oracle RAC

1For Oracle RAC configuration, it is not practical to extend half on premises and half in the cloud. Reason

is that RAC need super-low latency network connections between nodes. There's another way to do this,

though. When creating an Oracle Database Cloud Service database deployment, you can create an

Oracle Data Guard configuration. Then you can create a sort of active-passive cluster using Oracle Data

Guard which the standby in the cloud.

Oracle Data Guard enables Oracle databases to survive disasters and data corruptions by providing a

comprehensive set of services that creates, maintains, manages, and monitors a standby database.

Oracle Data Guard maintains the standby database as a copy of the primary database. If the primary

database becomes unavailable because of a planned or an unplanned outage, you can switch the

standby database to the primary role, minimizing the downtime associated with the outage. Figure 3

shows active and passive clustering using Oracle Data Guard.

1 Oracle Automatic Storage Management Administrator's Guide. About Oracle Instances. February 2012.

https://docs.oracle.com/cd/E11882_01/server.112/e18951/asmcon.htm#OSTMG94053


Figure 3) Oracle Database with RAC and Data Guard (graphic supplied by Oracle).

6 Oracle ASM Best Practices

This section describes the following Oracle ASM configuration best practices:

• Use disk multipathing software to protect from path failure

• Set the processes initialization parameter

• Use disk labels

• Set the FAILGROUP_REPAIR_TIME or DISK_REPAIR_TIME disk group attribute appropriately

• Use ASMLib on supported systems

6.1 Use Disk Multipathing Software to Protect from Path Failure

Disk multipathing software aggregates multiple independent I/O paths into a single logical path. The path

abstraction provides I/O load balancing across HBAs and nondisruptive failovers when there is a failure in

the I/O path. You should use disk multipathing software with Oracle ASM.

1. When specifying disk names during disk group creation in Oracle ASM, use the logical device representing the single logical path. For example, when using Device Mapper on Linux 2.6, a logical device path of /dev/dm-0 might be the aggregation of physical disks /dev/sdc and /dev/sdh.

Within Oracle ASM, the ASM_DISKSTRING parameter should contain /dev/dm-* to discover the

logical device /dev/dm-0, and that logical device is necessary during disk group creation:

asm_diskstring='/dev/dm-*'


2. Create DISKGROUP DATA DISK.

'/dev/dm-0','/dev/dm-1','/dev/dm-2','/dev/dm-3';

6.2 Set the Processes Initialization Parameter

The Processes Initialization parameter affects Oracle ASM, but the default value is suitable. However, if

multiple database instances are connected to an Oracle ASM instance, you can use the formula listed in

Table 4.

Table 4) Oracle formula.

For < 10 Instances per Node For > 10 Instances per Node

Processes = 50 * (n + 1) Processes = 50 * MIN (n + 1, 11) + 10 *

MAX (n – 10, 0)

Where n is the number database instances connecting to the Oracle ASM instance.

6.3 Use Disk Labels

Disk labels ensure consistent access to disks across restarts. ASMLib is the preferred tool for disk

labeling.

6.4 Set the FAILGROUP_REPAIR_TIME or DISK_REPAIR_TIME Disk Group Attribute Appropriately

In Oracle Database, the disk group attribute FAILGROUP_REPAIR_TIME governs the amount of time an

entire failure group is offline before dropping the disks in the failure group. This defaults to 24 hours to

give enough time to repair more complex components. Change this time if the time to repair such

components will take longer than 24 hours but still less time than it would take to completely rebalance

the affected disk groups.

The DISK_REPAIR_TIME disk group attribute specifies how long a disk remains offline before Oracle

ASM drops the disk. If a disk is made available before the DISK_REPAIR_TIME parameter has expired,

the storage administrator can issue the ONLINE DISK command and Oracle ASM resynchronize the

stale data from the mirror side. The online disk operation does not restart if there is a failure of the

instance on which the disk is running. You must reissue the command manually to bring the disk online.

You can set a disk repair time attribute on your disk group to specify how long disks remain offline before

being dropped. The appropriate setting for your environment depends on how long you expect a typical

transient type of failure to persist. Set the DISK_REPAIR_TIME disk group attribute to the maximum

amount of time before a disk is definitely considered to be out of service.

6.5 Use ASMLib On Supported Systems

To improve manageability, use ASMLib on systems where it is available. ASMLib is a support library for

Oracle ASM. Although ASMLib is not required to run Oracle ASM, using ASMLib is beneficial because

ASMLib:

• Eliminates the need for every Oracle process to open a file descriptor for each Oracle ASM disk, thus improving system resource usage.

• Simplifies the management of disk device names, makes the discovery process simpler, and removes the challenge of having disks added to one node and not be known to other nodes in the cluster.

• Eliminates the affect when the mappings of disk device names change upon system restart.

Note: ASMLib is not supported on all systems.


Oracle ASM Operational Best Practices

Use the following Oracle ASM operational best practices:

• Use SYSASM for Oracle ASM authentication

• Set the rebalance power limit to the maximum limit that does not affect service levels

• Use a single command to mount multiple disk groups

• Use a single command to add or remove storage

• Check disk groups for imbalance

• Proactively mine vendor logs for disk errors

• Use the ASMCMD Utility to ease manageability of Oracle ASM

• Use Oracle ASM Configuration Assistant (ASMCA)

• Configure data layout

• Manage disaster recovery with log replay

• Use two-volume layout

Use SYSASM for Oracle ASM Authentication

The Oracle ASM instance is managed by a privileged role called SYSASM, which grants full access to

Oracle ASM disk groups. Using SYSASM enables the separation of authentication for the storage

administrator and the database administrator. By configuring a separate operating system group for

Oracle ASM authentication, you can have users that have SYSASM access to the Oracle ASM instances

and do not have SYSDBA access to the database instances.

Set Rebalance Power Limit to the Maximum Limit that Does Not Affect Service Levels

Higher Oracle ASM rebalance power limits make a rebalance operation run faster but can also affect

application service levels. Rebalancing takes longer with lower power values, but consumes fewer

processing and I/O resources that are shared by other applications, such as the database.

After performing planned maintenance, for example adding or removing storage, it is necessary to then

perform a rebalance to spread data across all of the disks.

There is a power limit associated with the rebalance. You can set a power limit to specify how many

processes perform the rebalance. If you do not want the rebalance to affect applications, then set the

power limit lower. However, if you want the rebalance to finish quickly, then set the power limit higher. To

determine the default power limit for rebalances, check the value of the ASM_POWER_LIMIT initialization

parameter in the Oracle ASM instance.

If the power clause is not specified in an ALTER DISKGROUP statement, or when rebalance is run

implicitly when you add or drop a disk, then the rebalance power defaults to the value of the

ASM_POWER_LIMIT initialization parameter. You can adjust the value of this parameter dynamically.

Use a Single Command to Mount Multiple Disk Groups

Mounting multiple disk groups in the same command ensures that disk discovery runs only one time,

which increases performance. Disk groups that are specified in the ASM_DISKGROUPS initialization

parameter are mounted automatically at Oracle ASM instance startup. To mount disk groups manually,

use the ALTER ISKGROUP…MOUNT statement and specify the ALL keyword:

ALTER DISKGROUP ALL MOUNT;

Note: The ALTER DISKGROUP…MOUNT command only works on one node. For cluster installations, run the srvctl start diskgroup -g command.


Use a Single Command to Add or Remove Storage

Oracle ASM permits you to add or remove disks from your disk storage system while the database is

operating. When you add a disk to a disk group, Oracle ASM automatically redistributes the data so that it

is evenly spread across all disks in the disk group, including the new disk. The process of redistributing

data so that it is also spread across the newly added disks is known as rebalancing. By executing storage

maintenance commands in the same command, you ensure that only one rebalance is required to incur

minimal affect to database performance.

Check Disk Groups for Imbalance

You should periodically check disk groups for imbalance. Occasionally, disk groups can become

unbalanced if certain operations fail, such as a failed rebalance operation. Periodically checking the

balance of disk groups and running a manual rebalance, if needed, ensures optimal Oracle ASM space

utilization and performance.

Use the following method to check for disk group imbalance: To check for an imbalance from an I/O

perspective, query the statistics in the V$ASM_DISK_IOSTAT view before and after running a large

SQL*Plus statement. For example, if you run a large query that performs only read I/O, the READS and

BYTES_READ columns should be approximately the same for all disks in the disk group.

Proactively Mine Vendor Logs for Disk Errors

You should proactively mine vendor logs for disk errors and have Oracle ASM move data off the bad disk

spots. Disk vendors us usually provide disk-scrubbing utilities that notify you if any part of the disk is

experiencing problems, such as a media sense error. When a problem is found, run the ASMCMD

utility REMAP command to move Oracle ASM extents from the bad spot to a good spot.

Note: This best practice is only applicable for data that is not accessed by the database or Oracle ASM instances, because in that case Oracle ASM automatically moves the extent experiencing the media sense error to a different location on the same disk. In other words, use the ASMCMD utility REMAP command to proactively move data from a bad disk spot to a good disk spot before that data is accessed by the application.

Use ASMCMD Utility to Ease Manageability of Oracle ASM

Use the ASMCMD utility to ease the manageability of day-to-day storage administration. Use the

ASMCMD utility to view and manipulate files and directories in Oracle ASM disk groups and to list the

contents of disk groups, perform searches, create, and remove directories and aliases, display space

usage. Also, use the ASMCMD utility to backup and restore the metadata of the disk groups (using the

md_backup and md_restore commands).

Note: As a best practice to create and drop Oracle ASM disk groups, use SQL*Plus, ASMCA, or Oracle Enterprise Manager.

Use Oracle ASM Configuration Assistant

2Oracle ASMCA supports installing and configuring Oracle ASM instances, disk groups, volumes, and

Oracle Automatic Storage Management Cluster File System (Oracle ACFS). In addition, you can use the

ASMCA CLI as a silent mode utility.

2 Oracle Automatic Storage Management Administrator's Guide. Oracle ASM Configuration Assistant. February 2012.

https://docs.oracle.com/cd/E11882_01/server.112/e18951/asmca.htm#OSTMG50000


Configure Data Layout

The simplest layout is to isolate the data files into one or more dedicated volumes. They must be

uncontaminated by any other file type. Use this approach to make sure that the data file volumes can be

rapidly restored with a SnapRestore operation without destroying an important redo log, control file, or

archive log.

SAN has similar requirements for data file isolation within dedicated volumes. With an operating system

such as Microsoft Windows, a single volume might contain multiple datafile LUNs, each with an NTFS file

system. With other operating systems, there is generally a logical volume manager as well. For example,

with Oracle ASM, the simplest option would be to confine disk groups to a single volume that can be

backed up and restored as a unit. If additional volumes are required for performance or capacity

management reasons, creating an additional disk group on the new volume results in easier

management.

If these guidelines are followed, you can schedule Snapshot copies directly on ONTAP without having to

perform a consistency group Snapshot copy. The reason is that Snapshot copy-optimized backups do not

require that data files be backed up at the same time.

A complication arises in situations such as an ASM disk group that is distributed across volumes. In these

cases, a cg-snapshot must be performed to make sure that the ASM metadata is consistent across all

constituent volumes.

Caution: Verify that the ASM spfile and passwd files are not in the disk group hosting the data files.

This interferes with the ability to selectively restore datafiles and only datafiles.

Manage Disaster Recovery with Log Replay

The replication procedures for an Oracle database are essentially the same as the backup procedures.

The primary requirement is that the snapshots that constitute a recoverable backup must be replicated to

the remote storage system. A recoverable backup can be created with the hot backup process or by

leveraging snapshot-optimized backups.

The most important requirement is isolation of the data files into one or more dedicated volumes. They

must be uncontaminated by any other file type. The reason is to make sure that datafile replication is

wholly independent of replication of other data types such as archive logs.

Assuming the data files are encapsulated into dedicated volumes, the next question is how to manage the

redo logs, archive logs, and control files. The simplest approach is to place all those data types into a

single volume. The benefit of this approach is that replicated redo logs, archive logs, and control files are

perfectly synchronized. There is no requirement for incomplete recovery or using a backup control file,

although it might be desirable to also script creation of backup control files for other potential recovery

scenarios.

Use Two-Volume Layout

Figure 4 shows the two-volume layout, which is the most common approach. From a DBA perspective, it

might seem unusual to collocate all.


Figure 4) Two-volume layout.

Where to Find Additional Information

To learn more about the information that is described in this document, review the following documents

and/or websites:

• ONTAP 9 Documentation Center http://docs.netapp.com/ontap-9/index.jsp

• ONTAP and OnCommand System Manager Documentation Resources page https://www.netapp.com/us/documentation/ontap-and-oncommand-system-manager.aspx

• NetApp Product Documentation https://docs.netapp.com

• Oracle Automatic Storage Management Administrator's Guide https://docs.oracle.com/cd/E11882_01/server.112/e18951/toc.htm

Version History

Version Date Document Version History

Version 1.1 July 2019 Revisions to the deployment procedures and license requirements.

Version 1.0 March 2019 Initial release.

http://docs.netapp.com/ontap-9/index.jsp

https://www.netapp.com/us/documentation/ontap-and-oncommand-system-manager.aspx

https://docs.netapp.com/

https://docs.oracle.com/cd/E11882_01/server.112/e18951/toc.htm


Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer’s installation in accordance with published specifications.

Copyright Information

Copyright © 2019 NetApp, Inc. All Rights Reserved. Printed in the U.S. No part of this document covered by copyright may be reproduced in any form or by any means—graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—without prior written permission of the copyright owner.

Software derived from copyrighted NetApp material is subject to the following license and disclaimer:

THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp.

The product described in this manual may be protected by one or more U.S. patents, foreign patents, or pending applications.

Data contained herein pertains to a commercial item (as defined in FAR 2.101) and is proprietary to NetApp, Inc. The U.S. Government has a non-exclusive, non-transferrable, non-sublicensable, worldwide, limited irrevocable license to use the Data only in connection with and in support of the U.S. Government contract under which the Data was delivered. Except as provided herein, the Data may not be used, disclosed, reproduced, modified, performed, or displayed without the prior written approval of NetApp, Inc. United States Government license rights for the Department of Defense are limited to those rights identified in DFARS clause 252.227-7015(b).

Trademark Information

NETAPP, the NETAPP logo, and the marks listed at http://www.netapp.com/TM are trademarks of NetApp, Inc. Other company and product names may be trademarks of their respective owners.

TR-4760-0719

http://mysupport.netapp.com/matrix

http://www.netapp.com/TM

solution delivery guide - netapp.com · • move logical interfaces (lifs) along with volumes for...

Documents