bestpracticesforusing infoscaleenterprisewith oraclerac
TRANSCRIPT
Best Practices for UsingInfoScale Enterprise withOracle RAC
Last updated: 2019-09-17
Legal NoticeCopyright Ā© 2019 Veritas Technologies LLC. All rights reserved.
Veritas and the Veritas Logo are trademarks or registered trademarks of Veritas TechnologiesLLC or its affiliates in the U.S. and other countries. Other names may be trademarks of theirrespective owners.
This product may contain third-party software for which Veritas is required to provide attributionto the third-party (āThird-Party Programsā). Some of the Third-Party Programs are availableunder open source or free software licenses. The License Agreement accompanying theSoftware does not alter any rights or obligations you may have under those open source orfree software licenses. Refer to the third-party legal notices document accompanying thisVeritas product or available at:
https://www.veritas.com/about/legal/license-agreements
The product described in this document is distributed under licenses restricting its use, copying,distribution, and decompilation/reverse engineering. No part of this document may bereproduced in any form by anymeans without prior written authorization of Veritas TechnologiesLLC and its licensors, if any.
THE DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIEDCONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIEDWARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE ORNON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCHDISCLAIMERS ARE HELD TO BE LEGALLY INVALID. VERITAS TECHNOLOGIES LLCSHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES INCONNECTION WITH THE FURNISHING, PERFORMANCE, OR USE OF THISDOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION ISSUBJECT TO CHANGE WITHOUT NOTICE.
The Licensed Software and Documentation are deemed to be commercial computer softwareas defined in FAR 12.212 and subject to restricted rights as defined in FAR Section 52.227-19"Commercial Computer Software - Restricted Rights" and DFARS 227.7202, et seq."Commercial Computer Software and Commercial Computer Software Documentation," asapplicable, and any successor regulations, whether delivered by Veritas as on premises orhosted services. Any use, modification, reproduction release, performance, display or disclosureof the Licensed Software and Documentation by the U.S. Government shall be solely inaccordance with the terms of this Agreement.
Veritas Technologies LLC500 E Middlefield RoadMountain View, CA 94043
http://www.veritas.com
Technical SupportTechnical Support maintains support centers globally. All support services will be deliveredin accordance with your support agreement and the then-current enterprise technical supportpolicies. For information about our support offerings and how to contact Technical Support,visit our website:
https://www.veritas.com/support
You can manage your Veritas account information at the following URL:
https://my.veritas.com
If you have questions regarding an existing support agreement, please email the supportagreement administration team for your region as follows:
[email protected] (except Japan)
DocumentationMake sure that you have the current version of the documentation. Each document displaysthe date of the last update on page 2. The latest documentation is available on the Veritaswebsite:
https://sort.veritas.com/documents
Documentation feedbackYour feedback is important to us. Suggest improvements or report errors or omissions to thedocumentation. Include the document title, document version, chapter title, and section titleof the text on which you are reporting. Send feedback to:
You can also see documentation information or ask a question on the Veritas community site:
http://www.veritas.com/community/
Veritas Services and Operations Readiness Tools (SORT)Veritas Services andOperations Readiness Tools (SORT) is a website that provides informationand tools to automate and simplify certain time-consuming administrative tasks. Dependingon the product, SORT helps you prepare for installations and upgrades, identify risks in yourdatacenters, and improve operational efficiency. To see what services and tools SORT providesfor your product, see the data sheet:
https://sort.veritas.com/data/support/SORT_Data_Sheet.pdf
Best practices for usingInfoScale Enterprise withOracle RAC
This document includes the following topics:
ā Introduction
ā Intended audience
ā Acronyms used in this document
ā InfoScale support for Oracle RAC
ā Benefits of using InfoScale Enterprise with Oracle RAC configurations
ā Typical InfoScale Enterprise configuration
ā System configuration recommendations
ā InfoScale Enterprise network configurations in Oracle RAC environments
ā Storage recommendations
ā Database storage recommendations
ā Volume design recommendations
ā File system recommendations
ā I/O fencing recommendations
ā Cluster management recommendations
ā Disaster recovery considerations
ā References
IntroductionThis document is a compilation of the best practices that are recommended for thedeployment and administration of Veritas InfoScale Enterprise in OracleĀ® RACenvironments. The various configuration practices discussed in this paper includesystem configuration, network configuration, storage configuration, Oracle databaseconfiguration, disaster recovery (DR) configuration, installation, and upgrade.
Intended audienceThis document is intended to help system administrators, storage administrators,and database administrators deploy and administer InfoScale Enterprise with OracleRAC in production environments.
Acronyms used in this documentTable 1-1 List of acronyms used
DefinitionAcronym
Automatic Storage ManagementASM
Array Support LibraryASL
Cache FusionCF
Cluster File SystemCFS
Cluster Ready Services or Oracle ClusterwareCRS
Cluster Volume ManagerCVM
Distributed Lock ManagerDLM
Dynamic Multi-PathingDMP
Enclosure Based NamingEBN
Fast Application NotificationFAN
Fibre ChannelFC
Group Membership Services/Atomic BroadcastGAB
5Best practices for using InfoScale Enterprise with Oracle RACIntroduction
Table 1-1 List of acronyms used (continued)
DefinitionAcronym
Global Cluster OptionGCO
Global Cache ServiceGCS
Global Enqueue ServiceGES
Hardware Compatibility ListHCL
Internet ProtocolIP
Low Latency TransportLLT
LLT MultiplexerLMX
Oracle Cluster RegistryOCR
Oracle Cluster Synchronization Services DaemonOCSSD
Oracle Disk ManagerODM
Persistent Group ReservationPGR
Real Application ClusterRAC
Small Computer System InterfaceSCSI
Storage FoundationSF
Transmission Control ProtocolTCP
Veritas Cluster ServerVCS
VCS Interprocess CommunicationVCSIPC
Veritas Cluster Server Membership ModuleVCSMM
Virtual Internet Protocol (address)VIP
Veritas I/O FencingVXFEN
Veritas Volume ReplicatorVVR
Veritas Volume ManagerVxVM
6Best practices for using InfoScale Enterprise with Oracle RACAcronyms used in this document
InfoScale support for Oracle RACVeritas InfoScale Enterprise leverages proprietary storage management and highavailability technologies to enable robust, manageable, and scalable deploymentof Oracle RAC on UNIX platforms. The solution stack comprises the followingcomponents: VCS, CVM, CFS, VxVM, VxFS, Veritas Oracle Disk Manager(VRTSodm), and Veritas Oracle Real Application Cluster support (VRTSdbac). TheCFS technology that provides the dual advantage of easy file system managementand the use of familiar operating system tools and utilities to manage databases.
Benefits of using InfoScale Enterprise with OracleRAC configurations
Using InfoScale Enterprise with Oracle RAC provides the following benefits:
ā Support for file system-based management. InfoScale Enterprise provides ageneric clustered file system technology for storing and managing Oracle datafiles as well as other application data.
ā The Flexible Storage Sharing (FSS) feature enables the creation of a sharedstorage infrastructure without a Storage Area Network (SAN).
ā Faster performance and reduced costs per I/O per second (IOPS) using SmartIO.SmartIO supports read caching for the VxFS file systems that are mounted onVxVM volumes, in several caching modes and configurations. SmartIO alsosupports block-level read caching for applications running on VxVM volumes.
ā Support for a standardized approach toward application and databasemanagement. Administrators can apply their expertise of technologies towardadministering InfoScale Enterprise.
ā Increased availability and performance using Dynamic Multi-Pathing (DMP).DMP provides wide storage array support for protection from failures andperformance bottlenecks in the Host Bus Adapters (HBA), Storage Area Network(SAN) switches, and storage arrays.
ā Simplified administration and monitoring of multiple InfoScale clusters usingVeritas InfoScale Operations Manager.
ā VCS OEM plug-in provides a way to monitor InfoScale resources from the OEMconsole.For more information, see the Veritas InfoScale Storage and AvailabilityManagement for Oracle Databases guide.
ā Improved file system access times using Oracle Disk Manager (ODM).
7Best practices for using InfoScale Enterprise with Oracle RACInfoScale support for Oracle RAC
ā Ability to configure Oracle ASM disk groups over CVM volumes to take advantageof DMP.
ā Enhanced scalability and availability with access to multiple Oracle RACinstances per database in a cluster.
ā Support for backup and recovery solutions using volume-level and filesystem-level snapshot technologies, StorageCheckpoints, andDatabase StorageCheckpoints.For more information, refer to the Veritas InfoScale Storage and AvailabilityManagement for Oracle Databases guide.
ā Support for space optimization using periodic deduplication in a file system toeliminate duplicate data without any continuous cost.For more information, see the Storage Foundation administrator's guides.
ā Ability to fail over applications with minimum downtime using VCS and CFS.
ā Prevention of data corruption in split-brain scenarios with robust SCSI-3 PGRbased I/O fencing or Coordination Point Server-based I/O fencing. The preferredfencing feature also enables you to specify how the fencing driver determinesthe surviving subcluster.To understand fencing how it helps prevent a split-brain scenario, refer to thearticle at:https://www.veritas.com/support/en_US/article.100021261
ā Support for sharing application data, in addition to Oracle database files, acrossnodes.
ā Support for policy-managed databases in Oracle RAC 11g R2 and later versions.
ā Support for container and pluggable databases in Oracle RAC 12c and laterversions.
ā Fast DR with minimal downtime and interruption to users. Users can transitionfrom a local high availability site to a wide-area DR environment with primaryand secondary sites. If a site fails, clients that are attached to the failed site canreconnect to a surviving site and resume access to the shared database.
ā (Solaris only) Support for configuring and administering multiple non-globalzones with high availability to Oracle RAC instances inside the non-global zone.
ā Verification of DR configuration using fire drill technology without affectingproduction systems.
ā Support for campus clusters with the following capabilities:
ā Consistent detach with Site Awareness
ā Site aware reads with VxVM mirroring
8Best practices for using InfoScale Enterprise with Oracle RACBenefits of using InfoScale Enterprise with Oracle RAC configurations
ā Monitoring of Oracle resources
ā Protection against split-brain scenarios
Typical InfoScale Enterprise configurationThis section illustrates the physical view of a basic two-node InfoScale cluster.
9Best practices for using InfoScale Enterprise with Oracle RACTypical InfoScale Enterprise configuration
Figure 1-1 Basic two-node InfoScale configuration
NIC
1HBA1 HBA2
Clusternode 2
Physical view of 2-node SFRACCluster
SAN
FCSw
itch
FCSw
itch
CoordinatorLUN 1
CoordinatorLUN 2 Coordinator
LUN 3DataLUNs
DataLUNs
DataLUNsStorage
ArrayStorageArray
StorageArray
Ethern
et
Switch
Ethern
et
Switch
Ethern
et
Switch
LAN
NIC
1 NIC
2HBA2
HBA1
Clusternode 1
NIC
2Client
Applications
Public Links (GigE)Private interconnects (GigE)FC Cables:
Legends
10Best practices for using InfoScale Enterprise with Oracle RACTypical InfoScale Enterprise configuration
System configuration recommendationsThe following recommendations provide guidelines for consistent configuration ofInfoScale cluster nodes in an Oracle RAC environment:
Oracle RAC requires that all nodes in the cluster have the samearchitecture. Clusters with mixed architectures, such as x86, SPARC,IA/PA, are not supported.
Architecture
The servers in the InfoScale cluster should have identical CPU speedsand memory.
CPU andmemoryconsiderations
The minimum requirement for Oracle RAC 12c is 8 GB of swap space.
The operating system requirement for minimum swap is two times thesize of RAM for swap space.
Between the minimum requirements of Oracle RAC and the operatingsystem, make sure that you meet the minimum requirement that is higher.For example, if the operating system requirement for minimum swapspace computes to 5 GB on your Oracle RAC 11g systems, make surethat you meet the minimum swap space requirement of Oracle RAC, thatis 8 GB.
Swap spacerequirements
The servers should have identical operating system versions and patchesOS version andpatches
All nodes in the cluster must have the same InfoScale version.InfoScale version
Use the latest Oracle patchset. The latest information on supported Oracledatabase versions is available on SORT at:
https://sort.veritas.com/sclcentral/database
InfoScale Enterprise supports different versions of the Oracle databasein a physical cluster. For example, you can install the Oracle 11g R2 andthe Oracle 12c database in the same cluster with Oracle 12c Clusterware.
Oracle version
All nodes in the cluster must have the same system time. The NetworkTime Protocol (NTP) daemon is recommended for synchronizing the timesettings on each node.
System time oncluster nodes
InfoScale Enterprise network configurations inOracle RAC environments
The network connectivity requirements for InfoScale Enterprise and Oracle RACare classified as follows:
11Best practices for using InfoScale Enterprise with Oracle RACSystem configuration recommendations
ā Private network for communication within the cluster
ā Public network for communication with the external world
Figure 1-2 Communication layers between two InfoScale nodes
RAC IPC
VCSMM
OCSSDPrivate IPfailover
GABVCSMM GAB
Private IP
Switch 1
LLT LLT
RAC IPC
Private IPfailover
Private IP
OCSSD
Switch 2
Node 1 Node 2Oracle ClusterwarecommunicationOracle RAC databasecommunication
The InfoScale Enterprise and Oracle RAC components communicate within thecluster over the private network.
The communications can be categorized as follows:
The cluster communication constitutes the communication between thevarious components of InfoScale EnterpriseāVCS, CFS and CVM.
This communication is multiplexed over multiple LLT links specified inthe LLT configuration file.
Clustercommunication
TheOracle Clusterware communication constitutes theOracle Clusterwaretraffic, which includes Cluster Synchronization Service (heartbeat) traffic.
The Oracle Clusterware communication uses a single logical connectionover a single subnet. This network needs to be protected against linkfailures to prevent split-brain scenarios.
OracleClusterwarecommunication
12Best practices for using InfoScale Enterprise with Oracle RACInfoScale Enterprise network configurations in Oracle RAC environments
The Oracle RAC database communication constitutes the RAC instancepeer-to-peer communication for Global Cache Service (GCS) and GlobalEnqueue Service (GES), earlier referred to as Cache Fusion (CF) andDistributed Lock Manager (DLM).
The bandwidth requirements for this communication may vary dependingon the number of databases configured. The network needs to beprotected against link failures to ensure uninterrupted database service.
Oracle RACdatabasecommunication(per database)
Network configuration recommendationsThe following practices are recommended for a resilient network setup:
ā Configure the private cluster interconnect over multiple dedicated gigabit Ethernetlinks. All single point of failures such as network interface cards (NICs), switches,and interconnects should be eliminated.In case of FSS environments, Veritas recommends that you use at least 10GbEethernet links for private interconnect.
ā The NICs used for the private cluster interconnect should have the samecharacteristics regarding speed, MTU, and full duplex on all nodes. Do not allowthe NICs and switch ports to auto-negotiate speed.
ā Configure non-routable IP addresses for private cluster interconnects.
ā The LLT peer inactivity timeout value should be set based on service availabilityrequirements and the propagation delay between the cluster nodes in case ofcampus cluster setup. This value indicates the interval after which InfoScale onone node declares the other node in the cluster dead, if there is no networkcommunication (heartbeat) from that node.The default value for the CSS miss-count in case of InfoScale Enterprise is 600seconds. The value of this parameter is much higher than the LLT peer inactivitytimeout so that the two clusterwares, VCS and Oracle Clusterware, do notinterfere with each otherās decisions on which nodes should remain in the clusterin the event of network split-brain. Veritas I/O fencing is allowed to decide onthe surviving nodes first, followed by Oracle Clusterware. The CSS miss-countvalue indicates the amount of time Oracle Clusterware waits before evictinganother node from the cluster, when it fails to respond across the interconnect.For more information, see the Oracle Metalink document: 782148.1
Public network configuration for Oracle RACIdentify separate public virtual IP addresses for each node in the cluster. Oraclerequires one public virtual IP address for the Oracle listener process on each node.Public virtual IP addresses are used by client applications to connect to the Oracledatabase and help mitigate TCP/IP timeout delays.
13Best practices for using InfoScale Enterprise with Oracle RACInfoScale Enterprise network configurations in Oracle RAC environments
Additionally, for Oracle 12c and later versions, you need a Single Client AccessName (SCAN) registered in Enterprise DNS that resolves to three IP addresses(recommended). Oracle Clusterware/Grid Infrastructure manages the virtual IPaddresses.
Private network configuration for Oracle RACOracle RAC requires a minimum of one private IP address on each node for OracleClusterware heartbeat.
(Linux only) You must use UDP IPC for the database cache fusion traffic. TheOracle UDP IPC protocol requires an IP address. Depending on your deploymentneeds, this IP address may be a dedicated IP address or one that is shared withOracle Clusterware.
For Oracle RAC 11g and later, you must use UDP IPC for the database cachefusion traffic.
(AIX, Solaris only) Veritas recommends using multiple private interconnects for loadbalancing the cache fusion traffic.
Note: The private IP addresses of all nodes that are on the same physical networkmust be in the same IP subnet.
The following practices provide a resilient private network setup:
ā In an InfoScale cluster, the Oracle Clusterware heartbeat linkmust be configuredas an LLT link. If Oracle Clusterware and LLT use different links for theircommunication, then the membership change between VCS and OracleClusterware is not coordinated correctly. For example, if only the OracleClusterware links are down, Oracle Clusterware kills one set of nodes after theexpiry of the css-misscount interval and initiates the Oracle Clusterware anddatabase recovery, even before CVM and CFS detect the node failures. Thisuncoordinated recovery may cause data corruption.
ā Oracle Clusterware interconnects need to be protected against NIC failures andlink failures. An option to support this need is to configure the Oracle Clusterwareinterconnects over bonded NIC interfaces. IP failover solutions such as IPMPcan also provide the first level of protection.
Note:With Oracle RAC 11.2.0.2 and later, InfoScale no longer supports theuse of the PrivNIC and the MultiPrivNIC agents to manage cluster interconnects.
For later Oracle RAC versions, Veritas recommends the use of alternativesolutions such as bonded NIC interfaces or Oracle High Availability IP (HAIP).
14Best practices for using InfoScale Enterprise with Oracle RACInfoScale Enterprise network configurations in Oracle RAC environments
ā Configure Oracle Cache Fusion traffic to take place through the private network.Veritas also recommends that all UDP cache-fusion links be LLT links.Oracle database clients use the public network for database services. Wheneverthere is a node failure or network failure, the client fails over the connection, forboth existing and new connections, to the surviving node in the cluster withwhich it is able to connect. Client failover occurs as a result of Oracle FastApplication Notification, VIP failover and client connection TCP timeout. It isstrongly recommended not to send Oracle Cache Fusion traffic through thepublic network.
ā Use NIC bonding to provide redundancy for public networks so that Oracle canfail over virtual IP addresses if there is a public link failure.
High availability for Oracle RAC private networkTable 1-2 High availability solutions that you may adopt for your Oracle
RAC private network
DescriptionOptions
If Oracle Clusterware interconnects are configured over IPMP, all theNICs that are configured under LLT must be configured under theIPMP group.
(Solaris only) UsingIPMP for OracleClusterware
Use a native NIC bonding solution to provide redundancy, in case ofNIC failures.
Make sure that a link configured under a aggregated link or NIC bondis not configured as a separate LLT link.
When LLT is configured over a bonded interface, do one of thefollowing steps to prevent GAB from reporting jeopardy membership:
ā Configure another NIC under LLT in addition to the bonded NIC.ā Add the following line in the /etc/llttab file:
set-dbg-minlinks 2
Using linkaggregation/ NICbonding for OracleClusterware
In Oracle RAC 11.2.0.2 and later, Oracle provides the High AvailabilityIP (HAIP) feature to support IP address failover. The purpose of HAIPis to perform load balancing across all active interconnect interfacesand to fail over the existing non-responsive interfaces to availableinterfaces. HAIP has the ability to activate a maximum of four privateinterconnect connections. These private network adapters can beconfigured during the installation of Oracle Grid Infrastructure, or afterthe installation using the oifcfg utility.
Using HAIP
15Best practices for using InfoScale Enterprise with Oracle RACInfoScale Enterprise network configurations in Oracle RAC environments
Storage recommendationsThe following practices are recommended for efficient functioning of the InfoScalestack in an Oracle RAC enviornment:
ā Distribute files between local and shared storage in the following manner foroptimal performance benefits:
Place the following binaries on local disks:ā InfoScale Enterprise binaries for Oracle RACā Oracle Clusterware binariesā Oracle binaries
Placing the Oracle binaries on local disks enables phased upgrade ofthe cluster.
Localstorage
Place the following binaries on shared disks:ā InfoScale Enterprise fencing coordinator disksā InfoScale Enterprise database storage management repositoryā Oracle RAC databaseā Oracle Clusterware registry (OCR) and voting diskā Database datafiles
Store the Oracle database files on CFS rather than on raw device orCVM raw device for easier management. Create separate clusteredfile systems for each Oracle database. Keeping Oracle database homeon separate mount points enables you to unmount the database formaintenance purposes without affecting other databases.If you plan to store the Oracle database on ASM, configure the ASMdisk groups over CVM volumes to take advantage of dynamicmulti-pathing.
ā Database recovery data (archive, flash recovery)Place redo and archived logs on CFS rather than on local file systems.
ā OCR and vote disksPlace the OCR and vote disks on CVM raw volumes to provide highavailability to vote disks with dynamic multi-pathing. Mirror the CVMvolumes that contain OCR and vote devices for redundancy.
Sharedstorage
ā CVM provides native naming (cXtXdX) as well as enclosure-based naming(EBN).Use enclosure-based naming for easy administration of storage. Enclosure-basednaming guarantees that the same name is given to a shared LUN on all thenodes, irrespective of the operating system name for the LUN.
16Best practices for using InfoScale Enterprise with Oracle RACStorage recommendations
Shared storage optionsInfoScale provides the following options for shared storage in Oracle RACenvironments:
ā CVMCVM provides native naming (OSN) as well as enclosure-based naming (EBN).Use enclosure-based naming for easy administration of storage. Enclosure-basednaming guarantees that the same name is given to a shared LUN on all thenodes, irrespective of the operating system name for the LUN.
ā CFSIf FSS is used, it should be mounted as CFS.
ā Local storageIf FSS is used, local storage can be used as shared storage. The local storagecan be in the form of Direct Attached Storage (DAS) or internal disk drives. Also,Veritas recommends that you use at least 10GbE ethernet links for privateinterconnect in FSS configurations.For details on FSS, refer to the Storage Foundation Cluster File System HighAvailability Administrator's Guide.
ā Oracle ASM over CVMASM provides storage for data files, control files, online and archive redo logfiles, and backup files only. It does not support Oracle binaries, trace files, alertlogs, export files, tar files, core files, Oracle cluster registry devices (OCR) andquorum device (voting disk), application binaries, and data.
Disk array requirementsMake sure you meet the following requirements for disk arrays:
ā InfoScale requires that coordinator disks for I/O fencing as well as data disksare SCSI-3 PR compatible.
ā Confirm the compatibility of your storage array with InfoScale Enterprise byreviewing the current compatibility list at:https://sort.veritas.com/hclcentral/diskarray
ā Confirm that the appropriate Array Support Library (ASL) is installed on all thecluster nodes by reviewing the array-specific modules list at:https://sort.veritas.com/asl
17Best practices for using InfoScale Enterprise with Oracle RACStorage recommendations
Database storage recommendationsImplement the following recommendations to ensure better performance andavailability of storage:
ā Use multiple storage arrays, if possible, to ensure protection against arrayfailures. The minimum recommended configuration is to have two HBAs foreach host and two switches.
ā Design the storage layout keeping in mind performance and high availabilityrequirements. Use technologies such as striping and mirroring.
ā Use appropriate stripe width and depth to optimize I/O performance.
ā Use SCSI-3 persistent reservations (PR) compliant storage.
ā Provide multiple access paths to disks with HBA/switch combinations to allowDMP to provide high availability against storage link failures and to provide loadbalancing.
ā Use Database Dynamic Storage Tiering (DBDST) to optimize the storage cost.Using DBDST, less frequently used data can bemoved to slower, less expensivedisks. This also permits frequently accessed data to be stored on faster disksfor quicker retrieval.
ā For Oracle 12c R2 or later, you must store the OCR and voting files on OracleASM disk groups. You must create the Oracle ASM disks group on the raw CVMvolumes during Oracle Grid installation.
Volume design recommendationsThe following recommendations ensure optimal layout of VxVM/CVM volumes:
ā Mirror the volumes across two or more storage arrays, if using VxVM mirrors.Keep the Fast Mirror Resync regionsize equal to the database block size toreduce the copy-on-write (COW) overhead. Reducing the regionsize increasesthe amount of Cache Object allocations leading to performance overheads.
ā Distribute the I/O load uniformly on all Cache Objects when you create multipleCache Objects.
ā Implement zoning on SAN switch to control access to shared storage. Be awarethat physical disks may be shared by multiple servers or applications and musttherefore be protected from accidental access.
ā Choose DMP I/O policy based on the storage network topology and theapplication I/O pattern.
ā Exploit thin provisioning for better return on investment.
18Best practices for using InfoScale Enterprise with Oracle RACDatabase storage recommendations
ā For Oracle RAC:
ā Separate the Oracle recovery structures from the database files to ensurehigh availability when you design placement policies.
ā Separate redo logs and place them on the fastest storage (for example,RAID 1+ 0) for better performance.
ā Use "third-mirror break-off" snapshots for cloning the Oracle log volumes.Do not create Oracle log volumes on a Space-Optimized (SO) snapshot.
ā Create as many Cache Objects (CO) as possible when you useSpace-Optimized (SO) snapshots for Oracle data volumes.
ā For ASM:
ā Use VxVM mirrored volumes with dynamic multi-pathing with externalredundancy to ensure high availability.
ā Do not use the VxVM volumes that are used by ASM for any other purposesuch as creation of file systems.
File system recommendationsThe following recommendations ensure an optimal file system design for databases:
ā Create separate file systems for Oracle binaries, data, redo logs, and archivelogs. This ensures that recovery data is available if you encounter problemswith database data files storage.
ā Always place archived logs on CFS file systems rather then local file systems.
ā If using VxVM mirroring, use ODM with CFS for better performance. ODM withSmartSync enables faster recovery of mirrored volumes using Oracle resilvering.
ā Do not enable ODM when databases are installed on ASM.
I/O fencing recommendationsIn an InfoScale cluster, Veritas fencing handles the split-brain scenarios even thoughOracle Clusterware provides I/O fencing with OPROCD. It is therefore mandatoryto enable fencing in InfoScale Enterprise.
The following practices are recommended:
ā Configure three LUNs as coordinator disks. Verify that the LUN is SCSI-3compliant using the vxfentsthdw (1M) utility.
ā Provide high availability to coordinator disks using DMP and multiple physicalaccess paths.
19Best practices for using InfoScale Enterprise with Oracle RACFile system recommendations
ā In the case of InfoScale Enterprise in a stretch cluster, place the third coordinatordisk at a third location to ensure cluster availability during split-brain scenariosinvolving site failure.You can use iSCSI devices as coordinator disks for I/O fencing.
Note: InfoScale Enterprise supports iSCSI devices for I/O fencing only whenthe disk policy is set to DMP. If you use iSCSI devices, make sure that the/etc/vxfenmode file has the disk policy set to DMP.
In FSS configurations, use CP Server-based fencing.
Cluster management recommendationsTable 1-3 List of agents supported in InfoScale installations for effective
cluster management
DescriptionAgent
Oracle database management
The VCS Oracle agent is recommended for managing Oracledatabases. VCS controls the Oracle database in this configuration.The configuration without VCS Oracle agent may be used only in asingle database setup.
VCS agent for Oracle
Oracle ASM disk group monitoring
The ASMDG agent monitors the Oracle ASM disk groups.
VCS agent forASMDG
Volume management
An InfoScale Enterprise configuration automatically configures theCVMCluster resource and the CVMVxconfigd resource. You mustconfigure the CVMVolDg agent for each shared disk group.
VCS agents for CVM
File system management
If the database uses CFS, configure the CFSMount agent for eachvolume in the disk group.
VCS agents for CFS
20Best practices for using InfoScale Enterprise with Oracle RACCluster management recommendations
Table 1-3 List of agents supported in InfoScale installations for effectivecluster management (continued)
DescriptionAgent
Oracle Clusterware management
The CSSD agent starts, stops, and monitors Oracle Clusterware. Itensures that the OCR, the voting disk and the private IP addressresources required by Oracle Clusterware are online before OracleClusterware starts.
Note: It is mandatory to use CSSD agent in InfoScale Enterpriseinstallations in Oracle RAC environments to ensure adequatehandling of inter-dependencies and thereby prevent the prematurestartup of Oracle Clusterware.
CSSD agent
Oracle Clusterware resource monitoring.
The CRSResource agent monitors the Oracle Clusterware resourcessuch as the virtual IP address, listener, and the Oracle databaseinstance. It provides an alternative mechanism for monitoring theOracle database in the absence of the VCS Oracle agent. It is usefulin scenarios where the database is not managed by VCS and theapplications need to be started using VCS after Oracle Clusterwarestarts the database.
CRSResource agent
Note: Make sure that you do not configure the following Oracle resources underVCS: SCAN IP, virtual IP, and listener.
Disaster recovery considerationsInfoScale Enterprise provides various DR configurations, such as campus clustersand global clusters, for multi-site clusters. In multi-site clusters, the nodes can beplaced in different parts of a building, in separate buildings, or in separate cities.The distance between the nodes depends on the type of disaster from whichprotection is needed and on the technology used to replicate data. InfoScaleEnterprise supports various replication technologies for data replication.
To protect clusters against outages caused by disasters, the cluster componentsmust be geographically separated.
21Best practices for using InfoScale Enterprise with Oracle RACDisaster recovery considerations
Campus clustersA campus cluster is also known as a stretch cluster or remote mirror configuration.In a campus cluster, the hosts and storage of a cluster span multiple sites separatedby a few miles.
Keep in mind the following best practices when you configure a InfoScale Enterprisecampus cluster:
ā Campus cluster sites are typically connected using a redundant high-capacitynetwork that provides access to storage and private network communicationbetween the cluster nodes. A single DWDM link can be used for both storageand private network communication.
ā Tag the disks or enclosures that belong to a site with the corresponding VxVMsite name. VxVM allocates storage from the correct site when creating or resizinga volume and when changing a volumeās layout if the disks in the VxVM diskgroup that contain the volume are tagged with the site name.
ā Tag each host with the corresponding VxVM site name. Make sure the readpolicy of the volumes is set to SITEREAD. This setting ensures that the reads onthe volumes are satisfied from the local siteās plex.
ā Turn on the allsites attribute for all volumes that have data required by theapplication, to make sure they are evenly mirrored. Each site must have at leastone mirror of all volumes hosting application data, including the FlashSnap logvolume.
ā Turn on the siteconsistent attribute for the disk groups and the volumes toenable site-aware plex detaches. Snapshot volumes need not be site-consistent.
ā In the case of a two-site campus cluster, place the third coordinator disk on thethird site. You may use iSCSI disk on the third site as an alternative to DarkFiber connected FC-SAN or a Coordination Point Server (CPS), as a thirdcoordination point.
ā Make sure that a DCO log version 20 or higher is attached to the volumes toenable Fast Resync operations.
ā Set the CVM disk detach policy as global or local for all disk groups containingdata volumes.For OCR and voting disk, it is recommended to have the disk group policy aslocal detach policy.
22Best practices for using InfoScale Enterprise with Oracle RACDisaster recovery considerations
Figure 1-3 Four-node InfoScale Enterprise campus cluster
LANLAN
Dark Fibre
SITE 1 SITE 2
SITE 3
Physical view of 4-node SFRACCampus Cluster
NIC
1 NIC
2HBA2
HBA1
Clusternode 1
NIC
1 NIC
2HBA2
HBA1
Clusternode 2
NIC
1 NIC
2HBA2
HBA1
Clusternode 3
NIC
1 NIC
2HBA2
HBA1
Clusternode 4
ClientApplications
ClientApplications
Ethern
et
Switch
Ethern
et
Switch
Ethern
et
Switch
Ethern
et
Switch
Ethern
et
Switch
Ethern
et
Switch
FCSw
itchFC
Switch
FCSw
itch
FCSw
itch
CoordinatorLUN 1 Data
LUNsStorage Array
CoordinatorLUN 2 Data
LUNsStorage Array
CoordinatorLUN 3 Data
LUNsStorage Array
FCSw
itch FCSw
itch
DWDM
Private Interconnects(GigE)
San 2 Connections
Legends
Dark FiberConnectionsSan 1 Connections
Public Links (GigE)
SAN SAN
23Best practices for using InfoScale Enterprise with Oracle RACDisaster recovery considerations
Global clustersGlobal clusters provide the ability to fail over applications between geographicallydistributed clusters when a disaster occurs.
Global clustering involves two steps:
1. Replication of data between the sites
2. Migration of the application when disaster occurs
The following aspects need to be considered when you design a DR solution:
ā The amount of data lost in the event of a disasterāRecovery Point Objective(RPO)
ā The acceptable recovery time after the disasterāRecovery Time Objective(RTO)
Data replication considerationsWhen you choose a replication solution, one of the important factors that you needto consider is the required level of data throughput. Data throughput is the rate atwhich the application is expected to write data. The impact of write operations onreplication are of more significance than that of the read operations.
In addition to the business needs discussed earlier, the following factors need tobe considered while choosing the replication options:
ā Mode of replication
ā Network bandwidth
ā Network latency between the two sites
ā Ability of the remote site to keep up with the data changes at the first site
Data replication optionsInfoScale supports various hardware and software replication technologies.
ā Hitachi True Copyā IBM Metro Mirrorā IBM SVCā EMC Mirrorview
Examples of hardwarereplication options
ā Volume Replicator (VVR)ā Oracle Data Guard
Examples of softwarereplication options
24Best practices for using InfoScale Enterprise with Oracle RACDisaster recovery considerations
ReferencesFor details on using InfoScale Enterprise with Oracle RAC, refer to the followingdocuments:
ā Storage Foundation for Oracle RAC Administrator's Guidehttps://www.veritas.com/content/support/en_US/doc/129684178-129684183-1
ā Storage Foundation for Oracle RAC Configuration and Upgrade Guidehttps://www.veritas.com/content/support/en_US/doc/129685393-129685398-1
25Best practices for using InfoScale Enterprise with Oracle RACReferences