EMC Solutions Practice/EMC NAS Solutions Engineering.EMC Corporation

RTP HeadquartersRTP, NC 27709

www.EMC.com

Oracle RAC 10gCelerra NS Series NFS

Reference Architecture GuideRevision 1.0

Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide

Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide

ii

Trademark/Copyright InformationCopyright © 2006 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

Part Number H2373

About this Document

Purpose ...................................................................................... vAudience..................................................................................... vScope ......................................................................................... vRelated documents..................................................................... vi

Chapter 1 Solution Overview

The business challenge................................................................ 2The technology solution............................................................... 2

Solution advantages ............................................................. 2

Chapter 2 Solution Architecture

Overall architecture ..................................................................... 6General characteristics.......................................................... 8

Network architecture ................................................................... 9VLANS .................................................................................. 9Switches............................................................................. 10EMC NS502 Data Mover ports .............................................. 10

Storage architecture .................................................................. 10High availability and failover ................................................ 11RAID type and RAID group configuration............................... 12Disk volume setup............................................................... 13MVM vs. AVM ...................................................................... 14File systems, exports and mount points ............................... 14

Contents

Oracle RAC 10g NS Series NFS Reference Architecture Guide iii

Contents

iv

Database server architecture ...................................................... 15Oracle Database 10g server network architecture.................. 15Oracle cluster ready services (CRS)....................................... 16NFS client............................................................................ 16High availability and failover................................................ 16

Application architecture ............................................................. 17

Chapter 3 Hardware and Software Resources

Hardware resources .................................................................. 20Software resources .................................................................... 21

Contents

This document provides an overview of the architecture of the EMC solution for Oracle RAC 10g on NS Series NFS developed jointly by the EMC Technology Solutions Practice and EMC NAS Solutions Engineering.

Purpose Information in this document can be used as the basis for a solution build, white paper, best practices document, or training.

Information in this document can also be used by other EMC organizations (for example, the technical services or sales organization) as the basis for producing documentation for a technical services or sales kit.

Audience This document is intended for EMC personnel, partners, and customers.

Scope This document describes the architecture of an EMC solution built and tested jointly by the EMC Technology Solutions Practice and EMC NAS Solutions Engineering, both located in Research Triangle Park, N.C.

Implementation instructions and best practices are beyond the scope of this document.

The Oracle RAC 10g Celerra NS Series NFS Validation Test Report provides detailed information about the tests performed to validate this solution.

Note: The validation test report is available only to EMC personnel and partners.

About thisDocument

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide v

Preface

vi

Related documents

The following documents provide additional, relevant information:

• Oracle Database 10g/Oracle RAC 10g Celerra NS Series NFS Best Practices Planning

• Oracle Database 10g/Oracle RAC 10g Celerra NS Series NFS Applied Technology

• Celerra Network Server documentation

• Oracle RAC 10g presentations and white papers on EMC Powerlink

Related documents

1

This chapter includes the following sections:

• The business challenge ..................................................................2• The technology solution .................................................................2

Solution Overview

Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide 1

Solution Overview

2

The business challengeMid-size enterprises face the same challenges as their larger counterparts when it comes to managing database environments. Typical challenges include cost, control, resource utilization, and scaling.

Oracle RAC 10g supports numerous functions ranging from manufacturing to online web applications. These deployments may be in smaller data centers that often support applications ranging from custom database applications to standard packages such as SAS, Siebel, PeopleSoft, and SAP, or they may be in large, centralized data centers

Mid-size enterprises need scalable and RAID-protected storage but must be extremely cost-conscious. They may lack the IT resources to deploy, manage, and maintain complex environments at the departmental level.

The technology solutionThe Oracle RAC 10g - Celerra NS Series NFS solution enables a mid-size enterprise to deploy an EMC network-attached storage (NAS) architecture with NFS connectivity for its Oracle Database 10g database applications with lower cost and complexity than direct-attached storage (DAS) or a storage area network (SAN).

Solution advantages The solution offers the following advantages:

Table 1-1 Solution advantages

Benefit Details

Lower total cost of ownership (TCO) Reduces acquisition, administration, and maintenance costs than equivalent DAS or SAN

Greater manageability Eases implementation, provisioning, and volume management

Simplified Real Application Clusters (RAC) implementation

Provides NFS-mounted shared file systems

High availability Can implement a clustering architecture that provides very high levels of data availability

The business challenge

Solution Overview

Increased flexibility Easily make databases, or copies of database, available (via remounts) to other servers

Improved protection Integrate both availability and backup

Benefits of EMC Information Lifecycle Management (ILM)

Implement tiered storage

Table 1-1 Solution advantages

Benefit Details

Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide 3

Solution Overview

4

The technology solution

2

This chapter includes the following sections:

• Overall architecture....................................................................... 6• Network architecture......................................................................9• Storage architecture..................................................................... 10• Database server architecture ........................................................ 15• Application architecture ............................................................... 17

SolutionArchitecture

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 5

Solution Architecture

6

Overall architectureThe following figures show the overall architecture of the Oracle RAC 10g NFS solution:

Each of the configurations include the following components:

• Client, RAC interconnect, and storage networks consisting of dedicated network switches and VLANs

• Oracle RAC 10g servers connected to the client, RAC interconnect, and storage networks

• Clients connected to the Oracle RAC 10g servers via the client network

• A Celerra NS Series (NS502) network file server connected to the Oracle RAC 10g servers via the storage network

Table 2-1 Oracle RAC 10g NFS Solution configurations

Figure Configuration

Figure 2-1 Consolidation

Figure 2-2 Backup

Figure 2-3 Protect

Overall architecture

Solution Architecture

Figure 2-1 Overall architecture of the Oracle RAC 10g NFS consolidation solution

Figure 2-2 Overall architecture of the Oracle RAC 10g NFS backup solution

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 7

Solution Architecture

8

Figure 2-3 Overall architecture of the Oracle RAC 10g - Celerra NS Series NFS protect solution

General characteristics Each solution configuration has the following general characteristics:

• Each configuration consists of either a two-node or four-node active/active Oracle RAC 10g RAC.

• Oracle database files, online redo log files, archived log files and the flashback recovery area each reside on their own NFS file system. Online redo log files are mirrored across two different file systems using Oracle software multiplexing.

• The control files are mirrored across the online redo log file systems.

• The voting disk and ocr files are mirrored across multiple NFS file systems.

Overall architecture

Solution Architecture

• NFS file systems are designed to satisfy the I/O demands of individual database objects, using either RAID 5, RAID 3, or RAID 1, as appropriate.

• IP connections are trunked on the storage and RAC interconnect networks to balance and distribute network I/O.

• All database files are stored on the EMC Celerra NS Series storage system, making server replacement relatively simple.

• Oracle RAC 10g for x86 or x86-64 runs under Red Hat Enterprise Linux.

• TCP/IP and NFS version 3 provide network connectivity and file system semantics.

Network architectureThis section describes the network architecture of the validated solution.

VLANS VLANs, or virtual local area networks, logically group devices that are on different network segments, or subnetworks. The validated solution uses three VLANs to segregate network traffic of different types. This improves throughput, manageability, application separation, high availability, and security.

Figure 2-1 on page 7 shows the three VLANs:

• A client VLAN supports connectivity between the Oracle Database 10g servers and the client workstations. The client VLAN also supports connectivity between the EMC Celerra NS Series and the client workstations to provide network file services to the clients. Control and management of these devices is also provided via the client network.

• A RAC interconnect VLAN network supports connectivity between the Oracle Database 10g servers for network I/O required by Oracle cluster ready services (CRS). Two network interface cards (NICs) are configured on each Oracle Database 10g server to the RAC interconnect network. Link aggregation is configured on the servers to provide load balancing and port failover between the two ports for this network.

• The storage VLAN uses the NFS protocol to provide connectivity between servers and storage. Each database server connected to the storage VLAN has two NICs dedicated to the storage VLAN. Link aggregation is configured on the servers to provide load balancing and port failover between the two ports.

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 9

Solution Architecture

10

Switches For all intents and purposes, the network “pipes” shown in Figure 2-1 on page 7 represent IP switches. In addition, the RAC interconnect connections are also on a dedicated switch. For real-world solution builds, it is recommended that these switches support Gigabit Ethernet (GbE) connections, jumbo frames, and port channelling.

EMC NS502 Data Mover ports

For example, Figure 2-4 illustrates the network ports on the rear of two EMC NS502 Data Movers.

Figure 2-4 EMC NS502 Data Mover ports and traffic types

Port csp handles administrative and control traffic. Port cge0 (the last character is a zero) and cge1 are aggregated and connected to the storage network. They handle all I/O required by the database servers to the datafiles, online redo log files, archived log files, control files, ocr file and voting disk. Ports cge2 and cge3 are left open for future growth.

Note: Celerra Manager can be used to create Ethernet channels, link aggregations, and fail-safe networks.

Storage architectureSetting up storage involves:

• Establishing the RAID levels

Storage architecture

Solution Architecture

• Allocating hot spares

• Creating disk volumes

• Combining disk volumes (stripes and slices) into metavolumes, or disk pools, which can span disks

• Creating file systems, which can be exported to servers

High availability and failover

The EMC Celerra NS Series has built-in high-availability features. These HA features allow the Celerra NS Series to survive various failures without a loss of access to the Oracle Database 10g database. These features protect against the following:

• Data Mover failure

• Network port failure

• Power loss affecting a single circuit connected to the storage array

• Storage processor failure

• Disk failure

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 11

Solution Architecture

12

RAID type and RAID group configuration

Two sets of RAID and disk configurations were tested. The first is shown in Figure 2-5.

Figure 2-5 Two FC shelf RAID configuration

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 1_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 1_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

1 + 1RAID 1

1 + 1RAID 1

Shelf 0_0 6 7 9851 2 3 40

4 + 1RAID 3

8 + 1RAID 3

Shelf 2_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

HotSpare

HotSpare

HotSpare

FC Disk

ATA Disk

Data Disk

HotSpare

FC Disk

ATA Disk

Data Disk

4 + 1RAID 5

12 131110 14

Storage architecture

Solution Architecture

The second is shown in Figure 2-6.

Figure 2-6 Three FC shelf RAID configuration

Disk volume setup Once the RAID groups are created, the EMC Celerra NS Series automatically creates disk volumes that are accessible to the Data Movers.

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 1_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 2_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

1 + 1RAID 1

1 + 1RAID 1

1 + 1RAID 1

1 + 1RAID 1

HotSpare

Shelf 0_0 12 1311106 7 9851 2 3 40 14

4 + 1RAID 3

8 + 1RAID 3

Shelf 3_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

HotSpare

HotSpare

HotSpare

FC Disk

ATA Disk

Data Disk

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 1_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 1_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 2_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

4 + 1RAID 5

4 + 1RAID 5

Shelf 2_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

4 + 1RAID 5

1 + 1RAID 1

1 + 1RAID 1

1 + 1RAID 1

1 + 1RAID 1

HotSpare

Shelf 0_0 12 1311106 7 9851 2 3 40 14

4 + 1RAID 3

8 + 1RAID 3

Shelf 3_0 5 6 7 8 9 12 1410 11 130 1 2 3 4

HotSpare

HotSpare

HotSpare

FC Disk

ATA Disk

Data Disk

HotSpare

FC Disk

ATA Disk

Data Disk

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 13

Solution Architecture

14

MVM vs. AVM MVM, or manual volume management, allows the greatest flexibility for the user to configure storage in whatever way he sees fit, including providing the highest performance. However, this is a more complex method for configuring storage.

AVM, or automatic volume management, on the other hand, is a simpler, more automatic way of configuring storage but is not as flexible as MVM in its configuration options.

For MVM, once the disk volumes are created, metavolumes are created from sliced or striped volumes. Each metavolume is set up to stripe across multiple disk volumes so that multiple disks can work concurrently. The metavolumes shown in Table 2-3 were created.

For disks managed using AVM, system-defined storage pools are used.

Table 2-2 shows the RAID configurations used for each solution.

File systems, exports and mount points

In the next step, the database file systems are created, which are exported to the servers. In the case of the testing covered here the file

Table 2-2 RAID configurations per solution

Solution Configuration

Consolidation MVM 3 FC shelves

Consolidation MVM 2 FC shelves

Consolidation AVM 2 FC shelves

Backup AVM 2 FC shelves

Protect AVM 2 FC shelves

Storage architecture

Solution Architecture

systems shown in Table 2-3 were created, exported, and mounted on the database servers.

Database server architectureThis section describes the database server architecture.

Oracle Database 10g server network architecture

Each Oracle Database 10g server has five network interfaces. Two interfaces connect to the storage network, using a link aggregation trunk. Two interfaces connect the server to the RAC interconnect network, enabling the heartbeat and other network I/O required by

Table 2-3 File system layout

file system / export AVM MVM

/datafs clar_r5_performance (system-defined storage pool consisting of all RAID 5 4+1 groups)

stripe_data (metavol consisting of all available RAID 5 groups)

/log1fs clar_r1 (system-defined storage pool consisting of all RAID 1 groups)

meta_log1 (metavol using half of the RAID 1 groups)

/log2fs clar_r1 (system-defined storage pool consisting of all RAID 1 groups)

meta_log2 (metavol using half of the RAID 1 groups)

/archfs clarata_r3 (system-defined storage pool consisting of all RAID 3 groups)

meta_arch (metavol using the 8+1 RAID 3 group)

/flashfs clarata_r3 (system-defined storage pool consisting of all RAID 3 groups)

meta_flash (metavol using the 4+1 RAID 3 group)

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 15

Solution Architecture

16

Oracle cluster ready services. One interface connects to the client network. Table 2-4 lists each interface and describes its use.

Oracle cluster ready services (CRS)

Oracle cluster ready services (CRS) is enabled on each of the Oracle Database 10g servers. The servers operate in active/active mode to provide local protection against a server failure and to provide load balancing.

CRS required files (including the voting disk and the ocr file) can reside on NFS volumes provided that the required mount point parameters are used. See Oracle Database 10g/Oracle RAC 10g Celerra NS Series NFS Best Practices Planning for more information on the mount point parameters required for the Oracle Clusterware files.

NFS client Each Oracle RAC 10g server uses the NFS protocol to connect to the Celerra NS Series network file server. NFS runs on top of TCP/IP.

High availability and failover

TCP/IP provides the ability to establish redundant paths for sending I/O from a networked computer to another networked computer. This approach uses the link aggregation protocol, commonly referred to as “trunking”. Redundant paths facilitates high availability and load balancing for the networked connection. Two NICs on each Oracle Database 10g server were used in the NFS connection, referred to above as the storage network. The RAC interconnect network was trunked in a similar manner.

Table 2-4 Application server network interface configuration

Interface port ID Description

eth0 Client network

eth1 RAC interconnect (trunked)

eth2 RAC interconnect (trunked)

eth3 Storage network (trunked)

eth4 Storage network (trunked)

Database server architecture

Solution Architecture

Application architectureThe Oracle Database 10g binary files, including the Oracle Cluster Ready Services software, were installed on the database servers’ local disks. Datafiles, online redo log files, archive log files, tempfiles, and CRS files resided on EMC Celerra NS Series NFS file systems. These file systems were designed (in terms of the RAID level and number of disks used) to be appropriate for each type of file.

Table 2-5 lists each file or activity type and indicates where it resides.

Table 2-5 Application file or activity types and locations

File or activity type Location

Database binary files database servers local disk

Datafiles /datafs

Tempfiles /datafs

Online redo log files mirrored across /log1fs and /log2fs

Archive log files /archfs

Flashback recovery area /flashfs

Control files mirrored across /datafs, /log1fs, and /log2fs

CRS ocr file mirrored across /log1fs and /log2fs

CRS voting disk mirrored across /datafs, /log1fs, and /log2fs

Oracle RAC 10g - Celerra NS Series NFS Reference Architecture Guide 17

Solution Architecture

18

Application architecture

3

This chapter includes the following sections:

• Hardware resources .....................................................................20• Software resources ...................................................................... 21

Hardware andSoftware Resources

Oracle RAC 10g NS Series NFS Reference Architecture Guide 19

Hardware and Software Resources

20

Hardware resourcesThe Oracle RAC 10g - Celerra NS Series NFS solution uses the following hardware resources.

Table 3-1 Hardware specifications

Equipment Quantity Configuration

Dell 1850 server Two for backup and protect, four for consolidation

Two 3.0-GHz Intel Pentium 4 dual-core processors8 GB of memory for backup and protect, 12 GB for consolidationTwo 73 GB 10K internal SCSI disksTwo onboard 10/100/1000 MB Ethernet NICsTwo additional 10/100/1000 MB Ethernet NICs (one single-port and one dual-port)

Dell 5324switch Three for each solution (one for client, one for RAC inteconnect and one for storage)

24 ports per switch

Celerra NS502 networked server

One Two Data MoversFour GbE network connections per Data MoverTwo or three FC shelvesOne ATA shelf30 or 45 146 GB FC disks (depending on configuration)Fifteen 320 GB ATA disksOne Control Station

Hardware resources

Hardware and Software Resources

Software resourcesThe Oracle RAC 10g - Celerra NS Series NFS solution uses the software resources in Table 3-2.

Table 3-2 Software specifications

Software title Number of licenses

Red Hat Enterprise Linux One per database server

Oracle RAC 10g Enterprise Edition One per database server

Celerra Manager Advanced Edition One per Celerra NS Server

Oracle RAC 10g NS Series NFS Reference Architecture Guide 21

Hardware and Software Resources

22

Software resources