migrated 13 tb ebs production to 5 node 11gr2 rac on...

Migrated 13 TB EBS Production to 5 node 11gR2 RAC on vSphere - A Deep Dive

Charlene Huang, Database Architect Eisen Kung, Senior Manager

2

Introduction

100% Virtualized

Oracle Architecture

Best Practices

Migration and Testing

Reporting and Backup Solution

Virtualization Benefits

Summary and Next Steps

Reference

Q&A

Agenda

Introduction

VMware IT – 100% Virtualized

Why Run Oracle RAC on vSphere?

Lower capital and operating costs

Maximize the use of hardware resources without compromising application

performance

Achieve 100% virtualization in IT Production footprint using VMware products

A major step toward Cloud-ready architecture

Increase agility, efficiency, scalability, and availability to support VMware rapid

growth

Efficient provisioning of new RAC cluster using VM template and Oracle cloning

Simplify Disaster Recovery (DR) solution using VMware Site Recovery

Manager (SRM)

Success Criteria

No compromise in performance after the transition from physical

to virtual

Minimal service disruption during the transition from physical to

virtual

Backup and reporting must be offloaded from production and

continue to meet current SLA

No reduction in resiliency

Must enhance/simplify current disaster recovery solution

Establishing a Baseline

• A snapshot of the physical code base which passed the functional,

performance, and resiliency test rigor was taken

• Use above code base to build the virtual environment

• Took month-end and quarter-end timings from the physical for the

Backup and Reporting Refreshes

VMware IT – Oracle Architecture

Oracle EBS Virtual Solution

Oracle EBS RAC Virtual Solution

Service APP SOA Portal CCM1 CCM2 CCM3 DCO

Preferred Node Node1

Node2 Node3 Node4 Node5

Node3

Node4

Node5

Available Node Node3

Node4

Node4

Node5 Node3

Node5

Node3

Node4

Application EBS

Web/Forms

Fusion

Middleware Portal

Concurrent Manager

Server 1

Concurrent Manager

Server 2

Concurrent Manager

Server 3

Oracle

Discoverer

Multiple custom services are defined in RAC to distribute the load across different nodes.

Node1 and 2 primarily handle OLTP; Node3, 4, and 5 handle batch jobs and reports.

OLTP Batch/Reporting

EBS Infrastructure Details

11

Database is a 5 node RAC cluster

• Oracle Linux Server release 6.2 with 24 vCPU and 256GB memory

• Oracle Grid infrastructure 11.2.0.3.0 64bit; Oracle Database 11.2.0.3.0 64bit

• Virtual RDM is used for database storage

• VMDK is used for OCR and Voting disks

• Jumbo Frames are used for Oracle interconnect

Three node concurrent manager servers

• Oracle Linux Server release 5.7 32 Bit with 8 vCPU and 32GB memory

Six node internal and one node external application server


Three node Oracle Discoverer servers (10.1.2.55.26)


EBS Infrastructure Details (cont)

12

Two node RAC Active Data Guard

• Oracle Linux Server release 6.2 64 bit with 12 vCPU and 64GB memory

• Oracle Grid infrastructure 11.2.0.3.0 64 bit; Oracle Database 11.2.0.3.0 64bit

• Virtual RDM is used for database storage

• VMDK is used for OCR and Voting disks

• Jumbo Frames are used for oracle interconnect

Reporting Environment

• Single node Database

o Oracle Linux Server release 6.2 64 bit with 12 vCPU and 64GB memory

• 3 node App and Discoverer servers

o Oracle Linux Server release 5.7 32 Bit with 4 vCPU and 8GB memory

• 1 node concurrent manager

o Oracle Linux Server release 5.7 32 Bit with 4 vCPU and 8GB memory

Backup Environment


Server and Hardware Diagram

13

Storage Diagram

14

ESXi Cluster

Dual Fabric SAN

PowerPath/VE Multi-Pathing

VMAX 40

With FAST VP

VPLEX RecoverPoint / SRM

VPLEX Quad Engine VS2

UID

HP ProLiantBL660c Gen8

UID


UID


UID


UID


UID


Hardware Configuration

15

Server Hardware

• 6 HP Blade BL660c Gen 8 Server

4 Sockets x 8 Cores (32 Cores, 64 Threads), 512 GB DDR3 Memory, 2 x 2-Port FlexibleLOM

• 2 HP Virtual Connect Flex-10/10D Modules

• Dual-Port 8Gb QLogic HBA

Storage Hardware (Not Dedicated)

• SAN Fabric: Brocade DCX 8510 (Dual-Fabric)

• EMC VPLEX VS2 Virtualized Storage (Front End)

• EMC Symmetrix VMAX 40K (Backend)

• FAST VP

VMware IT – Best Practices

Best Practice on vSphere Implemented

17

BIOS setting

• Disable power management

• Enable hyper-threading

• Enable hardware-assisted virtualization features (VT-x, EPT)

Memory setting

• Use huge memory pages in guest OS to reduce TLB overhead

(Configure HugePages the same as SGA size).

• Avoid an expensive swapping overhead in guest OS (Set

vm.swappiness = 0 in /etc/sysctl.conf)

• Memory Reservation - Reserve memory equal to the size of the Oracle

SGA, PGA, plus background processes

Best Practice on vSphere Implemented (cont)

18

Network

• Use VMXNET3 network adapter type

• Enable Jumbo Frame for the RAC Heartbeat Link

• Disable dynamic coalescing to reduce latency on the network links for

all vNICs (example: ethernetX.coalescingScheme = disabled)

IO related setting

• Use paravirtualized SCSI Controller (PVSCSI)

• Use "NOOP" IO scheduler per block device in guest OS

• Increase the limit on the maximum number of outstanding disk requests

per device

Best Practice on vSphere Implemented (cont)

19

Disk related configuration

• Virtual RDM

Disk creation only needs to be done once for the 1st VM in the RAC. The rest of

the VMs will reference the same VMDK pointer

Disk creation can be done either through the vSphere client or the ESXi shell

• VMDK

Shared VMDK disks need to be Thick Provision Eager Zeroed virtual disks

• Enable Multi-Writer for shared disk

• Set SCSI Bus Sharing to None

• Enable Disk UUID

VMware IT – Migration and Testing

Migrating to Virtual RAC with Minimal Downtime

21

• Install 11.2.0.3 Grid and database software on 5 node cluster on VM

• Apply the latest PSU patch

• Allocate the same amount of storage to 5 node RAC virtual cluster

• Use EMC TimeFinder clone to sync the data from physical to virtual

• Setup Data Guard on virtual RAC nodes

• During downtime, perform Data Guard switchover (30min database

downtime)

• Clean up FND nodes and run autoconfig on new database servers

• For CCM and application servers, shared appl_top is used. During

downtime, mount the same NAS share to new VMs, then follow Oracle

support note 233428.1 to complete the migration

Functional Testing (1)

22

Test Coverage Type of Test Portals considered in Execution

EBS Internal Modules EBS (Automation/Manual) testing Execute the existing EBS regression

suite.

SC,OM,AR,PO,AP,GL,CM,FA,VPP,

Eclipse,PER VM,Keystone, Niagara,

Term Lic.

3rd Party Integration

(which will interact w/

EBS System )

EBS Manual execution covering

major scenarios

EDI, Iexpenses, Bookings 2.0,Trade

Beam,Change Point,Blackline,

Discoverer Reports, OM/SC Reports,

EDI Conc. Program Reports & Email

validations, Workflow Emails, Cyber

source – Credit card Order

Functional Testing (2)

23

Test Coverage Type of Test Portals considered in Execution

End to End Smoke

Test for customer

facing portals that

interact with EBS DB

Portal (Automation/Manual)

Testing covering P1 test

scenarios only

EMS,ISV,OEM, DR, Admin, Support,

Flex, Downloads, VPP, Partner Eval,

Public EVAL, Beta.Next, Communities,

SCA, Model N

End to End Smoke

Test Scenarios from

SFDC

SFDC Manual Testing

Will be covering Smoke Test

scenarios only

Admin, EMS, NFR Orders, UCM,

Partner Sync, Update Contact, IVR

3rd Party Integration

(which will interact

EBS System )

Manual testing by BA Citi - Iexpense & CFX, Wellsfargo,

FieldGlass , CDW and Dell Punchout,

RevStream, Commissions, GetPaid

Custom.pll – iSupplier, Vertex

Validations, My Learn (PMO), Work

day, BI, E-Trade, Zoomax, HRMS --

User creations, MnA Interfaces

Performance Testing

24

Test load Detail

1XQE (normal day-to-day load

during quarter end)

• 350 Test Cases, 4,677 Atomic Transactions

• 2,000 Concurrent Users, 567 Orders/hour Placed

2XQE (double the day-to-day load

during quarter end)


• 4,000 Concurrent Users, 1,134 Orders/hour Placed

Product Launch (the load that is

experienced during a product

launch)


• 3,000 Concurrent Users, 1,432 Orders/hour Placed

Endurance Test (1X normal day

load, 72 Hours)


• 2,000 Concurrent Users, 567 Orders/hour Placed

Conducted with HP LoadRunner 9.1. Three load tests were executed at different load levels and

one endurance test was executed for 72 hours.

Resiliency Testing

25

• Public NIC Failure/ Interconnect NIC Failure

• Storage path failure (disable one path)

• Network driver failure on master or slave node

• Use vMotion to live migrate all RAC nodes to different hosts

• CRSD/EVMD/CRSDD Process Failure

• RDBMS Process Failure

• ASM Instance failure ( Unplanned)

• Scan Listener/ Listener Failure ( Unplanned )

• Multiple Instance Failure (Unplanned)

• Planned Instance Termination

• Rebooting RAC node/ Power-off one node

• CCM server failure

All test cases were performed under various system loads, with SoapUI PRO calls, HP LoadRunner 9.1 injecting 350 test cases and 4,677 atomic transactions using 2,000 concurrent users) & transactions generated using QTP (Quick Test Professional) Tool, and Mercury Interactive Functional Testing Tool

Utilization Comparison Considerations

26

Physical RAC 5 Node RAC on HP G7 blades, each Node has:

CPU = 20 core with hyper threading

Memory = 256GB

SGA Size = 38GB

Virtual RAC combined with Hardware Tech Refresh 5 Node RAC on HP Gen 8 blades, each Node has:

vCPU = 24

Memory = 256GB

SGA Size = 62GB

RAC Service Configuration OLTP transactions are configured to connect to Node 1, 2

Concurrent programs are configured to connect to Node 3, 4, 5

Discoverer and reporting are configured to connect to Node 3, 4, 5

Memory Utilization Percentage Comparison

27

0

5

10

15

20

25

30

35

40

VMWPROD1 VMWPROD2 VMWPROD3 VMWPROD4 VMWPROD5

Memory Utilization Percentage

Physical ME

Virtual ME

Virtual QE

Physical SGA is 38GB

Virtual SGA is 62GB

DB Single Block Disk Read (ms) Comparison

28

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

VMWPROD1 VMWPROD2 VMWPROD3 VMWPROD4 VMWPROD5

DB Single Block Disk Read (ms)

Physical ME

Virtual ME

Virtual QE

Batch Jobs Performance Comparison

29

0

100

200

300

400

500

600

ContractsStatus Change

by Range

VM AutoRenewal

NotificationTable Script

VM BookingMN.Next

Allocations

VMCommissions

LoadTransactions

VM ModifyUpgradedCovered

Product Lines

VM: KeystoneData Profile

ExtractProgram

VM: SCExceptions

XXRS:revStream

PentupRevenueReport

XXVM RDM ARTransactions

Average Run Time in Minutes

Physical ME

Virtual ME

Virtual QE

Challenges and Resolutions

30

• Network Latency between VMs – affects OEL 6.2 due to LRO setting.

Override the inbox vmxnet3 driver with the VMware driver from VMware

tools

• Maximum allowed outstanding disk requests per device affects IO

throughput. Increase from default 32 to 128

• vMotion with RDM – availability. Able to vMotion using virtual mode

RDM

• CRS Bug – availability – affect OEL 6.x. The maximum process value

of the OHASD bin is too small if the CRS is started manually (Add

“ulimit -u 16384” to ohasd and crswrap.sh.sbs). Refer to Oracle support

Documentation ID 1594606.1 for the detailed resolution

• High storage bandwidth utilization required additional ISLs in the trunk.

Operation Support Awareness

31

• Do not attempt to take a snapshot of any RAC nodes with shared

RDMs; doing so will cause VMs to power off.

• Storage vMotion does not work if there is shared disk.

(http://kb.vmware.com/kb/1005241)

• Storage vMotion works with RDMs when no disks are shared; however,

any custom disk label and folder will be removed

• Hot Add or Delete Shared Disks without downtime (with vSphere API)

VMware IT – Reporting and Backup Solution


2 Node Cluster

App

Discoverer

Async

Redo

Transport

Ad Hoc Query

Active

Data Guard

App3 App5 App6App4App1 App2

CCM2 CCM3CCM1

5 Node Cluster

Storage syncStorage sync

Backup serverReporting server

CCM

App1-3

CCM

SOA

Portal

Reporting Refresh Solution

34

Business Requirement: 2hr downtime window to perform daily refresh of reporting instance end to end.

A master script automatically performs: • Shutdown reporting instance.

• Perform storage sync.

• Recover and open database.

• Rename database to reporting instance name.

• Run autoconfig on database, concurrent manager, and application servers.

• Run custom update scripts to clean up CCM queue, e-mail alerts, etc.

• Bring up concurrent manager services and application services.


2 Node Cluster

App

Discoverer

Async

Redo

Transport

Ad Hoc Query

Active

Data Guard

App3 App5 App6App4App1 App2

CCM2 CCM3CCM1

5 Node Cluster

Storage syncStorage sync

Backup serverReporting server

CCM

App1-3

CCM

SOA

Portal

Backup Solution

36

Full backup on the dedicated backup server

• Install single-node 11.2.0.3 Grid infrastructure and database software.

• Allocate same amount of storage as production server.

• A master script automatically performs

storage sync from production host to backup host,

Mount Database

Perform the RMAN backup to Data Domain using EMC Networker.

13TB database completed in 5 hours

Archivelog backup twice daily on primary RAC nodes using EMC Networker.

Disaster Recovery Solution using VMware Site Recovery Manager (SRM)

37

VMware vSphere

VMware

vCenter Server

Site Recovery

Manager

VMware

vCenter Server

Site Recovery

Manager

VMware vSphere

Site A – Santa Clara, CA (Primary) Site B – Wenatchee, WA (Recovery)

Servers Servers

EMC RecoverPoint

• Recovery site is located 800 miles

Away from Primary site

• EMC RecoverPoint Replication

leveraging EMC VPLEX

RecoverPoint splitter

• Cisco OTV deployed to extend

Layer 2 network to the Recovery

datacenter

• Simplified recovery using VMware

vCenter Site Recovery Manager

VMware IT – Virtualization Benefits

Reduced Datacenter Footprint by 60%

39

# of

Blades

# of

chassis EBS RAC Reporting Active

Data

Guard

CCM SOA/IDM Backup

Physical 15 2 5 node

20 core

256G mem

1 node

12 core

96G mem

3 nodes

8 core

64G mem

2 nodes SOA

2 nodes IDM

12 core

96G mem

1 node

12 core

96G mem

Virtual 6 1 5 node

24 vcpu

256G mem

1 node

12 vcpu

64G mem

2 nodes

12 vcpu

64G mem

3 nodes

8 vcpu

32G mem

3 nodes SOA

3 nodes IDM

8 vcpu

32G mem

1 node

12 vcpu

64G mem

Reduced Physical Servers and Oracle Licensing

Products / Features Used

40

• VMware vMotion – VM mobility enables us to do online maintenance and

infrastructure upgrade.

• VMware DRS – provides automated resource balancing based on real time

resource usage. Anti-affinity rules deployed to achieve system distribution over

multiple hosts in an application cluster.

• VMware HA – automated system restart/recovery when host failure/isolation

occurs in the cluster.

• Hot-plug for CPU, memory and storage.

• SRM – used along with EMC RecoverPoint technology for disaster recovery

solution.

• vSphere API for automated disk management.

• vCenter Operations Manager for vSphere monitoring

VMware IT – Summary

Summary

42

Success Criteria AccomplishmentCombined the RAC migration to virtual with the

Hardware Tech Refresh

No compromise in performance after the transition

from physical to virtual.

Virtual EBS 5 node RAC database performance

equaled or better its physical predecessor

Minimal service disruption during the transition

from physical to virtual.

The downtime of the switchover from physical to

virtual was reduced to 30 mins.

Backups and reporting must be offloaded from

production and continues to meet current SLA

With the tech refresh, the 13TB database backup

was reduced from 11 hrs to 5 hrs

Reporting refresh was within the 2 hour SLA

window

No reduction in resiliency. Increased resiliency with addition of VMHA

Must have a disaster recovery solution. Achieved a unify disaster recovery solutio

Reference

43

• VMware vSphere distributed Switch Best Practices http://www.vmware.com/resources/techresources/10250

• VMFS Heap, PB Cache Slab, and Max Open Addressable Space: vSphere 2013 and Beyond https://wiki.eng.vmware.com/VMFS/AddrModesPBCacheAndHeapSizes#VMFS_Heap.2C_PB_Cache_Slab.2C_and

_Max_Open_Addressable_Space:_vSphere_2013_and_Beyond

• Disabling simultaneous write protection provided by VMFS using the multi-writer flag

(1034165) http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1034165

• Oracle DB on VMware Best Practices http://www.vmware.com/files/pdf/partners/oracle/Oracle_Databases_on_VMware_-_Best_Practices_Guide.pdf

• Oracle DB on VMware RAC deployment Guide http://www.vmware.com/files/pdf/solutions/oracle/Oracle_Databases_VMware_RAC_Deployment_Guide.pdf

• vSphere 5.5 performance best practices http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.5.pdf

• Tuning VM for latency-sensitive workload http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf

https://wiki.eng.vmware.com/VMFS/AddrModesPBCacheAndHeapSizes



http://www.vmware.com/files/pdf/partners/oracle/Oracle_Databases_on_VMware_-_Best_Practices_Guide.pdf





http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.5.pdf

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.5.pdf

Reference (cont)

44

• Pre-installation requirements for 11gR2 http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/prelinux.htm

• Installing 11g R2 Grid infrastructure http://docs.oracle.com/cd/E11882_01/install.112/e22489/crsunix.htm#CWLIN315

• Oracle CRS Bug 17301761 THE MAXIMUM PROCESS VALUE OF OHASD,BIN IS TOO

SMALL IF THE CRS STARTS MANUALLY Oracle support Documentation ID 1594606.1

• Adding a EBS application node when using a shared file system oracle support note 233428.1 section 5

Acknowledgement

Zaigui Wang -- IT vSphere Architect Kandy O'Mara -- IT Storage Architect Chanh Chi -- Director – IT Cloud Operations Sudhir Balasubramanian -- CoE Solutions Architect

migrated 13 tb ebs production to 5 node 11gr2 rac on...

Documents