provisioning oracle rac in a virtualized environment ......oracle rac sig web seminar thursday,...

Provisioning Oracle RAC in a

Virtualized Environment,

Using Oracle Enterprise

Manager

Oracle RAC SIG Web Seminar

Thursday, August 5, 2010

Kai Yu, Sr. System Engineer Consultant

Dell Global Solutions Engineering

• Kai Yu

• Senior System Engineer, Dell Oracle Solutions Lab 15 years Oracle DBA and Solutions Engineering

Specialized in Oracle RAC, Oracle EBS and OVM

Oracle ACE Director, Oracle papers author/presenter

IOUG Oracle RAC SIG President

• Rajat Nigam

• Senior Product Manager, Oracle Corporation

Focus areas Enterprise Manager’s Datacenter Automation solution–

Provisioning, Patching, Upgrade, Migration

End user experience of Enterprise Manager install and upgrade

About Authors

2

• Introduction to Oracle Enterprise Manager and Oracle VM

• Oracle Virtualization Infrastructure Implementation

• Provisioning Oracle 11g RAC using Oracle Enterprise Manager

• Extending Oracle 11g RAC using Oracle Enterprise Manager

• QA

Agenda

3

Introduction to Oracle Enterprise Manager and Oracle VM

4

• New Challenges for IT organization

• Quick deployment of Oracle service at low cost

• Deploy using tested and patched gold images

• An ability to response demand in short order

• Server consolidation and partition

• Oracle Enterprise Manager Grid Control

• Life cycle management of the entire IT stack

• Oracle Database/ RAC Provisioning and Patching

• Oracle VM Provisioning and Patching

5

Introduction to Oracle Enterprise Manager and Oracle VM

• Automated provisioning, patching, lifecycle management

• Gold images based & reference based deployment

6

RAC PROVISIONING

Oracle VM Technology

7

• Oracle VM Server: Xen Hypervisor, management domain dom0

Linux kernel with support of devices, IO, networking, etc.

• VM Server Pool: an autonomous region of VM servers

• VM: guest OS with applications running on domU

8

Oracle VM and Oracle RAC

9

VM and Virtualization Management

• VM Management Pack features

• Lifecycle Automation for Guest Virtual Machines

OVM Infrastructure Implementation

1

0

• Dell/Oracle Joint POC Project: Oracle Grid/ RAC Provisioning

Expand the Grid to include virtualization

Implement the Grid on Commodity hardware and SAN

storage

Grid control as the unified management for the grid

• Oracle Grid Architecture

Physical Grid and Virtual Grid

Based on Commodity Hardware Dell Blade Servers

• Focus of this presentation:

Establish the VM infrastructure for running 11g RAC on VM

Provision 11g RAC on VMs using Enterprise Manager Grid

Control


11

• Virtualization Infrastructure Design VM Servers: Dell blade servers

Shared storage: Dell EqualLogic iSCSI SAN

Unified Management: Grid Control with VM Management

12

• Prepare VM Infrastructure for RAC

VM server installation

Network configuration in VM servers

Configure Shared storage in VM servers

Configure Enterprise Manage Grid control

Enable VM management pack to Grid control

Create guest VMs for the RAC nodes

Configuration network on guest VMs

Present the shared storage to guest VMs as the virtual

storage for Oracle homes, OCRs/Votingdisks, ASM disk

volumes.


13

• Oracle VM server Installation

Enable virtualization for CPUs on BIOS

Install Oracle VM server 2.1.2

Configure dom0 memory by editing /boot/grub/menu.lst

Secure OVM agent is running: service ovs-agent status

• Oracle VM server network configuration



14

Customize the default Xen bridge configuration

Disable the default Xen bridge settings.

Create ifcfg-ethn, incfg-bond0, ifcfg-xenbr0, ifcfg-xenbr1

Restart the network services

• Oracle VM server shared storage configuration

Shared storage partitions added into OVM repositories

/dev/mapper/ovs_data4p1; /dev/mapper/ovs_data6p1

Additional disk partitions for guest VMs through physical

disks mappings for RAC shared disks

2 X OCRs, 3 X Votingdisks, ASM disk volumes

Detailed Information in Building an Oracle Grid with Oracle

VM on Dell™ Blade Servers and Dell EqualLogic iSCSI

Storage, Dell Oracle Solutions Whitepaper

http://go2.wordpress.com/?id=725X1342&site=kyuoracleblog.wordpress.com&url=http://www.dell.com/downloads/global/solutions/OracleGrid_leveraging_OVM.pdf?c=us&cs=555&l=en&s=biz&sref=http://kyuoracleblog.wordpress.com/publications/










15

• Grid Control Management Infrastructure

Install EM Grid control 10.2.0.3,Upgraded to 10.2.0.5

Enable VM Management Pack: metalink#781879.1

• Connect the VM servers to Grid Control

Connect VM servers to EM Grid control

Create VM server pool with the VM server



16

• Create Guest VMs using the VM template

Download/Register OVM_EL5U2_X86_64_PVM_10GB

Create VMs from the template


17

Specify the parameters Guest VM is created and running

for the guest VM on virtual server kblade10


18

• Present xenbr0 and xenbr1 as eth0 (public) and eth1(private)

vm.cfg: vif=['bridge=xenbr0,mac=00:16:3E:11:8E:CE,type=netfront',

'bridge=xenbr1,mac=00:16:3E:50:63:25,type=netfront', ]

• Add the additional disk and shared storages to the VMsAttached the shared disks from VM repositories


19

storage configuration

Partitions Size File system Raw devices

/dev/xvda1 104391 Os

/dev/xvda2 13526730 OS

/dev/xvdb1 15360000 /u0

/dev/xvdc1 21486906 ASM disk /dev/raw/raw1

/dev/xvdd1 297171 ocr1 /dev/raw/raw2

/dev/xvdd2 297202 votingdisk1 /dev/raw/raw3

/dev/xvdd3 297202 ASM spfile /dev/raw/raw4

/dev/xvde1 297171 ocr2 /dev/raw/raw5

/dev/xvde2 297202 votingdisk2 /dev/raw/raw6

/dev/xvde3 297202 votingdisk2 /dev/raw/raw7

Virtual disk partitions


20

vm.cfg: disk = [''file:/OVS/sharedDisk/racdb.img,xvdc,w!',

b. Attached the physical partitions to the guest VM

vm.cfg: disk = ['‘phy: /dev/mapper/vmracdbp1, xvdc,w!',

Provision Oracle 11g RAC using EM

21

• Install/configure Grid control Agent on RAC node VMs Grid control agent 10.2.5 installed on two guest VMs

Disable firewall services

Check host/cluster preferred credentials on the guest VMs


22

• Setup software library: DeploymentProvisioning

Administration: Software Library configuration

• Load the Oracle RAC shipped software to the software library

Download 11g software; DeploymentComponents


23

Upload Oracle Database binary and Oracle clusterware binary


24

• Prepare guest VM nodes for RAC provisioning Install/configure Grid control Agent on RAC node VMs

Prepare shared devices: 2 X OCR and 3 X votingdisk, ASM disk

/udev/rules.d/65-raw.rules and 89-raw.rules: map raw devices

Change kernel parameters for RAC install

Extend OS partition by adding another 20 GB disks using LVM

Prepare /etc/hosts to include public, private, VIPs: hostnames/IPs


25

• In Grid control, click the Deployments tab.

On the Deployments page, in the Deployment Procedure Manager

section, click RAC Provisioning Procedures.


26

• Provision Oracle RAC on guest VM

Start the provisioning procedure

Select source from Software library

Click Schedule deployment and specify the source of software

Provision Oracle 11g RAC Using EM

27

Select

Target

Host:


28

Target: clusterware/ASM RAC Database details


29

Shared Storage configuration for RAC

Submit the provisioning job and check the job status


30

Check the Job status and ensure it completed successfully

Results: 1)11g R1 Clusterware installed and configured

2) ASM home installed, the instance configured

3) 11g R1 RAC home installed and RAC Database

created and started up.

Extend Oracle 11g RAC using EM MANAGER

31

• Two ways to extend RAC One Click Extend Deployment Procedure

Clone from the existing RAC node’s Template

• Prepare the guest VM for the new node Create a new guest VM

Same network, disks, kernel settings, rpms. disable firewall

Install Grid control agent 10.2.0.5

Add host and cluster preferred credentials of Oracle users

• Schedule the provisioning procedure to extend the RACDeploymentProvisioning Deployment Procedure Manager RAC

Provisioning Procedure

Extend Oracle 11g RAC using EM

32

• Select “One Click Extend Cluster Database”, Click “Schedule

Deployment”


33

• Select the RAC database to extend

•Specify the reference host


34

Specify the new node to add and the user credentials


35

• Check the job status: procedure completed successfully


36

• Verify the RAC extension results: Check clusterware status:

[root@bvmrac2 bin]# ./crsctl check crs

Cluster Synchronization Services appears healthy

Cluster Ready Services appears healthy

Event Manager appears healthy

Check database service:[oracle@bvmrac2 bin]$ srvctl status database -d vmracdb

Instance vmracdb1 is running on node bvmrac1

Instance vmracdb2 is running on node bvmrac2

Summary

37

• Use Grid control to manage the virtual servers

• Use Grid Control Provisioning pack to provision 11g

RAC on VMs and extend the RAC to additional VM

node.

• Coming Oracle OpenWorld 2010 Presentations S316318: Oracle RAC/Oracle VM Automated Provisioning with Oracle

Enterprise Manager 11g

S316263: Monitoring and Diagnosing Oracle RAC Performance with

Oracle Enterprise Manager 11g

• Dell Oracle Whitepapers: www.dell.com/oracle

• Efficient Deployment of Oracle RAC in a Virtualized Environment Using

Enterprise Manager Provisioning Pack

• Building an Oracle® Grid with Oracle VM on Dell™ Blade Servers and

Dell EqualLogic™ iSCSI Storage,

http://i.dell.com/sites/content/shared-content/solutions/en/Documents/oracle-clusters-deployed-in-virtual-env-wp.pdf




http://www.dell.com/downloads/global/solutions/OracleGrid_leveraging_OVM.pdf?c=us&cs=555&l=en&s=biz





38

QA

Thank You and Visit my Oracle Blog at http://kyuoracleblog.wordpress.com/

39