11

Oracle EXADATA and ZFS NAS Appliance for Faster

Platform as Service

22

Pranilesh ChandLead Engineer - Database

Oracle Certified Professional 9i, 10g,11gSpeciality:

Performance TuningOracle RAC/HAOracle Exadata

Disaster Recovery

33

Agenda

• Technology Overview• Why we chose this solution• How we did it• Issues• Q&A

44

SUN Oracle Database Machine

• 2 Sun Fire™X4170 Oracle Database servers

• 3 Exadata Storage Servers (All SAS or all SATA)

• 2 Sun Datacenter InfiniBand Switch

• Preconfigured out of the box high performance machine

• Balanced performance configuration

• Oracle database 11.2 preinstalled

• No ASM Instance

• Smart Scan

55

Sun ZFS Storage 7420•Only NAS appliance that supports Hybrid Columnar Compression

•Cluster Ready System for HA

•Can be easily integrated with Oracle database

•ZFS Filesystem

•Inline de-duplication and compression to save space

•Filesystem snapshot and cloning

66

Why We Chose This Solution

77

Problem Presented

•Performance Slowness•Batch jobs taking 9hrs to run•User experience affected as some had to work in weekends•Database backups•OLTP reports not able to run and times out•Some important reports had to run in weekend as it took 14hrs

•No High Availability

•Database Refresh•Takes almost 16-20hrs to refresh

88

Problem Analysis

Performance•Database IO Bound•Server Low RAM

Environment Refresh

Cost

99

Solution Accepted

•Platform as a service to customer

•Oracle Exadata for production DB hosting•Consolidation of all production databases

•Sun ZFS and Sun Fire servers for test/dev environment•RAC environment to be created•Consolidation of all test/dev databases

1010

Exadata BenefitsData Warehouse Jobs•From 9hrs to 90mins•From 8hrs to 40mins•ETL now running in minutes from hours before

OLTP Performance•Reports running in minutes from hours

Business Improvement•Staff engagement•Cost

Faster RMAN Backup

Consolidation

1111

Oracle ZFS Appliance Benefits

Hybrid Columnar Compression

Quick Refresh of Test/Dev from standby database

Consolidated platform for test databases

Attached to Exadata via IB

Clone DB requires very small space

1212

How We Did It

1313

Backup: High Level Design

Backup Server

10G Ethernet

OEM Server

OEM Repository/RMAN Catalog

Exadata Server

SAN

Tape Backup

OEM agent

/backup (DNFS mount)

Compute node Operating SystemOracle RDBMS/RAC binariesOCR and voting disk backup Database RMAN backupsRegular IB configuration export

1414

Sample: OEL Backup Script# Create snapshot backup of filesystems /, /u01, /bootlvcreate -L1G -s -n root_snap /dev/VGExaDb/LVDbSys1e2label /dev/VGExaDb/root_snap DBSYS_SNAP mkdir -p /root/mnt mount /dev/VGExaDb/root_snap /root/mnt lvcreate -L5G -s -n u01_snap /dev/VGExaDb/LVDbOra1 e2label /dev/VGExaDb/u01_snap DBORA_SNAP mkdir -p /root/mnt/u01 mount /dev/VGExaDb/u01_snap /root/mnt/u01

# Backup the snapshots to the NFS mount /OS_backupcd /root/mnt # Check for any significant errors reported in the /tmp/backup_tar.stderr filetar -pjcvf /OS_backup/node1_backup.tar.bz2 * /boot --exclude node1_backup.tar.bz2 >/tmp/backup_tar.stdout 2>/tmp/backup_tar.stderr

# Remove snapshotscd /umount /root/mnt/u01umount /root/mnt /bin/rm -rf /root/mnt lvremove -f /dev/VGExaDb/u01_snaplvremove -f /dev/VGExaDb/root_snap

1515

Backup: IB Switch

1616

Monitoring: High Level Design

Exadata Storage cells registered in ASRExadata compute nodes registered in ASRCompute node and Storage Cell ILOM registered in ASR

Exadata Machine

OEM Server

OEM Agent

Breached SoftwareAlert Triggered Alert

Gen-I Engineer

SM

S S

ent

Oracle Support

Breached Hardware Alert

ASR Manager

SR

Details

1717

DR – High Level Design

ExadataExadata

10G Ethernet 10G Ethernet

OEM ServerOEM Server

OEM repository/ RMAN Catalog

OEM repository/ RMAN Catalog

Load

Bal

ance

r

Primary Site DR Site

Redo Shipping

RAC Database

OEM Agent OEM AgentDark Fiber

ZFS 7420

RAC Database

Redo Shipping

Test/DEV DBRac1

Rac2

IB

1818

NFS SharesKey Notes (762374.1)•ZFS Shares mounted as NFS on Exadata DB nodes

•DNFS configured for each share used by DB

•To configure DNFS, run the following for each Oracle Home•make -f ins_rdbms.mk dnfs_on•Restart each instance from the Oracle Home

•Add NFS shares to /etc/oranfstab for global accessserver: headapath: 192.168.20.10export: /export/db_logs mount: /u01/app/oracle/db/recoserver: headbpath: 192.168.20.11export: /export/db_data mount: /u01/app/oracle/db/data

•Once DNFS is enabled, the following message will be displayed in alert log file•Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0

•Following query can be used to check if DNFS is used•select * from v$dnfs_servers;

1919

NFS Mount Options (RAC)

MOSS: 359515.1

Operating SystemMount options for    Binaries

Mount options for Oracle Datafiles

Mount options for CRS Voting Disk and OCR

Sun Solaris

rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,

vers=3,suid

rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,forcedirectio, vers=3,suid

rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3,noac,forcedirectio

AIX (5L)

rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,

vers=3,timeo=600

cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600

cio,rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600

HPUX 11.23

rw,bg,vers=3,proto=tcp,noac,hard,nointr,timeo=600,rsize=32768,wsize=32768,suid

rw,bg,vers=3,proto=tcp,noac,forcedirectio,hard,nointr,timeo=600,rsize=32768,wsize=32768,suid

rw,bg,vers=3,proto=tcp,noac,forcedirectio,hard,nointr,timeo=600,rsize=32768,wsize=32768,suid

Windows Not Supported Not Supported Not Supported

Linux x86rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp, vers=3,timeo=600, actimeo=0

rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600

rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,actimeo=0,vers=3,timeo=600

Linux x86-64

rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,vers=3,timeo=600, actimeo=0

rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,actimeo=0, vers=3,timeo=600

rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0

Linux - Itanium

rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,vers=3,timeo=600, actimeo=0

rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,actimeo=0, vers=3,timeo=600

rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0

2020

DB Refresh1. Suspend managed recovery of standby DBPrimary DBSQL> alter system set log_archive_dest_state_2=DEFER sid=’*’; DR DBSQL> alter database recover managed standby database cancel;

2. Take snapshot of the of the filesystem

3. Resume managed recovery of standby DBPrimary DBSQL> alter system set log_archive_dest_state_2=ENABLE sid=’*’;DR DBSQL > alter database recover managed standby database using current logfile disconnect;

4. Create clone of snapshot

2121

DB Refresh5. Mount the filesystems on both the test RAC servers

6. Create a softlink to refer old location of standby redo logsUpdate db_file_name_convert and log_file_name_convert

7. Startup DB in mount state

8. Drop the standby redo logs

9. Activate the databaseSQL> alter database activate standby database ; SQL> shutdown immediate ; SQL> startup

10. Use NID to rename the database.

2222

DB Refresh

2323

Technical Issues

2424

Issues: Exadata

Exadata IPoIB protocol died•The cause is still not known fully•This happened when ZFS was hooked to Exadata via IB cable

Workaround•Oracle support has confirmed that ZFS can be hooked to Exadata via IB•Transmit queue length to be increased to 1000 for the IB interfaces#ifconfig ib0 txqueuelen 1000#ifconfig ib1 txqueuelen 1000

2525

Issues: ZFS NAS ApplianceMemory leak at the server heads•Cause known to be with SNMP service•Firmware to be patched until Oracle releases it•Workaround is to disable SNMP service

HCC on ZFS needs SNMP service to be enabled

2626

Q & A