optimize oracle 12c with zfs storage - · pdf fileoptimize oracle 12c with zfs storage mike...
TRANSCRIPT
OPTIMIZE ORACLE 12C WITH ZFS STORAGE
MIKE GANGLER, LEAD ORACLE DBA SECURE-24
Performance.Innova.on.Trust.WearetheManagedCloud.
2 Secure-24 Confidential – Internal – Copyright © 2015 Secure-24 and/or its affiliates. All rights reserved.
3
ABOUT MIKE GANGLER
• Oracle ACE with robust database creden5als • DBA for over 28 years, working with Oracle since version 4 • Lead and Senior Database Specialist at Secure-24 • Currently serving on the board of the Southeast Michigan Oracle
Professionals (SEMOP) group – www.meetup.com • Charter member of the Board of Directors for the Interna5onal Oracle Users
Group (IOUG) – www.ioug.org • Follow me on my Blog hQp://mjgangler.wordpress.com and on twiQer!
@mjgangler
Communi.es Educa.on
Join for as low as $150
SELECT Journal Resource Center IOUG Press Webinars & Podcasts IOUG Forum 5 Minute Briefing
Plus get access to IOUG’s content library, peer-to-peer networking, and more! Corporate options also available!
SOUTHEASTERN MICHIGAN ORACLE PROFESSIONALS (SEMOP)
• Technology and Database Professionals - hQp://www.meetup.com/SouthEast-Michigan-Oracle-Professionals/
– Contact: Mike Gangler – [email protected]
– Meet monthly – 2nd Tuesday of the month
TODAY’S DISCUSSION
6
Learn how Secure-24 leverages Oracle ZFS Storage, Oracle Intelligent
Storage Protocol (OISP), Templates and Oracle 12C RAC to implement a
Private Cloud for Oracle 12c RAC applica5ons. This pairing of Database and
Storage hardware through Oracle’s VCA has enabled Secure-24 to take
advantage of the unique integra5on capabili5es such as OISP which enables
automa5c and dynamic tuning of storage se^ngs such as record size.
Secure-24 Confidential – Internal – Copyright © 2015 Secure-24 and/or its affiliates. All rights reserved.
• Simplicity ” – Simplify the environment with new technology
• Scalability – Technology should be able to grow and scale as the company grows.
• Automa5on – Technology should have the ability to be automated and repeatable process
• Overall reduc5on of cost – Technology should drive down the overall cost
• Solu5on driven by Customer requirements and demands. – This is the reason for looking at the technology
SECURE-24 TECHNOLOGY GOALS
BUSINESS CASE
• Simplify Oracle 12c RAC deployments • Eliminate complexity around storage tuning • Decrease support 5me through transparency and simplicity • Fully Oracle Support technology stack
9
Hardware Specifica5ons Overview
ORACLE PRIVATE CLOUD APPLIANCE
OracleX86Servers OracleZFSStorageAppliance• Redundantcontrollers,highperformancevirtualstorage
NetworkInfrastructure• OracleFabricInterconnectF1-15(VirtualNetwork)• OracleSwitches• Fullyconfiguredfabricnetworking,nosetuprequired• Integrateswithexis.ngEthernetandstoragenetworks
Fabric Interconnect
Fabric Interconnect
Compute Nodes
Compute Nodes
Mgmt Nodes
Management Storage
InfiniBand Expansion
Ethernet Switch
OracleX86Servers• Upto25nodes• CPU:Upto900cores• Memory:Upto6.4TBaggregate• SupportsanyLinux,SolarisandWindowsapplica.ons
Netw
orking M
anagement
40Gb/sServer-to-Server
InfiniBand Expansion
» HighspeedInfiniBandnetwork–mul.ple40Gbps
» InfiniBandconnectedstorage
» Dedicatedmanagementsystemsincluded
» OracleVMandOracleLinuxincluded
» TestedandCer.fiedbyOracle
» So_wareDefinednetworkingthroughFabricInterconnect
• Fast “Power-on to Produc5on” - Automa5on – 2 hours vs. 1 month on legacy infrastructure
• Mixed workloads - Scalable – Oracle DB, Oracle Applica5ons, Microsoe, Custom apps
• More efficient Licensing – Reduc5on of cost – Trusted Par55ons Licensing
• Rapid Applica5on Deployment – Simplicity and automa5on – Pre-built Oracle VM templates
• Simplified Support - Simplicity – One Hand to Shake (One Throat to Choke)
• Lower Acquisi5on and Deployment Costs
• Impressive performance – Customer requirements
WHY WE SELECTED PRIVATE CLOUD APPLIANCE
ORACLE 12C RAC DESIGN OVERVIEW
• Oracle PCA and ZS3-2 Storage
• Single Node or mul5ple nodes with RAC
• Oracle 12c Enterprise Edi5on
• Oracle ZFS storage array with OS 8.3+
• Database connec5ons made from Oracle using Direct NFS
• ZFS Storage – Flash enabled performance storage
• OISP (Oracle Intelligent Storage Protocol) for storage op5miza5on
• No “ASM”
X3-2
VIR
TUA
L CO
MP
UTE
AP
PLIA
NC
ES P
REARPS
MAJ OR
MINOR
US E R
C RIT IC AL
Z FS S TORAGEZ S 3-2
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S PREARPS
MAJ OR
MINOR
US E R
C RIT IC AL
Z FS S TORAGEZ S 3-2
DESIGN OVERVIEW
• Local OS disks Stored on Oracle VM data repository
• Data, Logs, and Vo5ng disks presented via NFS and dNFS
VM 2
dNFS
VM
NAS,SAN,iSCSI
OracleVMServers
OracleVMServerPool
VM VMVM
SIVM 1
RAC RAC
ZFSStorage
ZFS POWERED
• OISP MOS Doc ID 1943618.1
• ZFS Admin Guide hQp://docs.oracle.com/cd/E51475_01/pdf/E52872.pdf
• ZFSHybridArraywith3typesofcaching• DRAM,SLCWriteFLASH,
MLCReadFLASH
• OISPOp.malRecordSize&Latency• TheOracledNFSclientpassestheop.mal
recordsizetotheZFSStorageApplianceforeachNFSv4writerequest
• TheOracledNFSclientpassestheop.malwritebiastotheZFSStorageApplianceforeachNFSv4writerequest..
ZFS SETUP
• Create our 3 shares
• OCR Vo5ng Disk – This will be raw NFS v3
– hQps://docs.oracle.com/database/121/CWADD/votocr.htm#CWADD92368
– We leave Write Bias as Latency and Record Size as 128K (Generic)
• Data and Logs files – This is where the magic of OISP comes in.
– Instead of having to create mul5ple different volumes for redo, archive, control, data, temp etc. We simple create 2 volumes for Logs and Data
– Write Bias and Record Size will be set via database cue’s through OISP and op5mized for the database.
STANDARD NFS ARCHITECTURE
DNFS AND OISP
LINUX CONFIGURATION SETUP
dNFS – Fairly simple example
• # cat oranfstab.example
• server: 10.10.10.10
• nfs_version: nfsv4
• export: /export/U02 mount: /u02
• export: /export/U03 mount: /u03 – dNFS is fairly straight forward in
12c setup guide • hQps://docs.oracle.com/database/121/CWLIN/
storage.htm#CWLIN279
– This is where NFSv4 is setup
– Many other op5ons are possible, but simple is possible too • Old paper, s5ll valid - hQp://www.oracle.com/
technetwork/ar5cles/directnfsclient-11gr1-twp-129785.pdf
– Most dNFS guides are for 11g, 12c is the same except dNFS is Na5vely part of 12c
– Troubleshoo5ng • (Doc ID 1464567.1)
DNFS LINUX CONFIGURATION SETUP
dNFS Enable:
Ln –s $ORACLE_HOME/lib/libnfsodm11.so $ORACLE_HOME/lib/libodm11.so
Disable
Ln –s $ORACLE_HOME/lib/libodmd11.so $ORACLE_HOME/lib/libodm11.so
DNFS LINUX CONFIGURATION SETUP
dNFS Verify• bash-3.2$ ps -ef | grep dbworacle 23126 1 0 06:53 ? 00:00:00 ora_dbw0_MYINST1oracle 23128 1 0 06:53 ? 00:00:00 ora_dbw1_MYINST1oracle 23246 22377 0 07:04 pts/0 00:00:00 grep dbw
• bash-3.2$ /usr/sbin/lsof -p 23126COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEoracle 23126 oracle cwd DIR 104,3 4096 14811513 /u02/app/oracle/product/12.1.0/dbsoracle 23126 oracle rtd DIR 104,3 4096 2 /oracle 23126 oracle txt REG 104,3 210824714 15892486 /u02/app/oracle/product/12.1.0/bin/oracle
• ...oracle 23126 oracle 32u IPv4 322388 TCP mydbhost:64553->mynfshost:nfs (ESTABLISHED) oracle 23126 oracle 33u IPv4 322389 TCP mydbhost:21549->mynfshost:nfs (ESTABLISHED)
-- dNFS setup guide • hQps://docs.oracle.com/database/121/CWLIN/storage.htm#CWLIN279
LINUX CONFIGURATION SETUP
Oracle Linux 6 – uek Kernel – U5lize dtrace in Linux
– Ksplice for non-disrup5ve kernel patching
– UEK kernel op5miza5on for Oracle DB
Setup Mount Points – Setup Vo5ng disk
• Must be NFSv3
• Not Part of dNFS
• Mount op5ons important (Doc ID 359515.1)
• rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,noac,vers=3,5meo=600,ac5meo=0
– Setup Logs and Data • Must be NFSv3 (dNFSv4 setup to come later)
• Mount op5ons important (Doc ID 359515.1)
• rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,ac5meo=0,vers=3,5meo=600
LINUX CONFIGURATION SETUP
Install Oracle Pre-Install RPM
– Op5onal but makes life easier
– # yum install oracle-rdbms-server-12cR1-preinstall
– This will download all required dependencies, create the oracle user id, modify sysctl.conf and modify limits.conf.
– hQps://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1
Test snmp (needed for OISP)
– Doc ID 1943618.1
– # snmpget -v1 -c public 10.10.10.10 1.3.6.1.4.1.42.2.225.1.4.2.0
SNMPv2-SMI::enterprises.42.2.225.1.4.2.0 = STRING: "Sun ZFS Storage 7330"
Setup for dNFS – Oracle wants us to tune for
dNFS, so we will
• # vi /etc/sysctl.conf • # • #
hQps://docs.oracle.com/database/121/CWLIN/storage.htm#CHDEFHFC
• #7.3.4 Checking TCP Network Protocol Buffer for Direct NFS Client
• # • net.ipv4.tcp_rmem = 4096
87380 4194304 • net.ipv4.tcp_wmem = 4096
16384 4194304
ORACLE 12C INSTALLATION Oracle 12c Enterprise Edi5on
– Install Oracle 12.1.0.2
Confirm dNFS and OISP setup
– $ egrep I '"Sun ZFS"|Direct' alert_*.log • Found NAS server Sun ZFS Storage 7420
• Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0
• Direct NFS: channel id [0] path [10.80.216.18] to filer [10.80.216.18] via local [] is UP
• Direct NFS: channel id [1] path [10.80.216.19] to filer [10.80.216.18] via local [] is UP
• Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS version 4
• Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS version 4
– “Found NAS server” indicates SNMP has found the ZFSSA. Requirement for OISP.
– ODM/Direct NFS indicates that dNFS has been ac5vated using NFSv4.
Apply required patch
– Patch 19630940
– Resolves issue with DB Analy5cs
TYPICAL DEPLOYMENT
Typical Process For Creating and Deploying New Applications
1to3weeks
1-2days1day1-5days1-2days1-5days Start
PCA RAPID DEPLOYMENT
• Pre-made, cer5fied virtual machine templates, including applica5on, opera5ng system, and database available at hQps://edelivery.oracle.com
• Save Weeks/Days in Configura5on Time
– Pre-built, pre-configured Templates
– Applica5on, Middleware, and DB
– Oracle DB, JD Edwards, EBS, PeopleSoe, and many more…. 90+ templates
Start-upinOracleVMPool
ImporttoOracleVMManager
OracleVMServers
ServerPool2Customize&Save
asGoldenImage
E-DeliveryFile
JDEdwards
DownloadfromOracle.com
File
JDE
VM1
JDE
APPLICATION TEMPLATES & ASSEMBLIES
DB
WebLogicServer
DeploymentPoliciesConfigura.onMetadata
SOA_PROD_1
OracleVMAssemblyProduc.onEnvironment:SOA_PROD_1
VM
VM
VM
VM
VM
VM
Deploy
Deployapplica.onintoOracleVMManageras‘GoldenImage’• Clone,ini.alize,andstart-upinstance• Upandrunninginhoursvs.daysorweekswithliilerisk
OracleVMServers
StorageRepository
ServerPool2
Deployed:Produc.onEnvironmentSOA_PROD_1
GuestAddi.ons• Allowsdirectintegra.onbetweenguestso_wareandvirtualiza.onlayer• Assistsinorchestra.onofcomplex,mul.-VMdeployments
OVF-based
3
NOTES
• Oracle Database 11g RAC with ZFS Storage Appliance and dNFS on SPARC: hQp://www.oracle.com/technetwork/server-storage/sun-unified-storage/documenta5on/oracledb-rac-sparc-zfssa-2414324.pdf
• Op5mizing Storage for Oracle Database 11gR2 with ZFS Storage Appliance: hQp://www.oracle.com/technetwork/server-storage/sun-unified-storage/documenta5on/oracle11gr2-zfssa-bestprac-2255303.pdf
• How to enable OISP on ZFS Storage Appliance for 12c: hQps://support.oracle.com/epmos/faces/DocumentDisplay?id=1943618.1
ANALYST DATA
• hQp://www.oracle.com/us/corporate/analystreports/industries/zs3-storage-appliance-2033590.pdf
VISIT SECURE-24’S TABLE!
• Enter for a chance to win a $250 Amazon Gie Card! • Meet with other S-24 execu5ves and technical resources • Discuss your organiza5on’s Cloud Strategy for 2016 • Learn more about our capabili5es with Oracle’s Virtual Compute Appliance
• Learn about our newest product launch: Engineering Desktops
