creating a maximum availability architecture with sparc supercluster
DESCRIPTION
Learn how to deploy the integrated capabilities of Oracle’s SPARC SuperCluster to meet the most-demanding requirements for very high levels of availability, scalability, performance, and disaster recovery. This session discusses how to create a maximum availability architecture with Oracle Real Application Clusters (Oracle RAC), Oracle Active Data Guard, Oracle Database’s Oracle Recovery Manager (Oracle RMAN) feature, Oracle Enterprise Manager Cloud Control, and Oracle Enterprise Manager Ops Center. The session also examines how to leverage the built-in redundancy and security of the SPARC SuperCluster and the ease of migration provided by Oracle Solaris 11 to reduce risk and time to implementation.TRANSCRIPT
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.2
Creating a High Availability Architecture with SPARC SuperClusterCarter WrightDatabase Engineer, Computer Knowledge, Inc.
Bruce ChapmanPrincipal Software Engineer, Oracle
Dean HalbeisenSolutions Manager, Oracle
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
ProgramAgenda
SPARC SuperCluster High Availability
Implementation
SPARC SuperCluster High Availability
Optimized Solutions High Availability
Examples
Next Steps / Resources
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.4
SPARC SuperCluster High Availability
Leverage the HA features built in to the SPARC SuperCluster– Oracle Real Application Cluster(RAC), Oracle Exadata Storage Server,
Oracle Automatic Storage Management(ASM), Network and Power redundancies
– Clustering of applications in Oracle VM Server For SPARC and Oracle Solaris Zones on SPARC T4-4 nodes
Use of Oracle Data Guard Use of Oracle RMAN Use of Sun ZFS Storage Appliance 7420 and ZFS Replication
Large Government Law Enforcement Agency Implementation of HA
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5
SPARC SuperCluster High Availability
12,000 to 15,000 concurrent users worldwide during peak hours
Database greater than 1.6TB and growing
1500 to 2000 SQL executions per second per RAC node during peak hours (~4000 SQL execs per second combined)
Greater than 1.5 million blocks per second read from each SGA during peak hours ( > 3 million blocks per second combined )
900K to 1.2MB of redo written per second (combined RAC nodes)
Query response time typically less than 300ms
Data Guard latency less than three seconds to standby site
ZFS replication latency less than three seconds to standby site
Each RAC node CPU utilization at peak load 15% to 20%
Workload Overview
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.6
SPARC SuperCluster High AvailabilitySPARC SuperCluster Configuration
ExternalServer1
ExternalServer3
ExternalServer2
10GbE ZFS 7420 StorageRMAN BackupsData Pump Dir
InfiniBandN
od
es 1
& 2
No
des
3 &
4
Infi
niB
and
Solaris 10
600G10K SAS
App SvrApp Zone
App Zone
Exadata Data Cells216TB RawASM High
72TB Usable3 X ASM Disk Groups
Solaris 10
App Zone
Solaris 11
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.7
SPARC SuperCluster High Availability
SuperCluster installation does provide a fully functional Oracle RAC install, but it has to be configured to meet your applications needs
– Database Hardening must be performed– Listener configurations (scan and local) can be modified– You configure your RAC Workload Services– Network settings – 10Gb and 1Gb interfaces only
TCP window, MTU’s Leave IB settings alone
– Backup and recovery is your responsibility– Application tuning is your responsibility
Database Configuration
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.8
SPARC SuperCluster High Availability
– OLTP or Data Warehouse (or both)– SGA Settings can be modified
We changed only a few settings
– SGA, PGA Aggregate Target– Data Guard parameters
SPARC SuperCluster uses Auto SGA and Auto PGA, not full AMM (see pg 18-18 11g Concepts Guide)
Can change the quantity and size of Redo Logs (if needed) Users, tablespaces, datafiles
– Roles and privileges– Datafile size and quantity
Database Configuration
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.9
SPARC SuperCluster High AvailabilityWorkload Services
Smaller Apps
Data Guard
Non-RAC COTS
Data Guard
Single Node Services
Main App 1
RMAN Service
Main App 2
Load Balanced Services
SCAN Listeners
SCAN + Service Name
Intranet
12K-15K Active Concurrent Users Accessed World Wide
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.10
SPARC SuperCluster High Availability
Data Guard setup– Physical Standby, High Performance Mode, Async
Multi-instance Primary, single instance standby Active Guard enabled at standby site Use standby for
– Reporting and Read-only access– Smoke testing application deployments
Make standby a temporary Primary and restore back to standby using Flashback Database
Use standby as the Last deployment before deployment to prod site– Other live applications to leverage investment
Data Guard on SPARC SuperCluster
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.11
SPARC SuperCluster High AvailabilityReplication Strategy
Primary SPARC SuperCluster
Standby SPARCSuperCluster
ZFS Storage Appliance 7420
Data Guard ReplicationDatabase Content
ZFS ReplicationExternal Content
ZFS Storage Appliance 7420
Latency to Remote Site <3 Seconds
Latency to Remote Site <3 Seconds
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.12
SPARC SuperCluster High Availability
Backupsets stored on ZFS 7420 (not in ASM)– Accessible during crisis
– Leave space to database in Exadata
– Backup file system mounted to both nodes for parallelism
– ZFS RAID 10
Use RAC Workload Service– Allows RMAN parallelism across RAC
rman target sys/<pw>@rman_svc
Improves speed
Parallelism 16 with 16 Channels allocated– RMAN will allocate 8 on node 1, 8 on node2
RMAN Configuration
ZFS 7420 StorageRMAN Backups
No
de
2N
od
e 1
Solaris 10
App Zone
Solaris 11
Solaris 10
App Zone
Solaris 11
InfiniBand
RM
AN
Ser
vice
s
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.13
SPARC SuperCluster High Availability
Level 0 weekly– 7 day recovery window
Level 1 nightly– Cumulative Incremental
Backup both primary and standby sites separately
RMAN Backup Operations
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.14
SPARC SuperCluster High Availability
Recovery Catalog is required Flashback database enabled at both sites db_unique_name required Backups taken at both sites
– BACKUP and RESTORE command uses TAG ‘system_name’ option when creating backups
Distinguishes each site in Recovery Catalog
Archivelog deletion policy on Primary– Ensures archivelogs are not deleted by RMAN before getting to standby site
CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY;
RMAN Configuration and Management in a RAC Environment
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.15
SPARC SuperCluster High Availability
Grid Control 12c– Good graphs and drill down capabilities
Custom Scripts (bash)– Exadata Cell monitoring –
No Grid 12c plug-in yet for SPARC SuperCluster with Exadata– Alert monitoring to send notifications to our NOC
Monitoring and Management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.16
SPARC SuperCluster High Availability
Early adopter of SPARC SuperCluster and it was not your common install being a customer engineered application, had to train a new team with clearance to work on the systems to perform the install
Purchased three SPARC SuperCluster systems in December 2011 Pre-Prod, Production, Standby DR site
First system delivered early February, next system two weeks later, third system two weeks after that
Each system was installed and running our application within three weeks of each delivery.
Within a 9 week period, we had three fully operational SuperClusters in three different data centers.
Went live production in July of 2012. Successful worldwide launch
Instantiated DR site with Oracle Data Guard shortly thereafter.
Time to Deployment for an Engineered Solution
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.17
SPARC SuperCluster High Availability
SPARC SuperCluster High Availability White papers on www.cknowledge.com
– How To Guide: RAC Workload Services – 11gR2– How To Guide: Building Data Guard– RMAN Quick Reference
Knowledge to Share
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.18
ProgramAgenda
SPARC SuperCluster High Availability
Implementation
SPARC SuperCluster High Availability
Optimized Solutions High Availability
Examples
Next Steps / Resources
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.19
SPARC SuperCluster: High AvailabilityHardware HA Features
Dual Power Distribution Units Both connected to each server or switch in the SuperCluster, each with redundant power supplies.
Ten Gigabit Ethernet NetworkingEach logical 10GbE interface has two physical ports for connection to separate external lab switches.
Infiniband Networking- 3 IB switches for availability / growth.
- Each logical IB connection to T4-4, Exadata DB Storage Cells, and 7320 Storage Appliance has two physical InfiniBand ports.
Mirrored Disks All OS Disks On T4-4 Domains are mirrored.
SPARC T4-4 HA FeaturesHot swap redundant PS, hot swap PCI-E, DIMM off lining, predictive self healing
7320 ZFS Storage Appliance-Two Storage head nodes each with redundant networking, SCSI.-Configurable RAID / spare options for shares. Database Storage CellsEach with redundant IB and configurable disk mirroring.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.20
SPARC SuperCluster: High AvailabilitySoftware HA
Multiple RAC instancesEnabling DB to tolerate and react to instance and node failures automatically with zero to near zero impact on applications.
Oracle Grid Infrastructure Clusterware and ASM provide abstraction of DB storage cell nodes and disks, also able to continue functioning despite disk or DB storage node failure.
Oracle Solaris Cluster One of several high availability options for application domains. Configurable number of Domains for horizontal scaling and failover.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.21
ProgramAgenda
SPARC SuperCluster High Availability
Implementation
SPARC SuperCluster High Availability
Optimized Solutions High Availability
Examples
Next Steps / Resources
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.22
T4-4 Node 4
Oracle Solaris
Oracle Solaris 11
Database Domain
GP Domain
T4-4 Node 3
Oracle Solaris
Oracle Solaris 11
Database Domain
GP Domain
InfiniBand Network
Oracle Exadata Storage Servers
10 GbE Data Center Network
T4-4 Node 2
Oracle Solaris
T4-4 Node 1
Oracle Solaris 11
Database Domain
GP Domain
OracleSolarisCluster
Oracle RAC
11g R2
High Availability
Scale
High Availability on SPARC SuperClusterOracle Optimized Solutions
Oracle Solaris
Oracle Solaris 11
Database Domain
GP Domain
T4-4 Node 2
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.23
High Availability / Fail Over Test ResultsExtremely fast failure detection and service recovery times
Detection
SECONDS
Web server node failure 110
61
OPMN/OC4J failure
General purpose domain failure
Database domain failure
0
Recovery
120
5 10
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.24
Driving Risk Down • Driving Innovation Up • Active Partnership with Oracle
Down • Driving Innovation Up • Active Partnership with Oracle • Driving Risk
Up • Active Partnership with Oracle • Driving Risk Down • Driving Innovation
Driving Risk Down • Driving Innovation Up • Active Partnership with Oracle
Down • Driving Innovation Up • Active Partnership with Oracle • Driving Risk
Up • Active Partnership with Oracle • Driving Risk Down • Driving Innovation
Driving Risk Down • Driving Innovation Up • Active Partnership with Oracle
Up • Active Partnership with Oracle • Driving Risk Down • Driving Innovation
Driving Risk Down • Driving Innovation Up • Active Partnership with Oracle
Down • Driving Innovation Up • Active Partnership with Oracle • Driving Risk
Up • Active Partnership with Oracle • Driving Risk Down • Driving Innovation
Driving Risk Down • Driving Innovation Up • Active Partnership with Oracle
Down • Driving Innovation Up • Active Partnership with Oracle • Driving Risk
Tested, Secure, Flexible
Driving Risk Down
Driving Innovation Up
Active Partnership with Oracle
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.25
ProgramAgenda
SPARC SuperCluster High Availability
Implementation
SPARC SuperCluster High Availability
Optimized Solutions High Availability
Examples
Next Steps / Resources
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.26
SPARC SuperCluster Conference SessionsSession Date Time Room
Accelerate Oracle E-Business Suite Deployment with SPARC SuperCluster Monday 12:15PM Moscone West - 2001
SPARC SuperCluster: Architecture, Roadmap, and Partner Services Opportunities Monday 12:15PM Marriott Marquis – Golden Gate B
My Oracle Support: Tips and Techniques for Getting the Best Hardware Support Monday 1:45PM Moscone West - 2001
Top 10 Database Performance Tips for SPARC Systems Running Oracle Solaris Monday 1:45PM Moscone West - 2000
Building a Flexible Enterprise Cloud Infrastructure on Oracle SPARC Systems Monday 4:45PM Moscone West - 2001
Database and Application Consolidation on SPARC SuperCluster Monday 4:45PM Moscone West - 2000
Oracle's SPARC Server Strategy Update Tuesday 10:15AM Moscone West - 2001
Oracle Engineered Systems Backup and Recovery Best Practices Tuesday 10:15AM Moscone South - 252
Transform Data Center TCO with Oracle Optimized Servers: A Customer Panel Tuesday 11:45AM Moscone West - 2001
Deep Dive into SPARC SuperCluster: Engineered for Extreme Efficiency Tuesday 1:15PM Moscone West - 2001
Accelerate Your SAP Landscape with the Oracle SPARC SuperCluster Tuesday 5:00PM Moscone West - 2001
High Performance Security for Oracle Applications Using SPARC T4 Systems Tuesday 5:00PM Moscone West - 2000
SPARC Processor Update Wednesday 10:15AM Moscone West - 2000
Creating a Maximum Availability Architecture with SPARC SuperCluster Wednesday 11:45AM Moscone West - 2000
Improve PeopleSoft HCM Performance and Reliability with SPARC SuperCluster Wednesday 5:00PM Moscone South - 200
Managing Sun Servers and Oracle Engineered Systems with Oracle Enterprise Manager Thursday 12:45PM Moscone West - 2000
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.27
SPARC SuperCluster DemosDemo Location
Accelerating SAP Queries and Reporting on SPARC SuperCluster Moscone South -102
Bullet Proof E-Business Suite on SPARC SuperCluster Moscone South -101
Bullet proof PeopleSoft on SPARC SuperCluster Moscone South-131
Consolidate your Datacenter: Enterprise Cloud Infrastructure on SPARC Moscone South-150
Explore the Benefits of Oracle SPARC SuperCluster Moscone South-104
Mission Critical Availability and Disaster Recover for Your Cloud Moscone South-116
Mission Critical SPARC-Powered Infrastructure as a Service Moscone South-157
Oracle SPARC SuperCluster: World’s First General Purpose Engineered System Moscone South-106
Oracle SPARC SuperCluster T4-4 to Improve Efficiency and Cut Costs Moscone South-105
The Data Center Superhero: Backup and Recovery Systems Moscone South-107
Transform an Enterprise Cloud Infrastructure: Oracle SPARC SuperCluster Moscone South-130
Transform and Optimize Revenue Management with SPARC SuperCluster Moscone South-108
Engineered System Showcase - SPARC SuperCluster Consolidation Moscone North Lower Lobby
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.28
Resources
Computer Knowledge White Papers– http://www.cknowledge.com
Oracle Maximum Availability Architecture Guides– http://www.oracle.com/technetwork/database/features/availability/maa-090
890.html
Optimized Solutions– http://www.oracle.com/us/solutions/oos/overview/index.html
In depth information
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.29
Graphic Section Divider
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.30
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.31