oracle rac sig - implementing a dual site extended distance rac cluster.pps
TRANSCRIPT
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
1/58
(c) Copyright DSP Global Limited. All rights reserved. 1
UKOUG 2007
Implementing a dual site extendeddistance RAC Cluster
Dev Nayak04/12/07
DSP Global Limited
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
2/58
(c) Copyright DSP Global Limited. All rights reserved. 2
Agenda
Historical Oracle platform incorporating HA and DR
The Next Generation Oracle platformLinux v HP/UX Solutions
What we built
Does it work?
Conclusion
Questions
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
3/58
(c) Copyright DSP Global Limited. All rights reserved. 3
Historical Oracle Platform incorporating HAand DR
Standard Oracle Platform
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
4/58
(c) Copyright DSP Global Limited. All rights reserved. 4
Historical Oracle Platform incorporating HAand DR
Standard Oracle Platform
The server is a key SPOF
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
5/58
(c) Copyright DSP Global Limited. All rights reserved. 5
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOF
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
6/58
(c) Copyright DSP Global Limited. All rights reserved. 6
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
7/58
(c) Copyright DSP Global Limited. All rights reserved. 7
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scaleable
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
8/58
(c) Copyright DSP Global Limited. All rights reserved. 8
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scaleable
Is Single Site
The site is a key SPOF
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
9/58
(c) Copyright DSP Global Limited. All rights reserved. 9
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scaleable
With Dataguard
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
10/58
(c) Copyright DSP Global Limited. All rights reserved. 10
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scalable
With Dataguard
Removes site as SPOF
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
11/58
(c) Copyright DSP Global Limited. All rights reserved. 11
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scalable
With Dataguard
Removes site as SPOFIs Active-Passive
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
12/58
(c) Copyright DSP Global Limited. All rights reserved. 12
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scalable
With Dataguard
Removes site as SPOFIs Active-Passive
Scalability is costly
Especially as each node requires alicense
Regardless of it being passive
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
13/58
(c) Copyright DSP Global Limited. All rights reserved. 13
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scalable
With DataguardRemoves site as SPOF
Is Active-Passive
Scalability is costly
Especially as each node requires alicense
Regardless of it being passive
Thus, typical DR site provides lowerservice level
l l l f
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
14/58
(c) Copyright DSP Global Limited. All rights reserved. 14
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scalable
With DataguardRemoves site as SPOF
Is Active-Passive
Scalability is costly
Especially as each node requires alicense
Regardless of it being passive
Thus, typical DR site provides lowerservice level
Requires Failover
Failover manual and failover time ishigh
Hi i l O l Pl f i i HA
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
15/58
(c) Copyright DSP Global Limited. All rights reserved. 15
Historical Oracle Platform incorporating HAand DR
Oracle RAC Platform
Removes server as SPOFIs Active-Active
Is Scalable
With Split-site ServiceGuardRemoves site as SPOF
Is Active-Passive
Scalability is costly
Thus, typical DR site provides lowerservice level
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
16/58
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
17/58
(c) Copyright DSP Global Limited. All rights reserved. 17
The Next Generation Oracle platform
Oracle Platform
Removes site as SPOFIs Active-Active
Is Scaleable
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
18/58
(c) Copyright DSP Global Limited. All rights reserved. 18
The Next Generation Oracle platform
Oracle Platform
Removes site as SPOFIs Active-Active
Is Scaleable
I.e. Oracle RAC Platform
that removes site as the SPOF
Answer: Stretch a RAC Platform!
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
19/58
(c) Copyright DSP Global Limited. All rights reserved. 19
The Next Generation Oracle platform
Oracle Platform
Removes site as SPOFIs Active-Active
Is Scaleable
I.e. Oracle RAC Platform
that removes site as the SPOF
Answer: Stretch a RAC Platform!
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
20/58
(c) Copyright DSP Global Limited. All rights reserved. 20
The Next Generation Oracle platform
Oracle Platform
Removes site as SPOFIs Active-Active
Is Scaleable
I.e. Oracle RAC Platform
that removes site as the SPOF
Answer: Stretch a RAC Platform!
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
21/58
(c) Copyright DSP Global Limited. All rights reserved. 21
The Next Generation Oracle platform
Oracle Platform
Removes site as SPOFIs Active-Active
Is Scaleable
I.e. Oracle RAC Platform
that removes site as the SPOF
Answer: Stretch a RAC Platform!
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
22/58
(c) Copyright DSP Global Limited. All rights reserved. 22
The Next Generation Oracle platform
Oracle Stretch RAC Platform Considerations
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
23/58
(c) Copyright DSP Global Limited. All rights reserved. 23
The Next Generation Oracle platform
Oracle Stretch RAC Platform Considerations
Inter-Site Connectivity
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
24/58
(c) Copyright DSP Global Limited. All rights reserved. 24
The Next Generation Oracle platform
Oracle Stretch RAC Platform Considerations
Inter-Site ConnectivityDisk Mirroring
Disk Mirroring
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
25/58
(c) Copyright DSP Global Limited. All rights reserved. 25
The Next Generation Oracle platform
Oracle Stretch RAC Platform Considerations
Inter-Site ConnectivityDisk Mirroring
RAC Cache Fusion
Cache Fusion
Disk Mirroring
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
26/58
(c) Copyright DSP Global Limited. All rights reserved. 26
The Next Generation Oracle platform
Oracle Stretch RAC Platform Considerations
Inter-Site ConnectivityDisk Mirroring
RAC Cache Fusion
Requires dedicated fibre
Cache Fusion
Disk Mirroring
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
27/58
(c) Copyright DSP Global Limited. All rights reserved. 27
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Cache Fusion
Disk Mirroring
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
28/58
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
29/58
(c) Copyright DSP Global Limited. All rights reserved. 29
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Cache Fusion
Disk Mirroring
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
30/58
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
31/58
(c) Copyright DSP Global Limited. All rights reserved. 31
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
Cache Fusion
Disk Mirroring
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
32/58
(c) Copyright DSP Global Limited. All rights reserved. 32
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
Cache Fusion
Disk Mirroring
Oops!
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
33/58
(c) Copyright DSP Global Limited. All rights reserved. 33
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
The 3rdvoting disk needs an
independent site
Cache Fusion
Disk Mirroring
3RDSite
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
34/58
(c) Copyright DSP Global Limited. All rights reserved. 34
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
The 3rdvoting disk needs anindependent site
Cache Fusion
Disk Mirroring
3RDSite
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
35/58
(c) Copyright DSP Global Limited. All rights reserved. 35
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
The 3rdvoting disk needs anindependent site
The 3rdSite acts as an arbiter
Cache Fusion
Disk Mirroring
3RDSite
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
36/58
(c) Copyright DSP Global Limited. All rights reserved. 36
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
The 3rdvoting disk needs anindependent site
The 3rdSite acts as an arbiter
Cache Fusion
Disk Mirroring
3RDSite
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
37/58
(c) Copyright DSP Global Limited. All rights reserved. 37
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
The 3rdvoting disk needs anindependent site
The 3rdSite acts as an arbiter
Only Requires NFS
Cache Fusion
Disk Mirroring
3RDSite
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
38/58
(c) Copyright DSP Global Limited. All rights reserved. 38
The Next Generation Oracle platform
Oracle Stretch RAC Platform
ConsiderationsCluster Voting Disks
Redundant for resilience
Odd number
Say we lose a site?
A node must be able toaccess at least half of thevoting disks to remain amember
The 3rdvoting disk needs anindependent site
The 3rdSite acts as an arbiter
Only Requires NFS
Doesnt have to be highlyavailable
Cache Fusion
Disk Mirroring
3RDSite
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
39/58
(c) Copyright DSP Global Limited. All rights reserved. 39
Agenda
Historical Oracle platform incorporating HA and DR
The Next Generation Oracle platformLinux v HP/UX
What we built
Does it work?
Conclusion
Questions
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
40/58
(c) Copyright DSP Global Limited. All rights reserved. 40
Linux v HP-UX
Red Hat Linux HP-UX
RDBMS Oracle 10gR2 RAC Oracle 9i/10gR2RAC
Cluster Software Oracle Clusterware HP ServiceGuard
Software Mirroring Oracle ASM MirrorDisk UX orVeritas CVM
Network Teaming Linux bonding HP ServiceGuard
Distance < 100km 10km
16 < 10km
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
41/58
(c) Copyright DSP Global Limited. All rights reserved. 41
Cost Comparison
Costs
Linux is 42% CheaperOracle Licensing on HP/UX/Itanium is the significant factor
Linux/Opteron
HP-UX/Itanium
Total CapEx 175,879 282,014
Total 3-year OpEx 79,465 159,688
Total 3-year Cap and Op Ex 255,343 441,703
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
42/58
(c) Copyright DSP Global Limited. All rights reserved. 42
Agenda
Historical Oracle platform incorporating HA and DR
The Next Generation Oracle platformLinux v HP/UX
What we built
Does it work?
Conclusion
Questions
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
43/58
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
44/58
(c) Copyright DSP Global Limited. All rights reserved. 44
What We Built
Item Extended Distance RAC
RDBMS Oracle 10gR2 release 10.2.0.3
Cluster Software Oracle 10gR2 Clusterware release 10.2.0.3
Mirroring and
Volume Manager
Oracle 10gR2 ASM release 10.2.0.3
MultipathingSoftware
EMC PowerPath release 4.5.1
Operating System Red Hat Enterprise Linux AS release 4update 4
NIC teaming Linux Channel bonding module
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
45/58
(c) Copyright DSP Global Limited. All rights reserved. 45
Cluster Interconnect
RAC Cache Fusion
No SPOF
Redundant NICs are bonded using
Linux bondingRedundant Switches at each site
Switch spanning preservesnetwork path between NICs duringswitch failure
lononl11
Eth1
Eth3
Bond1Private
IP:10.10
0.48.33
lononl12
Site A Site B
Eth1
Eth3
Bond1
Private
IP:10.
100.
48.
33
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
46/58
(c) Copyright DSP Global Limited. All rights reserved. 46
SAN
Separate SAN Fabric at each site
Redundant HBAs in each node
Redundant Storage controllers ineach Array
EMC PowerPath manages loadbalancing and failover of multiple
OS paths to each LUN
lononl11
HBA 0 HBA 1
ASM disks VotingDisk
OCR Disk
DATA diskgroup
RECO diskgroup
Storage Array
ASM disks VotingDisk
OCR Disk
DATA diskgroup
RECO diskgroup
Storage Array
lononl12
Site A Site B
HBA 0 HBA 1
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
47/58
(c) Copyright DSP Global Limited. All rights reserved. 47
Voting Disks
lononl11
Eth0
Eth2
Bond0
Public IP: 10.100.14.46
VIP: 10.100.14.64
HBA 0 HBA 1
NFS Voting Disk Server
NFS Vote Disk
NFS Voting DiskServer
VotingDiskStorage Array
VotingDiskStorage Array
Eth1
lononl12
Site A Site B
IndependentSite
Eth0
Eth2
Bond0
Public IP:10.100.14.48
VIP:10.100.48.65
HBA 0 HBA 1
/dev/raw/raw
1
/dev/raw/ra
w2
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
48/58
(c) Copyright DSP Global Limited. All rights reserved. 48
Database Mirroring
ASM Requires 1 failure group per siteto preserve inter-site mirroring
ASM doesnt use a Dirty Region Logand so doesnt provide partialresilvering
Oracle Software is on local storage
Facilitates rolling upgrades
lononl11
HBA 0 HBA 1
ASM disks VotingDisk
OCR Disk
DATA diskgroup
RECO diskgroup
Storage Array
ASM disks VotingDisk
OCR Disk
DATA diskgroup
RECO diskgroup
Storage Array
lononl12
Site A Site B
HBA 0 HBA 1
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
49/58
(c) Copyright DSP Global Limited. All rights reserved. 49
What We Built
Historical Oracle platform incorporating HA and DR
The Next Generation Oracle platformLinux v HP/UX
What we built
Does it work?
Conclusion
Questions
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
50/58
Is It Highly Available and Disaster
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
51/58
(c) Copyright DSP Global Limited. All rights reserved. 51
g yTolerant?
Simulated loss of a node by
rebooting it
Result:Remaining node continues
Database cache remasters
60 second brown outThis can be tuned
Cache Fusion
Is It Highly Available and Disaster
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
52/58
(c) Copyright DSP Global Limited. All rights reserved. 52
g yTolerant?
Simulated loss of the cluster
interconnect
Result:One node is evicted from the
cluster
One node remains
Database cache remasters
60 second brown out
This can be tuned
Cache Fusion
Is It Highly Available and Disaster
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
53/58
(c) Copyright DSP Global Limited. All rights reserved. 53
g yTolerant?
Simulated Data Centre loss by halting the node andremoving SAN access to the array
Result:Remaining nodes continues
Database cache remasters
60 second brown outThis can be tuned
ASM mirrors require rebuilding
Cache Fusion
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
54/58
(c) Copyright DSP Global Limited. All rights reserved. 54
What is the Performance Like?
40% better performance thana similar extended cluster built
for another customer
3700 transactions per second
Cluster Interconnect Ping time of 0.8ms
Blocks were being passed betweeninstances with a latency of < 2ms
Meets the objective of being fasterthan a disk access
Throughput exceeded 500mbps
Performance test was boundedby CPU The inter-site components, IO and
the cluster interconnect, were notprimary bottlenecks
Interconnect Throughput
0
100
200
300
400
500
600
700
14:13:51
14:14:31
14:16:23
14:17:03
14:17:43
14:18:23
14:19:04
14:19:44
14:20:24
14:21:04
14:21:45
14:22:25
14:23:05
14:23:45
14:24:26
14:25:06
14:25:47
14:26:27
14:27:07
14:27:48
14:28:28
14:29:09
14:29:49
14:30:29
14:31:10
Time
Mbps
Mbps
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
55/58
(c) Copyright DSP Global Limited. All rights reserved. 55
What We Built
Historical Oracle platform incorporating HA and DR
The Next Generation Oracle platformLinux v HP/UX
What we built
Does it work?
Conclusion
Questions
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
56/58
(c) Copyright DSP Global Limited. All rights reserved. 56
Conclusion
Extended distance RAC using Oracle Clusterware/Linux/AMD
Works!Platform is Disaster Tolerant and Highly Available
Failover requires no administrator intervention
Recovery from site failure in 60 seconds
Full utilisation of all resources
It is a high performance platform
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
57/58
(c) Copyright DSP Global Limited. All rights reserved. 57
Most Challenging 2 Node Cluster Ever
-
8/13/2019 Oracle RAC SIG - Implementing a dual site extended distance RAC Cluster.pps
58/58
(c) Copyright DSP Global Limited All rights reserved 58
Questions?