geographically distributed multi-master mysql clusters
DESCRIPTION
In this webinar, we discuss the multi-master capabilities of Continuent Tungsten to help you build and manage systems that spread data across multiple sites. We cover important topics such as setting up large scale topologies, handling failures, and how to handle data privacy issues like removing personally identifiable information or handling privacy law restrictions on data movement. We will conclude with a live demonstration of a distributed MySQL solution with Continuent Tungsten clusters working across multiple Amazon Web Services (AWS) availability zones and regions.TRANSCRIPT
©Continuent 2014
Geographically Distributed Multi-Master MySQL Clusters
Robert Hodges, CEO Jeff Mace, Director of Services
©Continuent 2014
Introducing Continuent
2
• The leading provider of clustering and replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance scaling and data management for MySQL
• Replication - Flexible, high-performance data movement
©Continuent 2014
Quick Continuent Facts
• Largest Tungsten clustering installation processes up to 700M transactions daily
• Over 100 customers including many household names
• Multiple multi-site, multi-master deployments in production
• ... And more are on the way
3
©Continuent 2014 4
Tungsten Clustering in Two Minutes
©Continuent 2014
Tungsten Database-as-a-Service
5
Tungsten clusters combine off-the-shelf open source DBMS servers into data services with: !
• 24x7 data access • Scaling of load on replicas • Simple management commands !...without app changes or data migration
Amazon US West
apache /php
GonzoPortal.com
Connector Connector
©Continuent 2014
Cluster Detailed View
6
Application
Tungsten Connector
Application
Tungsten Connector
Replicator
Db2
Replicator
Db3Db1
Replicator
Slave SlaveMaster
Manager Manager Manager
Monitoring and Control
Monitoring and Control
Data Service: sjc
©Continuent 2014 7
Tungsten Replication in Four Minutes
©Continuent 2014
Tungsten Replicator Overview
8
Master
(Transactions + Metadata)
Slave
THL
DBMS Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download transactions via network
Apply using JDBC
©Continuent 2014
Replication Pipelines
9
Extract Filter Apply
StageExtract Filter Apply
Stage
Pipeline
Master DBMS
Transaction History Log
In-Memory Queue
Slave DBMS
Extract Filter Apply
Stage
©Continuent 2014
Multiple Services Per Replicator
10
NYCReplicator
London
Service nyc
FrankfurtReplicator
Service fra
Replicator
Service nyc
Service fra
©Continuent 2014
Using Services for Multi-Master Replication
11
New York LondonReplicator
lhr (master)
jfk (slave)
Replicator
lhr (slave)
jfk (master)
©Continuent 2014 12
Introducing Multi-Site Multi-Master Clustering
©Continuent 2014
Why Multi-Site? Why Multi-Master?
13
• Get data close to global applications
• Avoid latency cost of commit between sites
• Keep running during intermittent network failures between sites
• Keep running if a site fails
• Enable simple recovery when it comes back
©Continuent 2014
Multi-Site Multi-Master Clustering: the Marketecture
14
No app or data coupling across regions
ApplicationApplications ApplicationApplications
Amazon Region Amazon Region
Live data and apps
in all regions
Multi-AZ deployment
Async replication between sites
Run in Amazon or on-premises
©Continuent 2014
Detailed Topology
15
US-West-1 US-East-1
master
slave east1
east1
west
west1
master
slave
Important Note: Continuent recommends 3 nodes for clusters
©Continuent 2014 16
Demo Setting up Multi-Master Clustering
across Sites
US-East-1 US-West-1
©Continuent 2014
Puppet Installation and Apply
17
$> puppet module install continuent/tungsten $> puppet apply ~/tungsten.pp
See https://github.com/continuent/continuent-vagrant for more examples
©Continuent 2014 18
Failures in Multi-Master Topologies
©Continuent 2014
Best Practice Deployment in Amazon
19
Multi-AZ clustering with asynchronous multi-master across regions
US-East-1
AZAZ
AZ AZ
EU- West-1
AZ
AZ AZ
US- West-1
AZ
AZ AZ AZAZ AZ AZAZ
Availability ZoneRegion
AZ AZ AZ
©Continuent 2014
Normal Multi-Master Configuration
20
master
slave
master
slaveeast1
east1
west
west1
US-West-1 US-East-1
©Continuent 2014
Failed master
21
master
master
master
slave
masterXeast1
east1
west
west1
US-West-1 US-East-1
©Continuent 2014
Failed Slave
22
master
slave
master
slaveX east1
east1
west
west1
US-West-1 US-East-1
©Continuent 2014
Failed Multi-Master Replicator
23
master
slave
master
slaveeast1
east1
west
west1X
US-West-1 US-East-1
©Continuent 2014
Full-Site Failure
24
master
slave
master
slaveeast1
east1
west
west1XUS-West-1 US-East-1
©Continuent 2014 25
Enabling Data Replication Policies with Filters
©Continuent 2014
What Are Replication Filters?
26
Extract Filter Apply
StageExtract Filter Apply
Stage
Master Pipeline
Master DBMS
Transaction History Log
In-Memory Queue
Inspect Change Drop
©Continuent 2014
Problem: Handling PII Data
27
CREATE TABLE `personal_info` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userid` varchar(250) DEFAULT NULL, `personal_code` varchar(250) DEFAULT NULL, `birth_date` date DEFAULT NULL, `email` varchar(250) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8
Requirement 1: Do not replicate personally identifiable information
©Continuent 2014
Problem: Application-Specific Rules
28
mysql> select * from personal_info; +----+--------+---------------+------------+-----------------+ | id | userid | personal_code | birth_date | email | +----+--------+---------------+------------+-----------------+ | 1 | fre11 | secret | 1985-01-16 | [email protected] | | 2 | bill | verysecret | 1995-01-16 | [email protected] | +----+--------+---------------+------------+-----------------+
Requirement 2: prevent userid ‘bill’ from replicating
©Continuent 2014
Adding Filters to Meet Requirements
29
Extract Filter(s) Apply
StageExtract Filter Apply
Stage
Master Pipeline
Master DBMS
Transaction History Log
In-Memory Queue
colnames killbill dropcolumns
dropcolumns.json
©Continuent 2014
Creating a dropcolumn.json File
[ { "schema": "test", "table": "personal_info", "columns": [ "personal_code", "birth_date", "email" ] } ]
30
Table name
Columns to delete
©Continuent 2014
Writing a Custom killbill.js Filter
function filter(event) {! data = event.getData();! if(data != null) {! for (i = 0; i < data.size(); i++) {! d = data.get(i);! if(d != null && d instanceof com.continuent.tungsten.replicator.dbms!.RowChangeData) {! rowChanges = d.getRowChanges();! for(j = 0; j < rowChanges.size(); j++) {! oneRowChange = rowChanges.get(j);! schema = oneRowChange.getSchemaName();! table = oneRowChange.getTableName();! if (schema == "test" && table == "personal_info") {! columnValues = oneRowChange.getColumnValues();! for (row = 0; row < columnValues.size(); row++) {! raw = columnValues.get(row).get(1).getValue();! user = new java.lang.String(raw);! if (user.equals("bill")) {! logger.info("Dropping transaction from bill!");! return null;!} } } } } } } }!
31
Drop transaction
©Continuent 2014
Killbill.tpl file for Filter Installation
# Drop transactions for bill. replicator.filter.killbill=com.continuent.tungsten.replicator.filter.JavaScriptF ilter replicator.filter.killbill.script=/opt/continuent/share/killbill.js
32
©Continuent 2014
Installing the Filters...
cp dropcolumn.json /opt/continuent/share cp killbill.js /opt/continuent/share cp killbill.tpl /opt/tungsten/tungsten-replicator/samples/conf/filters/default !/opt/continuent/tungsten/tools/tpm update \ --svc-extractor-filters=colnames,killbill,dropcolumn \ --property=replicator.filter.dropcolumn.definitionsFile=/opt/continuent/share/dropcolumn.json \ --start-and-report=true
33
©Continuent 2014
Resulting Data on Slave(s)
34
mysql> select * from personal_info; +----+--------+---------------+------------+-------+ | id | userid | personal_code | birth_date | email | +----+--------+---------------+------------+-------+ | 1 | fre11 | NULL | NULL | NULL | +----+--------+---------------+------------+-------+ 1 row in set (0.00 sec) !
Requirement 2: No bill!
Requirement 1: PII columns cleared
©Continuent 2014
Conclusion
35
©Continuent 2014
How Do I Get Started?
36
• Documentation for all products http://docs.continuent.com
• Download registration for Tungsten clustering: http://www.continuent.com/downloads/software
• Downloads for open source Tungsten Replicator http://code.google.com/p/tungsten-replicator/
• Vagrant project for setting up test topologies https://github.com/continuent/continuent-vagrant/tree/master/examples/MSMM
©Continuent 2014
Continuent Web Page: http://www.continuent.com
!
Tungsten Replicator: http://code.google.com/p/tungsten-replicator
Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs
560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected]