liberating your data from mysql: cross-database replication to the rescue!
DESCRIPTION
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!. Robert Hodges and Linas Virbalas Continuent, Inc. Introductions What is Tungsten? A Word About MySQL Replication Tungsten’s MySQL to PostgreSQL Replication Demo! Questions and Answers. Agenda. Introductions. - PowerPoint PPT PresentationTRANSCRIPT
© Continuent 2010
Liberating Your Data From Liberating Your Data From MySQL: Cross-Database MySQL: Cross-Database Replication to the Rescue!Replication to the Rescue!
Robert Hodges and Linas Virbalas
Continuent, Inc.
© Continuent 2010
AgendaAgenda
/ Introductions
/ What is Tungsten?
/ A Word About MySQL Replication
/ Tungsten’s MySQL to PostgreSQL Replication
/ Demo!
/ Questions and Answers
© Continuent 2010
IntroductionsIntroductions
© Continuent 2010
About ContinuentAbout Continuent
/ Our Value: Make open source as robust as commercial databases
• Enterprise capability• Cloud flexibility• Open source cost
/ Our Solution: Tungsten Database Clustering
/ Our Company:• Founded in Finland, HQ in USA• Venture-backed• US/European presence• Deep experience in databases and distributed systems
© Continuent 2010
What is Tungsten and How Does It Work?
© Continuent 2010
What Is Tungsten?
/ Tungsten creates data services using off-the-shelf open source databases to:
1. Ensure reliable, fast access to data2. Simplify common management tasks3. Enable disaster recovery
…Without changing applications
© Continuent 2010
Apache/Mod_PHP
libmysqlclient.alibmysqlclient.a
Connector
Tungsten 1.3 Data Service ArchitectureTungsten 1.3 Data Service Architecture
Apache/Mod_PHP
DBMSDBMS
Replicator
Manager
DBMSDBMS
Replicator
Manager
DBMSDBMS
Replicator
Manager
libmysqlclient.alibmysqlclient.a
MasterMasterSlaveSlave SlaveSlave
Connector
© Continuent 2010
Technology: Replication Pipelines
THLTHLSlaveSlaveDBMSDBMS
ApplierApplierExtractorExtractor FiltersFilters ApplierApplierExtractorExtractor FiltersFilters
StageStage StageStage
PipelinePipeline
Tungsten Replicator ProcessTungsten Replicator Process
Transaction History LogTransaction History Log
Master THLMaster THL(via network)(via network)
© Continuent 2010
So Why Are We Here?
Tungsten can Tungsten can replicate in real replicate in real
time from time from MySQL to MySQL to
PostgreSQLPostgreSQL
© Continuent 2010
Why Would You Want to Do That?
/ Feed MySQL merchant transactions to PostgreSQL applications
/ Feed PG-based data warehouse and reporting
/ Migrate MySQL data to PG with minimum downtime
/ Plus Scott McNealy says it’s good to share
MySQL App 1MySQL App 1MySQL App 2MySQL App 2
MySQL App 3MySQL App 3
PG AppPG App
© Continuent 2010
A Word About A Word About MySQL MySQL
ReplicationReplication
© Continuent 2010
Writeable slavesWriteable slavesSlave readable Slave
writeable
MySQL Statement Based + +
MySQL Row Based + +
MySQL Mixed + +
PostgreSQL WAL Shipping - -
PostgreSQL Streaming Replication (Hot Standby)
+ -
/ Tungsten for MySQL• You can both read and write to the DB operating in slave mode• This, potentially, allows you to break consistency• In some scenarios that is a plus
/ Tungsten for PostgreSQL• In Streaming Replication you can read from slaves• Still, you cannot write to them• In effect, you cannot have different data/structure on a slave
© Continuent 2010
Statement and Row Based ReplicationStatement and Row Based Replication
/ Statement Based Replication
/ Row Based Replication
CREATE TABLE t (id INT, name CREATE TABLE t (id INT, name VARCHAR(12));VARCHAR(12));
ALTER TABLE t ALTER TABLE t ADD COLUMN id2 INT;ADD COLUMN id2 INT;
INSERT INTO t VALUES (1, ‘Sun’, 999);INSERT INTO t VALUES (1, ‘Sun’, 999);
UPDATE t SET name = ‘Moon’;UPDATE t SET name = ‘Moon’;
CREATE TABLE t (id INT, name CREATE TABLE t (id INT, name VARCHAR(12));VARCHAR(12));
ALTER TABLE t ALTER TABLE t ADD COLUMN id2 INT;ADD COLUMN id2 INT;
{ INSERT, t, (1, ‘Sun’, 999) };{ INSERT, t, (1, ‘Sun’, 999) };
{ UPDATE, t, id == 1, (NULL, ‘Moon’, NULL) };{ UPDATE, t, id == 1, (NULL, ‘Moon’, NULL) };
© Continuent 2010
Logical vs. Physical ReplicationLogical vs. Physical ReplicationLogical Physical
MySQL Statement Based x
MySQL Row Based x
MySQL Mixed x
PostgreSQL WAL Shipping x
PostgreSQL Streaming Replication
x
Filters (data transformation) possible
+ -
Different data/structure on slave possible
+ -
© Continuent 2010
TungstenTungstenMySQL -> PostgreSQLMySQL -> PostgreSQL
ReplicationReplication
© Continuent 2010
MySQL -> PostgreSQL: Who’s Who?MySQL -> PostgreSQL: Who’s Who?
BinBinLogsLogs
Master DBMaster DB
(Binlogs (Binlogs enabled)enabled) Slave DBSlave DB
MySQL setup MySQL setup to run as to run as MySQL MySQL mastermaster
PostgreSQL setup PostgreSQL setup to accept writes – to accept writes –
Continuous Continuous Recovery is OFFRecovery is OFF
© Continuent 2010
MySQL -> PostgreSQL: Tungsten MySQL -> PostgreSQL: Tungsten Replicator Doing the JobReplicator Doing the Job
BinBinLogsLogs
Tungsten Replicator
Tungsten Replicator
Platform-Platform-Independent Independent SQL EventsSQL Events
Log RecordLog Record
Master DBMaster DB
(Binlogs (Binlogs enabled)enabled) Slave DBSlave DB
JDBC SQL JDBC SQL RequestsRequests
© Continuent 2010
MySQL -> PostgreSQL: Tungsten MySQL -> PostgreSQL: Tungsten Replicator ComponentsReplicator Components
BinBinLogsLogs
Master Replicator
MySQLExtractor
Transaction History Log
Slave Replicator
PostgreSQLApplier
Transaction History Log
Platform-Platform-Independent Independent SQL EventsSQL Events
Log RecordLog Record
Master DBMaster DB
(Binlogs (Binlogs enabled)enabled) Slave DBSlave DB
JDBC SQL JDBC SQL RequestsRequests
Filters Filters
© Continuent 2010
MySQL -> PostgreSQL: ChallengesMySQL -> PostgreSQL: Challenges
/ Data Type Differences
/ Default (Implicitly Defined) Schema Selection
/ SQL Dialect Differences• Statement Replication vs. Row Replication
/ Character Sets and Binary Data
© Continuent 2010
MySQL -> PostgreSQL: Data TypesMySQL -> PostgreSQL: Data Types
MySQL PostgreSQL
! TINYINT SMALLINT
SMALLINT SMALLINT
INTEGER INTEGER
BIGINT BIGINT
! CHAR(1) CHAR(5) = {‘true’, ‘false’}
CHAR(x) CHAR(x)
VARCHAR(x) VARCHAR(x)
DATE DATE
TIMESTAMP TIMESTAMP
! TEXT (diff. sizes) TEXT
! BLOB BYTEA
…
/ Note the type differences between MySQL and PG
© Continuent 2010
MySQL -> PostgreSQL: Default SchemaMySQL -> PostgreSQL: Default Schema
MySQL Implicit MySQL Explicit
CREATE SCHEMA s; CREATE SCHEMA s;
USE s;
! CREATE TABLE t (i int);
CREATE TABLE s.t (i int);
! INSERT INTO t (1); INSERT INTO s.t (1);
/ MySQL: Trivial to use `USE`
/ MySQL: Going without `USE` generates different events
/ PG: Extract the default schema from the event
/ PG: Set it before applying
MySQL PostgreSQL
USE s; > SET search_path TO s, "$user”;
© Continuent 2010
MySQL -> PostgreSQL: SQL DialectMySQL -> PostgreSQL: SQL Dialect
MySQL PostgreSQL
CREATE TABLE complex (id INTEGER AUTO_INCREMENT PRIMARY KEY, i INT);
CREATE TABLE complex (id SERIAL PRIMARY KEY, i INT);
CREATE TABLE dt (i TINYINT);
CREATE TABLE dt (i SMALLINT);
…
/ Differences between DDL and DML statement SQL dialects
/ Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle
/ Tungsten Replicator Filters come to the rescue!• Simple to develop Java or JavaScript extensions
• Event structure IN -> Filter -> Event structure OUT
© Continuent 2010
MySQL -> PostgreSQL: CharsetsMySQL -> PostgreSQL: Charsets
MySQL PostgreSQL
INSERT INTO embedded_blob (key, data) VALUES (1, ‘?\0^Es\0^\0\’’)
ARGH!!! (SQL statement fails)
create table xlate(id int, d1 varchar(25) character set latin1, d2 varchar(25) character set utf8);
ARGH!!! (no way to translate to common charset)
/ Statement replication: MySQL syntax is “permissive”/ Embedded binary / alternate charsets/ Different charsets for different clients
/ Row replication: database/table/column charsets may differ
/ Answer: Stick with one character set throughout; use row replication if you need to move binary data
© Continuent 2010
MySQL-> PostgreSQL: What’s Left?
/ We have covered the basics but there is more…
/ Initial provisioning from MySQL to PostgreSQL
/ Data transformation
/ Making it go *really* fast (think parallel replication)
© Continuent 2010
DEMODEMO
© Continuent 2010
Q & AQ & A
© Continuent 2010
HQ and Americas560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009
e-mail: [email protected]
EMEA and APACLars Sonckin kaari 1602600 Espoo, FinlandTel +358 50 517 9059Fax +358 9 863 0060
Contact InformationContact Information
Continuent Web Site:http://www.continuent.com