easy mysql replication setup and troubleshooting

Easy MySQL ReplicationSetup and Troubleshooting

Bob BurgessSalesforce ExactTarget Marketing Cloud (Radian6)

• Introduction• Description• Configuration Items• Commands• Setup• Beyond the Basics• Troubleshooting Faults• Gotchas• Closing

Slides

• I will post these slides on the conference site and slideshare

• No need to copy them down

Who Am I?

• MySQL user since 2007• Radian6: Social Media Monitoring• Now part of

Salesforce ExactTarget Marketing Cloud• Large Data Set• Dozens of Databases • Replication is CRUCIAL for us

Why Am I Doing This?

• Replication isn’t hard• Show simplest replication case• Share how we deal with common problems

Our First Replication

• Set up a separate database for running reports, to separate back-end from user-facing activity

• Alexander Rubin of MySQL Support in 2008

What is “Replication”?

Why Replication?

• Database Failure• Share Read Traffic (Scale-Out)• Reporting/Analytics Databases• Network Load• Backups

Replication Compared

• DRBD– Block-level– Replica is unusable until switched to

• Galera / Percona Cluster– More complicated– Not WAN-Friendly for High Transaction Rate

• Tungsten Replicator• Others..?

Replication Described

Master Replica


Master

table1table2

mysql

Update table1 set...Insert into table2...

Binlogs:•Events in commit order•Contains timestamps and auto-increment IDs

client

Replica

table1table2

IO Thread SQL Thread


mysql


log_bin=/data/log/mysql-bin

Config items

Master

table1table2

mysql


client

Replica

table1table2



mysql


server_id=1 server_id=2

Config items

(1) Master

table1table2

mysql


client

Replica (2)

table1table2



mysql

Replication Commands

• change master to• start slave• stop slave• reset slave• show slave status\G• show master status• mysqlbinlog• grant replication slave

Replication Commandschange master to... Which server? Port? Credentials? Log file & position?

(1) Master

table1table2

mysql

client

Replica (2)

table1table2



mysql


Replication Commands• change master to...

Which server? Port? Credentials? Log file & position?•change master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744;

Replication Commands

(1) Master

table1table2

mysql


client

Replica (2)

table1table2

mysql

Replication Commandschange master to master_host=‘Master’, master_port=3306, master_user=‘repl’, master_password=‘pass’, master_log_file=‘mysql_bin.000001’, master_log_pos=4563744;

(1) Master

table1table2

mysql


client

Replica (2)

table1table2

IO Thread


mysql

Replication Commandsstart slave;

(1) Master

table1table2

mysql


client

Replica (2)

table1table2



mysql

Replication Commandsstop slave;

(1) Master

table1table2

mysql


client

Replica (2)

table1table2


mysql

Replication Commandsreset slave;

(1) Master

table1table2

mysql


client

Replica (2)

table1table2

mysql

Slave_IO_State: Waiting for master to send event

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error:

Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Slave_IO_State: Waiting for

master to send event

Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:

Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

What it’s doing



Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0

Master_Server_Id: 1

Where it’s connected


Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3

Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0

Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Where the IO Thread has read up to

Master

table1table2

mysql


client

Replica

table1table2



mysql

Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843


Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843

Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0


Where the SQL Thread has executed up to (in relaylog)

Master

table1table2

mysql


client

Replica

table1table2



mysql

Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548


Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548

Relay_Master_Log_File: mysql-bin.000001

Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0

Last_Error: Skip_Counter: 0

Exec_Master_Log_Pos: 23456234

Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Where the SQL Thread has executed up to (corresponding position in Master’s binlog)

Master

table1table2

mysql


client

Replica

table1table2



mysql

Relay_Master_Log_File: mysql-bin.000001 Exec_Master_Log_Pos: 23456234


Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001

Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0


If the IO and SQL Threads are running


Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes

Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:

Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Filters in effect



Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key:

Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Lag (how long ago was timestamp of latest-processed query?)

Can be “NULL”



Last_Errno: 0

Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error:

Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Error (SQL Thread)



Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Replication Error (IO Thread)

Basic Setup Walk-Through

• Unique server_id values

my.cnf on master[mysqld]server_id=1

my.cnf on replica[mysqld]server_id=2


• Replication UserOn Master:

grant replication slave on *.* to ‘repl’ identified by ‘pass’;


• Point to the correct binlog position• On master:show master status;

...once both databases are equal• On Replica:change master to ...


• Start it up• On replica:start slave;

• Check it• On replica:show slave status\G

And that’s it.

Limiting Tables or Schemas

• If you don’t need all the tables• In my.cnf:replicate-do-table=schema.tableNamereplicate-skip-table=schema.otherTable

• If you don’t need all the schemas• In my.cnfreplication-do-db=schemaOnereplication_skip_db=schemaTwo

Limiting Tablesreplicate-do-table=schema.tableName

(1) Master

table1table2

mysql


client

Replica (2)

table1table2

IO Thread SQL Threadwith filter


mysql

Limiting Columns or Rows

• If you don’t need all columns• Subset of table for reporting or analytics


• Original tablecreate table x (a int primary key, b int, c int) engine=InnoDB;

• Blackhole table on replica like master tablecreate table x (a int, b int, c int) engine=blackhole;

• Target table on replicacreate table y (a int primary key, b int) engine=InnoDB;


• Before-insert trigger on replica to put data into second “real” table

• Only columns a and b, and only for b=1create trigger x_bi before insert on x for each row insert into y (a, b) select new.a, new.b from dual where new.b=1;


(1) Master

table1table2

mysql


client

Replica (2)

x y



mysql

TRG

Multi-Level

• Local and Remote DR Site• Spreading load over many servers

M R M R

M R

R

RR

R

R

R

R

...

Multi-Level

• Configlog_slave_updates=1

Multi-LevelMaster Replica

table1table2

table1table2

mysql IO Thread SQL Thread



Replica

table1table2




mysql

Types of Binlogs

• Statement-Based– Contains the actual statements run– Data drift is more likely (code modified by triggers,

etc)• Row-based– Replicates changes for each row– Mostly unreadable by humans– Unaffected by how the change is done (data drift less

likely)– mysqlbinlog -v

Fault: Lag

• Long-running transactions• High concurrency on master becomes single

session on replica• show slave status\G• Easy to alarm on

Fault: Stopping from Error

• Any error stops SQL Thread!• Duplicate Key• Unknown Function• slave_exec_mode=idempotent

...suppresses duplicate-key and no-key-found errors

• show slave status\G

Faults: Can’t Connect to Master

• Fault shows in slave status as soon as you start slave.

• Did you create replication slave user?• Correct user/password?• Correct hostname/port?• Master is running and reachable?

Fault: Can’t Find Binlog

• Clearly stated in the slave status• Can happen if replica has been stopped (or is

lagged) and the binlogs are removed from the master

• Execute from archiveOR• Rebuild replica and restart from a known good

binlog position

Faults: Configuration

• replication-do-table• replication-do-schema• log_slave_updates

Faults: server_id

• Must be unique!• If equal: “Seconds behind” jumps everywhere.

Gotcha: functions

• Functions will be run on the replica even if only appearing in a select on the master

• Make sure all functions exist on replica DB, even if only a dummy version

create function myFtn (a int, b int) returns int deterministic return 0;

Gotcha: Reconnection

• When a replica is stopped and reset, until the database is restarted, replication may reconnect to a master after “reset slave”

change master to master_port=1234;

Gotcha: Master Crash

• Replication becomes unstable if master crashes• Error: unknown instruction at position xxx• Often need to use mysqlbinlog to find a

suitable position in the binlog to restart replication from

stop slave; reset slave;change master to ...;start slave;

Gotcha: MySQL Versions

• Replication not completely compatible between versions (especially later=>earlier)

• 5.5 master to 5.1 replica: character set error “sometimes”

• Stops replication

Impossible

• One replica can have only one master(until 5.7)

• Can’t change table or schema filters dynamically

• Can’t put binlogs back on master server, for recovery purposes

The Book

Alternatives

• Tungsten Replicator from Continuent• Others?

Questions?

Thank you!

Bob [email protected]

Don’t forget your session evaluation!

easy mysql replication setup and troubleshooting

Data & Analytics

master master

master replica

repl master

event master

replication commands

mysql user

mysql s

easy mysql replication