zero downtime schema changes - galera cluster - best practices
DESCRIPTION
Database schema changes are usually not popular among DBAs or sysadmins, not when you are operating a cluster and cannot afford to switch off the service during a maintenance window. There are different ways to perform schema changes, some procedures being more complicated than others. Galera Cluster is great at making your MySQL database highly available, but are you concerned about schema changes? Is an ALTER TABLE statement something that requires a lot of advance scheduling? What is the impact on your database uptime? This is a common question, since ALTER operations in MySQL usually cause the table to be locked and rebuilt – which can potentially be disruptive to your live applications. Fortunately, Galera Cluster has mechanisms to replicate DDL across its nodes. In these slides, you will learn about the following: How to perform Zero Downtime Schema Changes 2 main methods: TOI and RSU Total Order Isolation: predictability and consistency Rolling Schema Upgrades pt-online-schema-change Schema synchronization with re-joining nodes Recommended procedures Common pitfalls/user errors The slides are courtesy of Seppo Jaakola, CEO, Codership - creators of Galera ClusterTRANSCRIPT
![Page 1: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/1.jpg)
G a l e r a C l u s t e rSchema Upgrades
Seppo JaakolaCodership
![Page 2: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/2.jpg)
2
www.codership.com
Agenda
● Schema Upgrades in Synchronous Cluster
● DDL Classification
● Schema Backwards Compatibility
● Alternatives for Schema Upgrades● TOI● RSU / desync● wsrep_desync / wsrep_on method● Dropping Node for DDL● pt-online-schema-change
● Summary
![Page 3: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/3.jpg)
3
www.codership.com
Schema Upgrades
Applications have evolution, and there will be needs to change database schema in new application revisions
With 24/7 online requirements, the schema upgrade must happen while the system is online
Synchronous database cluster, requires that all active nodes must have same data
– Note that schema may differ, as long as data content is logically same
We need to figure out way(s) to roll schema upgrades in online cluster with minimal disruption for the service
![Page 4: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/4.jpg)
4
www.codership.com
What is DDL?
SQL statements can be classified in several classes:
DML Data Manipulation Language
– Transactional data manipulations– SELECT, INSERT, UPDATE...
DDL Data Definition Language
– CREATE..., DROP..., ALTER...
DCL Data Control Language
– GRANT, REVOKE
TCL Transaction Control Language
…
http://en.wikibooks.org/wiki/MySQL/Language/Definitions:_what_are_DDL,_DML_and_DQL%3F
![Page 5: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/5.jpg)
5
www.codership.com
DML vs DDL
Transactional SQL
– All DML on InnoDB
– Can be rolled back
NON Transactional SQL
– DDL, DCL..., DML on non transactional SE
– Not possible to rollback
– Requires up-front locking
● Galera uses optimistic concurrency control, and depends on possibility to rollback a conflicting operation
● Only transactional SQL can be replicated through Galera transactional replication● For others (DDL, DCL...), we either have to skip replication or use up-front locking
But the bottom line is division between transactional and non- transactional statements:
![Page 6: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/6.jpg)
6
www.codership.com
DML vs DDL
Notes on some DDL:
TRUNCATE
– is DDL!
– Is fast to execute, but nevertheless has some impact
OPTIMIZE, REPAIR, ANALYZE
– Table admin commands are now replicated
CREATE / DROP INDEX
– Hold MDL on affected table, and can stall the replication
![Page 7: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/7.jpg)
Schema Backward Compatibility
![Page 8: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/8.jpg)
9
www.codership.com
Backward Compatibility
MySQLSchema v1
MySQLSchema v2
MySQLSchema v2
App v1 App v1 App v2
Schema Upgrade
App Upgrade
Old application version must be able to use the new schema
![Page 9: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/9.jpg)
10
www.codership.com
Backward Compatibility
MySQLSchema v1
MySQLSchema v1
MySQLSchema v2
App v1 App v2 App v2
Schema Upgrade
App Upgrade
New application version must be able to use the old schema
![Page 10: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/10.jpg)
11
www.codership.com
Backwards Compatibility
New/old application should be able to use both old and new schema
Application should be backwards compatible
ROW replication between old and new schema should be possible
Schema change should be backwards compatible
![Page 11: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/11.jpg)
12
www.codership.com
App Backwards Compatibility
New/old application should be able to use both old and new schema
– New application can have compatibility mode to detect the version of underlying database
– If old app cannot use new schema, the old app must connect to one node only, which will be the last to upgrade
Dropping tables or columns can be a problem
– But drops can be done delayed: e.g. in v2 -> v3 upgrade, obsolete v1 elements can be dropped as neither v2 or v3 app will use them any more
![Page 12: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/12.jpg)
13
www.codership.com
ROW Replication Compatibility
MySQL guarantees ROW replication event compatibility with some limitations
Newer MySQL versions tolerate more variation between source and target tables, check out this page for latest status:
http://dev.mysql.com/doc/refman/5.6/en/replication-features-differing-tables.html
● Source and target can have different number of columns● But columns must be in same order
● New columns in the end, and must have default values
● Some data type conversions are also supported
![Page 13: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/13.jpg)
14
www.codership.com
ROW Replication Compatibility
Table A
col-a col-b col-a col-b col-c
Table A
Insert into table-A(col-a,col-b) values (5,7)
![Page 14: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/14.jpg)
15
www.codership.com
ROW Replication Compatibility
Table A
col-a col-b
5 7
col-a col-b col-c
5 7 def
Table A
Insert into table-A(col-a,col-b) values (5,7)
Replication
![Page 15: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/15.jpg)
16
www.codership.com
STATEMENT Replication
In STATEMENT format, schema compatibility is not an issue
Galera does not currently support STATEMENT replication, but:
– Enabling STATEMENT replication is minor task
● Consistency is at risk
● Parallel applying must be limited (OFF, Database or Table level)
● STATEMENT replication, in general, is phasing out
– Supporting STATEMENT replication for schema upgrades, is one potential extension we are looking for
![Page 16: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/16.jpg)
Schema Upgrades in Galera Cluster
![Page 17: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/17.jpg)
18
www.codership.com
Schema Upgrades in Galera
Galera Cluster has two inbuilt methods for DDL replication:
– TOI – Total Order Isolation
– RSU – Rolling Schema Upgrade
The method of choice is declared by variable:
wsrep_osu_method = TOI | RSU
Pt-online-schema-change is valid tool for upgrades, these and other DDL replication alternatives are discussed in following chapters.
![Page 18: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/18.jpg)
Total Order Isolation
![Page 19: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/19.jpg)
20
www.codership.com
TOI
Total oder Isolation (TOI) is the default DDL replication method
● wsrep_osu_method = TOI
● “master node” detects DDL at parsing time and sends out replication event for the SQL statement before even starting the DDL processing
● DDL replication happens in STATEMENT format
● Every node in the cluster will process the replicated DDL at the same “slot” in the cluster transaction stream (Total Order)
![Page 20: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/20.jpg)
21
www.codership.com
MySQL
TOI Replication
a
Parser
Execution
Replication
G a l e r a R e p l i c a t io n
WS
ALTER TABLE t1...
MySQL
SeqnoSTATEMENT event
GaleraReplication
![Page 21: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/21.jpg)
22
www.codership.com
MySQL
TOI Replication
a
Parser
Execution
WS
ALTER TABLE t1...
MySQLParser
Execution
continue
apply
Seqno slot reached
GaleraReplication
![Page 22: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/22.jpg)
23
www.codership.com
TOI Replication
Pros– Strict consistency, all nodes will get same change
– No worries about schema backwards compatibility
Cons– Strict commit order will make every transaction to wait until DDL is
over
Usable, when:– DDL is short term
– Schema change will not be backwards compatible
– And/or there is maintenance window
Some future work in road map:– TOI replication commit order can be relaxed
– We are working on optimization based on this
![Page 23: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/23.jpg)
Rolling Schema Upgrade
![Page 24: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/24.jpg)
25
www.codership.com
RSU
➢ Rolling Schema Upgrade
wsrep_osu_method=RSU
➢ Will desynchronize the node from replication for the duration of following DDL command
➢ Incoming replication is buffered
➢ Nothing will be replicated out of the node
➢ When DDL is over, the node will automatically join back in cluster, and catch up missed transactions from the buffer
![Page 25: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/25.jpg)
26
www.codership.com
RSU
a
G a l e r a R e p l i c a t io n
SET GLOBAL wsrep_osu_method=RSUALTER TABLE t1...
MySQL MySQL MySQL
G a l e r a R e p l i c a t i o n
![Page 26: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/26.jpg)
27
www.codership.com
RSU
a
MySQL
MySQL MySQL
G a l e r a R e p l i c a t i o n
ALTER TABLE
WS WS
Slave queue
![Page 27: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/27.jpg)
28
www.codership.com
RSU
a
ALTER TABLE t1...
MySQL MySQL MySQL
G a l e r a R e p l i c a t i o n
WS WS
Slave queue
![Page 28: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/28.jpg)
29
www.codership.com
RSU
a
Pros
– DDL will not slow down cluster
– Automatic re-sync after DDL is over
Cons
– Has global effect, all sessions will be RSU'ed
– Schema change has to be backwards compatible
– Only one RSU operation at a time
– Rolling over cluster is manual operation
![Page 29: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/29.jpg)
wsrep_desync
![Page 30: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/30.jpg)
31
www.codership.com
wsrep_desync
Node can be set to omit flow control by:mysql> SET GLOBAL wsrep_desync=ON;
A session can be declared to not replicate anything by:mysql> SET wsrep_on=OFF;
● Running DDL in such a session, will result in local schema change, and processing of the DDL will not hold back cluster.
● However, all cluster transactions will be replicated to the node, and if there are conflicts, the DDL will be aborted.
wsrep_desync+wsrep_on method is good only for non-confliction operations
![Page 31: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/31.jpg)
32
www.codership.com
wsrep_desync+wsrep_on
a
SET GLOBAL wsrep_desync=ON;SET wsrep_on=OFF;ALTER TABLE t1...
MySQL
G a l e r a R e p l i c a t i o n
WS
Slave queue
WS
t1 t2 tn
![Page 32: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/32.jpg)
33
www.codership.com
wsrep_desync
We are currently working on making better use of desync mode. The goal is to protect local desynced transactions from replication aborts.
This way, the DDL will succeed even if there are conflicts with the cluster. However, cluster replication will pause at first such conflict and wait until DDL is complete.
But, this will be future extension, and available in some of following 3.* release.
![Page 33: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/33.jpg)
Drop a Node for DDL
![Page 34: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/34.jpg)
35
www.codership.com
Dropping Node for DDL
One way to do “manual RSU”, is to drop a node from cluster and run DDL on the stand-alone node.
Joining the node back must happen through IST, as we don't want SST to bring back the old schema.
Make sure to protect the node from any production connections! Load balancers should be configured first to isolate the node from unwanted connections.
Adjust your gcache size big enough to allow IST after the DDL is over.
![Page 35: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/35.jpg)
36
www.codership.com
Dropping Node for DDL
MySQL
G a l e r a R e p l i c a t i o n
Load Balancer
MySQL
1. configure LB
2. Drop node, e.g. set global wsrep_cluster_address=gcomm://
![Page 36: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/36.jpg)
37
www.codership.com
Dropping Node for DDL
MySQL
G a l e r a R e p l i c a t i o n
Load Balancer
MySQL
3. ALTER TABLE t1...
![Page 37: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/37.jpg)
38
www.codership.com
Dropping Node for DDL
MySQL
G a l e r a R e p l i c a t i o n
WSWS
Load Balancer
MySQL
4. Join backset wsrep_cluster_address...
IST
![Page 38: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/38.jpg)
39
www.codership.com
Dropping Node for DDL
MySQL
G a l e r a R e p l i c a t i o n
Load Balancer
MySQL
5. configure LB
![Page 39: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/39.jpg)
pt-online-schema-change
![Page 40: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/40.jpg)
41
www.codership.com
pt-online-schema-change
Tool in Percona Toolkit to run non blocking schema changes
http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html
1. Creates a shadow copy of target table
2. Installs triggers in source table to forward updates to target table
3. Copies source table data in small chunks to target table
4. Renames target table to replace the source table
Pt-osc does not replicate schema changes, but makes it possible to change schema without interfering with replication
However, pt-osc requires TOI to be enabled, and TOI replication will propagate the changes to whole cluster
![Page 41: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/41.jpg)
42
www.codership.com
pt-online-schema-change
a
updates
t1
t1-new
CREATE similar tableand ALTER
![Page 42: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/42.jpg)
43
www.codership.com
pt-online-schema-change
a
updates
t1
t1-new
Install trigger to forward updates to new table
![Page 43: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/43.jpg)
44
www.codership.com
pt-online-schema-change
a
updates
t1
t1-new
Copy data in chunks
![Page 44: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/44.jpg)
45
www.codership.com
pt-online-schema-change
a
updates
t1 t1-new
Copy data in chunks
![Page 45: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/45.jpg)
46
www.codership.com
pt-online-schema-change
a
updates
t1-oldt1
Rename tables
![Page 46: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/46.jpg)
47
www.codership.com
pt-online-schema-change
a
updates
t1Drop old table
![Page 47: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/47.jpg)
48
www.codership.com
pt-online-schema-change
a
Some Caveats:
● TOI requirement
– Pt-osc changes will be run against whole cluster with one go
– Could be relaxed, imo
● Triggers not supported
– Pt-osc installs new triggers in source table and does not allow any other triggers to exists in the table
● Foreign key support
– Two methods for dealing with FKs
– Rebuilding child table FK constraint may be needed
– FK constraint name will be different
![Page 48: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/48.jpg)
Summary
![Page 49: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/49.jpg)
50
www.codership.com
Codership
Schema upgrades require careful planning
➢ Find out backwards compatibility both from application and from ROW replication perspective
➢ Plan your upgrade process
➢ Rehearse with test cluster
Instant methods:
➢ TOI replication, pt-osc
➢ ROW replication backwards compatibility is not an issue
Rolling methods
➢ RSU, wsrep_desync/wsrep_on, node dropping
➢ Schema backwards compatibility required
![Page 50: Zero Downtime Schema Changes - Galera Cluster - Best Practices](https://reader031.vdocuments.site/reader031/viewer/2022012314/554dd26bb4c905d10e8b4912/html5/thumbnails/50.jpg)
Questions?
Thank you for listening!Happy Clustering :-)