running galera clusters street signs to watch for when · pivotal cloud foundry (pcf) 2+ years on...
TRANSCRIPT
![Page 1: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/1.jpg)
© Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0
September 2017
Highway SafetyStreet Signs to Watch for When Running Galera Clusters
Thanks to: http://www.safetysign.com/
![Page 2: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/2.jpg)
Get Comfortable!
Demo Starts
3
![Page 3: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/3.jpg)
Get Comfortable!
Demo Starts
2
![Page 4: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/4.jpg)
Get Comfortable!
Demo Starts
1
![Page 5: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/5.jpg)
Get Comfortable!
Demo Starts
Now
![Page 6: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/6.jpg)
![Page 7: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/7.jpg)
WHAT JUST HAPPENED?
Same app, different results
Marco Nicosia
● Principal Product Manager
● Pivotal Cloud Foundry (PCF)
● 2+ years on MySQL for PCF, all Galera
● We have another project working on Leader/Follower
![Page 8: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/8.jpg)
Agenda
Galera Highway Safety
☑ Demo 1: Deadlocks
❏ Purpose of this Talk
❏ Demo 2: DDL under TOI
❏ Demo 3: Large DMLs
❏ Demo 4: Crashing Galera
❏ Mitigations
![Page 9: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/9.jpg)
Cover w/ Image
Why Intermediate?
Not a “How to install & use”
Not an “Advanced tuning”
This is “Make an informed decision”
Share our experiences with Galera
Not an advertisement
![Page 10: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/10.jpg)
GOAL
Convince you that Galera is Great
… when used sparingly
![Page 11: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/11.jpg)
Marco’s Rules
#1 People Don’t Read
#2 People Don’t Listen
![Page 12: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/12.jpg)
Cover w/ Image
Why Show Us This?
Any app who ignores a deadlock is an idiot
… or just a new developer
True! But not precisely “MySQL”
![Page 13: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/13.jpg)
Cover w/ Image
Galera Limitations
● InnoDB only, no MyISAM
○ System tables are MyISAM
○ Must use CREATE USER/GRANT
● Doesn’t distribute locks
Learn more: http://galeracluster.com/documentation-webpages/limitations.html
![Page 14: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/14.jpg)
Cover w/ Image
Galera is Complex
Advertised as Multi-Master MySQL
Can be a great HA solution
… at the cost of complexity.
![Page 15: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/15.jpg)
Demo #2: Large DDL
Concept: DDL modes
TOI vs. RSU
Learn more:http://galeracluster.com/videos/galera-cluster-best-practices-ddls-and-schema-upgrades/
![Page 16: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/16.jpg)
![Page 17: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/17.jpg)
WHAT JUST HAPPENED?
● One writer issued a large (~4min) DDL
● All other writers fully blocked
● Total Order Isolation serializes all DDLs
● … AND blocks all commits
![Page 18: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/18.jpg)
Demo #3: Large DML
Concept: Flow Control
Learn more: http://galeracluster.com/documentation-webpages/nodestates.html
![Page 19: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/19.jpg)
![Page 20: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/20.jpg)
WHAT JUST HAPPENED?
● One active writer committed a large DML
● All other writers were blocked
● Two nodes fell significantly behind
● Cluster snaps back to normal
![Page 21: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/21.jpg)
Cover w/ Image
Distributed Reads
Myth: Fully synchronous replication
Fact: Certification-based is eventually consistent MySQL
![Page 22: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/22.jpg)
Cover w/ Image
Synch vs. Certification
Myth: Multiple Copies of Data
Guarantees a writeset will commit
Applied subsequent to certification
![Page 23: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/23.jpg)
Demo #4: Divergent Data
![Page 24: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/24.jpg)
![Page 25: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/25.jpg)
WHAT JUST HAPPENED?
Two nodes had nearly full disks
Single writer issued large DDL
Immediately failed on other two nodes
Leaving incompatible schema
Single writer eventually issued incompatible DML
Two nodes immediately crash and SST
![Page 26: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/26.jpg)
Mitigations
Conc;usion
![Page 27: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/27.jpg)
Mitigations
Technology
● Restrict traffic to one node!
● Dedicated clusters
● Force InnoDB
● Disable table locks
● Force primary keys
● Tune Galera
● Maintain headroom
Education
● TOI vs. RSU
● Max txn size
● Deadlocks
● Primary keys
● Critical Reads
● Flow control
![Page 28: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/28.jpg)
Cover w/ Image
Conclusions
Galera is powerful
Especially for those who cannot tolerate downtime
Not standard MySQL
Developers must be Galera-aware
![Page 29: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/29.jpg)
Special Thanks
● Our Customers● Laine Campbell● The Percona support team● Shatarupa Nandi● Ben Laplanche● Andrew Garner● Joseph Palermo
● Samuel Serrano● Morgan Fine● Urvashi Reddy● Difan Zhao● Rob Dimsdale● Christopher Hendrix● Andrew Crump
… and the Barley cluster!
![Page 30: Running Galera Clusters Street Signs to Watch for When · Pivotal Cloud Foundry (PCF) 2+ years on MySQL for PCF, all Galera We have another project working on Leader/Follower. Agenda](https://reader033.vdocuments.site/reader033/viewer/2022060210/5f04ac717e708231d40f239e/html5/thumbnails/30.jpg)
Marco & Pivotal Software
● Marco Nicosia, Twitter: @menicosia
● Pivotal Labs, Data and Cloud Foundry
● Over 2,000 employees in more than 20 locations globally
● Key customers include Comcast, Allstate, Ford, Citi, GE, Southwest, Verizon.
● Investors include GE, Dell, Ford, and Microsoft.
We’re Hiring:
San Francisco, New York, Toronto, Dublin, and London