mysql high availability - managing farms of distributed servers

MySQL High Availability:Managing Farms of Distributed Servers

Narayanan Venkateswaran

Alfranio Correia

Mats Kindahl

Safe Harbor Statement

Jan 2010

Sept 2014

2x Engineering Staf3x QA Staf2x Support Staf

The following is intended to outline our general product direction. It

is intended for information purposes only, and may not be

incorporated into any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be relied upon in

making purchasing decision. The development, release, and timing

of any features or functionality described for Oracle’s products

remains at the sole discretion of Oracle.

Agenda

Jan 2010

Sept 2014

➢ Building Reliable Systems➢ MySQL Fabric Overview

➢ Groups

➢ Failure Detection

➢ Integration with other HA Solutions

➢ High Availability @ Fabric Node

➢ Future Work

Causes of Downtime

Jan 2010

Sept 2014

● System Failures

● Server faults

● Software bugs or crashes

● Physical Disasters

● Scheduled Maintenance

● User Errors

Jan 2010

Sept 2014

High Availability is an integral part of designing a reliable system

High Availability Solutions

Jan 2010

Sept 2014

● Primary – Secondary

● E.g. MySQL Replication

● Shared Nothing Clusters

● MySQL Cluster

● Tightly Coupled Clusters

● DRBD

● WSFC

● Solaris Clustering

● Oracle VM High Availability

Agenda

Jan 2010

Sept 2014

➢ Building Reliable Systems

➢ MySQL Fabric Overview➢ Groups

➢ Future Work

MySQL Fabric

Jan 2010

Sept 2014

An extensible and easy-to-use framework for managing a farm of MySQL servers supporting high-availability and sharding.

What does all that mean?

● Management System– Manages a MySQL Farm– Distributed Framework

● Framework– Procedure execution– State store– Transaction Routing

● Extensible– High-availability groups– “Semi-automatic” sharding

● Written in Python● Latest Release 1.5.2

GA● Open Source

– You can participate– Suggest features– Report bugs– Contribute patches

● MySQL 5.6 is focus

Birds-eye View

High Availability Groups

Application

XML-RPCMySQL-RPC

Connector

Operator

MySQLFabricNode

DatabaseServers

High-Level Components

● Fabric-aware Connectors– Python, PHP, Java, .NET,

C– Enhanced Connector API

● MySQL Fabric Node– Manage information about

farm– Provide status information– Execute procedures

● MySQL Servers– Organized in high-

availability groups– Handling application data

High AvailabilityGroup

ApplicationConnector

Connector

MySQLFabricNode

MySQL Fabric Node Architecture

MySQL FabricFramework

Executor State Store(Persister)

MySQL-RPCAMQP XML-RPC

?Connector

Connector

Protocols

Extensions

BackingStore

Agenda

Jan 2010

Sept 2014

➢ MySQL Fabric Overview

➢ Groups➢ Failure Detection

➢ Future Work

Groups

Jan 2010

Sept 2014

● Group of Servers

– Hardware Redundancy– Data Redundancy

● Generic Concept

– Default Master-Slave configuration

– Supports generic High Availability methods

● DRBD● MySQL Cluster● MySQL Replication● Etc.

ndbdndbd

ndbd ndbd

Creating a MySQL Fabric Group

Jan 2010

Sept 2014

Creating a Group

Command:mysqlfabric group create <group_id> [--description=NONE]

E.g.mysqlfabric group create group-1 –description=”Example Group”

Creates a logical entity named group-1 that represents a farm of MySQL Servers in a HA configuration

MySQL Fabric Group – Add Servers

Jan 2010

Sept 2014

Adding Servers to a Group

Command:mysqlfabric group add <group_id> <address>

E.g.mysqlfabric group add group-1 server1.example.commysqlfabric group add group-1 server2.example.com

Adds two servers to group-1

MySQL Fabric Group – Create Master / Slave Setup

Jan 2010

Sept 2014

Adding Servers to a Group

Command:mysqlfabric group promote group_id [ --slave_id=NONE]

E.g.mysqlfabric group promote group-1 --slave_id=”server1.example.com”

Promote server1 as the master

Agenda

Jan 2010

Sept 2014

➢ Groups

➢ Failure Detection➢ Integration with other HA Solutions

➢ Future Work

Built-In Failure Detector

Jan 2010

Sept 2014

● Group level failure detector

● Monitor servers within groups

● On master failure– Mark master as faulty

– Trigger fail-over

● On slave failure– Mark slave as faulty

● Solution is only for servers managed by Fabric

Built-in Failure detector configuration

Jan 2010

Sept 2014

[failure_tracking]detections = 3detection_interval = 6detection_timeout = 1

detectionsNumber of times a server must fail an alive check before being marked as FAULTY.

detection_intervalInterval at which a server must be checked to be alive.

detection_timeoutElapsed time after which a server alive ping must timeout.

Distributed Failure Detection

Jan 2010

Sept 2014

● Connector reports error to Fabric

● Error used to update backing store

● Trigger Failover

– Error count reaches threshold

● The report API can be used by any generic process to report failures

Distributed Failure Detection – MySQL Fabric Configuration

Jan 2010

Sept 2014

[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0prune_time = 3600

notificationsThe number of error notifications required to mark a server FAULTY.

notification_clientsThe number of clients that need to report an error to mark a server FAULTY.

Distributed Failure Detection – MySQL Fabric Configuration

Jan 2010

Sept 2014

notification_intervalThe interval of interest in which we want to verify error notifications.

failover_intervalThe interval at which we can safely do a failover in a group without causing system instability.

prune_timeInterval at which we need to prune the error logs.

Distributed Failure Detection – Connector – Enabling Error Reporting

Jan 2010

Sept 2014

● The option to report errors is part of the Fabric configuration

● “report_errors” can be turned on as follows

● Causes an error during a connection to a server to be reported

fabric_config = { 'host': .., 'report_errors': True, }

cnx = mysql.connector.connect(fabric=fabric_config)

Distributed Failure Detection – Connector – Error Reporting

Jan 2010

Sept 2014

● The following errors are reported by default

● The above can be extended by setting extra_failure_report

REPORT_ERRORS = ( errorcode.CR_SERVER_LOST, errorcode.CR_SERVER_GONE_ERROR, errorcode.CR_CONN_HOST_ERROR, errorcode.CR_CONNECTION_ERROR, errorcode.CR_IPSOCK_ERROR,)

from mysql.connector.fabric import extra_failure_report extra_failure_report([error_code_0, error_code_1, ...])

Distributed Failure Detection – Connector – Error Reporting

Jan 2010

Sept 2014

● The following errors are reported by default

● The above can be extended by setting extra_failure_report

REPORT_ERRORS = ( errorcode.CR_SERVER_LOST, errorcode.CR_SERVER_GONE_ERROR, errorcode.CR_CONN_HOST_ERROR, errorcode.CR_CONNECTION_ERROR, errorcode.CR_IPSOCK_ERROR,)

from mysql.connector.fabric import extra_failure_report extra_failure_report([error_code_0, error_code_1, ...])

Distributed Failure Detection – Connector – Cache Invalidation

Jan 2010

Sept 2014

● Define which errors cause invalidation of the connector cache

● The following errors cause a cache invalidation by default

• The above can be extended by

RESET_CACHE_ON_ERROR = ( errorcode.CR_SERVER_LOST, errorcode.ER_OPTION_PREVENTS_STATEMENT,)

from mysql.connector.fabric import RESET_CACHE_ON_ERROR RESET_CACHE_ON_ERROR.append(error_code_0)

Distributed Failure Detection – Avoiding thundering herds

Jan 2010

Sept 2014

Application

1. Error

MySQL Fabric Node

Application

2. Error

3. Error

4.Error

● When to report errors ?

● Reporting errors at the session

level can cause flooding

● E.g. Orders Page

– Orders Page writes customer

orders to the database

– Every failing order will cause

a report

Distributed Failure Detection – Connector – Role

Jan 2010

Sept 2014

● MySQL User role for accessing to Fabric dump and reporting

● Adding users done using

● connector role

● Causes an error during a connection to a server to be reported

3 connector Role for MySQL Connectors + Access to dump commands + Reporting to Fabric

mysqlfabric user add <username> --role

Agenda

Jan 2010

Sept 2014

➢ Groups

➢ Integration with other HA Solutions➢ High Availability @ Fabric Node

➢ Future Work

Update Only

Jan 2010

Sept 2014

Scenario

● Using Fabric as a lookup server

● Servers managed outside Fabric

– For E.g. using DRBD

Requirement

● Update Fabric without affecting servers

Group Based and Virtual IP Based solutions

Jan 2010

Sept 2014

● Group Based Solutions

● Offer a notion of Group Membership

● Virtual IP Based Solutions

● Only one virtual server identified by a virtual IP Address

● E.g. WFSC, DRBD

Group Based System

Jan 2010

Sept 2014

● Notion of membership

● Dead servers automatically excluded from the group

● Use update_only to store topology information in Fabric

Group Based System

Jan 2010

Sept 2014

● Creating a Group● Adding Servers to a Group● Promote

Creating a Groupmysqlfabric group create "group-1"

Adding Servers to a Groupmysqlfabric group add group-1 server1.example.com --update_onlymysqlfabric group add group-1 server2.example.com -- update_only

Promotemysqlfabric group promote group-1 --slave_id=7bcb0804-41bb-11e4-b4d2-f4ce2963772b --update_only

Group Based System – Updating Failures

Jan 2010

Sept 2014

● Updating the state when all the servers are dead● Updating the state when the system is brought back

online

When all the servers are deadmysqlfabric threat report_error 7bcb0804-41bb-11e4-b4d2-f4ce2963772b --error=FAULTY --update_only

When the system is brought back onlinemysqlfabric group promote group-1 --slave_id=7bcb0804-41bb-11e4-b4d2-f4ce2963772b --update_only

Virtual IP System – Integrating DRBD

Jan 2010

Sept 2014

● Creating a Group● Adding Virtual IP to Group

Creating a Groupmysqlfabric group create "group-1"

Adding Servers to a Groupmysqlfabric group add group-1 IPAddress:Portno --update_only

● Add the virtual IP of the DRBD setup to the group definition.● The DRBD setup handles the maintanence of the cluster

Virtual IP System – Updating Failures

Jan 2010

Sept 2014

● Updating the state when all the servers are dead● Updating the state when the system is brought back

online

When all the servers are deadmysqlfabric threat report_error 7bcb0804-41bb-11e4-b4d2-f4ce2963772b --error=FAULTY --update_only

When the system is brought back onlinemysqlfabric group promote group-1 --slave_id=7bcb0804-41bb-11e4-b4d2-f4ce2963772b --update_only

Agenda

Jan 2010

Sept 2014

➢ Building Reliable Systems➢ MySQL Fabric Overview

➢ Groups

➢ High Availability @ Fabric Node➢ Future Work

HA @ The MySQL Fabric Node

Jan 2010

Sept 2014

● Fabric Node Fails

● Crash-Safe Procedures

● State Store Fails

● Safeguarding meta-data

● Both Fabric and State Store Fails

Crash Safe Procedures

Jan 2010

Sept 2014

● Each Fabric Operation is a procedure

● Procedure consists of a set of steps/jobs

● Steps/Jobs

● Can be Rolled Back

● Are Idempotent

State Store failure

Jan 2010

Sept 2014

● Storing Data in MySQL Cluster

● Storing Data in replicated Storage

Agenda

Jan 2010

Sept 2014

➢ Groups

➢ Future Work

Future Work

Jan 2010

Sept 2014

● Ideas for the future

Detecting and Restarting when Fabric Fails

Jan 2010

Sept 2014

● Detecting Fabric Failure

● Using Pacemaker

● Avoiding split-brain

● Re-Starting Fabric

● Providing script for restart

Handling Failure of MySQL Fabric and State Store

Jan 2010

Sept 2014

● Building a replicated state machine

● Multiple Fabric instance that can take over from any of

the other instances

● Uses Paxos/Raft-like implementation

Handling Connector Failover when Fabric goes down

Jan 2010

Sept 2014

● Alternate Fabric Addresses

– Connector has a list of potential addresses

– Try these addresses when a Fabric node fails

● Connector notices server failure

– PRIMARY failure

– SECONDARY failure

– All servers fail

Error reporting when Fabric goes down

Jan 2010

Sept 2014

● PRIMARY failure

– Report error to application

– Remove from cache

– Set group to read only

● SECONDARY failure

– Remove from cache

– Pick alternate server

● All server failure

– Report Error to Application

Reading for the Interested

● MySQL Forum: Fabric, Sharding, HA, Utilities

http://forums.mysql.com/list.php?144

● MySQL Fabric Documentationhttp://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html

● Migrating From an Unsharded to a Sharded Setup

http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html

● Configuring and running MySQL Fabrichttp://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html

Want to contribute?

● Check it– … and send us use-case and feature suggestions

● Test it– … and send comments to the forum

● Break it– … and send in bugs to http://bugs.mysql.com

Mats KindahlTwitter: @mkindahlhttp://mysqlmusings.blogspot.com

Alfranio CorreiaTwitter: @alfraniohttp://alfranio-distributed.blogspot.com

Keeping in Touch

Narayanan Venkateswaran

Twitter: @vn_tweetshttp://vnwrites.blogspot.com

Geert VanderkelenTwitter: @geertjanvdkhttp://geert.vanderkelen.org

Thank You !

Jan 2010

Sept 2014

mysql high availability - managing farms of distributed servers

Software

mysql sandbox · creating a replication system is equally...

mysql web reference architectures - fpa · •mysql...

mysql connector/j - college of computer and connector/j 3.1...

php and mysql php-mysql php and mysql

mysql strategy - itreegroup.eu · mysql strategy morten...

enterprise data -...

mysql replication update mysql 5.5 (ga) & mysql 5.6.2 (dev...

the colonies come of age -...

conference 2017 - bcnet...• production & development...

expresscluster x for linux quick start guide for mysql...

mysql...

mysql installation guide. mysql downloading mysql installer

gisもmysqlで！ mysql 5.7 gis機能をご紹介 · mysql...

what you don’t know whitepaper final.pdfdocker jenkins...

mysql binlog servers at booking.com

tac9516 best practices for designing farms and...

tr-4722: mysql database on netapp ontap best practices ·...

mysql最新動向と便利ツールmysql workbench...mysql...

linux network servers nginx, php-fpm, mysql rostislav...

citrix provisioning 1912 ltsr · citrixprovisioning1912ltsr...