continuent tungsten - scalable saa s data management
DESCRIPTION
The key needs of SaaS vendors include: i) managing multi-tenant architectures with shared DBMS, ii) maintaining customer SLAs for uptime and performance and iii) optimized, efficient operations.The key benefits Continuent Tungsten offers SaaS vendors are: i) high availability and protection from data loss, ii) simple, efficient cluster management and iii) enable complex database topologies.Tungsten offers high-availability, database cluster management and management of complex topologies for multi-tenant architectures.Tungsten high availability and data protection features include maintaining live copies with data consistency checking and tightly coupled backup/restore integration with cluster management tools.Tungsten cluster management allows SaaS vendors to migrate customers and perform system upgrades without downtime, thus enabling these maintenance operations during normal business hours.Tungsten also enables complex replication topologies, including data filtering and data archiving strategies, maintaining extra data copies for data-marts, routing different customers to different DBMS copies, and providing cross-site multi-master replication.TRANSCRIPT
© Continuent 2009
Scalable SaaS Data Scalable SaaS Data Management with Management with TungstenTungsten
Robert Hodges
CTO, Continuent, Inc.
© Continuent 2009
AgendaAgenda
/ Introductions
/ What’s So Hard about SaaS Data Management?
/ Introducing Tungsten Clustering
/ Implementing SaaS Data Management with Tungsten
/ Commercial Support for SaaS from Continuent
Definition:
SaaS = “Software as a Service”
© Continuent 2009
About ContinuentAbout Continuent
/ Our Business: Full service database management for SaaS applications
/ Our Value: • Ensure data are available when and where you need them • TCO less than 20% of comparable solutions
/ Our Technical Expertise• Database replication• Cluster management• Application connectivity• SaaS applications
© Continuent 2009
What’s So Hard about SaaS Data Management?
© Continuent 2009
The SaaS Business Problem
/ Meet customer SLAs• Keep applications available• Don’t lose or confuse customer data• Meet response time requirements
/ Enable growth• More customers• More volume from existing customers• New service offerings
/ Raise bottom-line profitability• Keep costs of operation as low as possible• Enhance add-on revenue
© Continuent 2009
Tenant Data
Tenant Data
Shared Data
Tenant Apps
Tenant Apps
Shared Data
Tenant Analytics
Tenant Analytics
Tenant and Shared Data
Successful SaaS Systems Are Complex
/ SaaS databases quickly evolve into “big league” topologies
© Continuent 2009
Core Technical Issue: Multi-Tenancy
/ Defining a flexible multi-tenant data architecture
/ Implementing easy-to-manage clusters
/ Scaling performance on shared databases
/ Protecting tenant data from loss or corruption
/ Handling failures efficiently
/ Upgrades and maintenance windows
/ And do it on economical open source databases!
© Continuent 2009
Introducing Tungsten Clustering
© Continuent 2009
What Is Tungsten?
/ Tungsten implements master/slave clusters to:• Protect data• Maintain high availability• Improve resource utilization• Raise performance
/ Install and set up in a few minutes
/ Integrated backup/restore and data integrity checks
/ Rule-driven management with automated failover
/ Simple procedures for updates and maintenance
/ SQL scaling for multi-tenant systems
© Continuent 2009
What’s Inside Tungsten?
/ Replication - Making copies• Tungsten Replicator -- Database-neutral, platform independent
master/slave replication
/ Connectivity -- Finding databases• Tungsten Connector -- Fast MySQL/PostgreSQL client to JDBC
proxying• Tungsten SQL Router --JDBC wrapper for high-performance and
transparent failover, load-balancing, and partitioning (no proxy required)
/ Management -- Administering the database• Tungsten Manager -- Distributed administration with autonomic,
rule-based configuration and no single point of failure• Tungsten Monitor -- Track resource status and
© Continuent 2009
Tungsten Clustering In ActionTungsten Clustering In Action
Master DBMaster DB Slave DBSlave DB
Master HostMaster Host Slave HostSlave Host
Application Server
SQL Router/Connector
Application Server
SQL Router/Connector
Management Client
Management Client
Replicator
Manager/Monitor
Replicator
Manager/Monitor
© Continuent 2009
Distributed Rule-Based Management
Broadcast commands Broadcast commands and monitoring dataand monitoring data
BusinessBusinessRulesRules
Local DBMS ResourcesLocal DBMS Resources
Local DBMS Local DBMS ResourcesResources
Manager #1(Coordinator)
Manager #3
Manager #2
Admin Client #1
Admin Client #3
Group Group CommunicationsCommunications
Admin Client #2
Local DBMS Local DBMS ResourcesResources
© Continuent 2010
Implementing SaaS Data Management with Tungsten
© Continuent 2010
Best-Practice Multi-Tenant Architecture
/ Separate global and tenant data• Global data are same for all customers• Tenant data belong to individual customer
/ Establish shards to hold tenants• Bucket tenant data into databases, schemas, or groups of tables
/ Define data services• Global data service plus a set of tenant services is typical
/ Master data in a single location for each service• Services have one master and 1+N slaves for HA/scaling
/ Add data services as you grow
/ TIP: Use database as basic sharding unit
© Continuent 2010
Typical SaaS Architecture
globalds1
tenantds1 tenantds4tenantds4tenantds2
Global tenant definitions and billing data
Per tenant data (usually many tenants per service)
globalds1globalds1 tenantds2tenantds2 tenantds4tenantds4tenantds3tenantds3tenantds1tenantds1
Master/slave pair for each service
Implementation
© Continuent 2010
Installing a Tungsten Cluster
Four easy steps:
1. Set up host prerequisites (MySQL, Ruby, Java)
2. Download release from www.continuent.com and un-tar into local directory
3. Run ‘configure’ script• Ensure MySQL is correctly configured• Configure cluster networking• Configure replication and choose backup method• Configure connectivity
4. Perform customizations (e.g., Master VIP setup)
Configure starts services automatically
© Continuent 2010
Advanced But Simple Management Tools
/ ‘cctrl’ is the Tungsten management client
/ List data sources in the cluster[LOGICAL] /cluster/demo/> ls
/ Check cluster liveness[LOGICAL] /cluster/demo/> cluster heartbeat
/ Ensure that replication is synchronized [LOGICAL] /cluster/demo/> cluster flush
/ Start and stop operating system services[LOGICAL] /cluster/demo/> service */mysql restart
/ Get help[LOGICAL] /cluster/demo/> help
© Continuent 2009
Multiple Routes to Databases
Java App ServerTungsten SQL Router
PHP Application
Tungsten Connector
libmysqlclient.a
Tungsten Cluster
MySQL JDBC Driver
Admin &Admin &MonitoringMonitoring
Admin &Admin &MonitoringMonitoring
Virtual IP Virtual IP AddressAddress
© Continuent 2010
Connecting Applications to Databases
/ Tungsten SQL Router embeds in Java applications• Requires Java library substitution• Very high performance, seamless planned failover
/ Tungsten Connector proxies any MySQL application• Also very high performance with seamless planned failover• No library changes
/ Master Virtual IP maintained by Tungsten• Failover is not seamless (broken connections)• Completely transparent to applications
/ Direct connection to database(s)• Use whatever you want• You are responsible for locating master/slaves yourself
© Continuent 2009
Scaling Reads on Slaves
/ Tungsten “session-consistency” distributes tenant reads to slaves if caught up with last tenant write
/ Works with tenants sharded by database or with independent database connections
Master DBMaster DB Slave DBSlave DB
SQL Router
Record tenant Record tenant write position write position
Read if slave caught Read if slave caught up with last tenant up with last tenant write write
© Continuent 2010
A Short Digression on Backups
MySQLMySQLServerServer
DBMS DBMS Data FilesData Files
Active HostActive Host
Applications
BackupBackup HOT/COLD: HOT/COLD: mysqldumpmysqldump
BackupBackupHOT: LVM snapshot HOT: LVM snapshot HOT: InnoDB Hot HOT: InnoDB Hot BackupBackupCOLD: File system copy COLD: File system copy
© Continuent 2010
Setting Up Backups
/ Tungsten provides multiple backup methods
/ Backups can run hot while DBMS is live or cold
/ Logical backups use native DBMS utilities• mysqldump, pg_dump, etc.• Can run hot on slaves• Slow to dump/load and in the case of MySQL may have bugs
/ LVM (Logical Volume Manager) backups take a snapshot of the file system
• Very fast; run hot on slaves• Tend to be large• Snapshot maintenance affects write performance
/ Script backups integrate arbitrary backup programs
/ TIP: Configure backups and test them regularly
© Continuent 2010
Running Backups With Tungsten
/ Hot backup using one-line command on slave[LOGICAL] > datasource centos5a backup lvm
/ Cold backups run on off-line master or slave[LOGICAL] > datasource centos5b offline[LOGICAL] > replicator centos5b offline[LOGICAL] > datasource centos5b backup[LOGICAL] > replicator centos5b online
/ Restore with one-line command[LOGICAL] > datasource centos5a restore
/ TIP: Keep data services small enough to backup and restore easily!
© Continuent 2010
Verifying Tenant Data Copies
/ Tungsten incremental checks verify data without stopping replication
• Implemented as replication events inside Tungsten Replicator• Compute MD5 on all or part of tables• Can do single tables or all tables in a database
/ Run checks from manager in combination with flush command[LOGICAL] /cluster/demo/> cluster check tenant1.*[LOGICAL] /cluster/demo/> cluster flush
/ Slaves can either fail or warn of failures
/ TIP: Install consistency checks as a cron job
/ TIP: Verify backups with restore + consistency check
© Continuent 2010
Handling Database Failures
/ Automatic policy mode performs master failover automatically whenever:
• A database crashes• A replicator fails or crashes• A server reboots or drops off the network
/ Manual policy mode lets you failover manually[LOGICAL] /cluster/demo/> failover
/ Failover procedure promotes most advanced slave• And points other slaves to the slave
/ Recover command brings back failed master[LOGICAL] /cluster/demo/> datasource centos5a recover
/ Slaves recover without admin intervention in automatic mode
© Continuent 2010
Performing Planned Maintenance
/ Tungsten supports “daylight maintenance”
/ To perform maintenance on a slave, just take it offline[LOGICAL] /cluster/demo/> set policy maintenance[LOGICAL] /cluster/demo/> datasource centos5b offline[LOGICAL] /cluster/demo/> replicator centos5b offline
/ To perform maintenance on a master, first do a switch to promote a new master[LOGICAL] /cluster/demo/> switch
/ Then perform maintenance using the slave procedure
/ Switch the master back when you are done
© Continuent 2010
Upgrading Applications and Databases
/ If you have a maintenance window, just upgrade the master and let replication take care of it1. Stop old applications and replace code2. Run upgrade script against master3. Flush and wait for slave to catch up (if necessary)4. Restart new applications
/ If you don’t have a maintenance window, upgrade a slave and then switch1. Take slave offline so that apps point to master only2. Upgrade the slave3. Turn on replication to slave so that it catches up4. Switch the master to the upgraded slave; take old master offline5. Restart applications with new code6. Upgrade old master and bring back online
• TIP: Test all upgrades in a pre-prod environment
© Continuent 2010
SaaS Problems and Tungsten Solutions
Problem Tungsten Features(s)Multi-tenant architecture “Best-practices” database design
Open source databases MySQL 5.0/5.1
Implementing easy-to-manage clusters
Simple installation; cctrl distributed commands
Performance scaling Tenant-based load balancing
Protecting tenant data from loss or corruption
Backups; live replicas; consistency checks
Handling failures Automated and manual failover; live replicas; recovery procedures
Application upgrades Upgrades via replication; slave upgrade
System maintenance Cctrl switch operation and offline/online commands
© Continuent 2010
Solution Roadmap for SaaS Applications
/ Continuent is dedicated to making MySQL the foundation for SaaS businesses
/ We will talk about these in our February 11 webinar “Secrets of Successful SaaS Data Architecture”
SaaS Feature Beta ProdParallel replication Q1 Q2-3Tenant management using shards
Q2 Q2-3
Multi-master replication Q2 Q2-3Cross-site cluster management
Q2 Q2-3
© Continuent 2010
Commercial Support for SaaS from Continuent
© Continuent 2010
Tungsten Products
Feature Community Enterprise
Advanced MySQL replication X X
Advanced management tools X X
SQL routing/proxying X X
Manual failover X X
Automatic failover X
Virtual IP support X
Custom backup/restore X
Built-in Jump-Start consulting X
8x5, 24x7 support X
Guaranteed bug fixes X
/ Tungsten Enterprise offers subscription-licensed supported clustering for SaaS applications
© Continuent 2010
Continuent ServicesContinuent Services
/ Practical consulting by industry experts with deep MySQL and SaaS industry background
/ Structured Jump-Starts• Hardware/OS setup, education, installation, production roll-out
/ Solution architecture workshops• Collaborative engagements to specify SaaS data management
requirements, survey problems, and lay out options for solutions
/ Proof-of-Concept implementations• Fast design and implementation of specific cluster configurations
© Continuent 2010
Tungsten Design Partner Program
/ Continuent is developing advanced SaaS features in collaboration with leading SaaS vendors
/ We offer expertise not only in database clustering but operation of SaaS applications
/ Partner with Continuent to “push the envelope” on scalable data management using open source databases
/ Please contact us today to find out more about joining our efforts
© Continuent 2010
Conclusion
/ SaaS applications have a complex database problem that requires “big league” architecture
/ Continuent offers a complete solution for SaaS data management to raise overall profitability:
• Raise SLAs by improving availability and performance• Improve management efficiency• Grow customers and product offerings through scalable database
architecture• Increase SaaS business profitability
/ Contact us to try Tungsten now!
© Continuent 2010
HQ and Americas560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009
e-mail: [email protected]
EMEA and APACLars Sonckin kaari 1602600 Espoo, FinlandTel +358 50 517 9059Fax +358 9 863 0060
Contact InformationContact Information
Continuent Web Site:http://www.continuent.com