lessons learned in building a highly scalable mysql database
DESCRIPTION
Lessons Learned in Building a Highly Scalable MySQL Database. Mariella Di Giacomo Daniel Chote Geoff Harrison The Hive. Outline. Introduction/Background Motivations, Project Requirements Architectural Design Proposed Solutions Performance Analysis Results MySQL and Other Optimizations. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/1.jpg)
Presented by,
MySQL AB® & O’Reilly Media, Inc.
Lessons Learned in Building a Highly Scalable
MySQL Database
Mariella Di Giacomo
Daniel Chote
Geoff Harrison
The Hive
![Page 2: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/2.jpg)
Outline
Introduction/Background Motivations, Project Requirements Architectural Design Proposed Solutions Performance Analysis Results MySQL and Other Optimizations
![Page 3: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/3.jpg)
The Hive
Started in 1998 (under a different name) Operations in 4 countries Developer-centric organization
![Page 4: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/4.jpg)
The Hive
The Hive’s collective vision is coding and collaborating from anywhere.
The Hive has a number of large scale projects that include video conferencing, VOIP, IM, and virtual whiteboard all connected through smart self-managing project tools.
![Page 5: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/5.jpg)
MySQL Installations at The Hive
70+ MySQL Server Installations Linux Operating System Mostly Commodity Hardware Used for production, batch processing,
replication and development
![Page 6: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/6.jpg)
The Next Generation
“What we need now is a brilliant idea …”
![Page 7: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/7.jpg)
Scalable Architecture
High Volume of users and access traffic Exponential growth of data access and storage Use of relatively commodity hardware
![Page 8: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/8.jpg)
Architectural Requirements
Store data related to millions of users Store some of the data in an XML format Store and Retrieve the data through an Open
Source Relational Database Management System (RDBMS)
![Page 9: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/9.jpg)
Architectural Requirements
Build a scalable architecture capable of handling content updates and data addition at fastest rate reasonably possible
A system that has as little service disruption as possible
Robustness, Flexibility, Scalability and Speed
![Page 10: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/10.jpg)
A Scalable Architecture
Scalable, robust, fast and flexible.Capable of handling data related to millions of users.Based on a data layout that maps all the data for each user to its individual table, organizing the users across multiple systems, databases and filesystems. Consists of homogeneous systems.Each system consists of two physical computers that synchronize the data through Master Master MySQL Replication.
![Page 11: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/11.jpg)
Hardware Architecture
The hardware architecture consists of:Pairs of AMD nodes (Opteron). Each node has 2 CPUs 2GHz with 2cores; it has 24 GB of RAM + 8 GB of swapEach computer system has two types of data storage devices: Serial Advanced Technology Attachment (SATA) and Solid State Drive (SSD). Each computer System is composed of Debian Linux Operating System (OS)
![Page 12: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/12.jpg)
Software Architecture We benchmarked 4 filesystems (Ext3, JFS, ReiserFS
and XFS). We chose MySQL as the RDBMS for the following
reasons:Open SourceSpeedData Storage CapabilitiesDatabase Design Fault ToleranceSecurity
![Page 13: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/13.jpg)
MySQL Database Design MySQL supports several storage engines. We chose MyISAM, for the following reasons:
Each table data is stored in a MYD fileEach table indices are stored in a MYI fileEach table file requires less disk space compared to other storage engines.Memory key buffer for the indexed data
Downsides of MyISAMGood for high volume of writes and/or read, but not both in high concurrency.
![Page 14: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/14.jpg)
MySQL Database Design
Our design takes advantage of the fact that there will not be high level of concurrency on the same table, while high level of concurrency will be present at database and server level.
Table Structure Design:Reduce and or combine indices on some fields, to take fewer resources when adding and modifying dataUse of NOT NULL fields where possibleUse the most appropriate, efficient field type for a given field
![Page 15: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/15.jpg)
Architecture
![Page 16: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/16.jpg)
Why that architecture ?
![Page 17: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/17.jpg)
Why that architecture ?
We had an idea ….. We validated the idea through benchmark
results and analysis We analyzed several MySQL Storage engines
(MyISAM, InnoDB, Maria, PBXT, Falcon) on 4 types of filesystems (Ext3, JFS, XFS, ReiserFS) using two system architectures (AMD and Intel), two types of data device technologies and integrated the benchmarks with some specific low-level optimizations.
![Page 18: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/18.jpg)
Benchmarks
Benchmark Configuration Benchmark Hardware Architecture Benchmark Topics
![Page 19: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/19.jpg)
Benchmark Configuration
The benchmark program is written in Perl. The same program is used for table creation, data insertion, update and selection.
The benchmark program spawns 100 parallel instances (clients) that execute Data Definition Language (DDL) and Data Manipulation Language (DML) statements against the MySQL server.
The statement response time has been taken on the server side and on the client side.
![Page 20: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/20.jpg)
Benchmark Configuration
All benchmarks have been run several times for the same configuration.
All the computer systems benchmarked have been set on an isolated network to lower noise as much as possible.
All computer systems are composed entirely of Debian Linux (Lenny) running 2.6.23 and 2.6.24
All systems have been compiled with latest MySQL 5.1.x
![Page 21: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/21.jpg)
Benchmark Hardware Architecture
The hardware architecture used for the benchmarks consists of:Pairs of AMD nodes (Opteron). Each node has 2 CPUs 2GHz with 2cores; 24 GB of RAM + 8 GB of swap2 Intel Xeon nodes. Each node has 2 CPUs 3GHz with 2cores and hyper-threading enabled (4 virtual cores each CPU); 24GB of RAM + 8 GB of swapEach system has identical SATA storage devices. The AMD nodes contain also two identical SSD drives that are raided via an add-on raid controller.
![Page 22: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/22.jpg)
Benchmark Hardware ArchitectureEach system always uses two dedicated storage devices for database benchmarks.
![Page 23: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/23.jpg)
Benchmark Topics
Data LayoutOrganize each user’s data in a separate tablePartition a table to contain the data associated to a set of users, using MySQL horizontal partitioning.
Database Organization20 and 100 databases with 2 million tables50 databases with 1 million tables
MySQL Storage Engines: MyISAM, InnoDB, Maria, Falcon, PBXT
![Page 24: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/24.jpg)
Benchmark Topics
Filesystem OrganizationOne data storage device for filesystem. Each filesystem contains 1 million tablesExt3, JFS, ReiserFS and XFS
Data Storage Device TechnologySATASSD
![Page 25: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/25.jpg)
Data Layout Scenarios
Two data layout scenarios are evaluated:Organize each user’s data in a separate tablePartition a table to contain the data associated to a set of users, using MySQL horizontal partitioning. In this case the total number of tables required is the total number of users divided by the maximum number of partitions allowed on a partitioned table.
![Page 26: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/26.jpg)
Data Layout Scenarios
Each Data Layout scenario has been benchmarked for:Dynamic Creation of 1 million to 2 million tables on one system. In the case of 1 million tables they would be physically located on only one filesystem (one hard drive). In the case of 2 million tables they would be spread across 100 or 20 databases. Each dedicated filesystem would be mapped to a physical storage device.Data Insertion and Update on the created tablesData Selection
![Page 27: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/27.jpg)
Table Creation
![Page 28: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/28.jpg)
Table Creation Configuration
Table creation has been executed using a stored procedure.
Each benchmark has been executed on an AMD and Intel system
Each benchmark has been evaluated with 4 distinct filesystem types.
The table creation benchmarks have been evaluated with write cache enabled, write back (WB) and write cache disabled, write through (WT) disk policies.
![Page 29: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/29.jpg)
Data Insert, Update and Query
![Page 30: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/30.jpg)
Inserts and Updates Configuration
The benchmark configuration for the data insert and update is equivalent to the one used for table creation.
Each benchmark has been executed only on AMD systems
Each benchmark has been evaluated with 4 distinct filesystem types.
The benchmarks have been executed using write cache enabled and write cache disabled.
![Page 31: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/31.jpg)
Inserts and Updates Configuration
While table creation would perform better using write cache disabled, initial benchmarks with DML operations using write cache disabled would perform worst compared to the equivalent with write cache enabled.
![Page 32: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/32.jpg)
Data Layout Scenarios
Two data layout scenarios are evaluated:Organize each user’s data in a separate table
SATA Device Technology•MyISAM•InnoDB•Falcon•Maria•PBXT
SSD Device TechnologyMyISAM
![Page 33: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/33.jpg)
Data Layout Scenarios
Data layout scenarios:Organize each user’s data in a separate table
SATA Device Technology
![Page 34: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/34.jpg)
MyISAM and SATA Benchmarks
Benchmarks shown in the coming graphs have been executed using the MyISAM Storage Engine
The MyISAM key_buffer is set to 4GB In each benchmark the system containing
MySQL server has two storage devices for the databases
![Page 35: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/35.jpg)
Table Creation
These graphs show the benchmark results derived from processing 2 million tables.
Table creation of 2 million tables was spread across 20 databases. The databases were split across two filesystems. Each filesystem was on a unique SATA drive. (The drives were set up as single volumes through an add-on raid controller)
![Page 36: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/36.jpg)
MyISAM using WB
![Page 37: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/37.jpg)
MyISAM using WT
![Page 38: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/38.jpg)
MyISAM using JFS
![Page 39: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/39.jpg)
MyISAM with XFS
![Page 40: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/40.jpg)
MyISAM with ReiserFS
![Page 41: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/41.jpg)
MyISAM with Ext3
![Page 42: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/42.jpg)
Table Creation with MyISAM
The benchmark results showed: When creating 2 million tables, approximately
36GB of space is used The highest processing rate was obtained using
the JFS filesystem on a volume (SATA via raid controller) with write cache disabled
The MyISAM key buffer did not play a major role The number of volumes would influence the
processing rate.
![Page 43: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/43.jpg)
Table Creation with MyISAM
Between 1 million tables on one filesystem versus 2 million tables on two filesystems, we did not see any major impact in number of tables created per second (provided that we would not reduce the number of storage devices).
1 million tables would generate a total of 3 million files, 2 million tables would double the number.
![Page 44: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/44.jpg)
MyISAM and SATA Benchmarks
Inserts and Updates on 1 million tables (50 databases)
Inserts and Updates on 2 million tables (20 and 100 databases)
![Page 45: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/45.jpg)
Inserts and Updates
The next graphs show the results of DML operations on 1 million tables created using MyISAM storage engine with 4 types of filesystems and using write cache enabled.
While table creation would perform better using write cache disabled, initial benchmarks with DML operations using write cache disabled result in generating response times 10% higher compared to the equivalent with write cache enabled.
![Page 46: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/46.jpg)
MyISAM and Ext3 and WB
![Page 47: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/47.jpg)
MyISAM and JFS and WB
![Page 48: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/48.jpg)
MyISAM and XFS and WB
![Page 49: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/49.jpg)
MyISAM and ReiserFS and WB
![Page 50: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/50.jpg)
MyISAM and SATA Benchmarks
Inserts and Updates on 1 million tables Inserts and Updates on 2 million tables
Benchmarks on DML operations on 2 million tables were stopped after a few tests due to their high response time, caused by heavy I/O operations on the volumes.
![Page 51: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/51.jpg)
MyISAM and SATA Results
The benchmarks run using MySQL MyISAM storage engine on SATA device technology showed, after several optimizations, that the average response time of a DML operation would be higher than one second on an environment with 100 concurrent clients.
![Page 52: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/52.jpg)
InnoDB and SATA Benchmarks
Table creation with 1 million tables
(50 databases) Table creation with 2 million tables
(20 databases) Inserts and Updates on 1 million tables Inserts and Updates on 2 million tables
![Page 53: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/53.jpg)
Table creation
These next graphs show the benchmark results derived from processing 2 million InnoDB tables.
The benchmarks have shown:There is a memory overhead with the table dictionary (approximately 6GB of memory for all of the table dictionaries).There is an overhead on the disk. Approximately 4 times the space (in comparison to MyISAM) is used to store a table, whether using single or multiple table spaces, or a single file per table.
![Page 54: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/54.jpg)
InnoDB on JFS
![Page 55: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/55.jpg)
Inserts and Updates with InnoDB
The next graph shows the results obtained with 1 million tables using InnoDB
The number of databases is 50 The filesystem considered is JFS The data storage devices are SATA The InnoDB buffer is set to 6GB, but not limited
to it
![Page 56: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/56.jpg)
InnoDB on JFS with WB
![Page 57: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/57.jpg)
InnoDB and SATA Results
The benchmarks were evaluated only with one filesystem type, JFS. They were not extended to other types because of the InnoDB overheads (memory required for table dictionary and disk space).
![Page 58: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/58.jpg)
Falcon and SATA Benchmarks
We evaluated a few benchmarks related to table creation and DML operations using Falcon as MySQL Storage engine.
The benchmarks were executed using 1 million tables.
Falcon storage engine was discarded because it is a relatively new storage engine and is not yet robust as MyISAM or InnoDB.
![Page 59: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/59.jpg)
Maria and SATA Benchmarks
We evaluated a few benchmarks related to table creation and DML operations using Maria Storage engine.
The benchmarks were executed using 1 million tables.
Maria storage engine was discarded because it is a new storage engine and is not yet optimized as MyISAM.
![Page 60: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/60.jpg)
Maria and SATA Benchmarks
One of the interesting features of Maria is the ability to be generate crash safe tables. Maria is a crash safe version of MyISAM.
It has all the main functionality of MyISAM engine, but includes recovery support, full logging, etc.
![Page 61: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/61.jpg)
PBXT and SATA Benchmarks
We evaluated a few benchmarks related to table creation using PBXT Storage engine.
The benchmarks were executed using 1 million tables.
PBXT storage engine was discarded because it showed some limitations on the number of tables that could be allocated.
![Page 62: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/62.jpg)
Data Layout Scenarios
Data layout scenarios:Organize each user’s data in a separate table
SSD Device Technology
![Page 63: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/63.jpg)
MyISAM and SSD Benchmarks
1 million tables using 50 databases 2 million tables using 100 and 20 databases
![Page 64: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/64.jpg)
Table Creation (WB)
![Page 65: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/65.jpg)
MyISAM and Ext3 and WB
![Page 66: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/66.jpg)
MyISAM and JFS and WB
![Page 67: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/67.jpg)
MyISAM and ReiserFS and WB
![Page 68: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/68.jpg)
MyISAM and XFS and WB
![Page 69: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/69.jpg)
MyISAM and SSD Benchmarks
After having run the benchmarks for 1 million tables, we noticed that Ext3 provided the best performance. We expect this is due to the fact that it does less IO optimization when compared to other filesystems.
Consequently the following benchmarks of 2 million tables have been executed only using the Ext3 filesystem and two SSD storage devices.
![Page 70: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/70.jpg)
MyISAM and SSD Benchmarks
1 million tables 2 million tables using 100 and 20 databases
![Page 71: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/71.jpg)
Table Creation: Ext3 and WB (100)
![Page 72: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/72.jpg)
MyISAM and Ext3 and WB (100)
![Page 73: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/73.jpg)
MyISAM and Ext3 and WB (100)
![Page 74: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/74.jpg)
MyISAM and Ext3 and WB (20)
![Page 75: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/75.jpg)
MyISAM and Ext3 and WB (20)
![Page 76: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/76.jpg)
MyISAM and SSD Results
We have evaluated 50 databases with 1 million of tables on a single SSD device, 20 and 100 databases with 2000000 of tables spread on two SSD devices.
The benchmarks showed that Ext3 with write cache enabled would perform better while doing data inserts, updates and selects with 100 concurrent clients.
![Page 77: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/77.jpg)
MyISAM and SSD Results
The benchmarks showed also a quite interesting improvement in response time in the chosen architecture.
The noop I/O scheduler performed better compared to the others available on Linux I/O.
![Page 78: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/78.jpg)
MyISAM SSD versus MyISAM SATA
The graph below compares DML statements with MyISAM using 100 databases with 1 million tables on SATA and SSD
![Page 79: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/79.jpg)
Data Layout Scenarios
Initially two data layout scenarios are evaluated:Organize each individual user’s data in a separate tablePartition a table to contain the data associated to a set of users, using MySQL horizontal partitioning. In this case the total number of tables required is the total number of users divided by the maximum number of partitions allowed on a partitioned table.
SATA Device Technology•MyISAM•InnoDB
![Page 80: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/80.jpg)
MySQL and Horizontal Partitioning
The next slides discuss the results of 2 million tables with horizontal partitioning and both the MyISAM and InnoDB storage engines.
The type of partitioning used was “Range” partitioning based on the primary key, unsigned integer field.
The number of partitions investigated was 10 and 1024.
![Page 81: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/81.jpg)
MyISAM with Horizontal Partitioning
The next graph shows the creation of 2 million tables using the MyISAM storage engine and horizontal partitioning
The initial number of partitions was set to 10. This implies that for each table will be created 2 extra files per partition, with a total of 22 files per table. In the case of 2 million users this would generate a total of 4.4 million files and 200,000 tables.
![Page 82: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/82.jpg)
MyISAM with Horizontal Partitioning
We investigated also the scenario with a number of partitions of 1024.
With 1024 partitions per table, one table would generate 2050 files, so in the case of 2 millions users approximately 4 millions of files and approximately 1954 tables.
The graph shows the results of table creation using a number of partitions of 10.
![Page 83: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/83.jpg)
Table Creation with WB
![Page 84: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/84.jpg)
MyISAM with Horizontal Partitioning
The drawback of using MyISAM versus MyISAM with partitioning was:Pros
The results for using a number of 10 partitions were slightly lower compared to those of the first data layout scenario.
ConsHigher level of management compared to the first data layout scenario and still a high number of files to be managedComplexity and overhead when using partitioning with a number of partitions set to 1024Some limitations on the possibility of using sub partitioning to distribute segments of data for a specified user.
![Page 85: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/85.jpg)
InnoDB with Horizontal Partitioning
InnoDB with partitioning was considered but was excluded for the following reasons:
Memory overhead due to table dictionary Disk Space overhead compared to MyISAM as
discussed earlier (4 times that of MyISAM).
![Page 86: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/86.jpg)
Optimizations
Why Optimize ?
Get more Performance with same Hardware. As your data grows, Performance may degrade.
What Optimize ?
Hardware Architecture, Operating System Components and MySQL.
Where Optimize ?
Monitor your system and applications and watch for possible bottlenecks.
![Page 87: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/87.jpg)
Optimizations
MySQL Optimizations
Other Optimizations
![Page 88: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/88.jpg)
MySQL Optimization
Server Compilation (query cache disabled, fast mutexes enabled, etc.)
Server Configuration and Tuning (memory buffers, number of open tables, number of open files, concurrency, etc.)
Table structure (Database Design) and allocation
Query handling
![Page 89: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/89.jpg)
MySQL Server Compilation
Compilation using appropriate optimizations for Linux (Debian) and the specific hardware architecture made improvements of 10% of speed over equivalent standard MySQL server binary.
![Page 90: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/90.jpg)
MySQL Configuration
MySQL assumes little about your hardware system configuration.
Optimal size of the MyISAM key_buffer used to allocate indices in memory.
Number of open tables at a given point (it plays an important role when dealing with millions of tables).
Max number of temporary tables
![Page 91: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/91.jpg)
MySQL Configuration
Number of connections per second, number of threads created per second, max number of connections
Thread concurrency Thread memory allocation Long query time
![Page 92: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/92.jpg)
MySQL Query Handling
Use EXPLAIN SELECT to improve queries Use Indexes Simplify some of the WHERE clauses Use UNION of SELECT instead of only one
SELECT with several conditions in the WHERE clause
Disabled query cache, provided that the queries would not take advantage of that
Avoid Table Scans
![Page 93: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/93.jpg)
Other Optimizations
File System tuning File System “mount” options tuning OS Linux Scheduler tuning OS Linux Kernel Swap tuning Tuning for Write-Heavy Loads
![Page 94: Lessons Learned in Building a Highly Scalable MySQL Database](https://reader036.vdocuments.site/reader036/viewer/2022062323/56815985550346895dc6c5c7/html5/thumbnails/94.jpg)
Q&A
Thanks for your attention! We would like to thank The Hive for making this
talk possible.