my first 100 days with a mysql dbms (wp)

COLLABORATE 15 – IOUG Forum Database

1 | P a g e “My F i r s t 100 day s w i th a MySQL DBMS” Wh i t e Pape r

My First 100 days with a MySQL DBMS

René Antúnez, The Pythian Group

ABSTRACT

TARGET AUDIENCE This document will benefit whoever is starting to use and administer a MySQL RDBMS, it covers the basic concepts and management tips to be able to start as a MySQL Administrator.

EXECUTIVE SUMMARY

WHAT IS A MYSQL RDBMS? MySQL is one, if not, the world’s most popular open source database and it is a main component of the LAMP (Linux,Apache,MySQL,Php) open source software stack. Being an open source software, there exists a commercial site (http://mysql.com) which includes links to webinars, whitepapers, and information on products and services and a developer zone (http://dev.mysql.com/) includes links to manuals, worklogs, podcasts, labs, articles, news and Planet MySQL. Unlike other RDBMS, MySQL uses a pluggable storage engine architecture, meaning that that a storage engine can be loaded and unloaded from a running MySQL server

MYSQL ARCHITECTURE As many other popular RDBMS’s the MySQL database system also operates client/server architecture. MySQL can be setup to run multiple instances with different server ids on different sockets and port numbers

• Server MySQL : mysqld is the daemon program that manages databases and tables. Mostly installed using binary pre-compiled for select Operating System. It is also available open source from different vendors such as Oracle, Percona and Mariadb. Default port for mysql is 3306 and can be configured

• Client programs: These are programs are connecting to mysqld server by sending requests using network or socket

connection. While there are many clients to be used for mysql connection MySQL provides pre-compiled command line utility called mysql.

With MySQL being the most popular open source DBMS in the world and with an estimated growth of 16 percent annually until 2020,we can assume that sooner or later an Oracle DBA will be handling a MySQL database in their shop. This beginner/intermediate-level session will take you through my journey of an Oracle DBA and my first 100 days of starting to administer a MySQL database, show several demos and all the roadblocks and the success I had along this path.

Reader will be able to: • What is the MySQL architecture and how does it relate to the Oracle Architecture. • Gain an insight into how to administer a MySQL DBMS and what tools are available for a MySQL DBA. • What are the main concerns regarding backups and replication with a MySQL DBMS.



MySQL consists of two-tier architecture while processing data to respond DML statements. MySQL’s architecture on storage engine and SQL tiers are different than conventional RDBMS systems. To understand the role of storage engines in MySQL you have to have a basic understanding of MySQL’s API architecture.

1) A query comes in (SQL interface) 2) Is parsed and the syntax is checked (Parser), 3) An execution plan is determined (optimizer) 4) Execution plan sent to the query processor who generates a queue of method calls (Handler calls to the API interface

of the various storage engines). 5) The storage engine handles the actual data/file/disk access.

How it’s done is left up to the individual storage engine. Because of the separation of the server from the storage engines, the various storage engines can have differing capabilities.

• Connection Tier: The MySQL server (mysqld) accepts connections based on the protocol used and authenticates them to allow access to Upper Tier. If connections established through this tier it’s passed to upper MySQL tier.

• Upper Tier: This tier has combined layers of SQL parser and optimizer. The server mysqld parses statements and uses

optimizers to help to determine best access method.

• Lower Tier: This tier is more of modular architecture than upper tier. At this tier MySQL has multiple storage engines those manage own tables. Each storage engine is responsible for managing associated. So bottom line is storage engines are pluggable to MySQL server. There are differences and restrictions between storage engines.

OPTION FILES MySQL reads startup options from option files, better known as configuration files and these lists specific startup options which allows the option set to persist after a server restart. One thing that we need to be aware of is that the option file by default can be read from a number of locations, so be sure to look in the manual for a full listing of where MySQL looks for



the option files. In Linux you normally see them in /etc/my.cnf and /etc/mysql/my.cnf, in Windows INSTALLDIR\my.ini and INSTALLDIR\my.cnf, be sure to read the latest manual for a full listing of where MySQL looks for the option files. Example of a global option file.

To find out which configuration files mysqld is loading on Linux, you can run the following shell command:

DISK AND MEMORY SPACE USAGE MySQL server uses disk space based on storage type selected. Top-level directory per database is created from mysql --datadir configuration.

a) Database directories: Database directory created per database to hold data files associated with storage engine types. This directory is created independent from storage engine type.

b) Table format files called (*.frm) files. This file keeps the definition (schema) of the table. Every table will have one .frm file of its own.

c) Data and index files created for tables under same top-level directory used for database. MyISAM storage engine creates one file per table for each data and index. So it will have three files per table under its database directory.

d) If using the InnoDB storage engine it uses tablespace to control data and index information for all of its tables. It is the most important and complex one amongst other engines provided. It’s the only engine provides ACID compliance as well as other advanced features like stored procedures and triggers.

[client] port=3306 socket=/tmp/mysql.sock [mysqld] port=3306 socket=/tmp/mysql.sock key_buffer_size=16M max_allowed_packet=8M [mysqldump] quick

$strace -e stat64 mysqld --print-defaults $lsof -p <mysqld_pid)



INSTALLING MYSQL There are different methods of installing MySQL, but the easiest way to do it, is through a yum repository, below is an example on a Linux 6.6

The generic Unix/Linux binary file layout of the installation is shown in the table below

Directory Contents

bin Client programs and mysqld server

data Log files and databases

docs Manual in Info format

man Unix manual pages

As root [root@localhost ~]# groupadd mysql [root@localhost ~]# useradd -r -g mysql mysql [root@localhost ~]# yum localinstall mysql-community-release-el6-6.noarch.rpm [root@localhost ~]# yum repolist enabled | grep "mysql.*-community.*" [root@localhost ~]# yum repolist all | grep mysql [root@localhost ~]# yum install mysql-community-server [root@localhost ~]# service mysqld start Starting mysqld:[ OK ] [root@localhost ~]# service mysqld status mysqld (pid 407) is running. [root@localhost ~]# mysql_secure_installation As local user [mysql@localhost Desktop]$ mysql -u root -p Enter password: … mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) mysql> CREATE USER 'rene'@'%' IDENTIFIED BY 'oracle'; Query OK, 0 rows affected (0.00 sec) mysql> grant all privileges on *.* to 'rene'@'%' with grant option; Query OK, 0 rows affected (0.01 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) mysql> exit



include Include (header) files

lib

Libraries

scripts

mysql_install_db

share

Misc support files including error messages, sample option files and SQL for database installation

sql-bench Benchmarks

MYSQL ENGINE TYPES As we mentioned before, MySQL uses a pluggable storage engine architecture, meaning that that a storage engine can be loaded and unloaded from a running MySQL server. Two of the most popular are Innodb and MyISAM . As of MySQL 5.5 and later Innodb is the default storage engine in a fresh MySQL installation.

DIFFERENCES OF INNODB AND MYISAM

a) InnoDB DML operations (add, update and delete data) is ACID (atomic, consistent, isolated and durable) model compatible.

b) InnoDB has foreign keys and relationship constraints while MyISAM does not c) InnoDB has granular locking level from table to row-level lock for inserting and updating while MyISAM implements

table-level lock. d) InnoDB has Stored Procedures and Triggers while MyISAM does not e) InnoDB allows some online DDL (alter) operations without locking MyISAM does not

mysql> SHOW ENGINES;

+--------------------+---------+----------------------------------------------------------------+--------------+------+-----+ | Engine | Support | Comment | Transactions | XA | Savepoints | +--------------------+---------+----------------------------------------------------------------+--------------+------+-----+

| FEDERATED | NO | Federated MySQL storage engine | NULL | NULL | NULL | | MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |

| MyISAM | YES | MyISAM storage engine | NO | NO | NO | | BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO | | CSV | YES | CSV storage engine | NO | NO | NO |

| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO | | ARCHIVE | YES | Archive storage engine | NO | NO | NO |

| InnoDB | DEFAULT | Supports transactions, row-level locking, and foreign keys | YES | YES | YES | | PERFORMANCE_SCHEMA | YES | Performance Schema | NO | NO | NO |

+--------------------+---------+----------------------------------------------------------------+--------------+------+-----+ 9 rows in set (0.28 sec)



MYSQL TROUBLESHOOTING LOGS As always logs are the best place to start troubleshooting MySQL has the following files:

• Error Log. - It contains information about errors that occur while the server is running (also server start and stop). If mysqld notices a table that needs to be automatically checked or repaired, it writes a message to the error log. On some operating systems it writes the stack trace if mysqld crashes.

• General Query Log. - This is a general record of what mysqld is doing (connect, disconnect, queries). It logs each SQL statement received from clients and can be very useful when you suspect an error in a client and want to know exactly what the client sent to mysqld. This log can grow quickly on a moderately busy server - so watch it carefully. This can be enabled or disabled:

• Slow Query Log. – It is your first line of offense for tuning queries Ιt consists of "slow" SQL statements (as

indicated by its name). It logs SQL statements that took more than the parameter long_query_time in seconds and required at least min_examined_row_limit rows to be examined. To disable or enable the slow query log, use the global system variable slow_query_log. Set slow_query_log to 0 (or OFF) to disable the log or to 1 (or ON) to enable it.

OBTAINING EXECUTION PLAN INFORMATION When you issue a query, the MySQL Query Optimizer tries to devise an optimal plan for query execution. The EXPLAIN statement provides information about how MySQL executes statements. You can see information about the plan by prefixing the query with EXPLAIN. EXPLAIN EXTENDED can be used to obtain additional execution plan information. As of MySQL 5.7.3, the EXPLAIN statement is changed so that the effect of the EXTENDED keyword is always enabled. The order of the tables given in the EXPLAIN output matters. Tables listed in the order that MySQL reads them while processing and performing the nested-loop joins. MySQL reads a row from the first table, and then finds a matching row in the second table, the third table, and so on. When all tables are processed, MySQL outputs the selected columns and backtracks through the table list until a table is found for which there are more matching rows. The next row is read from this table and the process continues with the next table.

If you detect that you have a problem with indexes not being used when you believe that they should be, run ANALYZE TABLE to update table statistics, such as cardinality of keys, that can affect the choices the optimizer makes.

SET GLOBAL general_log = 'OFF';

slow_query_log = 1 slow_query_log_file = /var/log/mysql/slow.log long_query_time = 10 log_queries_not_using_indexes = 1

mysql> explain extended select j1.c1 from j1, j2, j3 where j1.c1 = j2.c1 and j3.c1 = j1.c1; +----+-------------+-------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | j1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | j2 | ALL | NULL | NULL | NULL | NULL | 3 | Using where | | 1 | SIMPLE | j3 | ALL | NULL | NULL | NULL | NULL | 4 | Using where | +----+-------------+-------+------+---------------+------+---------+------+------+-------------+



LOGICAL AND PHYSICAL BACKUPS Backups are always important when managing data. MySQL has several ways to do a backup. LOGICAL BACKUP The first way is doing a logical backup; this will save the logical structure and the data content of our database. This type of backup is suitable for small databases. A logical backup does not include log and/or configuration files or other database related files. A very common tool is mysqldump, it will pull data out of the server, and write it down in logical format. One great advantage of using mysqldump is that is machine independent.

PHYSICAL BACKUP The backup consists of exact copies of database directories and files. Typically this is a copy of all or part of the MySQL data directory, so it normally will contain log and/or configuration files or other database related files. One very common way to do physical backups is through filesystem snapshot, but there is a database level lock that has to happen for these to be consistent. The steps for this type of backup are below:

a) FLUSH TABLES WITH READ LOCK; b) Take the filesystem snapshot c) UNLOCK TABLES;

You can also use MySQL Enterprise Backup, which is the “official” solution for hot backups of InnoDB tables, it will perform warm backup for other storage engines.

[mysql@localhost Desktop]$ mysqldump -u pythian – p $pw --extended-insert=true --add-drop-database --databases rene | gzip > /export/full_dump.gz

[root@localhost mysqlbackup ]# ./mysqlbackup --user=root -p --backup-dir=/home/admin/backups backup-and-apply-log ... 140904 12:34:54 mysqlbackup: INFO: MEB logfile created at /home/admin/backups/meta/MEB_2014-09-04.12-34-54_backup_apply_log.log -------------------------------------------------------------------- Server Repository Options: -------------------------------------------------------------------- datadir = /var/lib/mysql/ innodb_data_home_dir = innodb_data_file_path = ibdata1:12M:autoextend innodb_log_group_home_dir = /var/lib/mysql/ innodb_log_files_in_group = 2 innodb_log_file_size = 50331648 innodb_page_size = 16384 innodb_checksum_algorithm = innodb innodb_undo_directory = /var/lib/mysql/ innodb_undo_tablespaces = 0 innodb_undo_logs = 128 … innodb_undo_directory = /home/admin/backups/datadir innodb_undo_tablespaces = 0 innodb_undo_logs = 128 mysqlbackup: INFO: Unique generated backup id for this is 14098484948398421 mysqlbackup: INFO: Creating 14 buffers each of size 16777216. 140904 12:34:57 mysqlbackup: INFO: Full Backup operation starts with following threads 140904 12:34:57 mysqlbackup: INFO: Starting to copy all innodb files... … 140904 12:35:14 mysqlbackup: INFO: Apply-log operation completed successfully. 140904 12:35:14 mysqlbackup: INFO: Full backup prepared for recovery successfully. mysqlbackup completed OK!



MYSQL REPLICATION MySQL supports many replication types. The way MySQL replication works is keeping track of all updates in log files (index & binlog) on the Master (with log-bin enabled). The MySQL slave receives these binlog dumps and builds relay logs via I/O thread. SQL thread applies all changes to the MySQL slave.

MASTER - MASTER (CIRCULAR) Master - Master replication means MySQL servers (mysqld) on different hosts or same host with different port number (port) setup with unique (server-id) and binlog (log-bin) enabled MASTER - SLAVE (SINGLE - MULTI) One of the most common replication styles that can be setup is with one master and multiple slaves. Master - Slave replication means MySQL servers (mysqld) on different hosts or the same host with different port number (port) setup with unique (server-id) and binlog (log-bin) enabled on Master. MASTER - SLAVE - SLAVE (A-> B-> C->) This type of MySQL replication is similar setup as master-slave replication and called chain replication. The only difference is host B in the chain should have log_slave_updates (log-slave-updates) enabled on top of binlog (log-bin) variables. A (Master) - > B (Master-Slave) -> C (Slave)



REFERENCES a) MySQL 5.6 Manual

http://dev.mysql.com/doc/refman/5.6/en/ b) MySQL Architecture

https://thinkingmonster.wordpress.com/database/mysql/mysql-architecture/ c) A fast and furious guide to MySQL database engines

http://www.techrepublic.com/article/a-fast-and-furious-guide-to-mysql-database-engines/ d) MySQL Storage Engines

http://www.w3resource.com/mysql/mysql-storage-engines.php e) Using EXPLAIN to Write Better MySQL Queries

http://www.sitepoint.com/using-explain-to-write-better-mysql-queries/ f) MySQL Enterprise Backup User's Guide

http://dev.mysql.com/doc/mysql-enterprise-backup/3.11/en/intro.html g) Getting Started with MySQL - Ligaya Turmelle (Presentation) h) Pythian Internal Documents

my first 100 days with a mysql dbms (wp)

Technology