mariadb training - downloads · mariadb performance tuning mariadb training. mariadb performance...

MariaDB Performance TuningMariaDB Training

MariaDB Performance Tuning

Introduction

Tuning Overview

Collecting Information

Schema Tuning

Query Tuning

Common BottlenecksArchitecture

Server Configuration

MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Introduction

[email protected]

Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

Introducing MariaDB AbFounders from MySQL the

Company and Community Funded by Founders, Employees,

and Venture Capital Over 100 Employees Several former MySQL Employees

and Community Members in over 14 Countries

3

[email protected]


Personal IntroductionsInstructor

Name and Background

Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics

4

[email protected]


Class Schedule & Personal ConcernsStarting and Ending Times Planned Breaks On- Site

Location of Rest Rooms Smoking Areas Snacks and Drinks

LVC Classes Chat with Everyone

5

[email protected]


Course OutlineBasics & Core

Tuning Overview Collecting Information

Tweaking Schema Tuning Query Tuning Common Bottlenecks

Server & Storage Engines Architecture Server Configuration MyISAM InnoDB

Administration MariaDB Replication Maintenance Impact

6


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Tuning Overview

[email protected]


Tuning TermsPerformance Efficiency Scalability Benchmark Throughput Risk

Queuing SLA Response Time Capacity Resources Uptime

8

[email protected]


Value in Performance TuningEfficient Use of Hardware Resources

Anticipate Needs

Best Performance for Users Machines Cost less than User Time Competitive Advantage

Maintain Spare Infrastructure Capacity Be Prepared for Application Development Needs (New

Features) Allow for Unexpected Traffic Spikes or Other Changes

9

[email protected]


Tuning LevelsApplication

Primary Responsibility of Architects and Developers Secondary Responsibility of DBA for Input

Database Primary Responsibility of DBA — Tuning and Monitoring

Secondary Responsibility of Developers — Gain Insight and Understanding

10

[email protected]


Tuning LevelsServer

Primary Responsibility of System Administrator Secondary Responsibility of DBA — Knows Well Databases

Network Primary Responsibility of Network Administrator Secondary Responsibility of DBA — Replication or Clusters

11

[email protected]


Tuning Routine - When to Tune

Tune from Start in the SDLC Start Early to Ensure Schema is Well Constructed Test Queries on Real Data — Watch for Bottlenecks Over Tuning without Production Data or Traffic is Counter

Productive

Conduct Periodic Reviews of Production Systems Watch for Schema, Query and Significant Changes Check Carefully New Application Features

Monitor System Resources — Disk, Memory, Network, CPU

Eliminate the Obvious Potential Problems Be Proactive in Solving Problems

Emergency Response Tuning

12

[email protected]


Tuning Routine - When Not to Tune

Identify Objectives in Advance Adhere to Objectives

Decide on Best Use of Resources Tune Appropriately Don't Excessively Tune

Affect Adversely Data Integrity Where is Speed Most Important Where is Integrity Most Important Adhere to these Boundaries

13

[email protected]


Establish BaselineNumber of Queries Expected per Unit of Time Number of Concurrent Client Connections Expected

Average Number of Clients Peak Number of Concurrent Clients

Length of a Client Connection Expected — In Milliseconds Size of Dataset — Fit in Memory When all Requirements will Double

14

[email protected]


Construct TestsDetermine System Resource Limits and Design Test

Should Run Normally Should Exceed Server Resource Limits Should Simulate Nightly Back-up Load without Users

Noticing

Examine Impact of Configuration Changes Remove or Restrict Access to CPU core or Hard Drives Start another Process via cron that Consumes Suddenly

Memory Observe What Happens When a Caching Layer goes

Offline

Use Tests to Identify Easy Solutions

15

[email protected]


Examine Test ResultsTest System Identical to the Production System

Operating System (Version, Libraries, Patches)

Hardware (Number of Cores, Physical Memory, Disk Speed) Other Resources (Memory Available to MariaDB, Competing

Services) Test System Simulating Real Data and Traffic

Number of Clients (Concurrently, per Hour per Day, Peak Traffic)

Any Caching that might Skew Results

16

[email protected]


HardwareOne Service per Server is Ideal to Prevent Contention More CPU Cores is generally Good More Disk is usually Better

Large Datasets, Fast Disks are Ideal

RAM is usually Best — Traffic Dependent

More of Dataset in Memory, Fewer Slow Disk Operations

17


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact


[email protected]


Tools & StatisticsIdentify Accurately and Carefully Trouble Spots

Guessing is Rarely Useful

Gather Performance Stats with MariaDB and OS Tools SHOW Statements PERFORMANCE_SCHEMA CPU, Disk, Network, Memory, & Swap Stats

Retain Snapshots of Multiple Stats Data from a Single Point Shows very Little

Automate the Collection of Stats into Logs Can be Useful for Emergency Tuning

19

[email protected]


SHOW PROCESSLISTSnapshot of mysqld Activity

mysqld is Multi-Threaded, One Thread per

Client Connection (i.e., query, transaction) — a

"process" is a "thread"

Accumulate SHOW PROCESSLIST Snapshots to

build History of Thread Activities

20

Time: Length of Time Thread has been in Current State State: Indicates Thread Activity

SHOW PROCESSLIST \G

****** 1. row ******** Id: 9546 User: monyog Host: localhost:40600 db: NULL

Command: Sleep Time: 192 State: Info: NULL Progress: 0.000

[email protected]


Thread States

21

After Create

Analyzing

Checking Permissions

Checking Table

Cleaning Up

Closing Tables

Converting HEAP to MyISAM

Copy to tmp Table

Copying to Group Table

Copying to tmp Table

Copying to tmp table on Disk

Creating Index

Creating Sort Index

Creating Table

Creating tmp Table

Deleting from Main Table

Deleting from Reference Tables

discard_or_import_tablespace

End

Executing

Execution of init_command

Flushing Tables

Freeing Items

FULLTEXT initialization

Init

Killed

Locked

Logging Slow Query

Login

Manage Keys

NULL

Opening Tables, Opening Table

Optimizing

Preparing

Purging Old Relay Logs

Query End

[email protected]


Thread States

22

Reading from Net

Removing Duplicates

Removing tmp Table

Rename

Rename Result Table

Reopen Tables

Repair by Sorting

Repair Done

Repair with Keycache

Rolling Back

Saving State

Searching Rows for Update

Setup

Sorting for Group

Sorting for Order

Sorting Index

Sorting Result

Statistics

System Lock

Table Lock

Updating

Updating Main Table

Updating Reference Tables

User Lock

User Sleep

Waiting for (tables|tables|table flush)

Waiting for All Running Commits to Finish

Waiting for Commit Lock

Waiting for Global Read Lock

Waiting for lock_type Lock

Waiting for Release of Read Lock

Waiting on Cond

Waiting to Get Read Lock

Writing to Net

[email protected]


SHOW STATUS - Global or Session

Returns List of Internal Counters GLOBAL for System-Wide Status — Since Start-Up

SESSION for Local to Client Connection

FLUSH STATUS Resets Local Counters Monitor Changes to Counters to Identify Hot Spots Collect Periodically Status Snapshots to Profile Traffic

23

[email protected]


Status Counters

24

Aborted_clients

Aborted_connects

Binlog_cache_*

Bytes_received

Bytes_sent

Com_*

Compression

Connections

Created_tmp_disk_tables

Created_tmp_files

Created_tmp_tables

Delayed_errors

Delayed_insert_threads

Delayed_writes

Flush_commands

Handler_*

Innodb_*

Key_*

Last_query_cost

Max_used_connections

Not_flushed_delayed_rows

Open_*

Opened_files

Opened_table_definitions

Opened_tables

Prepared_stmt_count

Qcache_*

Queries

Questions

Rpl_status

Select_*

Slave_open_temp_tables

Slave_retried_transactions

Slave_running

Slow_launch_threads

Slow_queries

Sort_*

Ssl_*

Table_locks_immediate

Table_locks_waited

Tc_log_max_pages_used

Tc_log_page_size

Tc_log_page_waits

Threads_cached

Threads_connected

Threads_created

Threads_running

Uptime*

[email protected]


PERFORMANCE_SCHEMASimilar to INFORMATION_SCHEMA, but Performance Tuning Monitors MariaDB Server Events

Function Calls, Operating System Waits, Internal Mutexes, I/O Calls

Detailed Query Execution Stages (Parsing, Statistics, Sorting)

Some Features Storage Engine Specific

Monitoring Lightweight and Requires No Dedicated Thread Not a Simple, Quick Fix

Interpreting Output Requires knowledge of Internals Designed to be Used Iteratively with Successive Refinement

25

[email protected]


PERFORMANCE_SCHEMA Tables

26

SET PROFILING=ON; SHOW PROFILES; SHOW PROFILE FOR QUERY 1;

cond_instances

events_waits_current

events_waits_history

events_waits_history_long

events_waits_summary_by_instance

events_waits_summary_by_thread_by_event_name

events_waits_summary_global_by_event_name

file_instances

file_summary_by_event_name

file_summary_by_instance

mutex_instances

performance_timers

rwlock_instances

setup_consumers

setup_instruments

setup_timers

threads

[email protected]


External ToolsApacheBench htop httperf innotop iostat / vmstat / top maatkit MyBench MariaDB Benchmark Suite (sql-bench)

mysqladmin (extended-status) mysqlslap mytop statpack Super Smack SysBench

27


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Schema Tuning

[email protected]


Naming ConventionConsistent, Descriptive Names Reduce Mistakes in

Development and Maintenance Plural Form for Table Names (products)

Singular Form for Column Names (name_last, email)

Alphabetical Order for Link Tables (posts_users)

Long Names are Fine — Use Aliases in Queries

29

[email protected]


Choosing Data TypesUse Appropriate Data Type (INT for Numbers, VARCHAR)

Use Accurate Types where required (DECIMAL vs. DOUBLE)

Use Smallest Useful Type On-Disk Size does Not Equal In-Memory Size Variable Length Fields are often Padded Use NOT NULL, where Practical

A NULL field uses slightly More Disk and Memory (Depends on Storage Engine)

Use PROCEDURE ANALYSE( )

30

[email protected]


Data Types for Integers

31

Documentation on Integer Data Types: https://mariadb.com/kb/en/mariadb/data-types/

Use UNSIGNED when Appropriate

INT(n) Specifies Display Precision, Not Storage Precision

Size and Precision is Storage Engine Dependent Define Handling of Out-of-Range Values with sql_mode

Default Mode: Values are Truncated Silently Strict Mode: Errors are Generated

TINYINT

SMALLINT

MEDIUMINT

INTEGER,INT

BIGINT

https://mariadb.com/kb/en/mariadb/data-types/

[email protected]


String Data Types

32

Documentation on String Data Types: https://mariadb.com/kb/en/mariadb/string-data-types/

All String Data Types have a Character Set CHAR(n) — number of characters, not bytes, wide VARCHAR(n)— Changes to CHAR in Implicit Temporary Tables and mysqld internal buffers

TEXT —Not Supported by the MEMORY Storage Engine; Implicit Temporary Tables may Convert to MyISAM

CHAR

VARCHAR

TINYTEXT

TEXT

MEDIUMTEXT

LONGTEXT

https://mariadb.com/kb/en/mariadb/string-data-types/

[email protected]


Character Set & Collation

33

Documentation on Character Sets and Collation: https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

SELECT * FROM table1 ORDER BY col1 COLLATE latin1_german2_ci;

Character Set may be Global or for Schema, Table or Column

Multi-Byte Character Sets Increase Disk Storage and Working Memory Requirements (e.g, UTF-8 Requires 3 or 4 bytes per Character)

Collations affect String Comparison (Character Order) Collations can be Changed for Query:

https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

[email protected]


Date & Time Data Types

34

DATE — from 1000-01-01 to 9999-12-31 (YYYY-MM-DD)

TIME — from -838:59:59 to 838:59:59

DATETIME — Same Range with Time (YYYY-MM-DD HH:mm:ss) TIMESTAMP — Unix timestamp, in seconds from

1970-01-01 Many Apps Store UNIX_TIMESTAMP() values in unsigned integer

field

YEAR — Accepts YY or YYYY

DATE

TIME

DATETIME

TIMESTAMP

YEAR

Documentation on Date and Time Data Types: https://mariadb.com/kb/en/mariadb/date-and-time-data-types/Documentation on Microseconds in MariaDB: https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

SELECT CURTIME(4); +---------------+ | CURTIME(4) |

+---------------+ | 05:33:09.1061 | +---------------+

https://mariadb.com/kb/en/mariadb/date-and-time-data-types/

https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

[email protected]


Listing Acceptable Values

35

Documentation on ENUM: https://mariadb.com/kb/en/mariadb/enum/ Documentation on SET: https://mariadb.com/kb/en/mariadb/set-data-type/

CREATE TABLE colors (primary_colors ENUM('red', 'yellow', 'blue'));

ENUM is an Enumerated List of String Values — uses a 2-byte integer index

CREATE TABLE colors

(primary_colors SET('red','yellow','blue'));

INSERT INTO colors VALUES('red'), ('red,blue');

Simple Example for an ENUM Column

Simple Example for a SET Column

SET is a Specified List of String Values — Can Hold Multiple Specified Values

https://mariadb.com/kb/en/mariadb/enum/

https://mariadb.com/kb/en/mariadb/set-data-type/

[email protected]


Keys and IndexesMake PRIMARY KEY as Small as Practical InnoDB Stores Copy of PRIMARY KEY with each Index Entry

Secondary Indexes Grow with PRIMARY KEY

Use Surrogate PRIMARY KEY, Not Natural Ones AUTO_INCREMENT is like a Sequence AUTO_INCREMENT UNSIGNED NOT NULL Careful of AUTO_INCREMENT Contention for Pre-Plugin InnoDB

Consider Partial (Prefix) Index for String Indexes With Good Index Selectivity, Won't Affect Performance

Much Without Good Selectivity is a Different Problem

36

[email protected]


File StorageStore Files in BLOB Useful

Included in Back-Up and Replication Guaranteed to be Consistent with Other Data

Store Files in BLOB has Problems Database gets Too Large MariaDB Can't do Partial Field Read — MyISAM Reads Entire File InnoDB can Skip if Field Not Referenced in Query

Can Minimize by putting Files in Separate Table (1:1 relationship)

Store Files in File System and File Name and Path in Table Database is Smaller and More Efficient Organize Separate Back-Up, possibly with External Locking

for Consistency

37

[email protected]


Tuning Tables OverallMinimize Table Size on Disk and in Memory Archive Table Data if Possible and Appropriate Remove Duplicate or Unused Indexes

No need to Index a Column if it’s First Field in Another Index

Use Appropriate Data Types — Smaller is Better

Consider Sharding Large Tables across Multiple Servers

38

[email protected]


Normalization of TablesProcess of Optimizing and Factoring a Schema

1:1, 1:n and n:n 1NF / 2NF / 3NF

Makes a Data Set as Smaller by Eliminating Redundant or Duplicated Data

Problem if Data is Smaller than Key

Ensures Data Set Consistency and Integrity Often Improves Concurrency by Reducing Locking

Overhead - Not Always Increases Number of Tables and Associated

Maintenance

39

[email protected]


Denormalization of TablesComplete Normalization can Slow Queries More Complex JOIN Queries are Drains and Harder to Maintain

Adding Redundant Data Back to Simplify Problems with JOIN Queries

Combine 1:n Relationships Add Pre-Computed Columns Use Fake Materialized Views

Denormalization Adds Redundant Data to Schema Write Queries become More Complex or Numerous as Multiple

Locations must be Maintained

It's Easier to Normalize First, Then Denormalize when Appropriate

40

[email protected]


ViewsMERGE

Query and Original View SELECT are Merged Often the Fastest Option, as Base Table Indexes Used

TEMPTABLE View Data Materialized to Temporary Table, then Query

Executed Can be Slower, as Temporary Table Doesn't have Indexes

UNDEFINED Algorithm is Chosen on Fly Usually Defaults to TEMPTABLE

41

CREATE VIEW ALGORITHM = ...

AS SELECT ...


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Query Tuning

[email protected]


Query Tuning OverviewTry Not to Query Tune on Production Server Use Test Server with same Hardware and OS Use Copy of Production Data Set Query Frequency is as Important as Query Speed

Moderately Slow Queries are often a Bigger Problem than a Rarely Run Very Slow Query

43

[email protected]


IndexingIndexes Improve Read Performance

Without Index, MariaDB Must Read Every Row — a Full Table Scan

With Index, MariaDB can jump to Requested Rows Reduced I/O and Improving Performance Index Increase cost of Writes

Index Improve Writes since they Require Reads Find Balance

Index for Speed, but Avoid Indexing Excessively or Arbitrarily Remove Dead or Redundant Indexes

44

[email protected]


Index SizeKeep Indexes as Small as Practical

Faster since More Likely to Fit in Memory Rebuilds Faster after Writes PRIMARY KEY should be Minimum Useful Size

Use Partial Prefix Indexes for String Columns

May Slow Searches Slightly, but Reduce Index Size

Use Index Cardinality (Uniqueness Measure) Only If Necessary — Re-evaluate as Data Grows

Low Cardinality Indicates many Duplicates High Cardinality is More Useful

45

[email protected]


B-Tree IndexesBalanced Binary Search Trees holding Pointers to

Rows Precision depends on Storage Engine MyISAM stores within .MYD File Pointers to Rows (Offsets) InnoDB Primary Keys store Row — Other Indexes Store Copy

of Key

Good for many Operations Equality (= <>) Range Access (> < >= <= BETWEEN)

Partial String (i.e., LIKE) — String Constants without Leading % Sorting (ORDER BY) — Indexes are Pre-Sorted

46

[email protected]


Composite IndexesSpans Multiple Columns Increases Search Speed Best when All Columns in Index

Specified in Query Not Used with Latter Columns of Index

48

CREATE TABLE people (id INT AUTO_INCREMENT KEY,

name_first CHAR(20), name_last CHAR(20), city CHAR(20), KEY names(name_first,name_last,city));

SELECT * FROM people WHERE name_first = 'Bob' AND name_last = 'Smith';

SELECT * FROM people WHERE name_last = 'Smith';

Index Used

Index Not Used

[email protected]


Joining TablesJOIN is a Cartesian Product of Table Subsets with

Variations The Optimizer can Help

Orders Tables for Minimum Cost — Fewest Rows Examined Converts OUTER to INNER when Possible

Ensure Columns in ON and USING Clauses are Indexed

Restrict GROUP BY and ORDER BY to Columns in same Table

Consider JOIN Decomposition

49

[email protected]


Query AnalysisUse the Slow Log to find Problem Queries (--

log-slow-queries)

Use mysqldumpslow Utility for Manageable Reports

Use EXPLAIN to see how MariaDB Executes a Troublesome Query and if Indexes are Used

Use EXPLAIN EXTENDED and SHOW WARNINGS to see how MariaDB Rearranges a Query before Execution

50

Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/

EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 \G id: 1 select_type: SIMPLE

table: employees type: ALL possible_keys: NULL key: NULL

key_len: NULL ref: NULL rows: 299587 Extra: Using where

EXPLAIN EXTENDED SELECT …; SHOW WARNINGS;

https://mariadb.com/kb/en/mariadb/explain/

https://mariadb.com/kb/en/mariadb/explain/#explain-extended

https://mariadb.com/kb/en/mariadb/mysqldumpslow/

[email protected]


Fields from EXPLAINField Descriptionid Query identifierselect_type Type of SELECT performed

table The table in use (each table in a JOIN will use one row)

*type The table access strategy

possible_keys Any available indexes that could resolve the query

key Index MariaDB chose from possible_keyskey_len The portion of the chosen index to be used (width in bytes)ref Columns from the index, or a constant, usedrows Number of rows the query is expected to touchExtra Extra information relating to the access strategy

51

Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended

* See Next Slide for List of Types

https://mariadb.com/kb/en/mariadb/explain/

https://mariadb.com/kb/en/mariadb/explain/#explain-extended

[email protected]


Scan or Index Method Types

Field Description

const A constant value can be read once and cached

eq_ref One index access per outer query row

ref Multiple index accesses per outer query row

index_merge Multiple indexes used, merging into a single result set

range Multiple index accesses to return all rows within a range

index Full index scan (every index entry is read sequentially)

all Full table scan (every record is read sequentially)

Method Used to Access Table

52

[email protected]


EXPLAIN select_typeType of SELECT being Performed

53

Type of SELECT Description

SIMPLE Normal SELECT Statement

PRIMARY First (Outermost) SELECT in JOIN

UNION Subsequent Query in JOIN

DEPENDENT UNION Subsequent Query Dependent on an Outer Query Result

UNION RESULT Query is the Results Set from a UNION

SUBQUERY First SELECT in a Sub-Query

DEPENDENT SUBQUERY A Sub-Query that must be Re-Executed for Each Combination of Values in Outer Query

DERIVED Sub-Query in a FROM Clause

UNCACHEABLE SUBQUERY

Sub-Query that must be Re-Executed for Every Iteration of an Outer Query

UNCACHEABLE UNION Subsequent Query in a UNION belonging to an Uncacheable Subquery

[email protected]


EXPLAIN Extraconst row not found

Impossible HAVINGNo matching min/max rowNot existsSelect tables optimized awayOpen_trigger_onlyUsing filesortUsing join buffer Using

intersect(...)

Distinct Impossible WHEREno matching row in const

tableRange checked for each

record (index map: N)Skip_open_table Open_full_tableUsing indexUsing index conditionUsing sort_union(...)Using temporary

Using WHERE with pushed condition

Full scan on NULL keyImpossible WHERE noticed after

reading const tablesNo tables used Scanned N

databasesOpen_frm_only

unique row not found Using index for group-by Using UNION(...) Using where

54

[email protected]


MySQL OptimizerOptimizer calculates Best Plan

Can't Take too Long or Defeats Purpose

Many Possible Plans for Large Joins Greedy Searches used to Reduce

Possibilities Possible to Adjust Decision Parameters

Limit Number of Tables Considered (optimizer_search_depth)

Skip Plans Based on Row Count Estimates (optimizer_prune_level)

55

Documentation on optimizer_search_depth: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_search_depth Documentation on optimizer_prune_level: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_prune_level Article on Setting Optimizer Search Depth: https://mariadb.com/resources/blog/setting-optimizer-search-depth-mysql

https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_search_depth

https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_prune_level

https://mariadb.com/resources/blog/setting-optimizer-search-depth-mysql

[email protected]


Optimizer HintsGiving Hints can Affect the Optimizer’s

Choices Index Choices USE INDEX IGNORE INDEX FORCE INDEX

Buffering Methods for GROUP BY SQL_BUFFER_RESULT SQL_SMALL_RESULT SQL_BIG_RESULT

56

SELECT * FROM people FORCE INDEX names WHERE name_first = 'Bob';

Documentation on Forcing Indexes: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/ Documentation on Indexing with GROUP BY: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/#forcing-an-index-to-be-used-for-order-by-or-group-by

https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/

https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/#forcing-an-index-to-be-used-for-order-by-or-group-by

[email protected]


Slow Query LogContains Queries that took Longer

than long_query_time Seconds to Execute — Off by Default

Activate when First Using an Application to Monitor Slow Queries

Default Location is in Data Directory

Use mysqldumpslow to Process Log

Use also pt-query-digest as an Alternative to Process Log

57

Documentation on Slow Query Log: https://mariadb.com/kb/en/mariadb/slow-query-log-overview/ Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/ Documentation on pt-query-digest: https://www.percona.com/doc/percona-toolkit/2.1/pt-query-digest.html

[mysqld] log-slow-queries = ON long-query-time = 2

log-queries-not-using-indexes log-slow-admin-statements log-slow-vervosity = query_plan,explain

Excerpt from /etc/my.cnf.d/server.cnf

https://mariadb.com/kb/en/mariadb/slow-query-log-overview/

https://mariadb.com/kb/en/mariadb/mysqldumpslow/

https://www.percona.com/doc/percona-toolkit/2.1/pt-query-digest.html

[email protected]


ORDER BY NULLGROUP BY Triggers an Implicit ORDER BY

(filesort)

ORDER BY NULL Avoids It

58

EXPLAIN SELECT Continent, COUNT(*) FROM country GROUP BY Continent \G

... Extra: Using temporary; Using filesort

EXPLAIN SELECT Continent, COUNT(*) FROM country

GROUP BY Continent ORDER BY NULL\G ... Extra: Using temporary

[email protected]


ORDER BY RAND( )MariaDB Can’t Use an Index for Ordering Entire Results Set must be Generated in a Temporary

Table Combined with GROUP BY, it can kill servers

Used Often be Developers when an Application is New or Small

Doesn't Necessarily Scale Well

59

[email protected]


LIMIT ClausesLIMIT is Evaluated at End of Query Plan

ORDER BY with LIMIT needs to Sort All Rows

LIMIT(o, n) will still Parse o + n Rows LIMIT in Sub-Queries Before Sorting

Scan in Index Order and Apply LIMIT vs. Sort and LIMIT

60

[email protected]


SELECT * StatementSELECT with * Performs Badly

Increases Size of Results Set — Memory and Network Overhead on Client and Server

Increases Disk Activity for Storage Engines that can Read Partial Records (not MyISAM, but InnoDB)

Causes Development and Usage Problems Ambiguity with JOIN tables having same Column Names Problems when Schema has been Changed

61

[email protected]


Pagination — Common Scenarios

Scenario 1 Processes Data Set Twice Depends on Complexity of WHERE Clause Worse if User Controlled Filtering

Allowed

Scenario 2 The SQL_CALC_FOUND_ROWS Option Causes

Entire Results Set to be Generated The ORDER BY and LIMIT Clauses become

a Drain as Data Grows

62

SELECT SQL_CALC_FOUND_ROWS FROM table1

JOIN table2 USING(id) ORDER BY col2 LIMIT 4;

SELECT FOUND_ROWS();

SELECT COUNT(*)

FROM table1 WHERE col1 = 'something' AND col2 = 'something else';

SELECT * FROM table1

JOIN table2 WHERE col1 = 'something' ORDER BY col2 LIMIT 100;

Scenario 1

Scenario 2

[email protected]


Pagination — Limit Size of Data Set

Scenario 1 Archive Old Data to Another

Table Add an Index to Column (e.g.,

col_time)

Scenario 2 The First Column is Assumed to be

PRIMARY KEY Value Can Be a Constant, Re-

Generated Periodically

63

SELECT * FROM table1 WHERE id < 1000;

SELECT * FROM table1 WHERE col_time > NOW()

- INTERVAL 6 MONTH;

Scenario 1

Scenario 2

[email protected]


Pagination — Reduce Data Set in ORDER BY and LIMIT

Scenario 1 First column Assumed to be PRIMARY KEY The ORDER BY and LIMIT Clauses often Use a

Temporary Table Retrieving Only PRIMARY KEY keeps

Temporary Table Small Allows 'Using Index' — Especially with InnoDB

Scenario 2 A Safe, Fixed Size Range Access

64

SELECT id FROM table1 WHERE col1 = 'something'

ORDER BY col2 LIMIT 10;

SELECT col2, col4 FROM table1

JOIN table2 USING(id) WHERE id IN(10, 14, 28);

Scenario 1

Scenario 2

[email protected]


Sub-QueriesAdvantages of Sub-Queries before MariaDB 5.3 for Certain

SQL Constructs SELECT...FROM (( (SELECT filter|order|limit) JOIN SELECT

order|limit ) JOIN column)

...WHERE exists (SELECT... LIMIT1) to remove DISTINCT or GROUP BY

SELECT ( SELECT ) FROM to remove DISTINCT or GROUP BY

Disadvantages of Sub-Queries before MariaDB 5.3 for Certain SQL Constructs IN or NOT IN that can be Rewritten with JOIN or LEFT JOIN

WHERE IS NULL Replay for Each Row of Outer Table, No Materialization

VIEW is a Sub-Query with Additional Security Features

65


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Common Bottlenecks

[email protected]


Lack of CachingCaching can Occur at Many Levels

67

Application NetworkUser Session Data

Pre-Compiled, Static ContentProxy Server

Content Distribution Network memcached

MariaDB OS

Query Cache Storage Engine

File System Cache

[email protected]


Using Caches — Questions to Consider

Do Users Care if Counter on Web Page is Real Time? Can Any Content be Re-Generated Periodically in a

Controlled Fashion and Served from a Cache? Can the Application Scale Cheaply, Simply by Sharing

Caches between Servers? Can the Queries be Rewritten to be Cacheable in the

Query Cache, or their Results put in memcached?

For MyISAM tables, does the Server have Sufficient Memory to Operate a File System Cache?

68

[email protected]


Using CacheQuery Cache has to be Enabled to Use Use memcached if controlling Server - If not, try MEMORY tables

Web Application Use for Pagination (Page Count and id = N)

Use if Data Not Specific to User and Not Required to Update in Real Time (Static, or Mostly Static)

Use if User Specific Data lasts Longer than One Page Request

69

[email protected]


Slow at StartSlow at Start could be from Cold Index & Data

Caches — Try Preloading

InnoDB Clustered Index

InnoDB without PRIMARY Indexes For Speed, Try Preloading PK Ranges

in Parallel MyISAM Key Cache

70

SELECT count(*) FROM tbl WHERE idx_col LIKE '%0%';

LOAD INDEX INTO CACHE;

SELECT count(*) FROM tbl WHERE non_idx_col=0;

[email protected]


Temporary Tables on DiskWhen MEMORY Engine Can't Handle an Implicit

Temporary Table, it Converts to MyISAM on Disk Table contains TEXT or BLOB Columns - Not Supported Small Values for tmp_table_size or max_heap_table_size Record Size greater than 8K (VARCHAR Converting to CHAR)

SHOW STATUS, created_tmp_disk_tables Incrementing SHOW PROCESSLIST, Threads in "Creating tmp table on disk"

Move TEXT and BLOB Columns to another Table Make Large VARCHAR Columns as Small as Possible

Increase tmp_table_size and max_heap_table_size

71

[email protected]


Persistent ConnectionsSome Client Languages and Connection Pool Solutions

use Persistent Connections (PHP mysql_pconnect( ))

Not very Effective — a New Client Connection Simpler

Client Session Buffers can Stay Allocated too Long if the Client Pool of Application is not Efficient, Increasing Memory Usage

Transactions can be Held Open if Application is Built Poorly, Reducing Concurrency

72

[email protected]


Domain Name ServiceMariaDB uses DNS and Reverse DNS for IP and

Hostname with Client Connections A Slow DNS Server can Slow MariaDB Client

Connection Running a Local DNS Server Helps with Lookup Speed

and Security Use --skip-name-resolve to Tell MariaDB Not to do

DNS Lookup — Then Use Only IP Addresses in Permissions Tables

73


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Architecture

[email protected]


MariaDB Terrain

mysql

application (e.g., php)

mysqladmin

mysqld

Archive

InnoDB

MyISAM

cron

DBA

web user

socket file

tcp/ip

socket file

ssh & tcp/ip

Memory

SQL Tier — ClientsStorage Engine

TierServer Tier

Handler Interface

75

[email protected]


Client Connections

76

TCP/IP Connections available on All Platforms

--skip-networking (disables)

Socket Files available on Unix Platforms - fastest choice Named Pipe and Shared Memory available on Windows

--enable-named-pipe, --shared-memory

MariaDB Connections require little resources and easy to open.

Most use External Connection Pools (not needed usually) Set Global Client Connection Limit (max_connections=n)

[email protected]


Client Libraries and the Like

Information on MariaDB Connectors: https://mariadb.com/products/connectors-plugins

Various Libraries for connecting to MariaDB — Most are Wrappers for MariaDB C API, which uses MariaDB Network Protocol

PHP API, Perl DBI, Python, Ruby MariaDB Connector for Java Native C/C++ driver MariaDB ODBC Drive Embedded MariaDB (libmysqld)

77

https://mariadb.com/products/connectors-plugins

[email protected]


MariaDB Query Process

Thread Cache

Archive

InnoDB

Aria

Memory

Storage Engines

Query Cache

SQL Parser

Optimizer

Clients

78

[email protected]


Thread Cache

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Thread is Assigned to Each Connection

Threads may be Reused from the Thread Cache or Created

79

[email protected]


Connection Pool

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Main Thread Listens for Client Connections User Authentication based on Host, User, and

Password Client Buffers for Session Variables and Network

Communications

80

thread_handling = pool-of-threads thread_pool_size = 3

Excerpt from /etc/my.cnf.d/server.cnf

Documentation on Thread Pool: https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/#using-thread-pool-scheduler

https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/#using-thread-pool-scheduler

[email protected]


Query Cache

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/

SHOW VARIABLES LIKE 'query_cache_type';

Possible to Stipulate in a Query

Stores SELECT Query Result Sets

Useful for High Read, Low Write Servers

Cache Purged when Related Data Changed

SELECT SQL_CACHE * FROM ...

SELECT SQL_NO_CACHE * FROM ...

81

No Longer Recommended

https://mariadb.com/kb/en/mariadb/query-cache/

[email protected]


SQL Parser

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

Converts SQL Text to Binary Format Parses SQL into Tokens (i.e., keyword, table, field, value) Applies Grammar Rules to check Validity (Lexical

Scanner, Grammar Rules Module) Construct a Parse Tree to be passed to Optimizer

82

[email protected]


Optimizer

Thread Cache

Query Cache

SQL Parser

Optimizer

Storage Engines

The Optimizer Reads the Parse Tree and Determines an Execution Plan

Locate any Related Indexes Compare Efficiency of Index Access to Table Scan Determine JOIN order of Table

Eliminate Unnecessary Tables and WHERE clauses

Find Indexes that can handle GROUP BY and ORDER BY

83

[email protected]


Information Schema (information_schema)

Generated as Needed No On-Disk Presence Read-Only Tables

Some Accessible with SHOW statement

CHARACTER_SETSCOLLATION_CHARACTER_SET_APPLICABILITYCOLLATIONSCOLUMN_PRIVILEGESCOLUMNSENGINESEVENTSFILESGLOBAL_STATUS

GLOBAL_VARIABLESKEY_COLUMN_USAGEPARTITIONSPLUGINSPROCESSLIST SCHEMATAPROFILINGREFERENTIAL_CONSTRAINTSROUTINESSCHEMA_PRIVILEGES

SESSION_STATUSSESSION_VARIABLESSTATISTICSTABLE_CONSTRAINTSTABLE_PRIVILEGESTABLESTRIGGERSUSER_PRIVILEGESVIEWS

84

[email protected]


Methods of Accessing Information Schema

SHOW STATUS LIKE '%qcache%';

Using SHOW Statement

SELECT * FROM information_schema.global_status

WHERE VARIABLE_NAME LIKE '%qcache%';

Accessing Directly

85

[email protected]


Information Schema — InnoDB Tables

SHOW TABLES LIKE 'INNODB%'; SHOW TABLES LIKE 'XTRADB%';

SELECT r.trx_id AS waiting_trx_id, r.trx_mysql_thread_id AS waiting_thread,

r.trx_query AS waiting_query, b.trx_id AS blocking_trx_id, b.trx_mysql_thread_id AS blocking_thread, b.trx_query AS blocking_query FROM information_schema.innodb_lock_waits w

INNER JOIN information_schema.innodb_trx b ON b.trx_id = w.blocking_trx_id INNER JOIN information_schema.innodb_trx AS r ON r.trx_id = w.requesting_trx_id;

Locked & Locking QueriesINNODB_LOCKS

INNODB_TRX

INNODB_BUFFER_POOL_STATS

INNODB_TABLE_STATS

INNODB_INDEX_STATS

86


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact


[email protected]


Global Variables

GLOBAL Variables are System Wide Settings

Drawn from Configuration File at Start

Some can be Changed Dynamically by a SUPER user with SET

88

SET GLOBAL tmp_table_size = 32*1024*1024;

SHOW GLOBAL VARIABLES;

[email protected]


Session VariablesSESSION Variables are Client Specific

Settings

Default Values Same as Global Variables, but may be Changed for Client

89

SET SESSION tmp_table_size = 64*1024*1024;

SHOW GLOBAL VARIABLES;

[email protected]


Memory FootprintCheck if Server is Dedicated to MariaDB Check How much Memory does OS and Other Services

Need MariaDB Allocates Global and per Session Buffers Global Buffers are Once-Only Allocations at Start (MyISAM

Key Cache, InnoDB Buffer Pool, etc.)

Session Buffers are Allocated Multiple Times per Client Connection, Depending on Queries

MariaDB's Footprint needs to have Space, Depending on how many Concurrent Client Connections Expected

90

[email protected]


Global Memory — Buffers Allocated once at mysqld Start

91

io_read io_write

Global Memory Buffers are Allocated once when mysqld Starts table_open_cache table_definition_cache query_cache_size thread_cache

Permissions Tables Global Memory Buffers are also Allocated for Storage Engines

MyISAM: key_buffer_size InnoDB: innodb_buffer_pool_size, innodb_additional_mem_pool_size, innodb_log_buffer_size

Storage Engines may Allocate internally Other Memory

[email protected]


Session Memory

92

Buffers Allocated as required for each Client Connection Some Allocated more than once for Joins and Sorting Released when Query is Finished or Client Session Closed Persistent Connections need to be Reset Client Session Memory Variables:

thread_stack read_buffer_size join_buffer_size bulk_insert_buffer_size

binlog_cache_size net_buffer_length

read_rnd_buffer_size max_heap_table_size max_allowed_packet sort_buffer_size

tmp_table_size myisam_sort_buffer_size

[email protected]


Estimating Memory Usage

93

(sum(session_buffers) * max_connections)

+ sum(global_buffers) = total memory usage

Worst Case, Maximum Memory Usage Scenario:

Chances of All Clients Allocating All Possible Session Buffers at once is usually Small

Normal Practice is to Commit Over Memory Requirements

Out-of-Memory (OOM) may occur if mysqld is near System Memory Limits

Add Memory or Use a Crash-Safe Storage Engine

Use SWAP space, but Avoid letting MySQL Swap for Normal Load

Basic Formula for Estimating Memory Usage

[email protected]


File HandlesLogs and Replication Files need to be Open Client Connections each need a File (socket = file_handle)

Storage Engines may have Specific Needs: MyISAM uses a file per table.frm, and multiple for

table.MYI and table.MYD InnoDB needs only a few for Global Table-Space and

Logs (unless using innodb_file_per_table)

Implicit Disk-Based Temporary Tables are Files SELECT INTO OUTFILE uses a file

94

[email protected]


Temporary Disk SpaceMariaDB Uses Temporary Disk Space

Large Implicit Temporary Tables for Resolving Queries Some Sort Operations (ORDER BY, GROUP BY)

tmpdir defaults to system temporary location Some Unix platforms mount /tmp as tmpfs

slave_load_tmpdir for Replicating LOAD DATA INFILE Defaults to Location of tmpdir — Never use tmpfs Slave Exports from binlog to here, then LOAD DATA

INFILE

95

[email protected]


Disk SpeedUnless a Data Set fits Memory, Disk Speed is Important Disk Reads are Alleviated by Caching Some Storage Engines Implement their own Caches Some Rely on OS Disk Caching Mechanism Disk Writes are Often a Bottleneck Battery Backed Writes are useful for Fast fsync( ) Consider Disk Access Patterns — Multiple Disks to Separate

Random & Sequential Disk IO Random: Most Data Access — Can be Query Dependent Sequential: Transaction and Replication Logs

96

[email protected]


Group CommitInnoDB Flushes the Redo Log for a Transaction before it’s Committed Group Commit makes one Write to Logs for Multiple Transactions, instead of Individual Flushes Group Commit as of MariaDB 5.3

97

Graphs taken from Facebook Article show MySQL version 5.1.52Benchmark Articles on Group Commit with MariaDB: https://www.facebook.com/note.php?note_id=10150211546215933; https://www.facebook.com/note.php?note_id=10150261692455933 Documentation on Group Commit: https://mariadb.com/kb/en/mariadb/group-commit-for-the-binary-log/

https://www.facebook.com/note.php?note_id=10150211546215933

https://www.facebook.com/note.php?note_id=10150261692455933

https://mariadb.com/kb/en/mariadb/group-commit-for-the-binary-log/

[email protected]


Server CPUSince mysqld is multi-threaded, it can

use many CPU cores A Client Connection is a single thread —

a Query uses only One Core

Query Concurrency is very Schema and Traffic Dependent

98


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

MyISAM

[email protected]


Advantages & Disadvantages of MyISAM

Indexes are Cached by mysqld in key_buffer Don't Disable — System Tables use MyISAM Storage

Engine (InnoDB in Future)

100

Not Transactional

Table-Level Locking

Disadvantages

Poor Crash Recovery

(Relies on OS to Flush Data to

Disk)

Data Caching Relies on OS Disk Cache

High Read, Low Write

Traffic

Concurrent Writes

Advantages

Many Column Data Types

Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

[email protected]


Architecture

101

[email protected]


Architecture

102

[email protected]


Key BufferOne Key Buffer by Default (key_buffer_size)

For MyISAM Only, Set Key Buffer to 25 - 50% of Memory (max 4GB)

Don't set key_buffer_size to 0 - Used for Internal Table Operations

Consider Key Buffer Hit and Miss Rates Ideal Values are Application Specific Better to have More Reads than Writes

Run Multiple Key Buffers Allows More than 4GB Dedicating a Key Buffer to a Single Table Prevents Index

Flushing

103

[email protected]


Delayed Key WritesMyISAM Supports Delayed Key Writes Data is Written Immediately to Disk Index Writes are Postponed (delay_key_writes)

ON needs DELAY_KEY_WRITE in Table Definition ALTER TABLE...DELAY_KEY_WRITE Rebuilds Whole Table

After Crash, Index Files must be Rebuilt myisam_recover_options = DEFAULT, BACKUP, FORCE,

QUICK

104

[email protected]


InnoDB & MyISAM Row Formats

105

Fixed-Length Data Types Less Data File Fragmentation Heaviest Disk Space Usage

Static

Fixed and Variable Length Data Types More Data File Fragmentation More Efficient Disk Space Usage

Dynamic

Decompressed as Needed Minimal Disk Space Usage Read-Only

Compressed

INT, CHAR, etc.

VARCHAR, TEXT, etc.

105

[email protected]


Concurrent InsertsMyISAM leaves Gaps in Data File when Deletes Rows

No Automatic Defragmentation

MyISAM tries to fill Gap with Data from INSERT MyISAM has no MVCC Need to Lock the Table to Avoid tripping a SELECT

Concurrent Inserts allows INSERT to add Data Concurrently with SELECT

Not Equivalent of InnoDB's MVCC, but can Improve Concurrency

106

concurrent_inserts = 0 Concurrent Inserts Disabled

concurrent_inserts = 1 Allow Concurrent Inserts when a Table has no gaps (Default)

concurrent_inserts = 2 Allow even When a Table has Gaps

[email protected]


TrapsMyISAM always Reads Whole Row from Disk, Not Partial

Row Large, Infrequently Needed Fields may be Better

Elsewhere in one-to-one Relationship MyISAM used for Implicit Temporary Tables on Disk

Never Reduce MyISAM Resources to Zero (key_buffer_size, etc.)

Ensure tmpdir Location has Plenty of Space - Cartesian Products can be Large

Check Settings of myisam_sort_buffer_size and myisam_max_sort_file_size

MyISAM Repairs use Key Buffer when too Small

107


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

InnoDB

[email protected]


Advantages & Disadvantages of InnoDB

109

Read Uncommitted Read Committed Repeatable Read (no fantom rows) Serializable

Fully Transactional,

ACID Compliant

Data & Indexes Cached by mysqld

in Buffer Pool

Row-Level Locking for High-

Concurrency

Supports Four Isolation Levels

Supports Foreign Keys and Multi-

Version Concurrency

Control

Reliable Crash Recovery

Slower than MyISAM, but

Higher Concurrency

Uses a Global Table-Space File

and two Redo Log Files

Advantages Disadvantages

Isolation Levels

Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

109

https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

[email protected]


InnoDB Architecture

110

Additional Memory Pool

Buffer Pool *

Data Files * **

Redo Log (File 1/2)

Redo Log (File 2/2)

Memory

Disk

* Cached Data & Indexes, Stored in Pages ** Transaction Data Writes *** Data, Index, and Undo Log Files

INSERT, UPDATE, DELETE

1

COMMIT

CHECKPOINT

Log Buffer * *

2

3

110

[email protected]


Buffer PoolFixed Size Pool of Memory Pool by InnoDB

Row Data and Index Pages Adaptive Hash Index Insert Buffer Lock Structures Misc. Other Housekeeping

111

[email protected]


Buffer PoolLarger is Generally Better for innodb_buffer_pool_size

For Servers with Only InnoDB Tables, 80% of Available Memory

MyISAM is Used Internally and Needs Resources along with OS

Dirty Buffer Pool Pages are Flushed in Background Allows Changes to be Merged and Written Sequentially to Disk Applications with Plenty of Writes, or Write Spikes, may need

Adjusting Use innodb_max_dirty_pages_pct to Control Frequency of

Flushing Dirty Pages

InnoDB Plugin can Disable the Adaptive Hash Index Mostly Improves Performance, but Not Always

112

[email protected]


IndexesInnoDB Indexes are Clustered

Row Data is Stored with Primary Key

Use Short Surrogate Primary Keys (INT UNSIGNED AUTO_INCREMENT)

Secondary Index Entries hold a Copy of PK — Large PK Increases All

Table Index Statistics are Vague for InnoDB Gathered by a Series of Random Index Dives It's Possible for Stats to be Inaccurate and Cause

Optimizer to Choose Bad Query Plan Run Periodically ANALYZE TABLE to Regenerate Stats

113

[email protected]


InnoDB Clustered Index

114

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER PRIMARY KEY ROW DATA

leaf

114

[email protected]


InnoDB Secondary Index

115

root

node node

leafleaf leaf

PAGE (16KB) PAGE (16KB) PAGE (16KB)

B-Tree

HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY

leaf

115

[email protected]


Transaction LogConverts Transactions from Random I/O to Sequential

Transaction Data is Written to Transaction Log for Durability and Speed - Not Directly to Data Files

Log Files Rotate as a Circular Buffer with Data Merged to Disk in Background

116

fsync() Frequency

Variable Setting Write and Flush Action

innodb_flush_log_at_trx_commit = 0 Flush on Checkpoint

innodb_flush_log_at_trx_commit = 1 Flush at Commit (Default)

innodb_flush_log_at_trx_commit = 2 Write at Commit, but No Flush

[email protected]


Transaction LogLarge innodb_log_file_size (Default 5MB) Results

Usually in Less I/O and Better Performance Crash Recovery Delay Increases with Log File Size

Increasing innodb_log_buffer_size (Default 1MB) May Reduce Log File I/O for Large Transactions

117

[email protected]


Isolation LevelsInnoDB Supports All Four Transaction Isolation Levels*

READ UNCOMMITTED READ COMMITTED REPEATABLE READ SERIALIZABLE

* Listed in Order of Increasing Performance Potential

Increasing Isolation Protection means More Locks Needed

Reduces Query Concurrency Increases Memory Usage

Set Isolation Level on Per-Session Basis for Performance for Transactions Requiring More Protection

Leave Bulk of Traffic to Operate at Lower Isolation Level

118

[email protected]


The InnoDB CoreVariable innodb_thread_concurrency Sets Number of

Threads that may Access simultaneously InnoDB Core

Set Based on Application and Traffic

Start about Double number of CPU Cores Increase Slowly to Watch for System Spikes or General

Thread Thrashing, Reduced Through-Put

119

[email protected]


The InnoDB Core

Variable innodb_flush_method Controls How Data is Written to Disk

Default is a Normal fsync( ) Operation

Leads to "Double Buffering" of Data (Buffer Pool and OS Disk Cache)

O_Direct Allows InnoDB to Use Direct I/O

Bypasses any OS Cache Changing Flush Method should be Tested Thoroughly Some Combinations of Device, Filesystem, OS may

Not Benefit

120

[email protected]


TablespacesInnoDB Stores Tables in Tablespace Files

Single Global Table Space is Used by Default Use innodb_file_per_table to Use Separate

Tablespaces Easier Maintenance More File Handles Required — Two Additional for Each

Table

121

[email protected]


InnoDB Plugin

New Pluggable version of Original Built-In InnoDB

Better Diagnostics Benchmarking Shows Faster & Higher

Throughput Optional Compressed File Format

Configurable IO Threads on All Platforms

MariaDB 5.5 Uses InnoDB Plugin by Default

122

CREATE TABLE… ROW_FORMAT=COMPRESSED;

[email protected]


InnoDB General TipsDo Bulk Data Loads

Do in PK Order (Sequential Ascending) Batch to Allow Background Threads Keep Pace

Avoid Large ROLLBACK Operations No Equivalent of Log and Insert Buffers Exist to Reduce

Rollback Disk I/O

RAID with Battery Back-Up Unit is Preferable Durability Benefit and some Drivers Allow Disk Sync

Operations Return after Writing Only to BBU Cache

Use when possible Newer InnoDB Plugin or XtraDB Many former Bottlenecks have been Fixed Supports Compression

123


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

MariaDB Replication

[email protected]


Purpose of MariaDB Replication

125

Load Balancing (Scaling SELECT Queries) Move Slow, Heavy Queries to Slave Take Slave Off-Line to Make Back-ups

Multiple Data Centers Need Fast Reads Gain Redundancy (High Availability)

Fail Over - Promote Quickly a Slave to Master Fail Over Isn’t Automatic — Requires External Monitoring

Minimal Downtime for Upgrades or Schema Changes Apply Changes to a Slave Promote Slave to Master and Redirect Traffic Apply Changes to Master and Switch

[email protected]


Replication Terrain

126

mysqld Data Storage

Master

Binary Log

mysqld Data Storage

Slave 1

Relay Log

Data Storage

Slave 2/Master 2

Relay Log

Binary Log

mysqldSlave 2A

Slave 2B

Slave 2C

INSERT UPDATE DELETE

CREATE ALTER DROP

Dump Thread

Client Threads

SQL Thread

IO Thread

IO Thread

IO Thread

[email protected]


MariaDB Replication Factors

127

One Master, Multiple Slaves No true Multi-Master Solution, but Circular Replication

Close to Real Time, but Asynchronous Semi-Synchronous Replication Mode Crash-Safe Slaves with Transactional Storage Engines

A Slave may also be a Master Set log_slave_updates in Configuration File Apply Optionally Replication Filtering Rules or Storage

Engine Changes on Intermediate Slaves

[email protected]


Replication — Threads

128

Master binlog Dump Thread Pushes binlog Events to Slave Visible in SHOW PROCESSLIST as "Binlog Dump"

Slave IO Thread — Visible in SHOW SLAVE STATUS

Requests and receives binlog events from the Master Writes them to the local relay log

Slave SQL Thread — Visible in SHOW SLAVE STATUS

Reads the Relay Log and Executes Queries on Local Data Checks the Query Result Codes Match those Recorded by Master

Slave Multiple Execution Threads Multi-Threaded Slave separates events based on Database Names Updates are Applied in Parallel, Not Sequence

[email protected]


Parallel Replication

129

Documentation on Parallel Replication: https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

Replication Process on Slaves Events Received from Master by IO Thread and Queued

in Relay Log Each Relay Log Entry is Retrieved by the SQL Thread Each Transaction is Applied to the Slave

On Non-Parallel Systems, Application Performed Sequentially by SQL Thread

On Parallel Systems, Application Performed in Pool of Separate Replication Worker Threads

https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

[email protected]


Topologies

130

Master to Slave Simplest Solution and Used Most Widely Allows Off-Loading of SELECT Traffic to Slave

Master1 to MasterN ... to Master1 (circular) Servers Replicate in a Circle, with binlog Events

Traversing the ring until Originating Server Does Not Alleviate Heavy Write Load Needs Careful setup of server-id and

auto_increment_offset, auto_increment_increment settings

Master to Slave to Slaves Can build Complex Trees Useful for Replication Rules or Storage Engine Changes

[email protected]


Master Configuration

131

GRANT REPLICATION SLAVE ON *.* TO 'maria_replicator'@'52.34.19.24' IDENTIFIED BY 'rover123';

Enable Binary Log — Choose Binary Log Format

Set server-id in Configuration File to Unique Value

Create Replication User Account on Master

Make a Consistent Snapshot of Data on Master

mysqldump -p -u admin_backup --master-data --flush-logs \ --all-databases > full-dump.sql

[email protected]


Slave Configuration

132

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/ Documentation on CHANGE MASTER TO: https://mariadb.com/kb/en/mariadb/change-master-to/

CHANGE MASTER TO

MASTER_HOST='35.161.145.71', MASTER_PORT=3306, MASTER_USER='maria_replicator', MASTER_PASSWORD='rover123';

Set server-id in Configuration File to Unique Value

Add read-only in Configuration File to Prevent Writes Set Optionally Replication Rules — Covered Later in Class

Restart MariaDB

Load Data from Master

Execute START SLAVE on Slave

mysql -p -u root < full-dump.sql

https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

https://mariadb.com/kb/en/mariadb/change-master-to/

[email protected]


Monitoring Replication

133

Documentation on SHOW MASTER STATUS: https://mariadb.com/kb/en/show-master-status/ Documentation on SHOW SLAVE STATUS: https://mariadb.com/kb/en/mariadb/show-slave-status/

SHOW MASTER STATUS;Check Regularly Status on Master — Includes binlog number and position

Check More Often Status of Replication on Slave

SHOW SLAVE STATUS \G

Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes

Slave_SQL_Running: Yes Last_Errno: 0 Last_Error: Seconds_Behind_Master: 300

https://mariadb.com/kb/en/show-master-status/

https://mariadb.com/kb/en/mariadb/show-slave-status/

[email protected]


Replication Files

134

Documentation mysqlbinlog: https://mariadb.com/kb/en/mariadb/using-mysqlbinlog/

Binary Log Files (Master)

Master Records Write-Queries to File Rotated when Flushed or Periodically to New Log File — File

Name Pattern (.000001)

Relay Log File (Slave)

Record of Master binlog Events Rotated when Flushed or Periodically — File Name Pattern (.

000001)

Replication Configuration Recorded in master.info (Slave)

Name of Relay Log File Recorded in relay-log.info (Slave)

https://mariadb.com/kb/en/mariadb/using-mysqlbinlog/

[email protected]


Slave Configuration Files

135

52.89.128.176 maria_replicator rover123 3306 60

0 ...

master.info

196803 mariadb-bin.000040

196513 30217 9 ...

relay-log.info

[email protected]


Replication File Maintenance & Back-Ups

136

Replication Files Updated & Purged Automatically — Don’t Edit or Move Manually

Include Replication Files when Making Binary Back-ups

Use --raw option with mysqlbinlog to Back-up Binary Log

[email protected]


Binary Log FormatStatement Based (SBR) — Original Queries are Replicated

Least Data Sent over Wire and Tested for Years Non-Deterministic Statements Executed on Slave — Slave Load is

Increased vs. RBR

Row Based (RBR) - Table Rows are Replicated

Only Non-Deterministic Statements Executed on Master — Slave Load is Reduced vs. SBR

More Data sent over Wire — Not Supported by All Engines

Mixed (default) - Smart Switching between SBR and RBR Checksum (--binlog-checksum) in Binary and Relay Logs to detect Errors

Includes Errors in Memory, Disk, Network and Database Can be Implemented for each Slave

137

Documentation on Binary Log Format: https://mariadb.com/kb/en/mariadb/binary-log-formats/

[mysqld] binlog-format=MIXED

https://mariadb.com/kb/en/mariadb/binary-log-formats/

[email protected]


Slave Filtering Rules — Database Level

138

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

Exclude Specific Databases (e.g., mysql)

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (), REPLICATE_DO_DB = (sales,inventory);

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (mysql);

Include Specific Databases

Excluding can Cause Problems with Joins

SET GLOBAL replicate_ignore_db = ''; SET GLOBAL replicate_do_db = 'sales,inventory';

SET GLOBAL replicate_ignore_db = 'mysql';


[email protected]


Slave Filtering Rules — Table Level

139

Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

Ignore Specific Tables

CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (employees), REPLICATE_DO_TABLE = (employees.names, employees.contacts);

CHANGE REPLICATION FILTER REPLICATE_IGNORE_TABLE = (employees.salary);

CHANGE REPLICATION FILTER

REPLICATE_IGNORE_DB = (sales), REPLICATE_DO_TABLE = (sales.europe_%), REPLICATE_IGNORE_TABLE = (sales.europe_uk_%);

Include Specific Tables

Wildcards for Multiple Tables


[email protected]


MariaDB Replication - Asynchronous

140

Master Doesn’t Wait for Slaves IO Thread may be Slow to Receive binlog Packets

Network Congestion or Disconnects

SQL Thread may be Slow in Processing Relay Log Events

Load on Slave or Network Problems

[email protected]


Semi-Synchronous

141

Documentation on Semi-Synchronous Replication: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

Implemented with an optional Plug-In A COMMIT on Master can Wait for a Slave to

Acknowledge it’s Received Transaction Master Waits for Slave to Write Transaction

to Relay Log, Not to Execute Transaction — Slave SQL Thread may still Lag

One Slave Response needed for Master to Continue (Semi-Synchronous, Not Synchronous)

Can Affect Significantly Performance of Master

INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';

INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';

https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

[email protected]


Lagging Slave

142

Documentation on Time-Delayed Replication: http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html

When Slave SQL Thread is Slow or Disabled due to Errors, Slave is said to Lag behind Master

Slave SQL Thread must Execute Serially Queries that were Executed in Parallel on the Master

Slave Multiple Execution Threads Multi-Threaded Slave separates events based on

Database Names Updates are Applied in Parallel, Not Sequence

Time-Delayed Replication - Set Provides a Buffer to Stop Replication of Mistakes

CHANGE MASTER TO MASTER_DELAY = seconds;

http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.htm

[email protected]


Troubleshooting Problems with Replication

143

Check Slave Error Log for Errors affecting Replication Look for Disconnects from Network Problems Binary or Relay Log Event Corruption will cause Slave

SQL Thread to Stop Different Query Error Codes on Slave indicate it’s Not

Synchronized with Master

Tools like Maatkit can help with Replication Troubleshooting and Recovery

May Need to Rebuild Slave from a fresh Snapshot (Back-up of Master or Another Slave)


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Maintenance Impact

[email protected]


OverviewMaintenance Tasks can Impact Heavily

a System Backups, Table and Index Optimization,

Schema DDL, etc. Some can Alleviate such Impact

145

Type of Operation PerformedHot Operation Warm Operation Cold Operation

No Locks Required Possible High Disk/Network I/O Runs usually Concurrently with Normal Traffic

System must be in Read-Only Mode Impact Depends on Application and Timing Related to Traffic

System must be Stopped Completely Results in Complete Down Time for Application

[email protected]


Back-UpsBinary Back-Up is Usually Faster than Logical Back-Up

Faster Often for Creation and Restoration May be Transparent to Application, Depending on

Method Often a Warm Operation (Read Lock Only) — Consider Storage

Engine

Incremental Back-Ups can be very Low Impact Synchronized with Binary Log Full Back-Up Daily and Periodic binlog Snapshots

Require Only FLUSH LOGS Slower to Restore than Full Back-Up

146

[email protected]


Table & Index OptimizationOPTIMIZE is Warm Operation (Read Lock)

Perform during Low Traffic or Scheduled Maintenance Time

OPTIMIZE LOCAL is Non-Replicated Version Can be Applied to Slaves one-by-one so Overall

Cluster Impact is Minimal and Predictable

ANALYZE is Warm operation (Read Lock)

Impact is Storage Engine Specific MyISAM is Equivalent to Full Table Scan InnoDB is Fast Series of Random Index Dives

147

[email protected]


Schema AlterationsMost DDL Commands are Warm Operations DDL Involves Generally a Table Rebuild

Table is Read Locked DDL is Performed on Copy of Table Two Tables are Swapped Lock is Released

pt-online-schema-change Might Help

DDL Operations can be a Long Process Best Performed in Scheduled Maintenance Time Test Carefully on Staging System Changes can be Made on Slave First, then Promoted to Master

148


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Conclusion

[email protected]


On-Line ResourcesDocumentation Knowledge Base Open-Source Community

Post Questions on Forums Ask Questions on IRC — FreeNode

150

MariaDB Documentation and Knowledge Base: http://mariadb.com/kb

http://mariadb.com/kb

[email protected]


Other Training CoursesPublic In-Person Courses Private In-Person Courses Live Virtual Training Courses Self-Paced On-Line Courses

151

Training Courses: https://mariadb.learnupon.com/store Upcoming Courses: https://mariadb.learnupon.com/store/sessions

https://mariadb.learnupon.com/store

https://mariadb.learnupon.com/store/sessions


Introduction

Tuning Overview


Schema Tuning

Query Tuning



MyISAM

InnoDB

MariaDB Replication

Maintenance Impact

Thanks for Participating

mariadb training - downloads · mariadb performance tuning mariadb training. mariadb performance...

Documents