mysql performance optimization #jdnl13
Post on 28-Jan-2018
213 Views
Preview:
TRANSCRIPT
MYSQL PERFORMANCE OPTIMIZATION(MYSQL PRESTATIE-OPTIMALISATIE)
ALS JE DIT MOEST LEZEN, DAN BEN JE IN DE VERKEERDE KAMER…
JOOMLADAY NETHERLANDS 2013
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 1
INTRODUCTION
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 2
INTRODUCTION
• Eli Aschkenasy
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 2
INTRODUCTION
• Eli Aschkenasy
• themodularway.com
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 2
INTRODUCTION
• Eli Aschkenasy
• themodularway.com
• Oracle certified (doesn’t really mean anything)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 2
INTRODUCTION
• Eli Aschkenasy
• themodularway.com
• Oracle certified (doesn’t really mean anything)
• GE sourcing database project
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 2
INTRODUCTION
• Eli Aschkenasy
• themodularway.com
• Oracle certified (doesn’t really mean anything)
• GE sourcing database project
• Agenda
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 2
AGENDA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
AGENDA
• Introduction (5min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
• Query Optimization (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 3
STRATEGIC OVERVIEW
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 4
STRATEGIC OVERVIEW
• Find smart people – Learn from them
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 4
STRATEGIC OVERVIEW
• Find smart people – Learn from them
• Find less knowledgeable people – Teach them (great design
benefits)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 4
STRATEGIC OVERVIEW
• Find smart people – Learn from them
• Find less knowledgeable people – Teach them (great design benefits)
• Talk to the business people! – Become their marketing specialist
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 4
STRATEGIC OVERVIEW
• Find smart people – Learn from them
• Find less knowledgeable people – Teach them (great design benefits)
• Talk to the business people! – Become their marketing specialist
• Benchmark – Do It !!!!!!!!!!
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 4
STRATEGIC OVERVIEW
• Find smart people – Learn from them
• Find less knowledgeable people – Teach them (great design benefits)
• Talk to the business people! – Become their marketing specialist
• Benchmark – Do It !!!!!!!!!!
• Decide early
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 4
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
• Query Optimization (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 5
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – STRATEGIC
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 6
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – STRATEGIC
• DB Engines (InnoDB vs. MyISAM) – not including MySQL 5.6
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 6
InnoDB
• Transactional
• Hot (Online) Backup
• Crash Safe(er)
MyISAM
• Full-Text Indexing
• Compression
• Low(er) Space Consumption
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 7
InnoDB
• Transactional
• Hot (Online) Backup
• Crash Safe(er)
MyISAM
• Full-Text Indexing
• Compression
• Low(er) Space Consumption
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 7
• Always!
• Transactional
InnoDB
• Transactional
• Hot (Online) Backup
• Crash Safe(er)
MyISAM
• Full-Text Indexing
• Compression
• Low(er) Space Consumption
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 7
• Always!
• Transactional
• Logging
• Read Only
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – STRATEGIC
• DB Engines (InnoDB vs. MyISAM) – not including MySQL 5.6
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 8
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – STRATEGIC
• DB Engines (InnoDB vs. MyISAM) – not including MySQL 5.6
• Data Types (Numbers, Strings, Special)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 8
NUMBERS STRINGS SPECIAL
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 9
NUMBERSINT(1) vs. INT(20) ? (trick question)
UNSIGNED vs. SIGNED
FLOAT isn’t accurate but fast
DECIMAL in MySQL<4.1 calculated
as FLOAT
DECIMAL needs additional byte to
store decimal point
Think if BIGINT could be used even
for precise calculations
(x*1’000’000)
PLEASE, PLEASE, use unsigned
integer as primary index. (we’ll
talk about it later again).
STRINGS SPECIAL
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 9
NUMBERSINT(1) vs. INT(20) ? (trick question)
UNSIGNED vs. SIGNED
FLOAT isn’t accurate but fast
DECIMAL in MySQL<4.1 calculated
as FLOAT
DECIMAL needs additional byte to
store decimal point
Think if BIGINT could be used even
for precise calculations
(x*1’000’000)
PLEASE, PLEASE, use unsigned
integer as primary index. (we’ll
talk about it later again).
STRINGSUse CHAR for fixed length columns
US States is the best example
State code are perfect for
TINYINT UNSIGNED
MD5 Hash values are
another good candidate for
CHAR usage
VARCHAR requires length byte!
VARCHAR allocates space in chunks
so don’t be overly generous
TEXT is problematic in SORTs
Trick: Use SUBSTRING(x, fixed) to
alleviate the problem
SPECIAL
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 9
NUMBERSINT(1) vs. INT(20) ? (trick question)
UNSIGNED vs. SIGNED
FLOAT isn’t accurate but fast
DECIMAL in MySQL<4.1 calculated
as FLOAT
DECIMAL needs additional byte to
store decimal point
Think if BIGINT could be used even
for precise calculations
(x*1’000’000)
PLEASE, PLEASE, use unsigned
integer as primary index. (we’ll
talk about it later again).
STRINGSUse CHAR for fixed length columns
US States is the best example
State code are perfect for
TINYINT UNSIGNED
MD5 Hash values are
another good candidate for
CHAR usage
VARCHAR requires length byte!
VARCHAR allocates space in chunks
so don’t be overly generous
TEXT is problematic in SORTs
Trick: Use SUBSTRING(x, fixed) to
alleviate the problem
SPECIALAvoid NULL if possible
Harder for MySQL to
optimize query
Use TIMESTAMP instead of DATETIME
Limit 2038
EST
TIMESTAMP uses less
space
(I hardly ever use TIMESTAMP…)
BIT (actually string form)
Don’t use ENUM or SET
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 9
CODE EXAMPLES - BIT
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 10
mysql> CREATE TABLE bittest(a bit(8));mysql> INSERT INTO bittest VALUES(b'00111001');mysql> SELECT a, a + 0 FROM bittest;
+------+-------+
| a | a + 0 |
+------+-------+
| 9 | 57 |
+------+-------+
CODE EXAMPLES - ENUM
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 11
mysql> CREATE TABLE enum_test(e ENUM('fish', 'apple', 'dog') NOT NULL);mysql> INSERT INTO enum_test(e) VALUES('fish'), ('dog'), ('apple');
mysql> SELECT e + 0 FROM enum_test; mysql> SELECT e FROM enum_test ORDER BY e;
+-------+
| e + 0 |
+-------+
| 1 |
| 3 |
| 2 |
+-------+
CODE EXAMPLES - ENUM
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 11
mysql> CREATE TABLE enum_test(e ENUM('fish', 'apple', 'dog') NOT NULL);mysql> INSERT INTO enum_test(e) VALUES('fish'), ('dog'), ('apple');
mysql> SELECT e + 0 FROM enum_test; mysql> SELECT e FROM enum_test ORDER BY e;
+-------+
| e + 0 |
+-------+
| 1 |
| 3 |
| 2 |
+-------+
+-------+
| e |
+-------+
| fish |
| apple |
| dog |
+-------+
CODE EXAMPLES – DATETIME NOT NULL
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 12
DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00‘
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
• Query Optimization (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 13
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – NORMALIZATION
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 14
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – NORMALIZATION
• Normalized data is non-redundant
(Text book says this is the best option)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 14
DESIGN CONSIDERATIONSSCHEMA OPTIMIZATION – NORMALIZATION
• Normalized data is non-redundant
(Text book says this is the best option)
• Reality introduces de-normalization
Welcome to Summary and Cache Tables
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 14
CODE EXAMPLES – SUMMARY TABLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 15
mysql> CREATE TABLE msg_per_hr (hr DATETIME NOT NULL,cnt INT UNSIGNED NOT NULL,PRIMARY KEY(hr)
);
mysql> SELECT SUM(cnt) FROM msg_per_hr stored 23 hour
WHERE hr BETWEEN CONCAT(LEFT(NOW(), 14), '00:00') - INTERVAL 23 HOUR concat rounds to nearest hr.
AND CONCAT(LEFT(NOW(), 14), '00:00') - INTERVAL 1 HOUR;mysql> SELECT COUNT(*) FROM message first hour
WHERE posted >= NOW() - INTERVAL 24 HOURAND posted < CONCAT(LEFT(NOW(), 14), '00:00') - INTERVAL 23 HOUR;
mysql> SELECT COUNT(*) FROM message last hourWHERE posted >= CONCAT(LEFT(NOW(), 14), '00:00');
CODE EXAMPLES – CACHE TABLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 16
mysql> DROP TABLE IF EXISTS my_cache_new, my_cache_old;mysql> CREATE TABLE my_cache_new LIKE my_cache;-- populate my_cache_new as desiredmysql> RENAME TABLE my_cache TO my_cache_old, my_cache_new TO my_cache;
CODE EXAMPLES – COUNTER TABLECONCURRENCY ISSUE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 17
mysql> CREATE TABLE hit_counter (cnt INT UNSIGNED NOT NULL
) ENGINE=InnoDB;
mysql> UPDATE hit_counter SET cnt = cnt + 1;
mysql> SELECT cnt FROM hit_counter;
CODE EXAMPLES – COUNTER TABLECONCURRENCY SOLUTION
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 18
mysql> CREATE TABLE hit_counter (slot TINYINT UNSIGNED NOT NULL PRIMARY KEY,cnt INT UNSIGNED NOT NULL
) ENGINE=InnoDB;
mysql> INSERT INTO hit_counter VALUES(0,0),(1,0),(2,0),(3,0),…(99,0);
mysql> UPDATE hit_counter SET cnt = cnt + 1 WHERE slot = RAND() * 100;
mysql> SELECT SUM(cnt) FROM hit_counter;
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
• Query Optimization (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 19
INDEXINGHASH INDEX
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 20
INDEXINGHASH INDEX
• InnoDB creates adaptive hash indexes on frequent queries
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 20
INDEXINGHASH INDEX
• InnoDB creates adaptive hash indexes on frequent queries
• Main downside is that index doesn’t help sorting
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 20
INDEXINGHASH INDEX
• InnoDB creates adaptive hash indexes on frequent queries
• Main downside is that index doesn’t help sorting
• Hash indexes can’t speed up range queries (WHERE age > 18)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 20
INDEXINGHASH INDEX – HOW TO – INCLUDING HASH COLLISION
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 21
mysql> SELECT id FROM url WHERE url="http://www.joomladagen.nl";
CREATE TABLE pseudohash (id INT UNSIGNED NOT NULL AUTO_INCREMENT,url VARCHAR(255) NOT NULL,url_crc INT UNSIGNED NOT NULL DEFAULT 0,PRIMARY KEY(id)
);
CREATE TRIGGER pseudohash_crc_ins BEFORE INSERT ON pseudohash FOR EACH ROW BEGINSET NEW.url_crc=crc32(NEW.url);
CREATE TRIGGER pseudohash_crc_upd BEFORE UPDATE ON pseudohash FOR EACH ROW BEGINSET NEW.url_crc=crc32(NEW.url);
mysql> SELECT id FROM url WHERE url_crc=CRC32("http://www.joomladagen.nl") AND url="http://www.joomladagen.nl";
INDEXINGINDEX PROBLEMS
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 22
SELECT * FROM Orgchart SELECT a, c FROM Orgchart SELECT a, c FROM OrgchartWHERE lft - rgt = 1; WHERE lft - rgt = 1; WHERE lft =(rgt – 1);
SELECT name FROM people mysql> ALTER TABLE people ADD KEY (idx_name(6));WHERE name = ‘Hans’;
CREATE TABLE t ( KEY(c1,c2,c3) ? KEY(c1,c3) Specificity!c1 INT,c2 INT,c3 INT,KEY(c1),KEY(c2),KEY(c3)
);
INDEXINGINDEX PROBLEMS
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 22
SELECT * FROM Orgchart SELECT a, c FROM Orgchart SELECT a, c FROM OrgchartWHERE lft - rgt = 1; WHERE lft - rgt = 1; WHERE lft =(rgt – 1);
SELECT name FROM people mysql> ALTER TABLE people ADD KEY (idx_name(6));WHERE name = ‘Hans’;
CREATE TABLE t ( KEY(c1,c2,c3) ? KEY(c1,c3) Specificity!c1 INT,c2 INT,c3 INT,KEY(c1),KEY(c2),KEY(c3)
);
SELECT cc FROM payment WHERE staff_id = 2 AND customer_id = 584;
INDEXINGINDEX PROBLEMS
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 22
SELECT * FROM Orgchart SELECT a, c FROM Orgchart SELECT a, c FROM OrgchartWHERE lft - rgt = 1; WHERE lft - rgt = 1; WHERE lft =(rgt – 1);
SELECT name FROM people mysql> ALTER TABLE people ADD KEY (idx_name(6));WHERE name = ‘Hans’;
CREATE TABLE t ( KEY(c1,c2,c3) ? KEY(c1,c3) Specificity!c1 INT,c2 INT,c3 INT,KEY(c1),KEY(c2),KEY(c3)
);
SELECT cc FROM payment WHERE staff_id = 2 AND customer_id = 584;KEY(staff_id,customer_id) ?
INDEXINGINDEX PROBLEMS
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 22
SELECT * FROM Orgchart SELECT a, c FROM Orgchart SELECT a, c FROM OrgchartWHERE lft - rgt = 1; WHERE lft - rgt = 1; WHERE lft =(rgt – 1);
SELECT name FROM people mysql> ALTER TABLE people ADD KEY (idx_name(6));WHERE name = ‘Hans’;
CREATE TABLE t ( KEY(c1,c2,c3) ? KEY(c1,c3) Specificity!c1 INT,c2 INT,c3 INT,KEY(c1),KEY(c2),KEY(c3)
);
SELECT cc FROM payment WHERE staff_id = 2 AND customer_id = 584;KEY(staff_id,customer_id) ?SELECT SUM(staff_id = 2), SUM(customer_id = 584) FROM payment\G*************************** 1. row ***************************SUM(staff_id = 2): 7992SUM(customer_id = 584): 30
INDEXINGINDEX PROBLEMS
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 22
SELECT * FROM Orgchart SELECT a, c FROM Orgchart SELECT a, c FROM OrgchartWHERE lft - rgt = 1; WHERE lft - rgt = 1; WHERE lft =(rgt – 1);
SELECT name FROM people mysql> ALTER TABLE people ADD KEY (idx_name(6));WHERE name = ‘Hans’;
CREATE TABLE t ( KEY(c1,c2,c3) ? KEY(c1,c3) Specificity!c1 INT,c2 INT,c3 INT,KEY(c1),KEY(c2),KEY(c3)
);
SELECT cc FROM payment WHERE staff_id = 2 AND customer_id = 584;KEY(staff_id,customer_id) ?SELECT SUM(staff_id = 2), SUM(customer_id = 584) FROM payment\G*************************** 1. row ***************************SUM(staff_id = 2): 7992SUM(customer_id = 584): 30ALTER TABLE payment ADD KEY(customer_id, staff_id);
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
WHERE age BETWEEN 18 AND 25ORDER BY rating ASC
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
WHERE age BETWEEN 18 AND 25ORDER BY rating ASCMySQL can’t use added index if primary index uses range criterion
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
WHERE age BETWEEN 18 AND 25ORDER BY rating ASCMySQL can’t use added index if primary index uses range criterion
KEY(sex, country)
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
WHERE age BETWEEN 18 AND 25ORDER BY rating ASCMySQL can’t use added index if primary index uses range criterion
KEY(sex, country)NO Selectivity!!!
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
WHERE age BETWEEN 18 AND 25ORDER BY rating ASCMySQL can’t use added index if primary index uses range criterion
KEY(sex, country)NO Selectivity!!!
KEY(sex, country)Assumption: all searches will include sex and most will include country
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 23
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
WHERE age BETWEEN 18 AND 25ORDER BY rating ASCMySQL can’t use added index if primary index uses range criterion
KEY(sex, country)NO Selectivity!!!
KEY(sex, country)Assumption: all searches will include sex and most will include country
Trick: AND sex IN('m', 'f')
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 24
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 24
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
(sex, country, age)(sex, country, region, age)(sex, country, region, city, age)
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 24
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
(sex, country, age)(sex, country, region, age)(sex, country, region, city, age)
Using the IN() trick, we can implement just the(sex, country, region, city, age) index.
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 24
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
(sex, country, age)(sex, country, region, age)(sex, country, region, city, age)
Using the IN() trick, we can implement just the(sex, country, region, city, age) index.
Why is age at end?
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 24
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
(sex, country, age)(sex, country, region, age)(sex, country, region, city, age)
Using the IN() trick, we can implement just the(sex, country, region, city, age) index.
Why is age at end?Remember our range problem?MySQL uses indexes from left to right unit the first range query
Trick: Convert WHERE age BETWEEN 18 and 25 toWHERE age IN(18,19,20,21,22,23,24,25)
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 25
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
SELECT age, country, …. , name FROM profilesWHERE sex=‘F‘ORDER BY ratingLIMIT 100000, 10;
Retrieving 100’010 rows, discarding 100’000If data load per row is 15kb calculated data is 18.32Mb and we’re discarding 18.31Mb!
SELECT age, country, …. , name FROM profilesINNER JOIN (
SELECT id FROM profilesWHERE x.sex='M‘ORDER BY rating LIMIT 100000, 10
) AS xUSING(id);
INDEXINGEXAMPLE
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 25
CREATE TABLE profile(id INT UNSIGNED NOT NULL AUTO_INCREMENT,sex CHAR(1) NOT NULL,age TINYINT NOT NULL,country VARCHAR(255) NOT NULL,region VARCHAR(255) NOT NULL DEFAULT ‘’,city VARCHAR(255) NOT NULL DEFAULT ‘’,color_hair VARCHAR(255) NOT NULL DEFAULT ‘’,color_eyes VARCHAR(255) NOT NULL DEFAULT ‘’,name
……
rating TINYINT NOT NULL DEFAULT 1,PRIMARY KEY(id)
);
KEY(sex, rating)SELECT age, country, …. , name FROM profiles
WHERE sex=‘F‘ORDER BY ratingLIMIT 100000, 10;
Retrieving 100’010 rows, discarding 100’000If data load per row is 15kb calculated data is 18.32Mb and we’re discarding 18.31Mb!
SELECT age, country, …. , name FROM profilesINNER JOIN (
SELECT id FROM profilesWHERE x.sex='M‘ORDER BY rating LIMIT 100000, 10
) AS xUSING(id);
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
• Query Optimization (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 26
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
All I want is the ingredients for a StroopwafelThis Query returns all columns for all three tables!
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
All I want is the ingredients for a StroopwafelThis Query returns all columns for all three tables!
SELECT recipes.ingredient.* FROM recipes.ingredient...;
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
All I want is the ingredients for a StroopwafelThis Query returns all columns for all three tables!
SELECT recipes.ingredient.* FROM recipes.ingredient...;
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
All I want is the ingredients for a StroopwafelThis Query returns all columns for all three tables!
SELECT recipes.ingredient.* FROM recipes.ingredient...;
SELECT img, name, …, comment FROM commentsWHERE user_id = 123983;
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
All I want is the ingredients for a StroopwafelThis Query returns all columns for all three tables!
SELECT recipes.ingredient.* FROM recipes.ingredient...;
SELECT img, name, …, comment FROM commentsWHERE user_id = 123983;
What, did the image and name change in the last 2microseconds?!?
QUERY OPTIMIZATIONTOO MUCH DATA
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 27
SELECT * FROM OrgchartWHERE lft - rgt = 1;
SELECT * FROM recipes.ingredientINNER JOIN recipes.recipe_ingredient USING(ingredient_id)INNER JOIN recipes.recipe USING(recipe_id)
WHERE recipes.recipe.name = ‘Stroopwafel';
All I want is the ingredients for a StroopwafelThis Query returns all columns for all three tables!
SELECT recipes.ingredient.* FROM recipes.ingredient...;
SELECT img, name, …, comment FROM commentsWHERE user_id = 123983;
What, did the image and name change in the last 2microseconds?!?
SELECT img, name, …, FROM commentsWHERE user_id = 123983;SELECT comment FROM commentsWHERE user_id = 123983;
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 28
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 28
DELETE FROM messagesWHERE created < DATE_SUB(NOW(),INTERVAL 3 MONTH);
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 28
DELETE FROM messagesWHERE created < DATE_SUB(NOW(),INTERVAL 3 MONTH);
<?php…$rows_affected = 0;do {
$rows_affected = do_query("DELETE FROM messagesWHERE created < DATE_SUB(NOW(),INTERVAL 3 MONTH)LIMIT 10000");
} while $rows_affected > 0;
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
SELECT * FROM tag WHERE tag='joomladagen';SELECT * FROM tag_post WHERE tag_id=1234;SELECT * FROM post WHERE post.id IN(123,456,567,9098,8904);
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
SELECT * FROM tag WHERE tag='joomladagen';SELECT * FROM tag_post WHERE tag_id=1234;SELECT * FROM post WHERE post.id IN(123,456,567,9098,8904);
WHY? DENORMALIZED ?!?
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
SELECT * FROM tag WHERE tag='joomladagen';SELECT * FROM tag_post WHERE tag_id=1234;SELECT * FROM post WHERE post.id IN(123,456,567,9098,8904);
WHY? DENORMALIZED ?!?
• Query cache validation of 3 tables instead of one
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
SELECT * FROM tag WHERE tag='joomladagen';SELECT * FROM tag_post WHERE tag_id=1234;SELECT * FROM post WHERE post.id IN(123,456,567,9098,8904);
WHY? DENORMALIZED ?!?
• Query cache validation of 3 tables instead of one• Lock contention (?)
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
SELECT * FROM tag WHERE tag='joomladagen';SELECT * FROM tag_post WHERE tag_id=1234;SELECT * FROM post WHERE post.id IN(123,456,567,9098,8904);
WHY? DENORMALIZED ?!?
• Query cache validation of 3 tables instead of one• Lock contention (?)• Query index lookup of IN() is better than with JOINS
QUERY OPTIMIZATIONDIVIDE AND CONQUER
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 29
SELECT * FROM tagJOIN tag_post ON tag_post.tag_id=tag.idJOIN post ON tag_post.post_id=post.id
WHERE tag.tag=‘joomladagen';
SELECT * FROM tag WHERE tag='joomladagen';SELECT * FROM tag_post WHERE tag_id=1234;SELECT * FROM post WHERE post.id IN(123,456,567,9098,8904);
WHY? DENORMALIZED ?!?
• Query cache validation of 3 tables instead of one• Lock contention (?)• Query index lookup of IN() is better than with JOINS• Potential for application caching
AGENDA
• Introduction (5min)
• Strategic Overview (5min)
• Design Considerations – Schema Optimization – Strategic (10min)
• Design Considerations – Schema Optimization – Normalization
(10min hopefully)
• Indexing (5min)
• Query Optimization (10min)
JOOMLADAY NETHERLANDS 2013 - ELI ASCHKENASY - @ELIASCHKENASY 30
top related