storage methods for nonstandard data patterns
TRANSCRIPT
-
Storage Methodsfor
Nonstandard Data PatternsB O B B U R G E S S
D ATA B A S E O P E R AT I O N S E N G I N E E R
-
2
AgendaIntro
Choices Available To You
Choices I Tested
Testing Methods
What I Saw When Loading Data
What I Saw When Using Data
Storage Engine Comparison
Conclusions
-
3
W H O I S T H I S G U Y ?
-
4
Bob Burgess
Developer, then DBA
1997, Oracle & Sybase
2007, MySQL at Radian6 => Salesforce
2015, MySQL at Shopify
-
2006 Platform to let merchants easily sell online and in person 243,000+ Active stores / $14B in sales / 150 Countries Large MySQL Installation Hiring!
5
-
6
Dont Panic
All slides will be posted to the conference site & Slideshare
Test scripts and test data will be posted to Github bobburgess
-
7
Why Am I Doing This?
I cant be the only one
Surprising results
-
8
C H O I C E S
-
9
Normal Tables
A few numbers and average-size varchar columns
Couple of indexes
Artificial or random primary key
-
10
Unusual Tables
Very tiny rows
Very large rows
Large random primary key
-
Unusual Tables Very tiny rows
Very large rows
Large random primary key
11
Multiple TB
Two bigints
Generally-increasing primary key
InnoDB
-
Unusual Tables Very tiny rows
Very large rows
Large random primary key
12
De-duplication
SHA1 hash
binary(20)
Completely random primary key
InnoDB
-
Unusual Tables Very tiny rows Very large rows
Large random primary key
13
De-duplication
SHA1 hash
binary(20)
Completely random primary key
InnoDB
Yves Trudeau (Percona)
Auto-increment PK
Index on 1st 5 bytes of binary(20) hash
Trigger to check for Duplicates
-
14
Storage Engines
MySQL
client
disk
-
15
Storage Engines
MySQL
client
disk
MySQL
client
disk
Storage Engine
-
16
Storage Engines
MySQL
client
disk
MySQL
client
disk
Storage Engine
-
17
Storage Engines
MySQL
client
disk
MySQL
client
disk
Storage Engine
-
18
Choices Storage: Solid-state vs. Spinning Rust
Arrays of Storage (RAID levels)
Storage Engine
suitability for data shape
suitability for access patterns
adaptability to slower storage
-
19
MY T E S T SC H O I C E S T E S T E D ( VA R I A B L E S )
-
20
Primary Key
Auto-Increment (Sequential) Integer
Random Integer
Random Large (20 bytes)
Auto-Increment (Sequential) Integer With Trigger
-
21
Data Shape
-
22
Data Shape: Tiny Rows
create table tinyrows ( id bigint primary key, payload bigint);
-
23
Data Shape: Tiny Rows
create table tinyrows ( id bigint auto_increment primary key, payload bigint);
-
24
Data Shape: Tiny Rows
create table tinyrows ( id bigint auto_increment primary key, payload bigint, index i (payload));
-
25
Data Shape: Average Rowscreate table averagerows ( id bigint primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50));
-
26
Data Shape: Average Rowscreate table averagerows ( id bigint auto_increment primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50));
-
27
Data Shape: Average Rowscreate table averagerows ( id bigint auto_increment primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50), index i1 (num0), index i2 (char1), index i3 (char3), index i4 (num2));
-
28
Data Shape: Average Rowscreate table averagerows ( id binary(20) primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50), index i1 (num0), index i2 (char1), index i3 (char3), index i4 (num2));
-
29
Data Shape: Average Rowscreate table averagerows ( newid bigint auto_increment primary key, real_id binary(20), num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50), index id_helper (real_id(6)), index i1 (num0), index i2 (char1), index i3 (char3), index i4 (num2));
-
30
Data Shape: Average Rows
create trigger averagerows_bi before insert on averagerows for each row begin select count(*) into @c from averagerows where real_id = new.real_id; if @c != 0 then set @m=concat('Duplicate entry ''',hex(new.real_id),''' for key ''PRIMARY'''); signal SQLSTATE '23000' set message_text=@m, mysql_errno=1062; end if; end;
-
31
Data Shape: Average Rows
create trigger averagerows_bi before insert on averagerows for each row begin select count(*) into @c from averagerows where real_id = new.real_id; if @c != 0 then set @m=concat('Duplicate entry ''',hex(new.real_id),''' for key ''PRIMARY'''); signal SQLSTATE '23000' set message_text=@m, mysql_errno=1062; end if; end;
Do we have this one already?YES
Raise an error
-
32
Data Shape: Large Rows
create table largerows_ai ( id bigint primary key, payload1 varchar(2000), payload2 varchar(2000), payload3 varchar(2000), payload4 varchar(2000));
-
33
Data Shape: Large Rows
create table largerows_ai ( id bigint auto_increment primary key, payload1 varchar(2000), payload2 varchar(2000), payload3 varchar(2000), payload4 varchar(2000));
-
34
Data Shape: Large Rows
create table largerows_ai ( id bigint auto_increment primary key, payload1 varchar(2000), payload2 varchar(2000), payload3 varchar(2000), payload4 varchar(2000), index i1 (payload1(40)), index i2 (payload2(767));
-
35
Storage Engines
-
36
Storage Engine: InnoDB
Very Mature
Well understood
Continually updated
Supported by Galera Cluster
-
37
Storage Engine: TokuDB Was separately-licensed; bought by Percona last year
Built into Percona Server now
Extreme data compression
Online table alters
Other benefits
Not supported by Galera Cluster
-
38
Storage Engine: Deep Separately-licensed product
Distributed as a complete package, based on 5.6
Does data compression
Self-tuning using constant data analysis
Other benefits
Not supported by Galera Cluster
-
39
Filesystem
ext4, xfs, others
Journaling uses IO and can be disabled
-
40
T E S T I N G M E T H O D S
-
41
HardwareLoad Generator Database
1 Gbe
Dell 2950-III 32 GB RAM Two 4-core 3GHz Xeon processors SSD and HDD storage Ubuntu 14.04 Measured SSD IOPS (fio: random RW / 5G file): 9750 read, 3254 write
Desktop 8 GB RAM Intel i7 processor SSD storage Ubuntu 14.04
-
42
SoftwareLoad Generator Database
Ubuntu 14.04 Percona Server 5.6.28 deepSQL 5.6.28-21214
Ubuntu 14.04 go v.1.2.1
-
43
Data Loading in go for speed
Loop until stopped: Fill buffer with 1000 (or 10000) rows Send buffer to LOAD DATA INFILE
Very low CPU and network on load-gen box
Concurrency of 1 and 20
github.com/bobburgess
http://github.com/bobburgess
-
44
Data Usage (Work that DB) in go for speed
Loop until stopped: Select by PK Select by each secondary index
(column starts with random value) Insert a row of random data
Very low CPU and network on load-gen box
Concurrency of 20, 60, and 100
github.com/bobburgess
http://github.com/bobburgess
-
45
DB Server Measurements
in bash cause Im lazy (and speed not required)
Loop until stopped: Take measurements Output measurements Remember old values (for differential measurements) sleep for 1 sec minus the time taken for the measurements
Very lightweight
github.com/bobburgess
-
46
DB Server Measurements
Rows per second inserted & selected
Disk IOs per second, read & write
Bytes per second, read & write
Size of mysqld process
Size of data on disk
CPU %, User-time & System-time
-
47
DB Server Measurements
Rows per second inserted & selected
Disk IOs per second, read & write
Bytes per second, read & write
Size of mysqld process
Size of data on disk
CPU %, User-time & System-time
Measure of throughput
Possible to use slower storage?
Possible to use slower storage?
Wasteful of memory? Bloated?
Possible to use less storage?
Lower = can handle more load
-
48
Testing Flow
Start DB monitor on DB server (output to file)
Start up required processes on load-gen box
Check DB server for unexpected activity
Check load-gen box for unexpected activity
Check load-gen box for CPU or network bottlenecks
Stop test processes when data set large enough
-
49
Plotting the Data
R
Define columns columns
-
50
Plotting the Data Example: Two plots with running-average lines
par(mar=c(5, 9, 4, 6) + 0.1)options(scipen=15)Xmin=1Xmax=14432Ymin=0Ymax=120smooth=100# Firstplot(seq(Xmax), avg_ai_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], col="red", pch=".", xlab="", ylab="", ylim=c(Ymin,Ymax), type="p", axes=FALSE, main="")axis(2,ylim=c(Ymin,Ymax), col="red", las=1)mtext("IOPS - Write", side=2, line=3)box()lines(seq(Xmax), filter(avg_ai_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], rep(1/(smooth+1),(smooth+1)), sides=2), col="red", lwd=2)par(new=TRUE)# Secondplot(seq(Xmax), avg_rnd_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], col="blue", pch=".", xlab="", ylab="", ylim=c(Ymin,Ymax), type="p", axes=FALSE)lines(seq(Xmax), filter(avg_rnd_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], rep(1/(smooth+1),(smooth+1)), sides=2), col="blue", lwd=2)# Finish upaxis(1,at=seq(0,Xmax,60),labels=seq(0,Xmax/60))mtext("Minutes", side=1, col="black", line=2)legend("topright",c("Auto-Inc PK","Random PK"),fill=c("red","blue"))
-
51
R E S U LT S
-
InnoDB
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
56
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
57
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
58
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
59
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
60
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
61
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
62
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
200,000
-
63
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
64
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
65
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
66
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
67
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
68
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
69
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
70
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
71
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
72
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
73
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
82
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
83
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
84
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
85
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
?!
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
93
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
94
-
95
InnoDBBulk-Load
Single-Session 1 Multi-Session 0
Primary Key
Auto-Inc 1 Random 1 Large 0
-
TokuDB
96
-
97
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
98
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
99
-
100
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
101
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
102
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
103
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
40-60
-
104
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
105
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
106
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
107
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
108
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
109
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
110
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Difference with InnoDB is not as great
-
111
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
112
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
113
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
114
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
115
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
116
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
117
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
118
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
119
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
120
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
121
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
122
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
123
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
124
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
125
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
126
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
127
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
128
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
129
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
130
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
131
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
132
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
133
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
134
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
135
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
136
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
137
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
138
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
139
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Same shape as PK-Only, 1 Session
-
146
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
147
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
148
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
deepSQL
153
-
154
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
InnoDB
TokuDB
-
157
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
158
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
161
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
162
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
163
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
164
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
165
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
166
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
167
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
168
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
171
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
172
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
173
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
174
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
175
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
176
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
177
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
178
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
179
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
180
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
181
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
182
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
183
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
184
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
185
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
186
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
InnoDB
TokuDB
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
194
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
195
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
196
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
197
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
198
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
199
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
200
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
201
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
202
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
203
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
204
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
-
205
Usage Performance
-
206
Usage Performance Each row size
Auto-Increment PK
Inserted by multiple threads
disk files of 2X or 3X cache size
-
207
Usage Performance
Loop this block: Several selects One Insert
20, 60, 100, 200 concurrent
Count Com_insert and Com_select on server side
-
208
Usage Performance
Average Rows InnoDB TokuDB Deep
Rows 3,264,053 6,742,470 5,324,183
Avg. Row Length 366 174 180
~2 GB on disk
-
209
Tiny Rows (Select)
20 sessions 60 sessions 100 sessions 200 sessions
-
210
Tiny Rows (Insert)
-
211
Average Rows (Select)
-
212
Average Rows (Insert)
-
213
Large Rows (Select)
-
214
Large Rows (Insert)
-
215
N O M O R E C H A R T S !
-
216
What I Learned
Vast differences depending on use-case
Some patterns not as bad as first thought (i.e. Random PK)
Some patterns worse than first thought (i.e. with bytes/row)
-
217
Take-Aways
Test for your use-case
Dont assume you already know how the DB will respond
Dont trust anyone, especially me. Test it yourself!
-
218
T H A N K Y O U !S L I D E S W I L L B E O N L I N E
B O B . B U R G E S S @ S H O P I F Y. C O M G I T H U B . C O M / B O B B U R G E S S
P L E A S E F I L L O U T A N E VA L U AT I O N !
is hiring!