sharding architectures 777
TRANSCRIPT
-
8/6/2019 Sharding Architectures 777
1/38
Sharding ArchitecturesInternational PHP Conference 2008
-
8/6/2019 Sharding Architectures 777
2/38
Who the f*** is talking?
David Soria Parra
PHP since 7 yearsSoftware Engineer at Sun Microsystems
Sun Microsystems Web Stack
PHP Source Committer Various other OSS Projects (Mercurial, etc)
mailto:[email protected]:[email protected] -
8/6/2019 Sharding Architectures 777
3/38
Agenda
Problem
Possible Solutions ...and what notWhat is sharding?
Implementation
OptimizationMigration
-
8/6/2019 Sharding Architectures 777
4/38
Problem
ScalabilityWhat kind of scalability?
High read and write trafc
Huge amount of data
-
8/6/2019 Sharding Architectures 777
5/38
Solutions
-
8/6/2019 Sharding Architectures 777
6/38
Master/Slave replication
-
8/6/2019 Sharding Architectures 777
7/38
Master/Master replication
-
8/6/2019 Sharding Architectures 777
8/38
Master/Master Master/Slave
Master / Master
unstable / no ofcial supportreplication not distribution
Master / Slave
no write optimizationreplication not distribution
-
8/6/2019 Sharding Architectures 777
9/38
Sharding
-
8/6/2019 Sharding Architectures 777
10/38
What is sharding?
Application sideimplementation
Application side aggregation
Splitting data across databases
Parallelization
-
8/6/2019 Sharding Architectures 777
11/38
Sharding is not...
sharding is...
NOT replication / backup
NOT clustering
NOT just spreading
-
8/6/2019 Sharding Architectures 777
12/38
Implementation
-
8/6/2019 Sharding Architectures 777
13/38
-
8/6/2019 Sharding Architectures 777
14/38
user ID: 9
-
8/6/2019 Sharding Architectures 777
15/38
user ID: 9
9 % 3 = 0
-
8/6/2019 Sharding Architectures 777
16/38
user ID: 9
9 % 3 = 0
-
8/6/2019 Sharding Architectures 777
17/38
user ID: 9
9 % 3 = 0
-
8/6/2019 Sharding Architectures 777
18/38
Shard
Tables with a lot of writes
Tables with a lot of reads
Related data on one shard
-
8/6/2019 Sharding Architectures 777
19/38
Splitting: Tables
-
8/6/2019 Sharding Architectures 777
20/38
Splitting: Tables
-
8/6/2019 Sharding Architectures 777
21/38
Splitting: Data
-
8/6/2019 Sharding Architectures 777
22/38
-
8/6/2019 Sharding Architectures 777
23/38
Data distribution
-
8/6/2019 Sharding Architectures 777
24/38
-
8/6/2019 Sharding Architectures 777
25/38
Modulo
-
8/6/2019 Sharding Architectures 777
26/38
Modulo
2
4 6
-
8/6/2019 Sharding Architectures 777
27/38
Lookup table
-
8/6/2019 Sharding Architectures 777
28/38
Optimizing
-
8/6/2019 Sharding Architectures 777
29/38
Optimizing
caches (memcache)
model optimization
normalization? no!
connection pools
persistant connections (using mysqli+mysqlnd)
-
8/6/2019 Sharding Architectures 777
30/38
Optimizing II
aggregation daemons
persistent
MySQL Proxy with LUA
asynchronous queries (mysqlnd)
-
8/6/2019 Sharding Architectures 777
31/38
Optimizing III
-
8/6/2019 Sharding Architectures 777
32/38
Problems
Joins, Unions, Intersections
Grouping
Selecting and projecting on groups
Aggregation
Primary keysReferential integrity (foreign keys)
(De-)Normalization
-
8/6/2019 Sharding Architectures 777
33/38
Problems
Global tables
Sharing
Tagging
Invitations
SearchLots of relations (due to normalization) between tables
-
8/6/2019 Sharding Architectures 777
34/38
Migration
Find the bottleneck
Refactoring of DB model needed?
Prepare implementation
prepare DBA layer
write unit testssplit existing data
-
8/6/2019 Sharding Architectures 777
35/38
Migration II
Setup shards
Migrate your code
Test, Test, Test
Check if everything runs smooth
MySQL proxyDTrace
-
8/6/2019 Sharding Architectures 777
36/38
Scale!
-
8/6/2019 Sharding Architectures 777
37/38
-
8/6/2019 Sharding Architectures 777
38/38
Questions?
[email protected]://blog.experimentalworks.net
mailto:[email protected]:[email protected]:[email protected]:[email protected]