stanford/mit/berkeley avoidance alan fekete alan fekete university of sydney mike franklin ali...
Post on 03-Jul-2018
215 Views
Preview:
TRANSCRIPT
COORDINATION
AVOIDANCE
IN
DATABASE
SYSTEMS
Peter BailisStanford/MIT/Berkeley
Alan FeketeUniversity of Sydney
Mike FranklinAli Ghodsi
Joe HellersteinIon StoicaUC Berkeley
Slides of Peter Bailis' VLDB’15 talk
Serializable transactions are not as widely deployed as you might think…
Supported?DefaultActian Ingres NO YES
Aerospike NO NONPersistit NO NONClustrix NO NONGreenplum NO YES
IBM DB2 NO YESIBM Informix NO YES
MySQL NO YESMemSQL NO NO
NMS SQL Server NO YESNuoDB NO NO
NOracle 11G NO NONOracle BDB YES YES
Oracle BDB JE YES YESPostgreSQL NO YESSAP Hana NO NO
NScaleDB NO NONVoltDB YES YES
VLDB 2014
Serializable transactions are not as widely deployed as you might think…
Supported?DefaultActian Ingres NO YES
Aerospike NO NONPersistit NO NONClustrix NO NONGreenplum NO YES
IBM DB2 NO YESIBM Informix NO YES
MySQL NO YESMemSQL NO NO
NMS SQL Server NO YESNuoDB NO NO
NOracle 11G NO NONOracle BDB YES YES
Oracle BDB JE YES YESPostgreSQL NO YESSAP Hana NO NO
NScaleDB NO NONVoltDB YES YES
VLDB 2014
WHY?
“post on timeline”
“accept friend request”
serializability: equivalence to some serial execution
very general!
r(y) w(x←1)
r(x) w(y←1)
very general!…but restricts concurrency
serializability: equivalence to some serial execution
serializability: equivalence to some serial execution
r(x)=0
w(x←1)w(y←1)r(y)=0
very general!…but restricts concurrencytransactions cannot make progress independently
Serializability requires Coordination
CONCURRENT EXECUTION
IS NOT SERIALIZABLE!
1. Decreased performance
transactions cannot make progress independentlySerializability requires Coordination
Two-Phase Locking
Optimistic Concurrency Control Pre-Scheduling
Multi-Version Concurrency Control BlockingWaitingAborts
Costs of CoordinationBetween Concurrent Transactions
2 3 4 5 6 7 8Number of Servers in Transaction
0
200
400
600
800
1000
1200
Max
imum
Thr
ough
put (
txns
/s)
Number of Servers in Transaction
+OR +CA +IR +SP +TO +SI +SYParticipating Datacenters (+VA)
2
4
6
8
10
12
Max
imum
Thr
ough
put (
txn/
s)
Local datacenter (Amazon EC2)Based on[Bobtail, Xu et al., NSDI 13]
Multi-datacenter (Amazon EC2)Based on[HAT, Bailis et al., VLDB 14]
For conflicting transactions
1. Decreased performance» due to waiting, communication delays, aborts» exacerbated in distributed environment!
2. Decreased availability during failures
transactions cannot make progress independentlySerializability requires Coordination
Costs of CoordinationBetween Concurrent Transactions
Serializability ⇒ Consistency
“usernames should be unique”
“account balances should remain positive”
“there should only be one administrator”
*
Goal: maintain application-level correctness criteria
Consistency via Coordination
Serializability ⇒ Consistency
“usernames should be unique”
“account balances should remain positive”
“there should only be one administrator”
*Serializability Consistency ⇒Goal: maintain application-level
correctness criteria
Serializability ⇒ Consistency
“usernames should be unique”
“account balances should remain positive”
“there should only be one administrator”
*Serializability Consistency ⇒Goal: maintain application-level
correctness criteria
Approach: use application-level criteria as basis for concurrency control
This paper’s contributions: 1. The I-confluence test (ICT):
Necessary, sufficient criterion for safe, coordination-free execution
2. ICT for SQL-based operations3. Empirical speedups for real workloads
Semantic concurrency control is back, works, and is needed!
Constraint OperationEquality, Inequality Any
Generate unique ID Any
Specify unique ID Insert
> Increment
> Decrement
< Decrement
< Increment
Foreign Key Insert
Foreign Key Delete
Secondary Indexing Any
Materialized Views Any
Typical databaseconstraints and operations
(SQL)
adopt-a-hydrant alchemy_cms amahi bostonrb boxroom brevidy browsercms bucketwise calagator canvas-lms carter chiliproject citizenry comas comfortable-mexican-sofa communityengine
copycopter-server danbooru diaspora discourse enki fat_free_crm fedena forem fulcrum gitlab-ci gitlabhq govsgo heaven inkwell insoshi jobsworth
juvia kandan linuxfr.org lobsters lovd-by-less nimbleshop obtvse onebody opal opencongress opengovernment openproject piggybak publify radiant railscollab redmine
refinerycms ror_ecommerce rucksack saasy salor-retail selfstarter sharetribe skyline spot-us spree sprintapp squaresquash sugar teambox tracks tryshoppe wallgig
CONSTRAINTS MORE COMMON 37x
adopt-a-hydrant alchemy_cms amahi bostonrb boxroom brevidy browsercms bucketwise calagator canvas-lms carter chiliproject citizenry comas comfortable-mexican-sofa communityengine copycopter-server danbooru diaspora discourse enki fat_free_crm fedena forem fulcrum gitlab-ci gitlabhq govsgo heaven inkwell insoshi jobsworth juvia kandan linuxfr.org lobsters lovd-by-less nimbleshop obtvse onebody opal opencongress opengovernment openproject piggybak publify radiant railscollab redmine refinerycms ror_ecommerce rucksack saasy salor-retail selfstarter sharetribe skyline spot-us spree sprintapp squaresquash sugar teambox tracks tryshoppe wallgig zena
67 projects 1.77M LoC 1957 tables
9986 total; avg. 5.1 per table
259 total; avg. 0.13 per table
[SIGMOD 2015]
WHAT THE APPLICATION SAYS
“post on timeline”
“accept friend request”
write readwrite
readwrite
writeread
write
write write
readwrite
WHAT THE DATABASE HEARS
readread
read
read
read
read
TODAY:ENFORCEMENT
VIA
write readwrite
readwrite
writeread
write
write write
readwrite
WHAT THE DATABASE HEARS
readread
read
read
read
read
WHAT THE APPLICATION SAYS
“no duplicate users”
CAN WE USE CONSTRAINTSTOAVOID
WHAT THE APPLICATION SAYS
“no duplicate users”
constraint
WHAT THE DATABASE HEARS
constraint
constraint
constraint
constraint
constraint
constraintconstraint
“no duplicate users”
CAN WE USE CONSTRAINTSTOAVOID
Some Definitions
• Replica R belonging to database D is I-valid if I(R) = true
• A system is globally I-valid if all replicas are always I-valid
• A transaction set T is I-confluent if for all reachable states Di and Dj with a common ancestor, the result of merging Di and Dj (Di U Dj) is I-valid
I-confluence Theorem
A globally I-valid system can execute a set of transactions T with coordination-freedom, transactional availability, convergence if and only if T is I-confluent with respect to I.
CONSTRAINT: User IDs are unique OPERATION: Add users MERGE: Set union
{{Stu,ID=1}, {Ann,ID=1}}
Constraint violated!
{}
MERGE
add {Stu,ID=1}
add {Ann,ID=1}
Key idea: Check if constraints can be violated by “merging” independent operations
ICT: Invar iant Confluence Test
Key idea: Check if constraints can be violated by “merging” independent operations
CONSTRAINT: User IDs are positive OPERATION: Add users MERGE: Set union
{{Stu,ID=1}, {Ann,ID=1}}
Constraint holds!
{}
MERGE
add {Stu,ID=1}
add {Ann,ID=1}
ICT: Invar iant Confluence Test
Key idea: Check if constraints can be violated by “merging” independent operations
OUR CONTRIBUTION:
Generalizes classic partitioning-based indistinguishability arguments
Theorem. A globally I-valid system can execute a set of transactions T with coordination-freedom, transactional availability, and convergence if and only if T are I-confluent with respect to I.
ICT ⟺ safe, coordination-free execution possible
ICT: Invar iant Confluence Test
Constraint Operation OK?
Equality, Inequality Any ???
Generate unique ID Any ???
Specify unique ID Insert ???
> Increment ???
> Decrement ???
< Decrement ???
< Increment ???
Foreign Key Insert ???
Foreign Key Delete ???
Secondary Indexing Any ???
Materialized Views Any ???
Typical databaseconstraints and operations(SQL)Under set merge
Constraint Operation OK?
Equality, Inequality Any Y
Generate unique ID Any Y
Specify unique ID Insert N
> Increment Y
> Decrement N
< Decrement Y
< Increment N
Foreign Key Insert Y
Foreign Key Delete Y*
Secondary Indexing Any Y
Materialized Views Any Y
Typical databaseconstraints and operations(SQL)Under set merge
14/16 CONSTRAINTS PASS ICTTPC-C
scale to
over 25xbest listed result
0 50 100 150 200
2M4M6M8M
10M12M14M
Tota
l Thr
ough
put (
txn/
s)
0 50 100 150 200Number of Servers
0
20K
40K
60K
80K
Thro
ughp
ut (t
xn/s
/ser
ver)
6-11x faster than ACID/serializability
8 16 32 48 64Number of Warehouses
40K
100K
600K
Thro
ughp
ut (t
xns/
s)
Coordination-Avoiding Serializable (2PL)
adopt-a-hydrant alchemy_cms amahi bostonrb boxroom brevidy browsercms bucketwise calagator canvas-lms carter chiliproject citizenry comas comfortable-mexican-sofa communityengine copycopter-server danbooru diaspora discourse enki fat_free_crm fedena forem fulcrum gitlab-ci gitlabhq govsgo heaven inkwell insoshi jobsworth juvia kandan linuxfr.org lobsters lovd-by-less nimbleshop obtvse onebody opal opencongress opengovernment openproject piggybak publify radiant railscollab redmine refinerycms ror_ecommerce rucksack saasy salor-retail selfstarter sharetribe skyline spot-us spree sprintapp squaresquash sugar teambox tracks tryshoppe wallgig zena
67 projects 1.77M LoC 1957 tables
9986 total; avg. 5.1 per table
259 total; avg. 0.13 per table
86.9% PASS ICT[SIGMOD 2015]
Serializable transactions have largelyfailed
Supported?DefaultActian Ingres NO YES
Aerospike NO NONPersistit NO NONClustrix NO NONGreenplum NO YES
IBM DB2 NO YESIBM Informix NO YES
MySQL NO YESMemSQL NO NO
NMS SQL Server NO YESNuoDB NO NO
NOracle 11G NO NONOracle BDB YES YES
Oracle BDB JE YES YESPostgreSQL NO YESSAP Hana NO NO
NScaleDB NO NONVoltDB YES YES
VLDB 2014
Conclusions• Serializable transactions suffer from fundamental
coordination overheads
• By reasoning about application correctness criteria instead of read-write traces, we can avoid coordination
• I-confluence is a necessary and sufficient condition for preserving criteria under coordination-free execution
• Real apps are I-confluent and allow huge speedups
top related