Extracting more concurrency from distributed transaction

Rococo: Extract more concurrency from distributed transactionsShuai Mu, Yang Cui, Yang Zhang, Wyatt Lloyd, Jinyang LiTsinghua University, New York University, University of Southern California, Facebook11What Large Web Sites Need

170 million sold/day36.8 million sold/day$169 billion trade/dayScalable Storage w/ Transactions!

22What is a Distributed TransactionItemStockiPhone 6 Plus1iPhone Case1iPhone=1Case=1BEGIN_TXif (iphone > 0) { iphone--;} if (case > 0) { case--;}END_TXif (case > 0) { case--;}

if (iphone > 0) { iphone--; }

3Transactions should be strictly serializable!Otherwise 3Loss of serializability = angry customer

44Serializability is Costly under Contention55OCC Aborts Contended TransactionsIf (iphone > 0) { iphone--;}if (case > 0) { case--;}

iPhone=1Case=1

If (iphone > 0) { iphone--;}if (case > 0) { case--;}

Execute6if (iphone > 0) { iphone--;} if (case > 0) { case--;}if (iphone > 0) { iphone--;} if (case > 0) { case--;}Two-Phase Commit (2PC)62PL Blocks Contended TransactionsIf (iphone > 0) { iphone--;}if (case > 0) { case--;}

iPhone=1Case=1

If (iphone > 0) { iphone--;}if (case > 0) { case--;}

7Execute

if (iphone > 0) { iphone--;} if (case > 0) { case--;}if (iphone > 0) { iphone--;} if (case > 0) { case--;}2PC

7Achieve serializabilityw/o aborting or blocking*

* for common workloads

8

8Rococos ApproachIf (iphone > 0) { iphone--;}if (case > 0) { case--;}

iPhone=1Case=1

If (iphone > 0) { iphone--;}if (case > 0) { case--;}

Defer piece execution to enable reordering

9if (iphone > 0) { iphone--;} if (case > 0) { case--;}if (iphone > 0) { iphone--;} if (case > 0) { case--;}9Rococo Overview: Key techniquesTwo-phase protocolMost pieces are executed at the second (commit) phase

Decentralized dependency tracking Servers track pieces arrival orderIdentify non-serializable ordersDeterministically reorder pieces

Offline workload checkingIdentifies safe workloads (common)Identifies small parts that need traditional approaches (rare)10Enable piece reordering for serializabilityAvoid aborts for common workloads10#1 Two-phase protocolStart PhaseCommit PhaseSend pieces to servers w/o executing themEstablish a final order and execute piecesReorder for serializabilitySet up a provisional order1111depdepdepT2:T1:#2 Dependency Tracking: Start Phase if ... iphone--if ... case--

iPhone=1

if ... iphone--if ... case--

12if ...iphone--if ... case--if ... iphone--if ... case--T2T1depCase=1T1T2T1T2T1T1T2T212if ... case--

if ... case--

depdepdep#2 Dependency Tracking: Commit Phase if ... iphone--

iPhone=1

if ... iphone--13T2T1depCase=1T1T2T2T1T1T2T1T2T1T2T2T1T2T1Sort the cycle by any deterministic order13oid = next_oid ++if ... iphone--if ... case--orderline.insert(oid, )iPhoneCaseorderline.insert(

oid,)Problem: Not Every Piece is DeferrableIntermediate Results Calls for Immediate Execution

next_oidorderline14if ... iphone--if ... case--oidoid = next_oid ++ 14a = next_a++; b = next_b++;ol_a.insert(a, );ol_b.insert(b, );a = next_a++; b = next_b++;ol_a.insert(a, );ol_b.insert(b, );Immediate Pieces are Naughty!

a = next_a++b = next_b ++

b = next_b ++1. Common workloads have few immediate pieces2. Pre-known workload Offline checknext_a = 0next_b = 0 15a = next_a++15S(ibling)-edge linking pieces within a transaction#3: Offline Checking: Basic

T1T2iphonecaseiphonecaseT1T1T2Item_TableItem_TableItem_TableItem_TableT1C(onflict)-edge linking pieces w/ conflicting accessAn SC-cycle consists of both S-edge and C-edgeSC-cycles represent potential non-serializable executions that require 2PL/OCC1616#3: Offline Checking: Enhanced

SC-cycles with deferrable pieces can be safely reordered!17T1T2OidTableItemTableOidTableItemTableItemTableItemTableDeferrableImmediate17depdepdepdepIncorporating Immediate Pieces

18oidiPhoneCasenext_oidorderline18iphonecaseorderlineoidiphonecaseorderlineoidoidiphoneiphoneImmediate dependencyT2T1orderlineorderlineT2T1T2T1T2T1casecaseEvery cycle contains deferrable pieces, ensured by offline checkingDeferrable dependencyA cycle with deferrable pieces is safe to reorder18

Merged PiecesRead-only TransactionsReducing Dep. SizeFault ToleranceOverlapping Trans.

Deferred ExecutionDecentralized Dependency TrackingOffline Checking 1919EvaluationHow does Rococo perform under contention?

How does Rococo scale?2020Workload: Scaled TPC-COne warehouse, many districtsPartitioned by districts - all transactions distributed5~155~155~155~155~155~155~155~1521New order(45%)Payment(43%)Delivery (4%)Read-only: OrderStatus(4%), StockLevel (4%)21Rococo Has Higher ThroughputAborts due to conflicts in validationAborts due to deadlock detectionIncreasing graph size228 servers, 10 districts per server Main source of contention is next_oid of each district22Rococo Avoids Aborts173~645ms152~392ms61~66ms2323Rococo Scales OutAbortsBlockingAlmost Linear2410 districts per server, fixed # of items contention grows slowly24Related WorkIsolation LevelConcurrency Control Mech.RococoStrict-Serial.RococoLynx [SOSP13]Serial.Origin OrderingGranola [SIGMOD10]Serial.Timestamp basedPercolator [OSDI10] S.I.OCCWalter [SOSP11]P.S.I.W-W PreclusionCOPS [SOSP11]CausalDependency TrackingSpanner [OSDI12]Strict-Serial.2PLHStore [VLDB07]Strict-Serial.OCCCalvin [SIGMOD12]Strict-Serial.Pre-ordered Locking25Decentralized dependency trackingCentralized sequencing layer, difficult to scale25ConclusionTraditional protocols perform poorly w/ contentionOCC aborts & 2PL blocks

Rococo defers execution to enable reorderingStrict serializability w/o aborting or blocking for common workloads

Rococo outperforms 2PL & OCCWith growing contentionScales out 2626Thank you!

Questions?

https://github.com/msmummy/rococo27

Poster tonight!27