S-Paxos: Eliminating the Leader Bottleneck

Martin Biely, Zarko Milosevic, Nuno Santos, André Schiper

Ecole Polytechnique Fédérale de Lausanne (EPFL)

Switzerland

October 9, 2012

Context: State Machine Replication

Nuno Santos Context and Motivation 2

Replicated Service

Service Service Service

Ordering protocol (Paxos)

Clients

The Paxos Protocol

Nuno Santos Context and Motivation 3

• Observation: leader receives and sends more messages than the followers• Potential system bottleneck…

• Paxos is a leader-based protocol• A distinguished process (leader) coordinates the others (followers)

Paxos Performance

Nuno Santos Context and Motivation 4

Experimental settings•JPaxos – implementation of Paxos in Java (protocol shown previously)•n=3, request size=20 bytes, CPU 2x2cores @2.2Ghz

The bottleneck in Paxos is typically the leader

Paxos is Leader-centric• Leader-centric protocol

• The leader does considerably more work than the followers• Therefore, the leader is prone to being the system bottleneck

• Paxos and most leader-based protocols are also leader-centric

Nuno Santos Context and Motivation 5

Leader-based vs Leader-centric• Note that leader-based ≠ leader-centric

• Leader-based – algorithmic concept, leader is a distinguished process

• Leader-centric – resource usage, leader is a bottleneck

Nuno Santos Context and Motivation 6

Question: do leader-based protocols like Paxos must also be leader-centric?

S-PAXOS OVERVIEWLeader-based but not leader-centric

Nuno Santos 7

Why Paxos is Leader-centric• Leader does the following

• Receives requests from clients• Coordinates protocol to order requests• Replies to clients

• Followers do much less• Receive client requests from leader• Acknowledge order proposed by leader

• Underlying problem: unbalanced resource utilization• Leader runs out of resources (CPU, network bandwidth)• While followers are lightly loaded

Nuno Santos S-Paxos Overview 8

S-Paxos: A Balanced Paxos Variant• S-Paxos balances workload across replicas

• Leader and followers have similar resource usage• The full resources of all replicas become available to the ordering

protocol

• S-Paxos is leader-based but not leader-centric

• Combines several well-known ideas in a novel way• All replicas handle client communication• All replicas disseminate requests• Ordering done on IDs

Nuno Santos S-Paxos Overview 9

S-Paxos key ideasDistribute client communication

• Commonly used in practice• For instance, ZooKeeper

• But by itself, still leader-centric• Leader runs the ordering protocol on

requests (Phase 2a messages of Paxos) • Followers have to forward requests to

leader• And hence, sends requests to other

followers

Nuno Santos S-Paxos Overview 10

All replicas handle client communication

S-Paxos key ideasDistribute request dissemination

• Note that Phase 2a messages have a dual purpose• Dissemination of requests• Establishing order

• All replicas disseminate requests• Ordering performed on IDs

Nuno Santos S-Paxos Overview 11

S-Paxos separates dissemination from ordering

S-Paxos Architecture and Data Flow

Nuno Santos S-Paxos Overview 12

S-Paxos balances work among replicas• Client communication and request

dissemination usually the bulk of the load• In S-Paxos this task is performed by all

replicas

• Leader still has to coordinate ordering protocol• But IDs are small messages• So leader has minimal additional overhead

Nuno Santos S-Paxos Overview 13

• Two levels of batching to further reduce load on leader• Dissemination layer: batch client requests and use ordering layer to

order ids of batches• Ordering layer: usual Paxos batching, in this case batches of batch

ids.

Benefits in the presence of faults• Faster view change

• Since IDs are small, Phase 1 of Paxos completes quickly

• Failures affecting the leader have less impact on throughput• Ordering protocol is interrupted, but dissemination protocol

continues among working replicas• When a correct leader emerges, it can quickly order the IDs of the

requests that were disseminated while there was no leader

Nuno Santos S-Paxos Overview 14

DISSEMINATION LAYER PROTOCOL

Nuno Santos 15

Dissemination Layer Overview• Dissemination layer tasks

1) Receive requests from clients

2) Disseminate requests and IDs to all replicas

3) Initiate ordering of IDs

4) Execute requests in the order established for IDs

• Challenges• Once an ID is decided, the corresponding

request must remain available in the system• Coordinate view change between ordering and

dissemination layers to ensure that ids are ordered once-and-only once

Nuno Santos Dissemination Layer Protocol 16

2 2

1

3 4

Overview of the Protocol

Nuno Santos Dissemination Layer Protocol 17

PERFORMANCE EVALUATION

Nuno Santos 18

Performance Evaluation• S-Paxos implemented on top of JPaxos, a Java implementation of

Paxos

• Experiments compare • JPaxos (leader-centric)• S-Paxos (non leader-centric)

• Testbed: Grid 5000 (helios cluster)• CPU: 2x2-cores @ 2.2Ghz• Network: 1Gbit Ethernet

• Experimental parameters• Request size: 20 bytes• Batch size

• S-Paxos: dissemination layer 1450 bytes, ordering layer: 50 bytes• JPaxos: 1450 bytes

• Null service

Nuno Santos Experimental Evaluation 19

Load Distribution: Average CPU utilization

Nuno Santos Experimental Evaluation 20

JPaxos S-Paxos

Performance with Increasing Number of Clients (n=3)

Nuno Santos Experimental Evaluation 21

Throughput Response time

Scalability

Nuno Santos Experimental Evaluation 22

Throughput

Throughput with crashes

Nuno Santos Experimental Evaluation 23

Crash of the leader

False suspicions

Nuno Santos Experimental Evaluation 24

• Leader is (wrongly) suspected every 10 seconds

Conclusion

Nuno Santos 25

A leader-based protocol does not need to be leader-centric

S-Paxos: balances the workload across replicas

Benefits•Better performance for the same number of replicas•Better scalability with the number of replicas•Better performance in the presence of faults

ADDITIONAL SLIDES

Nuno Santos 26

Discussion

Nuno Santos Dissemination Layer Protocol 27

• Broadcast of <request,ID>: best effort, no retransmission• Avoids cost of reliable broadcast on requests• Recovering from partial delivery (message loss/crashes):

• Request does not become stable - client timeouts and retransmits• Request becomes stable – after ID is decided, replicas poll other

replicas for request

• Broadcast of <Ack,ID>: retransmission• Ensures that once a request is stable, it will be proposed• Almost free in practice: acks are small and can be piggybacked on

other messages.