coding for atomic shared memory emulation

Erasure Coding for Distributed Storage

• Locality, Repair Bandwidth, Caching and Content Distribution– [Gopalan et. al 2011, Dimakis-Godfrey-Wu-Wainwright- 10, Wu-

Dimakis 09, Niesen-Ali 12]




• Queuing theory– [Ferner-Medard-Soljanin 12, Joshi-Liu-Soljanin 12, Shah-Lee-

Ramchandran 12]




• Queuing theory– [Ferner-Medard-Soljanin 12, Joshi-Liu-Soljanin 12, Shah-Lee-

Ramchandran 12]


This talk: Theory of distributed computingConsiderations for storing data that changes

6

Consistency: Value changing, get the “latest” version

Failure tolerance, Low storage costs, Fast reads and writes

7

Shared Memory Emulation - History

Atomic (consistent) shared memory

• [Lamport 1986]• Cornerstone of distributed

computing and multi-processor programming

8



Emulation over distributed storage

systems



• “ABD” algorithm [Attiya-Bar-Noy-Dolev95], 2011 Dijsktra Prize,

• Amazon dynamo key-value store

[Decandia et. al. 2008]• Replication-based

9




systems

Costs of emulation






• Low cost coding based algorithm

• Communication and storage costs

(This talk) • [C-Lynch-Medard-Musial 2014],preprint available

10




systems

Costs of emulation








• [C-Lynch-Medard-Musial 2014],preprint available(This talk)

12

Write

Readtime

13

Write

Readtime

14

Atomicity [Lamport 86]

aka linearizability. [Herlihy, Wing 90]

Write

Readtime

15

Write

Read



time

16

Write

Read



time

17

Write

Read



time

Atomic

18

Atomic

Not atomic

Write

Read



time

time

time

19




systems

Costs of emulation








• [C-Lynch-Medard-Musial 2014],preprint available(This talk)

20

• Client server architecture, nodes can fail (no. of server failures is limited)

• Point-to-point reliable links (arbitrary delay).

• Nodes do not know if other nodes fail

• An operation should not have to wait for others to complete

Distributed Storage Model

Servers

Write Clients Read Clients

21


• Point-to-point reliable links (arbitrary delay)




Servers


22


• Point-to-point reliable links (arbitrary delay).




Servers


23


Servers

Requirements and cost measure

Design write, read and server protocols such that

• Atomicity

• Concurrent operations, no waiting.

Communication overheads: Number of bits sent over links Storage overheads: (Worst-case) server storage costs

24

The ABD algorithm (sketch)

Servers


Quorum set: Every majority of server snodes. Any two sets intersect at at least one nodesAlgorithm works if at least one quorum set is available.

25


Write:Send time-stamped value to every server; return after receiving sufficeint acks.

Read: Send read query; wait for sufficient responses and return with latest value.

Servers:Store latest value from server; send ackRespond to read request with value

Servers


26


Write:Send time-stamped value to every server; return after receiving acks from quorum.

Read:: Send read query; wait for sufficient responses and return with latest value.

Servers:Store latest value; send ackRespond to read request with value

Servers

ACK

ACK

ACK

ACK

ACK

ACK


27

The ABD algorithm (sketch)QueryQueryQueryQueryQueryQuery

QueryWrite Clients Read Clients


Read: Send read query; wait for sufficient responses and return with latest value.


Servers

28


Servers


Read: Send read query; wait for quorum of responses; return with latest value.



29


Servers


Read: Send read query; wait for quorum responses; send latest value to quourm; latest value.



30


Servers


Read: Send read query; wait for acks from quorum responses; send latest value to servers; return latest value after receiving acks from quorum.Servers:Store latest value from server; send ackRespond to read request with value


ACK

ACK ACK

ACK

ACK

ACK

The ABD algorithm (summary)

• The ABD algorithm ensures atomic operations.

• Operations terminate is ensured as long as a majority of nodes do not fail.

• Implication: A networked distributed storage system can be used as shared memory.

• Replication to ensure failure tolerance.

ABD

Storage

Communication(read)

Communication(write)

Performance Analysis

• f represents number of failures• a lower communication cost algorithm in [Fan-Lynch 03]

33




systems

Costs of emulation








(This talk)• [C-Lynch-Medard-Musial 2014],

preprint available

Shared Memory Emulation – Erasure Coding

• [Hendricks-Ganger-Reiter 07, Dutta-Guerraoui-Levy 08, Dobre-et.al 13, Androulaki et. al 14]

• New algorithm, a formal analysis of costs

• Outperforms previous algorithms in certain aspects• Previous algorithms incur infinite worst-case storage costs• Previous algorithms incur large communication costs

35

Erasure Coded Shared Memory

36


Example:(6,4) MDS code

• Value recoverable from any 4 coded packets

• Size of coded packet is ¼ size of value

Smaller packets,smaller overheads

37

• Value recoverable from any 4 coded packets

• Size of coded packet is ¼ size of value

• New constraint, need 4 packets with same time-stamp


Smaller packets,smaller overheads

Example:(6,4) MDS code

38

Quorum set: Every subset of 5 server snodes. Any two sets intersect at 4 nodesAlgorithm works if at least one quorum set is available.

Coded Shared Memory – Quorum set up

Servers


39

Coded Shared Memory – Why is it challenging?

Servers


40

Coded Shared Memory – Why is it challenging?

Servers

QueryQuery

Query

Query

Challenges: reveal elements to readers only when enough elements are propagated discard old versions safely

Solutions: Write in multiple phases Store all the write-versions concurrent with a read

Servers store multiple versions


Coded Shared Memory – Protocol overview

Write:Send time-stamped value to every server; send finalize message after getting acks from quorum; return after receiving acks from quorum.

Read: Send read query; wait for time-stamps from a quorum;Send request with latest time-stamp to servers; decode and return value after receiving acks from quorum.

Servers:Store the coded symbol; keep latest δ codeword symbols and delete older ones; send ack. Set finalize flag for tag on receiving finalize message.Respond to read query with latest finalized tag.Finalize the requested tag; respond to read request with codeword symbol.




Servers:Store the coded symbol; keep latest δ codeword symbols and delete older ones; send ack. Set finalize flag for time-stamp on receiving finalize message. Send ack.Respond to read query with latest finalized tag.Finalize the requested tag; respond to read request with codeword symbol.




Servers:Store the coded symbol; keep latest δ codeword symbols and delete older ones; send ack. Set finalize flag for tag on receiving finalize message.Respond to read query with latest finalized tag.Finalize the requested tag; respond to read request with codeword symbol.



Read: Send read query; wait for time-stamps from a quorum;Send request with latest time-stamp to servers; decode and return value after receiving acks/symbols from quorum.

Servers:Store the coded symbol; keep latest δ codeword symbols and delete older ones; send ack. Set finalize flag for tag on receiving finalize message.Respond to read query with latest finalized tag.Finalize the requested time-stamp; respond to read request with codeword symbol if it exists, else send ack.



Read: Send read query; wait for time-stamps from a quorum;Send request with latest time-stamp to servers; decode and return value after receiving acks/symbols from quorum.

Servers:Store the coded symbol; keep latest δ codeword symbols and delete older ones; send ack. Set finalize flag for time-stamp on receiving finalize message.Respond to read query with latest finalized tag.Finalize the requested time-stamp; respond to read request with codeword symbol if it exists, else send ack.


• Use (N,k) MDS code, where N is the number of servers

• Ensures atomic operations

• Operations terminate is ensured as long as o Number of failed nodes smaller than (N-k)/2o Number of writes concurrent with a read

smaller than δ

Performance comparisons

ABD Our Algorithm

Storage

Communication(read)

Communication(write)

• N represents number of nodes, f represents number of failures• δ represents maximum number of writes concurrent with a read

48

Proof Steps

• After every operation terminates, - there is a quorum of servers with the codeword symbol - there is a quorum of servers with the finalize label - because every pair of servers intersects in k servers,

readers can decode the value

49

Proof Steps

• After every operation terminates, - there is a quorum of servers with the codeword symbol - there is a quorum of servers with the finalize label - because every pair of servers intersects in k servers,

readers can decode the value

• When a codeword symbol is deleted at a server– Every operation that wants that time-stamp has terminated– (Or the concurrency bound is violated)

50

Main Insights

• Significant savings on network traffic overheads

- Reflects the classical gain of erasure coding over replication

• (New Insight) Storage overheads depend on client activity• Storage overhead proportional to the no. of writes concurrent

with a read• Better than classical techniques for moderate client activity

51

Future Work – Many open questions

Refinements of our algorithm- (Ongoing) More robustness to client node failures

Information theoretic bounds on costs- New coding schemes

Finer network models- Erasure channels, different topologies, wireless channels

Finer source models- Correlations across versions

Dynamic networks

52




Finer network models- Erasure channels, different topologies, wireless channels

Finer source models- Correlations across versions

Dynamic networks

53

Storage costs

ABD

Our algorithm

Number of writes concurrent with a read

Storage Overhead

What is the fundamental cost

curve?

54




Finer network models, finer source models- Erasure channels, different topologies, wireless channels- Correlations across versions

Dynamic networks

55




Finer network models, finer source models- Erasure channels, different topologies, wireless channels- Correlations across versions

Dynamic networks

- Interesting replication based algorithm in [Gilbert-Lynch-Shvartsman 03]

- Study of costs in terms of network dynamics

coding for atomic shared memory emulation

Documents