distributed storage allocations for optimal delay

Distributed Storage Allocationsfor Optimal Delay

Derek Leong1, Alexandros G. Dimakis2, Tracey Ho1

1California Institute of Technology2University of Southern California

ISIT 20112011-08-02

Distributed Storage Allocations for Optimal Delay / 2

How should we store a data objectover a set of mobile nodes,

subject to a given storage budget,so as to achieve the optimal recovery delay?

When is coding beneficial?When will uncoded replication suffice?

Motivation


Network Model Consider a network of n mobile storage nodes Assume that the number of contacts between any given pair of

nodes follows a Poisson distribution with rate parameter ¸;the time between contacts is therefore described by an exponential distribution with mean 1/¸

Introduction


Storage Allocation A source node creates a data object of unit size, and subsequently

disseminates an encoded representation of it to other nodes for storage, subject to a given total storage budget T

At the end of the dissemination process, node 1 stores x1 amount of data, node 2 stores x2 amount of data, and so on, such that

Introduction


Recovery by a Data Collector At some time after the completion of the data dissemination process,

a data collector node begins to recover the data object by contacting other nodes and accessing the coded data stored in them

We make the simplifying assumption that the stored data is instantaneously transmitted on contact

Let random variable D denote the recovery delay incurred by the data collector

Introduction


Objectives We seek an allocation (x1; …; xn) of the given budget T that

produces the optimal recovery delay; specifically, we considertwo objectives involving the recovery delay D:

(i) maximization of the probability of successful recovery by agiven deadline d, or recovery probability

(ii) minimization of the expected recovery delay

Therefore, for each objective, we need to find

(i) an optimal allocation of the given budget over the nodes, and

(ii) an optimal coding scheme

that jointly optimize the objective

Introduction


Introduction

1 2 3 4 5

t1 t2

s

A. G. Dimakis, P. B. Godfrey, Y. Wu, M. J. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” Trans. Inf. Theory, Sep 2010.

A. Jiang, “Network coding for joint storage and transmission with minimum cost,” in Proc. ISIT, Jul 2006.

Objectives Using an appropriate code, successful recovery occurs whenever

the data collector accesses at least a unit amount of data(= size of the original data object)


Objectives Therefore, assuming the use of an appropriate code (e.g. random

linear coding, MDS code), we can express the recovery delay D as

,

where is the set of all nodes contacted by thedata collector by time d

Introduction


Let W1, …, Wn be i.i.d. random variables denoting the times at which the data collector first contacts nodes 1, …, n, respectively, where

Observe that the data collector contacts each node by the specified deadline d > 0 independently with probability p¸ ; d given by

It follows that the probability of contacting exactly a subset r of the n nodes by time d is ; the recovery probability can therefore be obtained by summing over all subsets r that allow successful recovery:

Maximizing Recovery Probability

Wi ~ Exponential(¸)


Discussion between R. Karp, R. Kleinberg, C. Papadimitriou, E. Friedman, and others at UC Berkeley, 2005 Found examples for the suboptimality of symmetric allocations Conjectured that there exists a symmetric optimal allocation when

the number of nodes n → ∞

S. Jain, M. Demmer, R. Patra, K. Fall, “Using redundancy to cope with failures in a delay tolerant network,” SIGCOMM 2005 Considered the allocation of a transmission budget over different

routes in a DTN Experimentally evaluated the performance of symmetric allocations

along with other heuristics Related theoretical claims and proofs incomplete/inaccurate

Recovery Probability: Related Work RECAP


1 2 3 4 5

A 7/15 7/15 7/15 7/15 7/15

B 7/6 7/6 0 0 0

C 2/3 2/3 1/3 1/3 1/3

0.79012

0.88889

0.90535C

RecoveryProbability

n = 5 nodes, access probability p¸ ; d = 2/3, budget T = 7/3

Recovery Probability: Illustrative Example RECAP


We are particularly interested in symmetric allocations because they are easy to describe and implement

Successful recovery for the symmetric allocation occursif and only if at least out of the m nonempty nodesare accessed

Therefore, the recovery probability of is given by

Recovery Probability: Optimal Symmetric Allocation

D. Leong, A. G. Dimakis, and T. Ho, “Symmetric allocations for distributed storage,” in Proc. GLOBECOM, Dec 2010.

RECAP


Recovery Probability: Optimal Symmetric AllocationThe problem is nontrivial even when restricted to symmetric allocations…

number of nonempty nodes in the symmetric allocation

The recovery probability for the symmetric allocation is

RECAP



Maximal spreading (with coding) is optimal among symmetric allocations when the contact rate ¸ or recovery deadline d is sufficiently large:

Minimal spreading (uncoded replication) is optimal among symmetric allocations when the contact rate ¸ or recovery deadline d is sufficiently small:

D. Leong, A. G. Dimakis, and T. Ho, “Symmetric allocations for distributed storage,” in Proc. GLOBECOM, Dec 2010.

RECAP

PROPOSITION 1

If , then either or is an optimal symmetric allocation.

PROPOSITION 2

If , then is an optimal symmetric allocation.



maximal spreading

(with coding)

is optimal among

symmetric allocations

minimalspreading(uncoded

replication)

is optimal among

symmetric allocations

other symmetric allocations may be optimal in the gap

When exactly, we observe numerically that minimal spreading is optimal among symmetric allocations

for most values of T ;the optimal symmetric allocation

changes continually over the intervals

while is optimal for

RECAP


By considering the derivative of wrt d, we obtain the following expression for the expected recovery delay:

CONJECTURE: A symmetric optimal allocation always exists for any n and T

Observe that given n, ¸, and T, the optimal allocation depends only on n and T, but not ¸; this is in contrast with the maximization offor which the optimal allocation depends on all parameters n, ¸, T, and d

Minimizing Expected Recovery Delay


Expected Delay: Related Work T. Spyropoulos, K. Psounis, C. S. Raghavendra,

“Spray and Wait: An efficient routing scheme for intermittently connected mobile networks,”ACM SIGCOMM Workshop on DTN 2005 Spray a fixed number of uncoded replicas into the network, and wait

for one of them to come into contact with the data collector Showed that this fixed budget approach performs very well compared

to other heuristics


number of nonempty nodes in the symmetric allocation

Finding the optimal symmetric allocation…

The expected recovery delay for the symmetric allocation is

Expected Delay: Optimal Symmetric Allocation


If T is an integer (i.e. ` = 1), then , which correspondsto minimal spreading (uncoded replication), is optimal

As the fractional part of T increases (i.e. ` increases), the amount of spreading (with coding) in the optimal symmetric allocation increases

RESULT 1

Suppose , where .

If , then is an optimal symmetricallocation.

If , then either or is an optimal symmetric allocation.

We are able to characterize the optimal symmetric allocation completely:



Proof Idea: Eliminating candidates for the optimal symmetric allocation…

1. We can show that an optimal m* can be found from among candidates:

2. For , where , the expected recovery delay is given by

3. Using a geometrical argument, we show that the choice ofminimizes the expected recovery delay among all ,where

4. To demonstrate the optimality of , i.e. , we apply the following bounds for the harmonic number Hn:



We apply our theoretical insights to the design of a simple data dissemination and storage protocol for a delay tolerant network

Simulations allow us to capture the transient dynamics of the data dissemination process, and its interaction with the data recovery process

Our goal is to understand how different symmetric allocations perform under different circumstances: Random waypoint mobility model vs real-world mobility traces Low vs high mobility Low vs high connectivity Starting recovery immediately vs after some time

Simulation Study


Our protocol extends SPRAY AND WAIT by allowing nodes to storecoded packets that are each 1/ w the size of the original data object

Successful recovery occurs when the data collector accesses at leastw such packets (choosing w = 1 produces the original protocol)

Different symmetric allocations of the given budget T can be realized by changing the value of parameter w

Simulation Study: Generalized Spray and Wait


Simulation Study: Random Waypoint: Key Observations Number of wireless mobile nodes n = 100 Plots show how the required wait time varies with

the desired recovery probability PS

Each line represents a specific choice of parameter

Recovery probability performance is consistent with our analysis:phase transition in the optimal symmetric allocation is clearly

discernable in most plots

Expected recovery delay performance is consistent

with our analysis:minimal spreading is optimal in most plots

High-mobility scenario plots appear to be vertically

scaled versions of the baseline scenario plots:

speeding up of time

Effect of increased connectivity appears less

straightforward,e.g. phase transition not

evident for recovery starting at time 0:

data dissemination process impeded by

greater interference?

In the high recovery probability regime, maximal spreading

(with coding) can lead to a significant

reduction in the required wait time

Recovery start time appears to have a limited impact on how different

allocations perform relative to each other:

most noticeable effect of starting recovery at time 0

is the reduced spread in performance


Simulation Study: Mobility Traces: Key Observations Number of wireless taxi cabs n = 100 Plots show how the required wait time varies with



Plots show distinct “jumps”

in wait times:reduced mobility of cabs at night

Despite nonideal conditions, many of our previous

observations still apply here In the high recovery

probability regime, maximal spreading

(with coding) can lead to a significant

reduction in the required wait time


The optimal symmetric allocations are not the same for both objectives…

(i) Maximization of Recovery Probability :For any budget T, there is a phase transition from a regime whereminimal spreading (uncoded replication) is optimal to a regime wheremaximal spreading (with coding) is optimal, as the access probability p(or the deadline d) increases

(ii) Minimization of Expected Recovery Delay :With the averaging over both regimes, minimal spreading (uncoded replication) turns out to be optimal whenever the budget T is an integer; the amount of spreading in the optimal symmetric allocation increases with the fractional part of T

Performance gap between minimal spreading and maximal spreading can be quite substantial , e.g. for the required wait time in both the low and high recovery probability regimes

Summary: Theoretical Analysis


Results of the simulation study are consistent with our analytical findings

Provides clear evidence that the choice of storage allocation can have a significant impact on the recovery delay performance

Shows how mobility, connectivity, and recovery start time may affect performance

Summary: Simulation Study


The simple contact model assumed here can be generalized to the case where a variable amount of data is transmitted during each contact between nodes

Allow nonuniform contact rates ¸ i between the data collector and individual nodes

Future Work


Thank you!


Additional Simulation Results


Simulation Study: Random Waypoint (Budget T = 5) Number of wireless mobile nodes n = 100 Plots show how the required wait time varies with




Simulation Study: Mobility Traces (Budget T = 5) Number of wireless taxi cabs n = 100 Plots show how the required wait time varies with



distributed storage allocations for optimal delay

Documents

optimal recovery delay

recovery delay d

joint storage

distributed storage

given storage budget

expected recovery delay

stored data

data collectorat