Locality-Aware Locality-Aware Content DistributionContent Distribution

Danny Bickson, Danny Bickson,

Dahlia Malkhi,Dahlia Malkhi,

David RabinowitzDavid Rabinowitz

Lecture outlineLecture outline

Problem statementProblem statement Previous workPrevious work The Julia algorithmThe Julia algorithm Algorithm deploymentAlgorithm deployment

Problem StatementProblem Statement Large scale content distributionLarge scale content distribution High speed data disseminationHigh speed data dissemination Application layer multicastApplication layer multicast

The problem: We would like to transfer a The problem: We would like to transfer a file file FF, from a source , from a source ss to a group of to a group of nn recipients.recipients.

MeasurementsMeasurements

Minimize worst completion timeMinimize worst completion time Minimize total work (the number of bits * Minimize total work (the number of bits *

link distance) over all linkslink distance) over all links Average fair sharing ratioAverage fair sharing ratio ConnectivityConnectivity

Naïve solutionsNaïve solutions

* * Client / ServerClient / Server* Mirroring / replication* Mirroring / replication* Multicast tree* Multicast tree

State-of-the-art solutionsState-of-the-art solutions

SRMSRM SplitStream / Coopnet / BayeuxSplitStream / Coopnet / Bayeux BulletBullet FastReplicaFastReplica Zigzag / TMesh / Narada Zigzag / TMesh / Narada BitTorrentBitTorrent

Julia AlgorithmJulia Algorithm

Load balanceLoad balance Fair sharingFair sharing Optimal finishing timeOptimal finishing time Local transfer of most parts of the fileLocal transfer of most parts of the file

Locality motivationLocality motivation

Stretch – Overlay distance / Underlying network distance

Locality motivationLocality motivation

First phase

Locality motivationLocality motivation

Phase 2 Phase 2

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

Round 0

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8

Round 1 –

Exchange 1 part along longest links

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8

Round 2

1,3,5,7 2,4,6,8 1,3,5,7 1,3,5,7 1,3,5,72,4,6,8 2,4,6,8 2,4,6,8

Julia algorithmJulia algorithm

1 2 3 4 5 6 7 8

1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8

Round logN – exchange half of the file along shortest links

1,3,5,7 2,4,6,8 1,3,5,7 1,3,5,7 1,3,5,72,4,6,8 2,4,6,8 2,4,6,8

1-8 1-8 1-8 1-8 1-8 1-8 1-8 1-8

Comparison SummaryComparison SummaryDownload time Download time (worst case)(worst case)

Number of Number of edgesedges

Total workTotal work

Application Application multicast treemulticast tree

K|F|logK|F|logkk(n)(n) n-1n-1 |F|D/2(n-1)|F|D/2(n-1)

SplitStream SplitStream protocolprotocol

|F|(log|F|(logkk(n)+1)(n)+1) (n-1)k(n-1)k (|D|n(|D|n1-1-

FastReplica FastReplica protocolprotocol

((2-12-1/k/k)|F|log)|F|logkk(n)(n) (n-1)*(n-1)*

(k+1)/2(k+1)/2

(|D|n(|D|n1-1-

Our protocolOur protocol |F||F| nlognlog22(n)(n) (|F|D log(|F|D log22(n))(n))

=1/log2(k)

The deploymentThe deployment

Distance estimationDistance estimation Level categorizationLevel categorization Node selection algorithmNode selection algorithm Chunk selection algorithmChunk selection algorithm

ImplementationImplementation

C++ client consisting of 15,000 lines of C++ client consisting of 15,000 lines of code.code.

Event queue model.Event queue model. Did both LAN and Planetlab experimentsDid both LAN and Planetlab experiments

Protocol MessagesProtocol Messages

Request file infoRequest file info Reply file infoReply file info Request chunkRequest chunk Reply chunkReply chunk Data / Error msgsData / Error msgs

Distance estimationDistance estimation

Distance Distance measurements are measurements are collected on the fly – collected on the fly – no spare bandwidth no spare bandwidth allocated.allocated.

Nodes are Nodes are categorized into 8 categorized into 8 levels levels

11 2Mbps2Mbps

22 1.5Mpbs1.5Mpbs

33 750Kbps750Kbps

44 500Kbps500Kbps

55 250Kbps250Kbps

66 100Kbps100Kbps

77 50Kbps50Kbps

88 UnknownUnknown

Node selection algorithmNode selection algorithm

Progress depended.Progress depended. Up to 25% progress – connect random Up to 25% progress – connect random

nodes.nodes. Up to 50% progress – connect close nodes Up to 50% progress – connect close nodes

with probability p1.with probability p1. Above 50% progress – connect close nodes Above 50% progress – connect close nodes

with probability p2.with probability p2.

Chunks selection Chunks selection algorithmalgorithm

Rarest firstRarest first RandomRandom Round robinRound robin MixedMixed

OptimizationsOptimizations

Pipelining of the Julia algorithm Pipelining of the Julia algorithm Message batchingMessage batching

The ENDThe END

Thank You!Thank You!