araneola: a scalable reliable multicast system for dynamic ......araneola: a scalable reliable...

11

Araneola: A Scalable Reliable Multicast System for Dynamic

Wide-Area Environments

Roie Melamed and Idit KeidarRoie Melamed and Idit KeidarTechnionTechnion

Roie Melamed and Idit KeidarRoie Melamed and Idit Keidar IBM Systems and Storage seminar, December 2005IBM Systems and Storage seminar, December 2005 22

OutlineOutline

•• AraneolaAraneola’’s overlays overlay–– MotivationMotivation–– Main featuresMain features–– Construction and maintenanceConstruction and maintenance–– FaultFault--tolerancetolerance–– Average cost per join/leaveAverage cost per join/leave–– Exploiting network proximity and bandwidth Exploiting network proximity and bandwidth

heterogeneityheterogeneity

•• MulticastMulticast–– Undisrupted service in the face of churnUndisrupted service in the face of churn


High Churn MeasurementsHigh Churn Measurements

•• Mean life time (or session duration) Mean life time (or session duration) –– On the On the MBoneMBone -- generally very short, e.g., 7 generally very short, e.g., 7

minutes in a typical multicastminutes in a typical multicast–– In a file sharing application In a file sharing application -- roughly one roughly one

hour hour

•• How do we cope with such user behavior How do we cope with such user behavior (without relying on infrastructure)?(without relying on infrastructure)?

•• Answer: by using a robust overlayAnswer: by using a robust overlay


Main FeaturesMain Features

•• FlexibilityFlexibility–– MultiMulti--purpose overlaypurpose overlay–– Support various types of optimizations Support various types of optimizations

•• ReliabilityReliability–– Undisrupted serviceUndisrupted service–– Under node and link failures Under node and link failures –– Despite churn Despite churn

•• Coping with high churn Coping with high churn –– Low constant cost for handling joins and failuresLow constant cost for handling joins and failures

•• ScalabilityScalability–– Low constant load on each nodeLow constant load on each node

•• Easy deployment without relying on infrastructureEasy deployment without relying on infrastructure


A KA K--Regular Random GraphRegular Random Graph

•• A random graph in which each node has A random graph in which each node has exactly k neighborsexactly k neighbors

•• The most robust graph for a given The most robust graph for a given average degreeaverage degree


Properties of KProperties of K--Regular Random Regular Random Graph (Graph (kk == 3)3)

•• EEach node has exactly k random neighbors ach node has exactly k random neighbors •• Logarithmic diameterLogarithmic diameter•• kk--connectedconnected•• Expander, remains highly connected following Expander, remains highly connected following

random removal of large subsets of edges or random removal of large subsets of edges or nodesnodes

•• Regularity allows degree to be as low as 3 Regularity allows degree to be as low as 3 instead of instead of log(Nlog(N) in a normal random graph) in a normal random graph


Known Constructions of KKnown Constructions of K--Regular Regular Random GraphsRandom Graphs

•• Each addition or removal of a single edge Each addition or removal of a single edge or node from the graph requires the or node from the graph requires the reconstruction of the graph anewreconstruction of the graph anew–– Highly inefficient in dynamic settingsHighly inefficient in dynamic settings–– Solution: an efficient approximation of a kSolution: an efficient approximation of a k--

regular graphregular graph


AraneolaAraneola’’s Basic (Random) s Basic (Random) OverlayOverlay

•• For For kk == 3, a3, approximate pproximate kk--regular random graphregular random graph::–– EEach node has either ach node has either kk or or kk+1 random neighbors +1 random neighbors –– Logarithmic diameterLogarithmic diameter–– kk-- connectedconnected–– Expander, remains highly connected following random Expander, remains highly connected following random

removal of large subsets of edges or nodesremoval of large subsets of edges or nodes–– Regularity allows degree to be as low as 3 or 4Regularity allows degree to be as low as 3 or 4

•• Each join or leave operation incurs sending only Each join or leave operation incurs sending only a total of about 3a total of about 3kk messagesmessages


AraneolaAraneola’’s Overlay: Main Featuress Overlay: Main Features

•• Very cheap maintenanceVery cheap maintenance–– Good performance with high Good performance with high churnchurn

•• Robust to link & node failuresRobust to link & node failures•• Balanced low degree basic overlayBalanced low degree basic overlay

–– Fair: no node contributes more than its shareFair: no node contributes more than its share

•• Logarithmic diameterLogarithmic diameter•• Readily extensible: basic overlay's low degree leaves Readily extensible: basic overlay's low degree leaves

ample bandwidth for adding links according to ample bandwidth for adding links according to application's requirementsapplication's requirements–– Communication with nearCommunication with near--by neighbors, available BW, by neighbors, available BW, QoSQoS

requirementsrequirements


Building and Maintaining Building and Maintaining the Basic Overlaythe Basic Overlay

•• Two tunable parametersTwo tunable parameters–– L (usually 5)L (usually 5)–– H (usually L+5)H (usually L+5)

•• A connect taskA connect task–– Each node connects to L random nodesEach node connects to L random nodes

•• A connect request is accepted A connect request is accepted iffiff the nodethe node’’s degree is below Hs degree is below H

•• Use gossipUse gossip--based membership service to choose random based membership service to choose random nodesnodes–– Empirically, this is good enoughEmpirically, this is good enough

•• Two reduction rulesTwo reduction rules–– Reduce each nodeReduce each node’’s degree to be L or L+1s degree to be L or L+1


The Connect TaskThe Connect Task

N1 N2

2

<CONNECT>

<CONNECT_OK>

0 31


The Connect Task: RedirectionThe Connect Task: Redirection

N1

N2

0

10

1

N356

<CONNECT><REDIRECT, N3>

<CONNECT>

<CONNECT_OK>


Steady State: Take ISteady State: Take I

•• If there is a time after which no joins, If there is a time after which no joins, leaves, or failures occur, eventually, each leaves, or failures occur, eventually, each nodenode’’s degree is between L and H s degree is between L and H

•• Reduction rules (next slides) will further Reduction rules (next slides) will further reduce node degrees reduce node degrees


Reduction Rule 1Reduction Rule 1

•• Removes the connection between neighboring Removes the connection between neighboring nodes with degrees > Lnodes with degrees > L–– Without reducing any node's degree to be below LWithout reducing any node's degree to be below L

N2 N1

7

<DISCONNECT>

< DISCONNECT_OK>

6 65


Steady State: Take IISteady State: Take II

•• If there is a time after which no joins, leaves, or If there is a time after which no joins, leaves, or failures occur, eventually, each nodefailures occur, eventually, each node’’s degree is s degree is between L and H, between L and H, and at most 50% of the nodes have degrees > Land at most 50% of the nodes have degrees > L

•• Reduction Rule 2 (next slide) further reduces Reduction Rule 2 (next slide) further reduces each nodeeach node’’s degree to be L or L+1s degree to be L or L+1


Reduction Rule 2Reduction Rule 2

n

hl

7

5

5

6

6

<CONNECT

_TO, h>

<CHANGE_CONNECTION, n>

<CONNECT>

<CONNECT_OK>

<DISCONNECT>

<DISCONNECT_OK>

6


Steady State: Take IIISteady State: Take III

•• If there is a time after which no joins, If there is a time after which no joins, leaves or failures occur, eventually, leaves or failures occur, eventually, each nodeeach node’’s degree is either L or L+1,s degree is either L or L+1,and at most 50% of the nodes have and at most 50% of the nodes have degree L+1degree L+1


EvaluationEvaluation

•• Araneola is implemented in Java using Araneola is implemented in Java using UDP/IPUDP/IP

•• Extensive evaluationExtensive evaluation–– Up to 10,000 nodes on 125 machines in Up to 10,000 nodes on 125 machines in

clustercluster–– Network proximity simulations on a WAN Network proximity simulations on a WAN

((PlanetLabPlanetLab))


Overlay ScalabilityOverlay Scalability

--9990.361000010000

5.015.016.129990.3380008000

5.015.015.9388--9990.4260006000

5.015.015.638891.4540004000

5.015.015.1677--889220002000

5.015.014.697791.410001000

5.015.014.1866--7791.8500500

avgavg#paths#paths

avgdistance

diameterdiameter%nodesdegree=5

#Nodes#Nodes


Fault Tolerance andFault Tolerance andGraceful Degradation to Node RemovalsGraceful Degradation to Node Removals

<1000,6><1000,5><2000,5><1000,4>


High Churn Experiments: High Churn Experiments: Diminishing Average Cost Per Join/LeaveDiminishing Average Cost Per Join/Leave

Static experiments

High-churn experiments

2222

Exploiting Available BandwidthExploiting Available Bandwidthfor Application Needsfor Application Needs


ObservationObservation

•• In order to achieve the important In order to achieve the important mathematical properties of mathematical properties of kk--regular regular random graphs, 3 random neighbors random graphs, 3 random neighbors sufficesuffice

•• Additional neighbors can be chosen Additional neighbors can be chosen according to application needs according to application needs –– Network proximity, available bandwidth, etc.Network proximity, available bandwidth, etc.


Example: Adding NearExample: Adding Near--By By NeighborsNeighbors

•• Each node connects to up to NB nearEach node connects to up to NB near--by by nodes in addition to its L/L+1 random nodes in addition to its L/L+1 random neighborsneighbors


HopHop--Count Statistics with Different Count Statistics with Different Selections of <L,NB>Selections of <L,NB>

5.545.5410.4610.4635.635.6<5,3><5,3>

3.823.8212.2512.2551.1851.18<3,5><3,5>

1.881.883.43.474.2374.23<0,6><0,6>

8.698.696.936.934.974.97<6,0><6,0>

5.215.2115.2715.2734.4334.43<3,3><3,3>

avgavg hop hop countcount

% of short % of short linkslinks

% of links on % of links on the same the same machinemachine

<L,NB><L,NB>


Resilience to Edge RemovalsResilience to Edge Removals<L, NB><L, NB>

<5,3> <3,5>

<6,0> <3,3>

<0,6>

2727

Multicast/Content Distribution Multicast/Content Distribution Using AraneolaUsing Araneola


Multicasting over Araneola: Main Multicasting over Araneola: Main FeaturesFeatures•• Scalability: Scalability:

–– Load, reliability, resilience to message loss, resilience to Load, reliability, resilience to message loss, resilience to simultaneous node failures, and overhead for handling join and simultaneous node failures, and overhead for handling join and leave events are all independent of the number of nodesleave events are all independent of the number of nodes

–– Scalable latency increases logarithmically with the group sizeScalable latency increases logarithmically with the group size

•• High reliability in the presence of sizable message loss High reliability in the presence of sizable message loss rates, simultaneous failures of a certain percentage of rates, simultaneous failures of a certain percentage of the nodes, and high churnthe nodes, and high churn

•• Fairness: all nodes send the same number of packetsFairness: all nodes send the same number of packets–– Due to overlayDue to overlay’’s propertiess properties

•• Low overhead: no redundant packets sentLow overhead: no redundant packets sent


Initial Data DisseminationInitial Data Dissemination

•• OneOne--toto--allall–– The source sends each data packet to a The source sends each data packet to a

random subset of the nodesrandom subset of the nodes

•• AllAll--toto--allall–– Each node sends each data packet to each of Each node sends each data packet to each of

its (overlay) neighborsits (overlay) neighbors


Phase I: Gossip Phase I: Gossip

N2N1

N4N3

N6

N5I have packets P1,

P3, P8, and P9

I have packets P1, P4, P6, and P7


Phase II: Process Gossip, Phase II: Process Gossip, Send RequestsSend Requests

N2N1

N4N3

N6

N5Send me packets P1,

P3, and P9

Send me packets P4, P6, and P7


Phase III: Send DataPhase III: Send Data

N2N1

N4N3

N6

N5Packets P1, P3, and

P9

Packets P4, P6, and P7


Message Propagation Rate,Message Propagation Rate,L= 5L= 5


Undisrupted Service with ChurnUndisrupted Service with Churn


ConclusionsConclusions

•• Flexible overlay for churnFlexible overlay for churn--resistant faultresistant fault--tolerant tolerant multicastmulticast

•• Undisrupted service under node and link failures Undisrupted service under node and link failures and despite high churn ratesand despite high churn rates

•• ScalableScalable–– Low constant load on each nodeLow constant load on each node

•• Build an expander using the minimal number of Build an expander using the minimal number of random linksrandom links–– leaves ample bandwidth for applicationleaves ample bandwidth for application--specific needs specific needs

araneola: a scalable reliable multicast system for dynamic ......araneola: a scalable reliable...

Documents