overlay neighborhoods for distributed publish/subscribe systems

Overlay Neighborhoods for Distributed Publish/Subscribe Systems

Reza Sherafat KazemzadehSupervisor: Dr. Hans-Arno Jacobsen

SGS PhD Thesis DefenseUniversity of TorontoSeptember 5, 2012

2

Content-Based Pub/Sub

Pub/Sub

S

SS SS

S

S

PPublish PP

P

SS

Subscribers

PPP

Publishers

3

Thesis Contributions

List of publications:[ACM Surveys] Dependable publish/subscribe systems (being submitted)[Middleware’12] Opportunistic Multi-Path Publication Forwarding in Pub/Sub Overlays[ICDCS’12] Publiy+: A Peer-Assisted Pub/Sub Service for Timely Dissemination of Bulk Content[SRDS’11] Partition-Tolerant Distributed Publish/Subscribe Systems[SRDS’09] Reliable and Highly Available Distributed Publish/Subscribe Service[ACM Transactions on Parallel and Distributed Systems] Reliable Message Delivery in Distributed Publish/Subscribe Systems Using Overlay Neighborhoods (being submitted)[Middleware Demos/Posters’12] Introducing Publiy (being submitted)

Dependability

Reliability

Ordered delivery

Fault-tolerance

4


Dependability

Reliability

Ordered delivery

Fault-tolerance

Multipath forwarding

Adaptive overlay mesh

Dynamic forwarding strategies

Efficient data structure


5


Dependability

Reliability

Ordered delivery

Fault-tolerance




Efficient data structures

Content Dissemination

Bulk content dissemination

Peer-assisted hybrid architecture

Dissemination strategies


6

Overlay Neighborhoods


Dependability

Reliability

Ordered delivery

Fault-tolerance




Efficient data structures

Content Dissemination


Peer-assisted hybrid architecture

Dissemination strategies


7

Publications:

[SRDS’11] Partition-Tolerant Distributed Publish/Subscribe Systems

[SRDS’09] Reliable and Highly Available Distributed Publish/Subscribe Service

[ACM Transactions on Parallel and Distributed Systems] Reliable Message Delivery in Distributed Publish/Subscribe Systems Using Overlay Neighborhoods (being submitted)

[ACM Surveys] Dependable publish/subscribe systems (being submitted)

[Middleware Demos/Posters’12] Introducing Publiy (being submitted)

DEPENDABILITY IN PUB/SUB SYSTEMSPart I

8

Dependable pub/sub systems

Challenges of Dependabilityin Content-based Pub/Sub Systems

The “end-to-end principle” is not applicable in a pub/sub system– Loose-coupling between publishers and subscribers (endpoints)– End-point cannot distinguish message loss from filtered

messages: This is especially true in content-based systems supporting flexible publication filtering

PS Pub/Sub

Middleware

✓✗✓✗?

Loss cannot be differentiated

from filtered pubs

Filtered out(not matching sub)

9


Overlay Neighborhoods

Primary network: An initial spanning tree– Brokers maintain

neighborhood knowledge– Allows brokers to transform

overlay in a controlled manner

d-Neighborhood knowledge(d is a config. parameter):– Knowledge of other brokers

within distance d– Knowledge of forwarding paths

within neighborhood

3-neighborhood

2-neighborhood

1-neighborhood

10


queue

Publication Forwarding Algorithm1. Received pubs are placed on a

FIFO msg queue and kept until processing is complete

2. All known subscriptions having interest in p are identified after matching

3. Forwarding path of the publication within downstream neighborhoods are identified

4. Publication is sent to closest available brokers towards matching subscribers

p

d-neighborhood

S

S

S

upst

ream

dow

nstr

eam

11


S

S

S

When There are Failures

• Broker reconnects the overlay by creating new links to neighbors of the failed brokers

• Publications in message queue are re-transmitted bypassing failed neighbors

• Multiple concurrent failed neighbors (up to d-1) are bypassed similarly

P

12


Impact of Mass Failures on ThroughputExperiment setup:• 500 brokers (failures injected at random brokers)• Measurement interval of 2 mins (aggregate publish rate changes depending number of

failures)

Expected # of deliveries w/o

failures

13



failures)

Actual deliveries

with failures


failures

14



failures)

Actual deliveries

with failures


failures

15



failures)

Actual deliveries

with failures


failures

16



failures)


failures

Actual deliveries

with failures

Low deliveries with d=1

17



failures)


18



failures)


19

Publications:

[Middleware’12] Opportunistic Multi-Path Publication Forwarding in Pub/Sub Overlays

OPPORTUNISTIC MULTI-PATHPUBLICATION FORWARDING

Part II

20

Multi-path publication forwarding

– Forwarding paths in the overlay are constructed in“fixed end-to-end” manner (no/little path diversity)

– This results in a high number of “pure forwarding” brokers

– Low yield (ratio of msgs delivered over msgs sent is small) Low efficiency

Problems in Existing Pub/Sub Systems

B CA D E PS✗ ✗ ✓✗✓

21


Monitor neighborhood

traffic

Selectively create additional soft links

Different Pub. Forwarding Strategies

Multi-Path Forwarding in a Nutshell

Actively utilize neighborhoods

A Soft links

• Conventional systems:Strategy 0Total msgs: 6

• Forwarding strategy 1Total msgs: 5

• Forwarding strategy 2Total msgs: 3

Different Forwarding Strategies

A B C

* *

*

p * *

*

A B C

* *

*

p * *

*

A B C

* *

*

p * *

*

23


Maximum System ThroughputExperiment setup:• 250 brokers• Publish rate of 72,000 msgs/min

S2 outperforms S0 by 90%

S1 outperforms S0 by 60%

24

Publications:

[ICDCS’12] Publiy+: A Peer-Assisted Publish/Subscribe Service for Timely Dissemination of Bulk Content

BULK CONTENT DISSEMINATION INPUB/SUB SYSTEMS

Part III

25


Applications Scenarios InvolvingBulk Content Dissemination

Fast replication of content:(video clips, pics)• Scalability• Reactive delivery• Selective delivery

Distributionof software

updates

P2P filesharing

File synch.

Replicationwithin CDN

Socialnetworks


Control layer

brokers

Data layer

subscribers

A case for a peer-assisted design

Control layer (for metadata)• P/S broker overlay• Distributed repository

maintaining users’subscriptions

Data layer (for actual data)• Form peer swarm• Exchange blocks

of data

Subs

crib

e

Subs

crib

e

Subs

crib

e

Subs

crib

e Subs

crib

e

Hybrid Architecture

27


Scalability w.r.t. Number of SubscribersNetwork setup:• 300 and 1000 clients• 1 source publishing 100 MB of content

28

Conclusion• We introduced the notion of overlay neighborhoods in

distributed pub/sub systems– Neighborhoods expose brokers’ knowledge of nearby neighbors and

the publication forwarding paths that crosses these neighborhoods

• We used neighborhood in different ways– Passive use of neighborhoods for ensuring reliable and ordered

delivery– Active use of neighborhoods for multipath publication forwarding– Bulk content dissemination

29

Thanks for your attention!

30

BONUS SLIDES IF NEEDED!EXTRAS

31

OVERLAY NEIGHBORHOODS

32

London

Toronto

Trader 1

Content-Based Publish/Subscribe

Pub/Sub

S

SS SS

S

S

PPublish PP

P

sub = [STOCK=IBM]

sub= [CHANGE>-8%]

NY

Trader 2Stock quote dissemination

application

33

Overlay neighborhoods

System ArchitectureTree dissemination networks: One path from source to destination• Pros:

– Simple, loop-free– Preserves publication order

(difficult for non-tree content-based P/S)• Cons:

– Trees are highly susceptible to failures

Primary tree: Initial spanning tree that is formedas brokers join the system

– Maintain neighborhood knowledge– Allows brokers to reconfigure overlay

after failures on the fly

∆-Neighborhood knowledge: ∆ is configuration parameterensures handling ∆-1 concurrent failures (worst case)

– Knowledge of other brokers within distance ∆ Join algorithm

– Knowledge of routing paths within neighborhood Subscription propagation algorithm

3-neighborhood

2-neighborhood

1-neighborhood

34


Overlay Disconnections

When there are d or more concurrent failures– Publication delivery may be interrupted– No publication loss

B BB B B

Remain connected

Subtree Subtree

B CA D E

Failed chain of d brokers

DisconnectedSubtrees are Disconnected

35


Experimental Evaluation

Studied various aspects of system’s operation:– Impact of failures/recoveries on delivery delay– Impact of failures on other brokers– Size of d-neighborhoods– Likelihood of disconnections– Impact of disconnections on system throughput

Discussed next

36


Publication Forwarding in Absence of Overlay Fragments

• Forwarding only uses subscriptions accepted brokers.

• Steps in forwarding of publication p:– Identify anchor of accepted subscriptions that match p– Determine active connections towards matching subscriptions’ anchors– Send p on those active connections and wait for confirmations– If there are local matching subscribers, deliver to them– If no downstream matching subscriber exists, issue confirmation towards P– Once confirmations arrive, discard p and send a conf towards p

Publications

ABCDEP SSubscriptions

☑

p

☑ ☑ ☑ ☑ ☑ ☑

CE

p p p p p

Deliver to localsubscribers

confconfconfconfconfconf

p

37


Publication Forwarding in Presence of Overlay Partitions

Key forwarding invariant to ensure reliability: we ensure that no stream of publications are delivered to a subscriber after being forwarded by brokers that have not accepted its subscription.

• Case1: Sub s has been accepted with no pid. It is safe to bypass intermediate brokers

conf

conf

conf

Publications

ABCDEP S

Subscriptionsp

C BD☑ ☑ ☑ ☑ ☑ ☑ ☑

p p

Deliver to localsubscribersconf

p

38


Publication Forwarding (cont’d)• Case2: Sub s has been accepted with some pid.

– Case 2a: Publisher’s local broker has accepted s and we ensure all intermediate forwarding brokers have also done so:

It is safe to deliver publications from sources beyond the partition.

conf

conf

conf

Publications

ABCDEP S

Subscriptionsp

C BD☑ ☑ ☑ ☑ ☑*

p p

conf

p

39


Publication Forwarding (cont’d)• Case2: Sub s has been accepted with some pid.

– Case 2a: Publisher’s local broker has accepted s and we ensure all intermediate forwarding brokers have also done so:

It is safe to deliver publications from sources beyond the partition.

conf

conf

conf

Publications

ABCDEP S

Subscriptionsp

C BD☑ ☑ ☑ ☑ ☑*

p p

Depending on when this link has been establishedeither recovery or subscription propagation ensure

C accepts s prior to receiving p

conf

p

40


Publication Forwarding (cont’d)• Case2: Subscription s is accepted with some pid tags.

– Case 2b: Publisher’s broker has not accepted s:

It is unsafe to deliver publications from this publisher (invariant).

Publications

ABCDEP S

Subscriptionsp

☑ ☑*

p p*

s was acceptedat S with the same pid tag

p p

p

Tag with pid

41


Overlay Fragments• When primary tree is setup, brokers communicate with their immediate neighbors in the

primary tree through FIFO links.

• Overlay fragments: Broker crash or link failures creates “fragments” and some neighbor brokers “on the fragment” become unreachable from neighboring brokers

• Active connections: At each point they try to maintain a connection to its closest neighbor in the primary tree.– Only active connections are used by brokers

ABCDEF SP D

pid1=<C, {D}>

Fragment detectorBrokers on the fragmentBrokers beyond

the fragmentBrokers onthe fragment

Active connection to E

?

x

42


Overlay Fragments – 2 Adjacent Failures

• What if there are more failures, particularly adjacent failures?

• If ∆ is large enough the same process can be used for larger fragments.

ABCDEF SP D

pid1=<C, {D}>

Brokers beyondthe fragment

Brokers onthe fragment

E

+ pid2=<C, {D, E}>

Active connection to F

43


Overlay Fragments - ∆ Adjacent Failures

• Worst case scenario: ∆-neighborhood knowledge is not sufficient to reconnect the overlay.

• Brokers “on” and “beyond” the fragment are unreachable.

No new active connection

ABCDEF SP D

pid1=<C, {D}>

Brokers beyondthe fragment

Brokers onthe fragment

E

pid2=<C, {D, E}>

F

+ pid3=<C, {D, E, F}>

44


FragmentsBrokers are connected to closest reachable neighbors & aware of nearby fragment identifiers.

• How does this affect end-to-end connectivity? For any pair of brokers, a fragment on the primary path between them is:

– An “island” if end-to-end brokers are reachable through a sequence of active connections

– A “barrier” if end-toe-end brokers are unreachable through some sequence of active connections

ABCDEF SP DEF

ABCDEF SP D

sourcedestination

destination source

45


Store-and-Forward

• A copy is first preserved on disk

• Intermediate hops send an ACK to previous hop after preserving

• ACKed copies can be dismissed from disk

• Upon failures, unacknowledged copies survive failure and are re-transmitted after recovery– This ensures reliable delivery but may cause delays while the machine is down

P P PPFromhere

Tohere

ackackack

46


Mesh-Based Overlay Networks [Snoeren, et al., SOSP 2001]

• Use a mesh network to concurrently forward msgs on disjoint paths

• Upon failures, the msg is delivered using alternative routes

• Pros: Minimal impact on delivery delay

• Cons: Imposes additional traffic & possibility of duplicate delivery

PPPP

Fromhere

Tohere

47


Replica-based Approach [Bhola , et al., DSN 2002]

• Replicas are grouped into virtual nodes• Replicas have identical routing information

PhysicalMachines

Virtual node

48


Replica-based Approach[Bhola , et al., DSN 2002]

• Replicas are grouped into virtual nodes• Replicas have identical routing information

• We compare against this approach

PP

PP

P

P

Virtual node


Problems with a Single Overlay Tree

• Tree provides no routing diversity

• Overloaded root– All traffic goes through a

single broker

• Under utilization: Not all availablecapacity is effectively used

Tree: Single path connectivitynot suitable for diverse

forwarding patterns

?

Unutilizedbandwidth capacity

Overloaded root


Related Work – Structured Topologies

• A topology is an interconnection between brokers: – Topology relatively stable: long-term connections– Most commonly a global/per-publisher spanning tree

• Topology adaptation change topology based on:– Traffic patterns [1,2] – optimize a cost function– Maintain acyclic property by adding + removing links

• Advantages:– Fixed topology enables high-throughput connections– Routes may be improved from a “course-grained” system-wide perspective

• Disadvantages:– Routes may never be optimal for individual broker pairs– Introduces pure forwarding brokers– Diversity of routing is not accounted for

[1] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks. In: FTDCS. (2003)[2] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks. In: FTDCS. (2003)

Re-configur

e

Tree A

Tree A’


Related Work – Unstructured Topologies

• No fixed topology exists: – Short-term links are created based on

message destination– [3] uses dissemination trees computed

at the publishers’ brokers

• Advantages:– Routes may be optimal– Zero pure forwarding brokers

• Disadvantages:– Link maintenance is difficult and on-demand– Global knowledge is required and no support

for subscription covering/merging– Scalability problems

[3] Cao, F., Singh, J.: MEDYM: Match-early and dynamic multicast for content-based publish-subscribe service networks. ICDCSW (2005)

52


Publication Forwarding StrategiesStrategy S1

• Publication is sent on intersection of primary paths towards matching subscribers

• Some pure forwarding brokers are bypassed

• Broker incurs no extra outgoing load

Strategy S2

• Publication is sent on as far as possible directly towards matching subscribers

• As many pure forwarding brokers as possible are bypassed

• Broker incurs high outgoing load

A B C

X

D

Y

Z

p X Y

Z

A B C

X

D

Y

Z

p X Y

Z

Local matching subscribers

Bypassed pure forwarder

Bypassed pure forwarder

Master vs. Working Routing Data Structures

• Overlay views captured by brokers’ d-neighborhoods are relatively static Master Overlay Map (MOM)

• Brokers link connectivity change dynamically, brokers need an efficient way to compute forwarding paths over the changing set of links Working Overlay Map (WOM) via Edge retraction

• MOM only contains brokers with a direct link - it acts as a quick cache

Master Overlay Map

Working Overlay Map

construct

54


Experimental Evaluation

Experimental setup– Various overlays

• Primary network size: 120, 250, 500 brokers• Fanout parameter: 3 and 10

– Datasets with sparse or dense matching distributions• Synthetic datasets based on Zipf distribution• Real world datasets constructed from Social Networking

user traces• Synthetic datasets with covering

55


Overlay Reconfiguration

56


Connectivity in the Overlay MeshExperiment setup:• 120 and 250 brokers• Fanout of 10

1000

100

100

10

1

Pair-

wise

for

war

ding

pat

hs

57


Impact of Broker Fanout on Subscription CoveringExperiment setup:• 500 brokers• Fanout of 5-25

58



59



60



61



62


Publication Hop CountExperiment setup:• 120 Brokers• Sparse publication/subscription workload• Publish rate of 1,800 msgs/sec Deliveries: 73,000 in 5 min

63



64



65



66



67


Publication Hop CountSparse Matching Workload Dense Matching Workload

Multi-path forwarding is more effective in sparse workloads

68


System Yield (measure of efficiency)Delivered

publicationsStrategy Number of pure

Pure ForwardersSystem Yield

73,000(Sparse Workload)

Strategy 0 91,000 44%

Strategy 1 42,000 63%

Strategy 2 29,000 71%

Delivered publications

Strategy Number of pure Pure Forwarders

System Yield

284,000(Dense Workload)

Strategy 0 195,000 59%

Strategy 1 104,000 73%

Strategy 2 69,000 80%

69


Maximum System ThroughputExperiment setup:• 250 brokers• Publish rate of 72,000 msgs/min


Data Exchange Using Network Coding

Blocks matrix(k * n)

Randomcoefficients Cused for coding

Decoding usesk linearlyindependent blocks

Segmentation Encode intodata packets DecodeSegment 3

Yi =

B1

B k

Segment 2

Yi =

Segment 1

Yi = B1

Bk

File

Segment 3

Yi =

B1

B k

Segment 2

Yi =

Segment 1

Yi = B1

Bk

FileCi Xi

Encode Transfer


Hybrid architectureRegional dissemination Cross-Regional dissemination

Control layer

Data layer

Matchingsubscribers

PList

Control layer

Data layerPublisher

PList

Codedpackets

Matchingsubscribers

Involves routing of notificationsin control layer

Information is immediately available at broker

72


Evaluation Results

• Experimental setup– UofT’s SciNet cluster with up to 1000 nodes– Peers have a capped uplink bandwidth

(100-200 KB/s)

• Network setup:– 5 Regions– 120, 300 or 1000 subscribers uniformly

distributed among regions

73


Content Serving PolicyNetwork setup:• 300 clients• 1 source publishes 100 MB of content

74



75



76



77


Impact of Packet LossNetwork setup:• 300 clients• 1 source publishes 100 MB of content

78



79



80


Impact of source Fanout on dissemination timeNetwork setup:• 300 clients• 1 source publishes 100 MB of content

81



82



83



84


Effectiveness of Traffic Shaping

REG 1 REG 2 REG 3 REG 4 REG 5

REG 1

REG 2

REG 3

REG 4

REG 5

Experiment setup:• 5 regions and 1000 clients (capped uplink bandwidth at 200 KB/s)• 1 sources publish 100 MB

85




REG 1 19.57%

REG 2

REG 3

REG 4

REG 5


Regional traffic

86




REG 1 19.57% 0.109% 0.109% 0.109% 0.108%

REG 2

REG 3

REG 4

REG 5


Regional traffic

Cross-regional traffic

87




REG 1 19.57% 0.109% 0.109% 0.109% 0.108%

REG 2 0.110% 19.57% 0.109% 0.109% 0.109%

REG 3 0.110% 0.110% 19.55% 0.108% 0.108%

REG 4 0.114% 0.114% 0.114% 19.57% 0.114%

REG 5 0.110% 0.110% 0.110% 0.111% 19.49%


Regional traffic

Cross-regional traffic

88


Traffic Sharing Among Competing Contentwith Uniform Popularity

Experiment setup:• 5 regions and 1000 clients (capped uplink bandwidth at 200 KB/s)• 15 sources (3 in each region) publish 100 MB with uniform popularity

1 TB of data

89


Traffic Sharing Among Competing Contentwith Different Popularity

Experiment setup:• 5 regions and 1000 clients (capped uplink bandwidth at 200 KB/s)• 15 sources (3 in each region) publish 100 MB • Content has 1x, 2x, and 3x popularity

Most popular

Least popular

Medium popularity

90


Contribution of PeersContribution of the source

Avg per segment: 136% content size

Contribution of subscribers

Overall avg: 102% of download size

Network setup:• 300 clients• 1 source publishes 100 MB of content

91


Comparison With BitTorrentExperiment setup:• 120 clients (capped uplink bandwidth at 200 KB/s)• 1 source publishes 100 MB of content

Upon release all clients start download

Within 1300 s download ends

92



[BT]: Polling intervalof 10 minutes

[BT]: Within 1700 s downloads end

93



Polling intervalof 2 seconds

[BT]: Within 1600 s downloads end

overlay neighborhoods for distributed publish/subscribe systems

Documents

pubsub overlaysicdcs12

pubsub service

despite failures