Computer Science

ROMA: Reliable Overlay Multicast with Loosely Coupled

TCP Connections

Gu-In Kwon and John Byers

Computer Science Dept.

Boston University

Computer ScienceIP Multicast

CMU

BerkeleyMIT

UCSD

routersend systemsmulticast flow

Highly efficient Challenges

Congestion Control Reliability

http://esm.cs.cmu.edu/Sigcomm2001/SigcommTalk.ppt

Computer Science

Overlay Multicast

CMU2

UCSD

MIT1

MIT2Overlay Tree

Berkeley

CMU1

MIT1

MIT2

CMU1

CMU2

CMU

BerkeleyMIT

UCSD

http://esm.cs.cmu.edu/Sigcomm2001/SigcommTalk.ppt

Traditional Performance Metrics• Stretch

Relative Delay Penalty• Stress

# of identical packet over a physical link

Computer Science

Quick deployment ISP’s reluctant to turn on IP Multicast

All multicast state in end systems Routers maintain per-group state in IP Multicast

Congestion control easier on “unicast” end to end connections

Potential Benefits over IP Multicast

http://esm.cs.cmu.edu/Sigcomm2001/SigcommTalk.ppt

MIT1

MIT2

CMU1

CMU2

CMU

BerkeleyMIT

UCSD

Computer Science

Past Work on Overlay Multicast

Yoid, Narada, Scattercast, Overcast, NICE, ALMI, RMX, PRM, ZIGZAG, OMNI …….

Target Application Non-reliable Streaming Application

Design Goal Low overhead on tree construction. Optimize the performance metrics.

Stretch and Stress

Computer Science

Reliable Content Delivery

Claim: TCP on each overlay link is sufficient for

reliable delivery.

A B C5Mbps 2Mbps

Available Bandwidth

Computer Science

Store-and-forward approach

Back-pressure mechanism If the application buffer is full, ask the parent to

reduce the sending rate to avoid the buffer overflow.

ALMI(USITS’01) and MCC(NGC’02).

A

B C

D E F

S

1Mbps

10Mbps 6Mbps

8Mbps

Single rate Multicast

Congestion Control

1Mbps

10Mbps1Mbps

1Mbps

1Mbps

Computer Science

ROMA Contributions

ROMA Reliable Overlay Multicast Architecture. Multirate. TCP on each overlay link. Forward-when-feasible. Digital Fountain Approach.

Performance evaluation of the chains of TCP. Loosely coupled TCP connections. Conventional wisdom on overlay network is not correct.

Conventional wisdom: Increased latency and loss rate will reduce the performance of overlay node comparing to the direct unicast.

Computer ScienceForward-when-feasible

Digital Fountain Approach.• A sender encodes n packets of original content into an unbounded set of encoding packets. • A receiver can reconstruct the original content by receiving any n distinct encoding packets.

Digital Fountain Approach(Byers, Luby, Mitzenmacher, Rege: SIGCOMM ’98)

EncodingStream

Received

Message

Sourcen

n

n Can recover filefrom any set of n encoding packets.

Transmission

Computer ScienceOverview ROMA

A B C5Mbps 2Mbps

Computer Science

Candidate Architectures

Limited Buffer Space solution Back-pressure mechanism

Unlimited Buffer Space Solution Use a disk as an extra buffer. Limitations.

A separate application buffer for each downstream. Substantial complexity to support I/O access. The overlay cannot be adaptively reconfigured.

MemoryMemory

Memory

Incoming TCP

Outgoing TCP

Computer Science

Adaptive reconfiguration

Adaptive reconfiguration of overlay network. Reconfigure when congestion or failures of

intermediate nodes occur.

A

B C

D

10Mbps

10Mbps

5Mbps

5MbpsX

1Mbps

Memory

Shapeshifter BCMR’02

Computer Science

Overlay Node Implementation

Transport Layer

Application Layer

Incoming TCP

Outgoing TCP

Application Layer Buffer

Computer Science

Modeling Chains of TCP Connections

Computer Science

Chains of TCP Connections

Loosely coupled TCP connections. An upstream TCP connection may or may not

affect the performance of a downstream TCP. A downstream connection never affects the

performance of an upstream connection.

Computer Science

Modeling Chains of TCP flows I

When the downstream transfer rate is slower than the upstream transfer rate.

The application layer buffer will grow without bound. Behaves like a normal TCP driven by an application

that always has data to send.

A B C5Mbps 2Mbps

Computer Science

Modeling Chains of TCP flows II

When the downstream transfer rate is faster than the upstream transfer rate.

B will periodically drain the application level buffer. The throughput to C is limited to that of the

upstream rate into B.

A B C2Mbps 5Mbps

Computer Science

Expected throughput on ROMA

S 1 2 iRTT_1

P_1

RTT_2 RTT_i

P_2 P_i

• Local network condition limits the throughput of overlay node.• OR the upstream connection limits the throughput of overlay node.

Computer Science

Other Measurement Studies

S. Savage et al. The end-to-end effects of Internet path selection, ACM

SIGCOMM 1999. There often exists detour route with lower aggregate loss

rate and shorter round-trip time than IP rounte. D. Andersen et al.

Resilient Overlay Networks, SOSP 2001. Use detour route both to improve performance and to

route around faults in the overlay.

• Conventional wisdom on overlay network follows this model. Increased latency and loss rate will reduce the performance of overlay node comparing to the direct unicast.

Computer Science

Example

Conservative Formula

Our Formula

A C 9 Mbps 9 Mbps

A B C 9Mbps 22.2Mbps

Computer Science

Experiments

PlanetLab 160 machines hosted by 65 sites. Linux. We use University hosts in the U.S. Abilene.

Computer Science

Multirate Reliable Multicast

• Slow link does not impact the performance either at upstream nodes, or at nodes in other regions of the tree

Effect of Link Stress

Computer Science

Multirate Overlay Multicast

Effect of Link Stress

Computer Science

Throughput Improvement

BU UIUC

GT26.4ms

0.0049%

54.7Mbps

15.8ms

0.0295%

37.2Mbps

33ms

0.0236%

19.9Mbps

Computer Science

How to construct the overlay tree

Construct the single-source widest-path tree. Maximize the minimum per-hop available

bandwidth to every destination. Simple variant of Dijkstra’s algorithm.

Weight on each hop is the available bandwidth.

Select the unvisited node with the widest path from the source.

Path width is measured by the minimum of the weights on the path.

Computer Science

Maximizing Overall Throughput

Computer Science

Throughput Advantage

Computer Science

Conclusion

ROMA New architecture for reliable distribution of large

content across an overlay network using TCP. Multiple-rate reception. Minimal amount of resources at the application

layer. Provides ability to adaptively reconfigure the

topology. Provides ability to speed up downloads with

collaborative peer-to-peer transfers. [BCMR’02]

Analysis of chains of loosely coupled TCP TCP chains offer an opportunity to increase

performance.