Murari Sridharan and Kun Tan

(Collaborators: Jingmin Song, MSRA & Qian Zhang, HKUST

Design goalsEfficiency

Improve throughput by efficiently using the spare capacity in the network

RTT fairnessIntra-protocol fairness when competing with

flows that have different RTTsTCP fairness

Must not impact performance of regular TCP flows sharing the same bottleneck

Stability

The Compound TCP approachSynergy between loss and delay based

approachesUsing delay to sense network congestionAdaptively adjust aggressiveness based on

network congestion levelOne flow, two components

Loss based component: cwnd (standard TCP Reno)

Scalable delay-based component: dwndTCP send window is controlled by

win = cwnd + dwnd

CTCP congestion controlVegas-like early congestion detector

Estimate the backlogged packets (diff) and compare it to a threshold, γ

Binomial increase when no congestion

Multiplicative decrease when loss

On detecting incipient congestionDecrease dwnd and yield to competing flows

ktwintwintwin )()()1(

1)()1( twintwin

CTCP congestion controlcwnd is updated as TCP Renodwnd control law

The above control law kicks in only when the flow is in congestion avoidance and cwnd >= 40 packets. No changes to slow start phase.

detected is loss if ,2/)1()(

if ,)(

if ,)1)(()(

)1(

cwndtwin

diffdifftdwnd

difftwintdwnd

tdwnd

k

Response function

10-10

10-9

10-8

10-7

10-6

10-5

10-4

10-3

10-2

10-1

10-1

100

101

102

103

104

105

106

107

108

109

Packet loss rate

Win

dow

CTCP (k=3/4)HSTCP TCP Reno STCP

CTCP window evolution

γ

D E F G

Loss Free Period

W(1-b)

Ws = cwnd + dwnd

Wm

B

dwnd = 0

cwnd

dwnd

Convergence and RTT fairnessTheorem 1: Two CTCP flows with same round

trip delay converge to fair share. Theorem 2: Let Th1 and Th2 present the

throughput of two CTCP flows with round trip times R1 and R2, respectively. Then, the following inequality satisfied

2

1

2

2

1

R

RTh

Th

u

CTCP

BTi

Regular TCP

l

m

TCP fairnessBandwidth stolen

Let P be the aggregated throughput of m regular TCP flows when they compete with l regular flows. Let Q be the aggregated throughput when competing with high-speed flows. The bandwidth stolen by high-speed protocol flows from regular TCP flows is

Theorem 3: CTCP is fair and will not steal bandwidth from competing flows when , where B is the bottleneck buffer size.

lm

B

P

QPBstolen

Effect of Gammaγ fixed at 30

packets. This works well on most

Delay component loses ability to detect early congestionAverage buffer

allocated for each flow is < γ

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5

Number of CTCP/HSTCP flows

Ba

nd

wid

th s

tole

n (

%)

CTCP HSTCP

Gamma tuning by emulationLoss based component of CTCP emulates the

behavior of regular TCP. The cwnd’s of competing flows converge and should be the same before hitting a packet loss.

At the end of every round, compute backlogged packets (Diff_reno) purely based on cwnd the loss based component.

On a packet loss, choose γ = 3/4 * Diff_reno. Update γ using an exponential moving average

Ensure γlow <= γ <= γhigh . Experimentally we have determined γlow = 5 , γhigh = 30

*)1(

Implementation & EvaluationWindows platform implementation

Microsecond resolution RTT timerDynamic memory management for sample

buffersDummyNet-based Test-bed

Iperf

Sender ReceiverRouter

Iperf

DummyNet

Giga EthernetSwitch

Giga EthernetSwitch

FreeBSD 5.3MS Windows MS Windows

Multi-supportTCP/IP stack

Multi-supportTCP/IP stack

Efficiency

0

100

200

300

400

500

600

700

0.01 0.001 0.0001 0.00001 0.000001 0

Packet loss rate

Th

rou

gh

pu

t (M

bp

s)

Regular TCP HSTCP CTCP

RTT fairnessInverse RTT ratio

1 2 3 6

Regular TCP 0.9 3.6 6.2 31.6

HSTCP 1 28.9 90.5 233.8

CTCP 1 2.2 4.1 9.5

TCP fairness – Effect of γ

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5

Number of CTCP/HSTCP flows

Ban

dwid

th s

tole

n (%

)

CTCP HSTCP CTCP-TUBE

0

100

200

300

400

500

600

700

800

900

1000

0.01 0.001 0.0001 0.00001 0.000001 0

Link packet loss rateT

hro

ug

hp

ut

(Mbp

s)

Regular TCP HSTCP CTCP-TUBE CTCP

TCP fairness – Varying buffer sizes

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

50 150 250 500 1000 2000

Buffer size (packets)

Ban

dwid

th s

tole

n

CTCP CTCP-TUBE

Intra-protocol fairness and convergence of γ

0

50

100

150

200

250

300

350

400

0 500 1000 1500 2000 2500 3000

Time (s)

Thr

ough

put

(Mbp

s)

0

5

10

15

20

25

30

35

0 500 1000 1500 2000 2500 3000

First 3 flows enter

Second 3 flows enter

Third 3 flows enter

Last 3 flows enter

Time (s)

Stability

Multiple bottlenecks – UtilizationM flows 2.5G/10ms 1G/30ms 2.5G/10ms

N flows K flows

N=400 TCP, K=50 TCPM is either 8 CTCP or TCP flows

0

500

1000

1500

2000

2500

3000

1 51 101 151 201 251 301

TCP CTCP

Multiple bottlenecks – Throughput

N=400 TCP, K=50 TCPM is either 8 CTCP or TCP flows

0

100

200

300

400

500

600

1 51 101 151 201 251

Th

rou

gh

pu

t (M

bp

s)

Time (s)

CTCP TCP

SummaryCTCP is a promising approach that achieves

good efficiency, RTT fairness and TCP fairness.

Implemented on Windows platform and verified the above properties in a range of environments. Validated on test-beds, Microsoft IT high-speed

links, Microsoft internal deployments, SLAC/Internet2/ESNet production links. (Refer to the “Testing TCP over Wide Area Networks” talk by Yee-Ting Lee)

ChallengesIn the absence of explicit feedback

Does delay correspond to congestion? Delay indicates incipient congestion which may or may not

result in packet loss. Is queue build up measureable at the end systems?

The granularity of the RTT samples are based off high-resolution hardware timers. Helps measure variations in baseRTT.

Can queueing delay be more accurately estimated than end-to-end loss probability? More fine grained rtt samples, loss events are rare.

Can we detect congestion in timescales of the order of RTT? Should we?