fast and cautious: leveraging multi-path diversity for ... · fast and cautious: leveraging...

Fast and Cautious:Leveraging Multi-path Diversity

for Transport Loss Recovery in Data Centers

Guo ChenYuanwei Lu, Yuan Meng, Bojie Li, Kun Tan, Dan Pei, Peng Cheng, Layong (Larry) Luo, Yongqiang Xiong,

Xiaoliang Wang, and Youjian Zhao

Motivationn Services care about the tail flow completion time (tail FCT)

¨ Large number of flows generated in each operation¨ Overall performance governed by the last completed flows

16/6/25 2

Large-scale web applicationhosted in

Data Center Network (DCN)

AppLogic

Respondingtoauserrequest

Motivationn Services care about the tail flow completion time (tail FCT)

¨ Large number of flows generated in each operation¨ Overall performance governed by the last completed flows

n But packet loss hurts tail FCT¨ Real case in a Microsoft Azure’s DCN

16/6/25 3

Spine switch 2% random drop rate -->

increase of 99th percentile latency of all users

DCNtaillatencyvisualization[Pingmesh (SIGCOMM’15)]

(a)Normal (b)Spinefailure

Outlinen Motivationn Packet Loss in DCNn Impact of Packet Lossn Challenge for Loss Recoveryn FUSO Designn Evaluationn Summary

16/6/25 4

Lossrateandlocationdistributionoflossy links(lossrate>1%)

Mean loss rate 4%

78% above ToR

Similar in 5 days

Packet Loss in DCN

16/6/25 5

1) Lossfrequently happens(theoveralllossrateislow)2) Mostlosseshappeninthenetworkinsteadoftheedge

n Loss characteristics¨ Measured in a Microsoft production DCN during Dec. 1st-5th, 2015

Packet Loss in DCNn Reasons causing loss

¨ Congestion lossØ Uneven load-balance

Ø Incast

¨ Failure lossØ Silent random drop

Ø Packet black-hole

Bursty;Transient

16/6/25 6

Complex;Hardtodetect

Greatly mitigated(e.g., 1%->0.01%)[Jupiter Rising SIGCOMM’15]

Common& Huge impacton performance[Pingmesh SIGCOMM’15]

Outlinen Motivationn Packet Loss in DCNn Impact of Packet Loss

¨ Why loss hurts the tail?¨ How hard loss hurts?

n Challenge for Loss Recoveryn FUSO Designn Evaluationn Summary

16/6/25 7

How TCP Handles Loss?n Fast recovery

¨ Wait for certain number of DACKs to detect the loss and retransmit

Ack1-2

DupAck3

Retran3

Sender Receiver

Ack1-2

Retran3

Timeout

Sender Receiver

n Timeout (RTO)¨ If not enough DACKs return, retransmit

after a timeout

RTO>> RTTe.g.RTO=5ms,RTT<100us[Pingmesh (SIGCOMM’15),DCTCP(SIGCOMM’10)]

Ack1-2

Retran3

Timeout

Sender Receiver

n Timeout (RTO)¨ If not enough DACKs return, retransmit

after a timeout

RTO>> RTTe.g.RTO=5ms,RTT<100us[Pingmesh (SIGCOMM’15),DCTCP(SIGCOMM’10)]

Encountering one RTO àdramatically increase the FCT

Timeout probability of flows with different sizes passing a path with different packet loss rate

10KB(testbed) 100KB(testbed)

100KB(analysis)

10KB(analysis)

Loss Incurs Timeout

1. 1% loss à more than 1% flows timeout2. Larger flows (e.g. 100KB)

a. timeout ratio sharply grows when loss rate > 1%16/6/25 11

99th FCT>RTO3%lossà ~10%timeout

n A little loss causes enough timeout to hurt the tail FCT

Timeout probability of flows with different sizes passing a path with different packet loss rate

10KB(testbed) 100KB(testbed)

100KB(analysis)

10KB(analysis)

Loss Incurs Timeout

1. 1% loss à more than 1% flows timeout2. Larger flows (e.g. 100KB)

a. timeout ratio sharply grows when loss rate > 1%16/6/25 12

99th FCT>RTO3%lossà ~10%timeout

n A little loss causes enough timeout to hurt the tail FCT

To avoid RTO

16/6/25 13

Challenge for TCP Loss Recoveryn Prior works add aggressiveness to congestion control to do

loss recovery before timeout (RTO)¨ Tail Loss Probe (TLP)

Ø transmit one prober after 2RTT

¨ Instant Recovery (TCP-IR)Ø generate an FEC packet for every group of packets (up to 16)

Ø FEC packets also act as probers, delayed 1/4RTT before sent

¨ Proactive/RepFlowØ Duplicate every packet/flow

16/6/25 14

[SIGCOMM’13,RFC5827]

[SIGCOMM’13,INFOCOM’14]

Challenge for TCP Loss Recoveryn How long to wait before sending recovery packets?

¨ For congestion lossØ Should delay enough in case of worsening congestion

16/6/25 15

Bursty：Lead to multiple consecutive losses

[Incast (WREN’09),DCTCP(SIGCOMM’10)]

Challenge for TCP Loss Recoveryn How long to wait before sending recovery packets?

¨ For congestion lossØ Should delay enough in case of worsening congestion

¨ For failure loss such as random dropØ Should recover as fast as possible, otherwise already increase the FCT

16/6/25 16

• Wait 2RTT is too costly• Accurate & high-precision RTT measurement is challenging

[TLPSIGCOMM’13,RFC5827]

How to accelerate loss recovery as soon as possible, under various loss conditions without causing congestion?

Brief Summaryn Loss easily incurs timeout to hurt the tailn To prevent timeout, prior works add fixed aggressiveness to

recover loss before timeoutn Hard to adapt to various loss conditions

¨ Should be fast for failure loss¨ Should be cautious for congestion loss

16/6/25 17

16/6/25 18

n Utilize the “good” paths to proactively conduct loss recovery for “bad” paths¨ Leveraging path diversity (multiple paths; a few encounter loss)

n Fast and Cautious¨ Fast

Ø Proactive (immediate) recovery for potential packet loss utilizing sparetransmission opportunity

¨ CautiousØ Strictly follow congestion control without adding aggressiveness

FUSO: Fast Multi-path Loss Recovery

16/6/25 19

ReceiverSender

Multi-path Transport Background

16/6/25 20

CWND2CWNDtotal

Multi-pathCongestionControl

DataDistribution

Sub-flows:Implicitly/Explicitlymappingtophysicalpaths

ReceiverSender

16/6/25 21

CWNDtotal

P2P3P4P5 P1

ReceiverSender

16/6/25 22

CWNDtotal

P2P3P4P5

ReceiverSender

16/6/25 23

CWNDtotal

P2P3P4P5

ReceiverSender

16/6/25 24

CWNDtotal

P2P3P4P5 P1

ReceiverSender

16/6/25 25

CWNDtotal

P2P3P4P5 P1

ReceiverSender

16/6/25 26

CWNDtotal

P3P4P5 P1P2

ReceiverSender

16/6/25 27

CWNDtotal

P4P5 P1

ReceiverSender

16/6/25 28

CWNDtotal

P4P5 P1

ReceiverSender

16/6/25 29

CWNDtotal

P4P5 P1P3P2

ReceiverSender

16/6/25 30

CWNDtotal

P4P5 P1P3P2

ReceiverSender

16/6/25 31

CWNDtotal

P4P5 P1P3P2

ReceiverSender

16/6/25 32

CWNDtotal

P1P3P2

ReceiverSender

16/6/25 33

CWNDtotal

P1P3P2

ReceiverSender

16/6/25 34

CWNDtotal

P1P3P2

ReceiverSender

16/6/25 35

CWNDtotal

P1P3P2

ACKP1P4P5

ACKP4&P5

ReceiverSender

16/6/25 36

CWNDtotal

P3P4P5 P1P2

ReceiverSender

16/6/25 37

CWNDtotal

P3P4P5 P1P2

LostSpareCWND

Nonewdata

ReceiverSender

16/6/25 38

CWNDtotal

P3P4P5 P1P2

Proactivelossrecovery

ReceiverSender

16/6/25 39

CWNDtotal

P3P4P5 P1P2

“Worst”sub-flow

“Best”sub-flow

ReceiverSender

16/6/25 40

CWNDtotal

P3P4P5 P1P2

“Worst”sub-flow

“Best”sub-flow

ReceiverSender

16/6/25 41

CWNDtotal

P3P4P5 P1P2

“Worst”sub-flow

“Best”sub-flow

ReceiverSender

16/6/25 42

CWNDtotal

P3P4P5 P1P2

“Worst”sub-flow

“Best”sub-flow

ReceiverSender

16/6/25 43

CWNDtotal

P3P4P5 P1P2

“Worst”sub-flow

“Best”sub-flow

ReceiverSender

Standard MPTCP

16/6/25 44

CWNDtotal

P3P4P5 P1P2

LostRetransmitafteranRTO

Sender

FUSO: Path Selection

16/6/25 45

CWNDtotal

Possibility of encountering loss

Sender

16/6/25 46

CWNDtotal

“Worst”sub-flow

n “Worst” Sub-flow¨ With un-ACKed data¨ Most likely having loss

Un-ACKed data

Sender

16/6/25 47

CWNDtotal

“Worst”sub-flow

n “Worst” Sub-flow¨ With un-ACKed data¨ Most likely having loss

n “Best” Sub-flow¨ With spare CWND¨ Least likely having loss

Spare CWND

“Best”sub-flow

FUSO in 1 Slide

n If (spare CWND) && (no new data)¨ Utilize the transmission opportunity to proactively recover¨ Use “good” paths to help “bad” paths

n Multi-path diversity offers many transmission opportunities¨ “Good” paths have spare window

16/6/25 48

AppData...

MultipathCongestionControl

SendtobestSub-Flow

P5P6P7R

Sparewindow

Un-ACKeddata

Sender

AppData

Recover

Receiver

Sub-Flow1

Sub-FlowN

Sub-Flow2P3

Sub-Flow1

Sub-Flow2

Sub-FlowN

Recoverypackets

FUSO Implementationn Implemented in Linux kernel; ~900 lines of code

16/6/25 49

https://github.com/1989chenguo/FUSO

16/6/25 50

Testbed Settingsn Network

¨ 1Gbps fabric & 1Gbps hosts; ECMP routing; ECN enabledn TCP

¨ Init_cwnd=16; min_RTO=5ms

16/6/25 51

99th FCT % of flows encountering

timeout

better

Testbed Resultsn Failure loss

¨ Random-drop

16/6/25 52

Reducing 99th FCT up to ~82.3%

Reducing the timeout flows up to 100%

Loss rate:0.125%-4%

Latency-sensitiveflows

better

Testbed Resultsn Congestion loss

¨ Incast

16/6/25 53

Concurrentresponses

Performs the best

Cautious

Testbed Resultsn Failure loss & Congestion loss

¨ From failure-loss-dominated to congestion-loss-dominated

16/6/25

Loss rate:2%

Adapt to various loss condition

better

Background long flows

Larger-scale Simulationsn Simulation settings

¨ NS2 simulator; 3-layer, 4-port FatTree

¨ 40Gbps fabric, 10Gbps host; 64 hosts, 20 switches

¨ Empirical failure generation

16/6/25 55

Backgroundlong flows

Random failure

better

Larger-scale Simulationsn Simulation settings

¨ NS2 simulator; 3-layer, 4-port FatTree fabric¨ 40Gbps fabric, 10Gbps host; 64 hosts, 20 switches¨ Empirical failure generation

16/6/25 56

Reducing the average FCT up to ~60.3%

Reducing the 99th FCT up to ~87.4%

16/6/25 57

Summaryn Loss hurts tail latency

¨ Loss is not uncommon¨ A little loss leads to enough timeout, hurting the tail

n Challenges for loss recovery¨ How to accelerate loss recovery under various loss conditions without

causing congestion?n Philosophy for FUSO

¨ To be fast & cautious are equally important¨ Fast: Proactive loss recovery utilizing spare transmission opportunity,

leveraging multipath diversity ¨ Cautious: Strictly follows congestion control without adding

aggressiveness16/6/25 58

Thanks

fast and cautious: leveraging multi-path diversity for ... · fast and cautious: leveraging...

Documents

cautious optimism: adopting blockchain to improve canadian...

be cautious before buy real soundcloud followers

“a cautious, sober love affair with humanity:”

treasury priorities 2014: cautious optimismcitigroup.com ›...

a cautious conservationists view of stewardship contracting

bicycle rules: safe, aware, and cautious

fast and cautious: leveraging multi-path diversity for...

difference between 'careful and cautious

concerned/cautious #1

2012 kpmg-retail-outlook-cautious-optimism-in-style

a cautious balance - soas university of london

indonesia: cautious calm in ambon - eth z indonesia --...

the cautious caterpillar

mechanisms,principles,and lorentz’s cautious...

vocabulary. mysterious, reserved, guarded, cautious...

marlborough cautious...

yuanwei lu 1,2 , peng cheng 2, jiansong zhang , enhongchen...

national happiness and genetic distance: a cautious...

7im passive moderately cautious model portfolio pdf...

south yemen under hasani: cautious pragmatism