netpilot: automating datacenter network failure mitigation xin wu, daniel turner, chao-chih chen,...

NetPilot: Automating Datacenter Network Failure Mitigation

Xin Wu, Daniel Turner, Chao-Chih Chen, David A. Maltz, Xiaowei Yang, Lihua Yuan, Ming Zhang

Presented by: Chen Li

2

Failures are Common and Harmful

• Network failures are common

10,000+ switches

3

Failures are Common and Harmful

• Network failures are common

• Failures cause long down times

Time from detection to repair (minutes)

Six-month failure logs of production datacenters

25% of failures take 13+ hours to repair

4

Failures are Common and Harmful

• Failures are common due to VERY large datacenters

• Failures cause long down times

• Long failure duration large revenue loss

How to Shorten Failure Recovery Time?

6

Previous Work

• Conventional failure recovery takes 3 steps

• Failure localization/diagnosis– [M. K. Aguilera, SOSP’03]– [M. Y. Chen, NSDI’04]– [R.R Kompella, NSDI ’05]– [P.Bahl, SIGCOMM’07]– [S. Kandula, SIGCOMM’09]…

Detection Diagnosis Repair

passiveping

active

7

Automating Failure Diagnosis is Challenging

• Root causes are deep in network stack

• Diagnosis involves multiple parties

8

Category Failure types Diagnosis & Repair

%

Software 21% Link layer loop Find and fix bugs

19%Imbalance overload 2%

Hardware 18% FCS error Replace cable 13%Unstable power Repair power 5%

Unknown 23% Switch stops forwarding N/A 9%Imbalance overload 7%Lost configuration 5%High CPU utilization 2%

Configuration 38%

Errors on multiple switches

Update configuration

32%

Errors on one switch 6%

• Six-month failure logs from several production DCNs

1. Root causes are deep in the network stack

2. Diagnosis involves multiple partiesFailure Diagnosis Requires

Human Intervention !

Can we do something other than failure diagnosis?

10

NetPilot: Mitigating rather than Diagnosing Failures

• Mitigate failure symptoms ASAP, at the cost of reduced capacity

Detection Diagnosis RepairAutomated Mitigation

11

NetPilot Benefits

• Short recovery time• Small network disruption• Low operation cost

Automated Mitigation

Detection Diagnosis Repair

12

Failure Mitigation is Effective

• Most failures can be mitigated by simple actions

• Mitigation is feasible due to redundancy

13

Category Failure types Mitigation Repair %Software 21%

Link layer loop Deactivate port Find and fix bugs

19%

Imbalance-triggered overload

Restart switch2%

Hardware 18%

FCS error Deactivate port Replace cable 13%

Unstable power Deactivate switch Repair power 5%Unknown 23%

Switch stops forwarding

Restart switch N/A 9%

Imbalance-triggered overload

Restart switch 7%

Lost configuration Restart switch 5%High CPU utilization

Restart switch 2%

Configuration 38%

Errors on multiple switches

n/a Update configuration

32%

Errors on single switch

Deactivate switch 6%

68% of failures can be mitigated by simple actions

14

Mitigation Made Possible by Redundancy

• Redundancy deactivation unlikely to partition / overload the network

ToR

AGG

CORE

Internet

15

Outline

• Automating failure diagnosis is challenging

• Failure mitigation is effective

• How to automate mitigation?

• NetPilot evaluations

• Conclusion

16

A Strawman NetPilot: Trial-and-error

Network failure

Roll back if necessary

No Failure mitigated? End

Yes

Execute an action

Localization

17

NetPilot: Challenges & Solutions

1. Blind trial-and-error takes a long time

Network failure

Roll back if necessary

NoFailure

mitigated? EndYes

Execute an action

LocalizationLocalization

Failure specific localization

18

NetPilot: Challenges & SolutionsNetwork failure

Roll back if necessary

NoFailure

mitigated? EndYes

Execute an action

Localization

Estimate impact

Localization

2. Partition/overload network

Impact estimation

19

NetPilot: Challenges & SolutionsNetwork failure

Roll back if necessary

NoFailure

mitigated? EndYes

Execute an action

Localization

Estimate impact

Rank actions

Localization

3. Different actions have different side-effects

Rank actions based on impact

20

Failure Specific Localization• Limited # of failure types• Domain knowledge improves accuracy

Failure types1. Link layer loop2. Imbalance-triggered overload3. FCS error4. Unstable power5. Switch stops forwarding6. Imbalance-triggered overload

7. Lost configuration8. High CPU utilization9. Errors on multiple switches

10. Errors on single switch

21

Example: Frame Check Sequence (FCS) Errors

• 13% of all the failures• Cut-through switching– Forward frames before checksums are verified

• Increase application latency

22

Localizing FCS Errorserror frames seen on L frames corrupted by L

frames corrupted by other links & traverse L

• xL: link corruption rate

• # of variables = # of equations = # of links

• Corrupted links: xL> 0

23

NetPilot OverviewNetwork failure

Roll back if necessary

NoFailure

mitigated? EndYes

Execute an action

Localization

Estimate impact

Rank actions

24

Impact Metrics

• Derived from Service Level Agreement (SLA)– Availability: online_server_ratio– Packet loss: total_lost_pkt– latency: max_link_utilization• Small link utilization small (queuing) delay

• Total_lost_pkt & max_link_utilization derived from utilization of individual links

25

Estimating Link Utilization

• # of flows >> redundant paths– Traffic evenly distributed under ECMP

• Estimate the load contributed by each flow on each link

• Sum up the loads to compute utilization

Impact Estimator

Action

Traffic Link utilization Topology

26

Link Utilization Estimation is Highly Accurate

• 1-month traffic from a 8000-server network– Log socket events on each server

• Ground truth: SNMP counters

27

NetPilot OverviewNetwork failure

Roll back if necessary

NoFailure

mitigated? EndYes

Execute an action

Localization

Estimate impact

Rank actionsChoose the action with the least impact

28

Outline• Automating failure diagnosis is challenging

• Failure mitigation is effective

• How to automate mitigation? – Localization impact estimation ranking

• NetPilot evaluations–Mitigating load imbalance–Mitigating FCS errors–Mitigating overload

• Conclusion

29

Load Imbalance

• Agga stops receiving traffic • Localize to 4 suspects

corea

Agga

coreb

Aggb

30

Mitigating Load Imbalance

0:00 0:05 0:10 0:15 0:20 0:250

5000000000100000000001500000000020000000000250000000003000000000035000000000

lag core_a->AR_a lag core_a->AR_b

lag core_b->AR_a lag core_b->AR_b

Time (minutes)

corea -> agga

coreb -> agga

corea -> aggb

coreb -> aggb

Agga stops receiving traffic

Detected & reboot coreb

Reboot corea Reboot Agga

Mitigation confirmed

Load evenly splitted

corea

Agga

coreb

Aggb

31

Fast FCS Error Mitigation

NetPilot:deactivates 2 links in 1 trial

within 15 minutes

Human operator:

after 11 trials in 3.5 hours, 2 out of 28 ports are deactivated

3.5 hours 15 minutes

32

Mitigating Link Overload• Mitigate overload by deactivating healthy links

core1

1.5 1.5

3

agg

core2

core1

agg

33

Mitigating Link Overload• Mitigate overload by deactivating healthy links– Many candidate links in production networks– Choose the link(s) with the least impact

core1

1.5 1.5

3

agg

core2 core1

1 1.5

3

agg

core2 core1

0 3

3

agg

core2

lost 0.5

34

Action Ranking Lowers Link Utilization

• Replay 97 overload incidents due to link failures

35

Conclusion

• Mitigation shortens failure recovery time– Simple actions are effective– Made possible by redundancy

• NetPilot: automating failure mitigation – Recovery time: hour minutes– Several mitigation scenarios deployed in Bing

36

Thank You!

Detection Diagnosis RepairNetPilot:

Automated Mitigation

netpilot@microsoft.com