the art of chaos engineering failure at netflix velocity · cannot connect to the netflix service....
TRANSCRIPT
![Page 1: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/1.jpg)
FAILURE AT NETFLIX VELOCITY
THE ART OF CHAOS ENGINEERING
![Page 2: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/2.jpg)
![Page 3: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/3.jpg)
![Page 4: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/4.jpg)
![Page 5: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/5.jpg)
![Page 6: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/6.jpg)
Cannot Connect to the Netflix Service
![Page 7: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/7.jpg)
![Page 8: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/8.jpg)
IMPACT LATENCY
0%
0Ms
![Page 9: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/9.jpg)
IMPACT LATENCY
1%
50Ms
![Page 10: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/10.jpg)
IMPACT LATENCY
50%
50Ms
![Page 11: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/11.jpg)
IMPACT LATENCY
100%
50Ms
![Page 12: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/12.jpg)
IMPACT LATENCY
25%
250Ms
![Page 13: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/13.jpg)
IMPACT LATENCY
50%
250Ms
![Page 14: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/14.jpg)
IMPACT LATENCY
75%
250Ms
![Page 15: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/15.jpg)
IMPACT LATENCY
100%
250Ms
![Page 16: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/16.jpg)
IMPACT LATENCY
50%
500Ms
![Page 17: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/17.jpg)
IMPACT LATENCY
0%
500Ms
![Page 18: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/18.jpg)
IMPACT LATENCY
0%
0Ms
![Page 19: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/19.jpg)
IMPACT LATENCY
-50%-500
Ms
![Page 20: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/20.jpg)
IMPACT LATENCY
0%
0Ms
![Page 21: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/21.jpg)
HYPOTHESIS
PROVEN
![Page 22: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/22.jpg)
LEARNINGSApplication Behavior
Blast Radius
Consistency
![Page 23: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/23.jpg)
You already know all that
![Page 24: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/24.jpg)
REMEMBER THE
GRAY AREA?
![Page 25: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/25.jpg)
{Systems | People}
don’t react predictably when exposed
to non-typical behavior.
![Page 26: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/26.jpg)
INCIDENT MANAGEMENT
![Page 27: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/27.jpg)
WHAT’S THE BIG DEAL? FIX IT AND MOVE ON!
![Page 28: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/28.jpg)
INCIDENTS ARE:Expensive
Stressful
Potentially Public
![Page 29: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/29.jpg)
INCIDENTS ARE:Expensive
Stressful
Potentially Public
![Page 30: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/30.jpg)
INCIDENTS ARE:Expensive
Stressful
Potentially Public
![Page 31: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/31.jpg)
INCIDENTS ARE:Expensive
Stressful
Public
![Page 32: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/32.jpg)
![Page 33: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/33.jpg)
![Page 34: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/34.jpg)
![Page 35: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/35.jpg)
WHAT CAUSES INCIDENTS?
![Page 36: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/36.jpg)
CHANGE
![Page 37: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/37.jpg)
THINGS WE CAN’T CONTROL
![Page 38: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/38.jpg)
THINGS WE DON’T CONTROL
![Page 39: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/39.jpg)
THINGS WE CAN CONTROL
![Page 40: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/40.jpg)
THAT ALL SOUNDS REALLLLLY NICE
![Page 41: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/41.jpg)
HOW?
![Page 42: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/42.jpg)
INCIDENT MANAGEMENT GOALS
Short
Unique
Never Forget
![Page 43: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/43.jpg)
![Page 44: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/44.jpg)
FOUR PARTS
![Page 45: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/45.jpg)
1 ENGAGEMENT
![Page 46: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/46.jpg)
2 COMMUNICATIONS
![Page 47: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/47.jpg)
3 COORDINATION
![Page 48: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/48.jpg)
4 MEMORIALIZATION
![Page 49: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/49.jpg)
TECHNIQUES
![Page 50: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/50.jpg)
SPECIALISTS
![Page 51: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/51.jpg)
BEFORE INCIDENTS
Education
Best Practices
Drilling
![Page 52: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/52.jpg)
DURING INCIDENTS
Incident Leader
![Page 53: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/53.jpg)
AFTER INCIDENTSMemorialization
Research Coordination
Incident Review
![Page 54: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/54.jpg)
NETFLIX INCIDENTS MANGERS
How We Think
![Page 55: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/55.jpg)
WE EXPECTFailure
Incidents
Empathy
![Page 56: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/56.jpg)
INCIDENT MANAGEMENT CONTEXT
People
![Page 57: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/57.jpg)
“Human operators have dual roles:
producers of & defenders against failure”
— Richard I Cook
![Page 58: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/58.jpg)
INCIDENT MANAGEMENT CONTEXT
Failure
![Page 59: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/59.jpg)
Constant Failures
Degraded is Normal
Any Change is a Gamble
![Page 60: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/60.jpg)
INCIDENT MANAGEMENT CONTEXT
Biases
![Page 61: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/61.jpg)
Hindsight
Single Attribution
Anchoring
![Page 62: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/62.jpg)
MEASURING SUCCESSShort & Less Impact
More Incidents
Better Team Engagement
![Page 63: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms](https://reader033.vdocuments.site/reader033/viewer/2022042000/5e6d2c9db00e4163856195fc/html5/thumbnails/63.jpg)
FAILURE AT NETFLIX VELOCITY
THE ART OF CHAOS ENGINEERING
Dave Hahn — Senior SRE, Netflix
@relix42