maintaining the netflix front door - presentation at intuit meetup
DESCRIPTION
This presentation goes into detail on the key principles behind the Netflix API, including design, resiliency, scaling, and deployment. Among other things, I discuss our migration from our REST API to what we call our Experienced-Based API design. It also shares several of our open source efforts such as Zuul, Scryer, Hystrix, RxJava and the Simian Army.TRANSCRIPT
![Page 1: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/1.jpg)
Maintaining the Front Door to Netflix
Daniel Jacobson@daniel_jacobson
http://www.linkedin.com/in/danieljacobsonhttp://www.slideshare.net/danieljacobson
![Page 2: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/2.jpg)
Global Streaming Videofor TV Shows and Movies
![Page 3: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/3.jpg)
More than 48 Million Subscribers
More than 40 Countries
![Page 4: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/4.jpg)
Netflix Accounts for >34% of Peak Downstream Traffic in North America
Netflix subscribers are watching more than 1 billion hours a month
![Page 5: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/5.jpg)
Netflix Accounts for >6% of Peak Upstream Traffic in North America
Netflix subscribers are watching more than 1 billion hours a month
![Page 6: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/6.jpg)
![Page 7: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/7.jpg)
![Page 8: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/8.jpg)
Team Focus:Build the Best Global Streaming Product
Three aspects of the Streaming Product:• Non-Member • Discovery• Streaming
![Page 9: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/9.jpg)
The Netflix API - Background
![Page 10: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/10.jpg)
Netflix API
![Page 11: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/11.jpg)
Netflix API Requests by AudienceAt Launch In 2008
Netflix DevicesOpen API Developers
![Page 12: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/12.jpg)
![Page 13: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/13.jpg)
![Page 14: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/14.jpg)
Netflix API
![Page 15: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/15.jpg)
Netflix API Requests by AudienceFrom 2011
Netflix DevicesOpen API Developers
![Page 16: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/16.jpg)
Current Emphasis of Netflix API
Netflix Devices
![Page 17: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/17.jpg)
Netflix API : Key Responsibilities
• Broker data between services and Devices
• Provide features and business logic
• Maintain a resilient front-door
• Scale the system
• Maintain high velocity
• Provide detailed insights into the system health
![Page 18: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/18.jpg)
Netflix API : Key Responsibilities
• Broker data between services and Devices
• Provide features and business logic
• Maintain a resilient front-door
• Scale the system
• Maintain high velocity
• Provide detailed insights into the system health
![Page 19: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/19.jpg)
APIs DoLots of Things!
![Page 20: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/20.jpg)
Data Gathering
Data Formatting
Data Delivery
Security
Authorization
Authentication
System Scaling
Discoverability
Data Consistency
Translations
Throttling
Orchestration
APIs DoLots of Things!
These are some of themany things APIs do.
![Page 21: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/21.jpg)
Data Gathering
Data Formatting
Data Delivery
Security
Authorization
Authentication
System Scaling
Discoverability
Data Consistency
Translations
Throttling
Orchestration
APIs DoLots of Things!
These three are at the core.All others ultimately
support them.
![Page 22: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/22.jpg)
Definitions
• Data Gathering– Retrieving the requested data from one or many local
or remote data sources
• Data Formatting– Preparing a structured payload to the requesting agent
• Data Delivery– Delivering the structured payload to the requesting
agent
![Page 23: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/23.jpg)
Meanwhile…
There are two players in APIs
![Page 24: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/24.jpg)
API Provider API Consumer
![Page 25: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/25.jpg)
API Provider
PROVIDES
API Consumer
CONSUMES
Traditional API Interactions
![Page 26: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/26.jpg)
API Provider
PROVIDES EVERYTHING
API ConsumerCONSUMES
WHAT IS PROVIDED
Everything means, API Provider does:• Data Gathering• Data Formatting• Data Delivery• (among other things)
Traditional API Interactions
![Page 27: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/27.jpg)
Why do most API providers provide everything?
• API design tends to be easier for teams closer to the source
• Centralized API functions makes them easier to support
• Many APIs have a large set of unknown and external developers
![Page 28: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/28.jpg)
Why do most API providers provide everything?
• API design tends to be easier for teams closer to the source
• Centralized API functions makes them easier to support
• Many APIs have a large set of unknown and external developers
At Netflix, we see it a different way…
![Page 29: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/29.jpg)
Data Gathering Data Formatting Data Delivery
API Consumer
API Provider
Separation of Concerns
To be a better provider, the API should address the separation of concerns of the three core functions
![Page 30: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/30.jpg)
Data Gathering Data Formatting Data Delivery
API ConsumerDon’t care how data is gathered, as long
as it is gathered
API ProviderCare a lot about how the data is
gathered
Separation of Concerns
![Page 31: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/31.jpg)
Data Gathering Data Formatting Data Delivery
API ConsumerDon’t care how data is gathered, as long
as it is gathered
Each consumer cares a lot about the format for that specific use
API ProviderCare a lot about how the data is
gathered
Only cares about the format to the extent it
is easy to support
Separation of Concerns
![Page 32: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/32.jpg)
Data Gathering Data Formatting Data Delivery
API ConsumerDon’t care how data is gathered, as long
as it is gathered
Each consumer cares a lot about the format for that specific use
Each consumer cares a lot about how payload
is delivered
API ProviderCare a lot about how the data is
gathered
Only cares about the format to the extent it
is easy to support
Only cares about delivery method to the
extent it is easy to support
Separation of Concerns
![Page 33: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/33.jpg)
Because of our separation of concerns, the Netflix API team is
enabled to focus on different charters
![Page 34: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/34.jpg)
Brokering Data to 1,000+ Device Types
![Page 35: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/35.jpg)
![Page 36: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/36.jpg)
![Page 37: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/37.jpg)
Screen Real Estate
![Page 38: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/38.jpg)
Controller
![Page 39: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/39.jpg)
Technical Capabilities
![Page 40: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/40.jpg)
One-Size-Fits-AllAPI
Request
RequestRequest
Request
Request
Request
RequestRequest
Request
Request
RequestRequest
Request
Request
Request
Request
![Page 41: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/41.jpg)
Courtesy of South Florida Classical Review
![Page 42: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/42.jpg)
![Page 43: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/43.jpg)
Resource-Based API
vs.
Experience-Based API
![Page 44: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/44.jpg)
Resource-Based Requests
• /users/<id>/ratings/title• /users/<id>/queues• /users/<id>/queues/instant• /users/<id>/recommendations• /catalog/titles/movie• /catalog/titles/series• /catalog/people
![Page 45: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/45.jpg)
OSFA API
RECOMMENDATIONS
MOVIE DATA
SIMILAR MOVIES
AUTH MEMBERDATA
A/B TESTS
START-UP
RATINGS
Network Border Network Border
![Page 46: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/46.jpg)
RECOMMENDATIONS
MOVIE DATA
SIMILAR MOVIES
AUTH MEMBERDATA
A/B TESTS
START-UP
RATINGS
OSFA API
Network Border Network Border
SERVER CODE
CLIENT CODE
![Page 47: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/47.jpg)
RECOMMENDATIONS
MOVIE DATA
SIMILAR MOVIES
AUTH MEMBERDATA
A/B TESTS
START-UP
RATINGS
OSFA API
Network Border Network Border
DATA GATHERING,FORMATTING,AND DELIVERY
USER INTERFACERENDERING
![Page 48: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/48.jpg)
![Page 49: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/49.jpg)
![Page 50: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/50.jpg)
Experience-Based Requests
• /ps3/homescreen
![Page 51: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/51.jpg)
JAVA API
Network Border Network Border
RECOMMENDATIONS
MOVIE DATA
SIMILAR MOVIES
AUTH MEMBERDATA
A/B TESTS
START-UP
RATINGS
Groovy Layer
![Page 52: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/52.jpg)
![Page 53: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/53.jpg)
RECOMMENDATIONSA
ZXSXX C CCC
MOVIE DATA
SIMILAR MOVIES
AUTH MEMBERDATA
A/B TESTS
START-UP
RATINGS
JAVA API
SERVER CODE
CLIENT CODE
CLIENT ADAPTER CODE(WRITTEN BY CLIENT TEAMS, DYNAMICALLY UPLOADED TO SERVER)
Network Border Network Border
![Page 54: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/54.jpg)
RECOMMENDATIONSA
ZXSXX C CCC
MOVIE DATA
SIMILAR MOVIES
AUTH MEMBERDATA
A/B TESTS
START-UP
RATINGS
JAVA API
DATA GATHERING
DATA FORMATTINGAND DELIVERY
USER INTERFACERENDERING
Network Border Network Border
![Page 55: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/55.jpg)
![Page 56: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/56.jpg)
Netflix API : Key Responsibilities
• Broker data between services and Devices
• Provide features and business logic
• Maintain a resilient front-door
• Scale the system
• Maintain high velocity
• Provide detailed insights into the system health
![Page 57: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/57.jpg)
![Page 58: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/58.jpg)
1000+ Device Types
![Page 59: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/59.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies Reviews A/B Test
Engine
Dozens of Dependencies
![Page 60: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/60.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 61: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/61.jpg)
Dependency Relationships
![Page 62: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/62.jpg)
2,000,000,000Incoming Requests Per Day
to the Netflix API
![Page 63: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/63.jpg)
30Distinct Dependent
Services for the Netflix API
![Page 64: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/64.jpg)
~500Dependency jars Slurped
into the Netflix API
![Page 65: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/65.jpg)
14,000,000,000Netflix API Outbound Calls
Per Day to those Dependent Services
![Page 66: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/66.jpg)
0Dependent Services with
100% SLA
![Page 67: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/67.jpg)
99.99% = 99.7%30
0.3% of 2B = 6M failures per day
2+ Hours of Downtime Per Month
![Page 68: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/68.jpg)
99.99% = 99.7%30
0.3% of 2B = 6M failures per day
2+ Hours of Downtime Per Month
![Page 69: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/69.jpg)
99.9% = 97%30
3% of 2B = 60M failures per day
20+ Hours of Downtime Per Month
![Page 70: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/70.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 71: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/71.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 72: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/72.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 73: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/73.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 74: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/74.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 75: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/75.jpg)
![Page 76: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/76.jpg)
Circuit Breaker Dashboard
![Page 77: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/77.jpg)
![Page 78: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/78.jpg)
Call Volume and Health / Last 10 Seconds
![Page 79: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/79.jpg)
Call Volume / Last 2 Minutes
![Page 80: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/80.jpg)
Successful Requests
![Page 81: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/81.jpg)
Successful, But Slower Than Expected
![Page 82: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/82.jpg)
Short-Circuited Requests, Delivering Fallbacks
![Page 83: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/83.jpg)
Timeouts, Delivering Fallbacks
![Page 84: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/84.jpg)
Thread Pool & Task Queue Full, Delivering Fallbacks
![Page 85: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/85.jpg)
Exceptions, Delivering Fallbacks
![Page 86: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/86.jpg)
Error Rate# + # + # + # / (# + # + # + # + #) = Error Rate
![Page 87: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/87.jpg)
Status of Fallback Circuit
![Page 88: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/88.jpg)
Requests per Second, Over Last 10 Seconds
![Page 89: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/89.jpg)
SLA Information
![Page 90: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/90.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 91: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/91.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 92: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/92.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 93: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/93.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
Fallback
![Page 94: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/94.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
Fallback
![Page 95: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/95.jpg)
Netflix API : Key Responsibilities
• Broker data between services and Devices
• Provide features and business logic
• Maintain a resilient front-door
• Scale the system
• Maintain high velocity
• Provide detailed insights into the system health
![Page 96: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/96.jpg)
![Page 97: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/97.jpg)
Netflix API : Requests Per Month
Jan-10 Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 -
5
10
15
20
25
30
35
Requ
ests
in B
illio
ns
50x growth in 18 months
![Page 98: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/98.jpg)
![Page 99: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/99.jpg)
AWS Cloud
![Page 100: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/100.jpg)
![Page 101: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/101.jpg)
Netflix API : Requests Per Month
Jan-10 Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 -
5
10
15
20
25
30
35
Requ
ests
in B
illio
ns
![Page 102: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/102.jpg)
Autoscaling
![Page 103: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/103.jpg)
Autoscaling
![Page 104: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/104.jpg)
Scryer : Predictive Auto Scaling
Not yet…
![Page 105: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/105.jpg)
Typical Traffic Patterns Over Five Days
![Page 106: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/106.jpg)
Predicted RPS Compared to Actual RPS
![Page 107: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/107.jpg)
Scaling Plan for Predicted Workload
![Page 108: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/108.jpg)
What is Scryer Doing?
• Evaluating needs based on historical data– Week over week, month over month metrics
• Adjusts instance minimums based on algorithms
• Relies on Amazon Auto Scaling for unpredicted events
![Page 109: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/109.jpg)
Results
![Page 110: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/110.jpg)
Results : Load Average
ReactivePredictive
![Page 111: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/111.jpg)
Results : Response Latencies
ReactivePredictive
![Page 112: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/112.jpg)
Results : Outage Recovery
![Page 113: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/113.jpg)
Results : AWS Costs
![Page 114: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/114.jpg)
Scaling Globally
![Page 115: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/115.jpg)
More than 48 Million Subscribers
More than 40 Countries
![Page 116: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/116.jpg)
ZuulGatekeeper for the Netflix Streaming Application
![Page 117: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/117.jpg)
Zuul *
• Multi-Region Resiliency
• Insights• Stress Testing• Canary Testing• Dynamic Routing
• Load Shedding• Security• Static Response
Handling• Authentication
* Most closely resembles an API proxy
![Page 118: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/118.jpg)
All of these approaches are designed to prevent failures…
![Page 119: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/119.jpg)
But sometimes the best way to prevent failures is to force them!
![Page 120: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/120.jpg)
![Page 121: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/121.jpg)
I randomly terminate instances
in production to identify dormant
failures.
Chaos Monkey
![Page 122: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/122.jpg)
Chaos Gorilla
I simulate an outage of an
entire Amazon availability zone.
![Page 123: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/123.jpg)
I simulate an outage in an AWS
region.
Chaos Kong
![Page 124: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/124.jpg)
I find instances that don’t adhere to best practices.
Conformity Monkey
![Page 125: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/125.jpg)
I extend Conformity Monkey to find
security violations.
Security Monkey
![Page 126: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/126.jpg)
I detect unhealthy instances and remove them from service.
Doctor Monkey
![Page 127: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/127.jpg)
I clean up the clutter and waste that runs in the
cloud.
Janitor Monkey
![Page 128: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/128.jpg)
I induce artificial delays and errors into services to determine
how upstream services will respond.
Latency Monkey
![Page 129: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/129.jpg)
![Page 130: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/130.jpg)
Netflix API : Key Responsibilities
• Broker data between services and Devices
• Provide features and business logic
• Maintain a resilient front-door
• Scale the system
• Maintain high velocity
• Provide detailed insights into the system health
![Page 131: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/131.jpg)
Personalization
EngineUser Info Movie
MetadataMovie Ratings
Similar Movies
API
Reviews A/B Test Engine
![Page 132: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/132.jpg)
Dependency Relationships
![Page 133: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/133.jpg)
![Page 134: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/134.jpg)
Testing Philosophy:
Act Fast, React Fast
![Page 135: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/135.jpg)
That Doesn’t Mean We Don’t Test
![Page 136: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/136.jpg)
Automated Delivery Pipeline
![Page 137: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/137.jpg)
Cloud-Based Deployment Techniques
![Page 138: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/138.jpg)
Current Code
In Production
API Requests from the Internet
![Page 139: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/139.jpg)
Single Canary InstanceTo Test New Code with Production Traffic
(around 1% or less of traffic)
Current Code
In Production
API Requests from the Internet
![Page 140: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/140.jpg)
Canary Analysis Automation
![Page 141: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/141.jpg)
Single Canary InstanceTo Test New Code with Production Traffic
(around 1% or less of traffic)
Current Code
In Production
API Requests from the Internet
Error!
![Page 142: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/142.jpg)
Current Code
In Production
API Requests from the Internet
![Page 143: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/143.jpg)
Current Code
In Production
API Requests from the Internet
![Page 144: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/144.jpg)
Current Code
In Production
API Requests from the Internet
Perfect!
![Page 145: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/145.jpg)
Stress Test with Zuul
![Page 146: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/146.jpg)
Current Code
In Production
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 147: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/147.jpg)
Current Code
In Production
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 148: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/148.jpg)
Error!
Current Code
In Production
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 149: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/149.jpg)
Current Code
In Production
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 150: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/150.jpg)
Current Code
In Production
API Requests from the Internet
Perfect!
![Page 151: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/151.jpg)
Stress Test with Zuul
![Page 152: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/152.jpg)
Current Code
In Production
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 153: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/153.jpg)
Current Code
In Production
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 154: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/154.jpg)
API Requests from the Internet
New Code
Getting Prepared for Production
![Page 155: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/155.jpg)
https://www.github.com/Netflix
![Page 156: Maintaining the Netflix Front Door - Presentation at Intuit Meetup](https://reader035.vdocuments.site/reader035/viewer/2022062418/554f4408b4c905423f8b4769/html5/thumbnails/156.jpg)
Maintaining the Front Door to Netflix
Daniel Jacobson@daniel_jacobson
http://www.linkedin.com/in/danieljacobsonhttp://www.slideshare.net/danieljacobson