net flix kafka seattle meetup
TRANSCRIPT
![Page 1: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/1.jpg)
![Page 2: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/2.jpg)
●○○
●○○○
![Page 3: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/3.jpg)
●●●
![Page 4: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/4.jpg)
A NETFLIX ORIGINAL SERVICE
![Page 5: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/5.jpg)
Stream Consumers
Router
EMR
FrontingKafka
EventProducer
ConsumerKafka
Management
HTTPPROXY
![Page 6: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/6.jpg)
Fronting Kafka Clusters Consumer Kafka Clusters
Number of clusters 24 15
Total number of instances 1700+ 1100+
Instance type d2.2xl i2.2xl
Replication factor 2 2
Retention period 8 to 24 hours 2 to 4 hours
![Page 7: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/7.jpg)
●○○○
●○
![Page 8: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/8.jpg)
●
○●
![Page 9: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/9.jpg)
●○
●
●●
○○
![Page 10: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/10.jpg)
●●
●
0.1% 0.5% 1% 5% Percent loss
![Page 11: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/11.jpg)
●●
○○
●
![Page 12: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/12.jpg)
●●
●
![Page 13: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/13.jpg)
●○○
● …
![Page 14: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/14.jpg)
![Page 15: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/15.jpg)
●
○ →
![Page 16: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/16.jpg)
●●
○○○ →
●
![Page 17: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/17.jpg)
![Page 18: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/18.jpg)
●●
○○○
![Page 19: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/19.jpg)
●○○○
●●
![Page 20: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/20.jpg)
●○ →
●○ →
●○ →
![Page 21: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/21.jpg)
●●
○●
○
![Page 22: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/22.jpg)
Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 0 1 1 2 2 3
N = Partition N for a topic with 2 replicas
0 ← Off line partition
![Page 23: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/23.jpg)
Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 1 2 0 1 2 3
N = Partition N for a topic with 2 replicas
No offline partition
![Page 24: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/24.jpg)
●
●○
●
●
![Page 25: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/25.jpg)
●○○○
●○○
![Page 26: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/26.jpg)
EventProducer
Kafka
Buffer exhausted and message drop Slow replication
Broker with networking problem
Disk read causes slow responses
X
X
X
![Page 27: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/27.jpg)
![Page 28: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/28.jpg)
28
![Page 29: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/29.jpg)
29
![Page 30: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/30.jpg)
●●
○○○
![Page 31: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/31.jpg)
●○○○
![Page 32: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/32.jpg)
●○○
●●●
![Page 33: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/33.jpg)
![Page 34: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/34.jpg)
●●
●●
![Page 35: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/35.jpg)
RouterFrontingKafka
EventProducer
XConsumer
Kafka
Copy topic metadata
Consumer
![Page 36: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/36.jpg)
●
Fully Automated
![Page 37: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/37.jpg)
![Page 38: Net flix kafka seattle meetup](https://reader031.vdocuments.site/reader031/viewer/2022020103/5aaae9be7f8b9a586f8b47a1/html5/thumbnails/38.jpg)