c* capacity forecasting (ajay upadhyay, jyoti shandil, arun agrawal, netflix) | cassandra summit...
TRANSCRIPT
![Page 1: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/1.jpg)
Capacity Forecast @ ScaleCDE, Cloud Database EngineeringNetflix.
![Page 2: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/2.jpg)
●CDE, Cloud Database Engineering ●Providing data stores as a service
○Cassandra,○ Dynomite, ○ Elasticsearch and RDS
Ajay Upadhyay Cloud Data Architect @ Netflix
Arun AgrawalSr. Software Engineer @
Netflix
Who are we?
![Page 3: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/3.jpg)
●Cassandra @ Netflix●Cassandra footprint ●Capacity planning lifecycle
●Forecasting the capacity
●Q and A
Agenda
![Page 4: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/4.jpg)
• 98% of streaming data is stored in Cassandra
• Data ranges from customer details to Viewing history / streaming bookmarks to billing and payment
Cassandra @ Netflix
![Page 5: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/5.jpg)
Cassandra Footprint
Hundreds C*
![Page 6: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/6.jpg)
Cassandra Footprint
Thousands
![Page 7: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/7.jpg)
Capacity Planning
•Able to predict
– Current usage and available capacity
– Resources needing upgrade– Life cycle of current configuration– Appropriate configuration for new
and existing App/Service
•Optimize – Under or over utilized resource– Increased business productivity
![Page 8: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/8.jpg)
Capacity Planning
Avoid:
• Impact on Business • No service or SLA
disruption• Un-planned
maintenance• Firefighting
![Page 9: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/9.jpg)
Life Cycle
Capture Requirement
RequirementAnalysis/
feasibility
Proxy or Simulate
Requirement
Monitoring /
Trending
New / Increased
traffic Optimization
![Page 10: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/10.jpg)
Capture Requirement
– IOPs and SLA– Maintenance overhead– Failover – Access pattern
![Page 11: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/11.jpg)
IOPs and SLAQuestions Response
Read OPS/sec [avg, peak] 5k - 10kRead Latency requirement 95th - 20ms
99th - 100ms Write OPS/sec [avg, peak] 1k - 2kWrite Latency requirement 95th - 20ms
99th - 100msNum Columns / Row 100
Avg col size / or avg row size 64kNum of rows 100 Mil
TTL [life Cycle of data] 365 Days
Data storeC*
Gutenberg publisher service
Gutenberg publisher serviceReadWrite
![Page 12: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/12.jpg)
Maintenance Overhead
Repairs / Compactions Y/N
Node replacement Y
Backup - Full / Incrementals
Y/N
TypeRespons
e
![Page 13: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/13.jpg)
Failover
Region Failover Y/N
SLA in case of region failover
Y/N
Questions Response
![Page 14: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/14.jpg)
Access Pattern
Questions ResponseRead Point read
All row readersColumn slices
Write Part existing rowNew rows
![Page 15: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/15.jpg)
Proxy/Simulate Traffic
– Proxy existing traffic – Simulate traffic
–NDBench– Generate actual /
synthetic traffic before final deployment using app
![Page 16: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/16.jpg)
Optimization
• Cache - Application level- Fronting cache engine before C*
- Stagger R - W operations if possible
![Page 17: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/17.jpg)
Cluster Sharding
![Page 18: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/18.jpg)
Trend AnalysisContinuous monitoring / trending on usage pattern
![Page 19: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/19.jpg)
New / Increased TrafficCapacity planning cycle begins
Capture
Requirement
RequirementAnalysis/
feasibility
Proxy or Simulate
Requirement
Monitoring /
Trending
New / Increased
traffic
Optimization
![Page 20: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/20.jpg)
Capacity Forecasting
![Page 21: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/21.jpg)
Arun AgrawalSr. Software Engineer
![Page 22: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/22.jpg)
Demo
![Page 23: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/23.jpg)
![Page 24: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/24.jpg)
![Page 25: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/25.jpg)
Metrics
Atlas
Previous Architecture
![Page 26: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/26.jpg)
Pain Points
•No support for complex relationships
•Hardware failure could fail leading to false positives
![Page 27: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/27.jpg)
Winston• Bridge between atlas and oncall• Complex relationship modeling
between metrics• Reduce false positives• Auto remediation platform
![Page 28: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/28.jpg)
Lesson Learnt•It might be already too late to fix the system.
•Reactive than proactive
![Page 29: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/29.jpg)
Requirements• Show us trend for the clusters. • Warn us of what is coming if
trend continues.• Give us time to scale their
cluster
![Page 30: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/30.jpg)
![Page 31: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/31.jpg)
Automic (UC4)
Architecture
![Page 32: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/32.jpg)
Aggregation• Daily • Instance Level• Cluster Level
•Instance Failures•Adding capacity over days
![Page 33: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/33.jpg)
Growth Criteriaf(x) of – Subscriber – Netflix content– # Viewing Sessions
![Page 34: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/34.jpg)
![Page 35: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/35.jpg)
![Page 36: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/36.jpg)
ARIMA– AR
•Regression on prior values–I•Data values are replaced with (x(i) - x(i-1))
–MA•Linear combination of error terms
![Page 37: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/37.jpg)
![Page 38: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/38.jpg)
![Page 39: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/39.jpg)
![Page 40: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/40.jpg)
![Page 41: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/41.jpg)
Future•Vector Auto Regression
•Automate manual judgement
![Page 43: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/43.jpg)
Q & A
![Page 44: C* Capacity Forecasting (Ajay Upadhyay, Jyoti Shandil, Arun Agrawal, Netflix) | Cassandra Summit 2016](https://reader036.vdocuments.site/reader036/viewer/2022062316/58eeb51a1a28ab9e4f8b45b9/html5/thumbnails/44.jpg)
You may not control all the events that happen to you, but you CAN decide not to be reduced by them.
-Maya Angelou