dead simple scalability patterns
TRANSCRIPT
- 1. Dead Simple Scalability Patterns Vedang Manerikar Platform Architect, Helpshift Fifth Elephant, 16th July 2015
- 2. Background: Helpshift +
- 3. Agenda Monitoring 101: Logs! Database Access Patterns Serialization and Deserialization Network Calls Integration Points
- 4. Agenda Monitoring 101: Logs! Database Access Patterns Serialization and Deserialization Network Calls Integration Points
- 5. Monitoring 101: Logs!
- 6. Monitoring 101: Logs! Request URL UUID Resource being queried Time taken (ms) Size of response (bytes) Human Identifiers for Data Store, Type of operation IMP
- 7. Monitoring 101: Logs! Top 5 slowest DB calls $ sort -k6 -r -n | cut -f3- -d Top 5 popular URLs $ sort -k4,4 -u | sort -k3 | cut -f 3-3 -d ' ' | uniq -c | sort -k1 -n -r Top 5 routes making the maximum number of DB calls. $ sort -k4 | cut -f2-4 -d ' ' | uniq - f1 -c | sort -k1 -n -r
- 8. Monitoring 101: Logs!
- 9. Agenda Monitoring 101: Logs! Database Access Patterns Serialization and Deserialization Network Calls Integration Points
- 10. Unbounded DB Calls Lock up Resources Eat up Memory Discovered when: Data Sludge Unexpected Use-case Fixed by: Batch Sizes Abstractions for Chunked Requests QA < Crazy Users
- 11. Indiscriminate DB Calls
- 12. Check Out: Batch requests to the same Data Source Cache previous results
- 13. Agenda Monitoring 101: Logs! Database Access Patterns Serialization and Deserialization Network Calls Integration Points
- 14. Serialisation / Deserialisation
- 15. Agenda Monitoring 101: Logs! Database Access Patterns Serialization and Deserialization Network Calls Integration Points
- 16. The Network is Wonderful
- 17. The Network
- 18. Avoid Network Calls Caching Precomputing Avoid
- 19. Agenda Monitoring 101: Logs! Database Access Patterns Serialization and Deserialization Network Calls Integration Points
- 20. Integration Points and Domino Effects App Server DB Network Calls Web Server Cascading Failures QUEUE Analytics Consumer
- 21. Integration Points and Domino Effects App Server DB Network Calls Web Server Cascading Failures QUEUE Analytics Consumer X X X X
- 22. Timeouts and Circuit Breakers Closed (everything is operational) Half-Open (Has it recovered?) Open (Resource has failed) Failure Wait for some time, in the meanwhile: - Fail Fast - Gracefully Degrade Attempt Reset Failure Success Resource Timeout
- 23. Check out:
- 24. Health Checks Am I Alive? Auto-Scaling
- 25. Revisiting our Stack App Server DB Network Calls Web Server Cascading Failures QUEUE Analytics Consumer CB, T, GD CB, T BP CB, T You Shall Not Pass CB: Circuit Breaker T: Timeouts BP: Back Pressure GD: Graceful Degradation
- 26. Take-Aways and Questions Designing Scalability > Testing Scalability Scalability is an important perspective [email protected] @vedang