Lessons Learnt from Backend Systems Development

Download Lessons Learnt from Backend Systems Development

Post on 08-Aug-2015

684 views

Category:

Technology

3 download

Embed Size (px)

TRANSCRIPT

<ol><li> 1. Lessons Learnt from Backend Systems Development by Josef Nevoral @ HotelQuickly </li><li> 2. About HotelQuickly Last minute hotel booking iOS, Android and BlackBerry applications In 15 countries and more than 200 cities 2 years old 85+ people in company, 25+ in product team </li><li> 3. 10+ years in software development In HotelQuickly for 2+ years Many roles in the past - Titanium, iOS, backend systems Currently leading backend systems development About me </li><li> 4. Backend systems in HQ </li><li> 5. Backend systems in HQ 100+ repositories in github (20+ public) 40+ third party services integrated 5 programming languages o php, javascript - NodeJS, python, ruby </li><li> 6. Lessons learnt Evolution of our development workflow From a monolithic app to microservices Asynchronous processing </li><li> 7. Evolution of our development workflow Lesson #1 </li><li> 8. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban 20 </li><li> 9. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban </li><li> 10. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban </li><li> 11. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban </li><li> 12. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban </li><li> 13. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban 20 </li><li> 14. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban 20 </li><li> 15. Key moments 11/2012 1st commit 03/2014 Code reviews 05/2014 Continuous integration 08/2014 QA 10/2014 Scrum sprints 10/2014 Retrospectives 01/2015 Kanban 20 </li><li> 16. From sprints to Kanban </li><li> 17. From a monolithic app to microservices Lesson #2 </li><li> 18. Why have we built a monolithic app? fast initial development not enough experience with large systems simplicity in the beginning for DevOps </li><li> 19. What problems do we face? Huge database (250+ tables) A single technology stack Scaling Technical debt Onboarding new joiners Backend Frontend Hotels extranet Tracking Payment engine Mobile API Intranet Notifications engine Callbacks </li><li> 20. Cut the monolith into pieces Backend === intranet Frontend Tracking Mobile API Callbacks Notifications engine Payments engine Hotel extranet </li><li> 21. What microservices bring us Single responsibility Free hand in technology Independent scaling Smaller chunks to digest for new joiners Reducing technical debt step by step New challenges </li><li> 22. Asynchronous processing Lesson #3 </li><li> 23. Booking a hotel (synchronous) API request from mobile app 1. Validate offer (is hotel still available?) 2. Verify that customer is not fraud 3. Charge customers credit card 4. Save information about the booking 5. Generate voucher and send confirmation to user (sms, email, push notification) 6. Issue loyalty points 7. Issue referral program rewards 8. Send confirmation to hotel (email, fax) 9. Send payment to hotel (generate credit card, send email/fax) 10.Return API response to customer </li><li> 24. Problems can be slow minor error can fail booking </li><li> 25. Booking a hotel (asynchronous) API request from mobile app 1. Validate offer (is hotel still available?) 2. Charge customers credit card 3. Save informations about the booking 4. Return API response to customer Processed on background Verify that customer is not fraud Generate voucher and send confirmation to user Issue loyalty points Issue referral program rewards Send confirmation to hotel Send payment to hotel </li><li> 26. Our solution Cron jobs (but it sucks) o need to keep the state using flags in database o waiting for cron (min 1 minute) Messaging queue system o fire and forget o highly scalable with microservices (workers) o very fast communication o Moving from Beanstalkd to AWS SQS (experimenting with RabbitMQ) </li><li> 27. Asynchronous processing was not answer to all our performance problems and everything 42but it helped </li><li> 28. Key takeaways become agile o sprints / kanban o retrospectives o daily updates have 2 pairs of eyes for everything (not just code) think in microservices move as much as possible to background processes </li><li> 29. Becoming agile - book about agile best practices 12factor App - methodology for building software-as-a- service apps Joel Spolsky test - 12 steps for better code codebunk.com - realtime pair coding queues.io - list of popular job and message queues Top 10 uses of queues Resources </li><li> 30. See you at the next HQ meetup Follow our blog - product.hotelquickly.com @hotelquickly @HQProduct josef.nevoral@hotelquickly.com </li></ol>