Lessons Learnt from Backend Systems
Development
by Josef Nevoral @ HotelQuickly
About HotelQuickly
● Last minute hotel booking● iOS, Android and BlackBerry applications● In 15 countries and more than 200 cities● 2 years old● 85+ people in company, 25+ in product team
● 10+ years in software development● In HotelQuickly for 2+ years● Many roles in the past - Titanium, iOS,
backend systems● Currently leading backend systems
development
About me
Backend systemsin HQ
Backend systems in HQ
● 100+ repositories in github (20+ public)● 40+ third party services integrated● 5 programming languages
o php, javascript - NodeJS, python, ruby
Lessons learnt
Evolution of our development workflow
From a monolithic app to microservices
Asynchronous processing
Evolution of our development workflow
Lesson #1
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
20
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
20
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
20
Key moments
11/20121st
commit
03/2014Code reviews
05/2014Continuous integration
08/2014QA
10/2014Scrumsprints
10/2014Retrospectives
01/2015Kanban
20
From sprints to Kanban
From a monolithic app to microservices
Lesson #2
Why have we built a monolithic app?
● fast initial development● not enough experience with large systems● simplicity in the beginning for DevOps
What problems do we face?
● Huge database (250+ tables)● A single technology stack● Scaling● Technical debt● Onboarding new joiners Backend
Frontend
Hotels extranet
Tracking
Paymentengine
Mobile APIIntranet
Notifications engine
Callbacks
Cut the monolith into pieces
Backend === intranet
Frontend
Tracking
Mobile API
Callbacks
Notifications engine
Payments engine
Hotel extranet
What microservices bring us
● Single responsibility● Free hand in technology● Independent scaling● Smaller chunks to digest for new joiners● Reducing technical debt step by step
New challenges
Asynchronous processing
Lesson #3
Booking a hotel (synchronous)API request from mobile app1. Validate offer (is hotel still available?)2. Verify that customer is not fraud3. Charge customer’s credit card4. Save information about the booking5. Generate voucher and send confirmation to user (sms, email, push
notification)6. Issue loyalty points7. Issue referral program rewards8. Send confirmation to hotel (email, fax)9. Send payment to hotel (generate credit card, send email/fax)10.Return API response to customer
Problems● can be slow
● minor error canfail booking
Booking a hotel (asynchronous)
API request from mobile app1. Validate offer
(is hotel still available?)2. Charge customer’s credit
card3. Save informations about
the booking4. Return API response to
customer
Processed on background● Verify that customer is not
fraud● Generate voucher and
send confirmation to user● Issue loyalty points● Issue referral program
rewards● Send confirmation to hotel● Send payment to hotel
Our solution
● Cron jobs (but it sucks)o need to keep the state using flags in databaseo waiting for cron (min 1 minute)
● Messaging queue systemo fire and forgeto highly scalable with microservices (workers)o very fast communicationo Moving from Beanstalkd to AWS SQS
(experimenting with RabbitMQ)
Asynchronous processing was not answer to all our performance problems and everything
42but it helped
Key takeaways
● become agileo sprints / kanbano retrospectiveso daily updates
● have 2 pairs of eyes for everything (not just code)● think in microservices● move as much as possible to background processes
● Becoming agile - book about agile best practices
● 12factor App - methodology for building software-as-a-service apps
● Joel Spolsky test - 12 steps for better code
● codebunk.com - realtime pair coding
● queues.io - list of popular job and message queues
● Top 10 uses of queues
Resources
See you at the next HQ meetup
Follow our blog - product.hotelquickly.com
@hotelquickly@HQProduct
