getting it booking right
TRANSCRIPT
Ge#ng it Booking right – Mobile apps
Iaroslav, Khramov
Team Leader & Android developer
Iaroslav Khramov TL & Android developer
About me
Making Booking right apps
ExperimenAng • Why • How • What do you achieve
Developing TesAng Releasing
Booking.com
• Founded and based in Amsterdam • Biggest accommodaAon plaKorm in the world • > 8,000 dedicated employees
Mobile apps: • iOS & Android • 7 app development teams • ~35 apps developers
Booking.com apps
Android • > 10M downloads * • raAng 4.6 out of 5 *
iOS • raAng “4+” out of 5 **
* according to Google Play Store on 23/03/2015 ** according to Apple Store on 23/03/2015
Facebook Vkontakte Instagram YouTube Gmail Inbox WhatsApp
4.0 / 5 * 4.2 / 5 * 4.5 / 5 * 4.1 / 5 * 4.3 / 5 * 4.2 / 5 * 4.4 / 5 *
Our challenges
• Users all around the world -‐ Millions of users -‐ Our user doesn’t have a unique profile -‐ High user expectaAons
• 40+ languages • > 600k properAes • expert opinion vs. users opinion
What do we want to achieve?
Product that is: • User friendly • TrusKul • Reliable • …
What do we want to achieve?
Product that is: • User friendly • TrusKul • Reliable • …
Provide our users the best experience before, during and after making a
booking
Are you sure?
• Are you sure that new design is good? • Are you sure this change will not break the app? • Are you sure customers will like our new feature? • Are you sure …. ?
Variant A Lek hand up
Variant B Right hand up
Can you please stand up
Lets play!
Which got more money on Kickstarter?
Smartphone Controlled Paper Airplane
Jot: CapaciAve Touch Stylus
A B
Winner: Version B
A B
Which got more money on Kickstarter?
Winner: Version B
A B
Which got more money on Kickstarter?
People with hand up - please take a seat
Which variation performed better?
A B
Which variation performed better?
Winner: Version A
“The truth is, someAmes it’s bener to have no proof than low proof.”
Which variation performed better?
Winner: Version A
“The truth is, someAmes it’s bener to have no proof than low proof.”
People with hand up - please take a seat
Which got more requests? A B
Winner: Version A
“Like most people, if you guessed that Version B won, you are wrong. Version A surprisingly had a 24% increase in submissions without using an image on the page.”
Which got more requests?
Winner: Version A
“Like most people, if you guessed that Version B won, you are wrong. Version A surprisingly had a 24% increase in submissions without using an image on the page.”
Which got more requests?
People with hand up - please take a seat
What is better for conversion? A B
What is better for conversion?
Winner: Version A
In this example removing the social bunons resulted in an increased conversion to the basket of 11.9%.
What is better for conversion?
Winner: Version A
In this example removing the social bunons resulted in an increased conversion to the basket of 11.9%.
People with hand up - please take a seat
More completions?
A B
More completions?
Winner: Version B
“…surely adding an eTrust image would improve form compleAons, it makes everyone feel safe and secure, doesn’t it? The answer is no. Although it’s only a slight rise of 12.6%, Version B, without the image, proved to be more effecAve.”
More completions?
Winner: Version B
“…surely adding an eTrust image would improve form compleAons, it makes everyone feel safe and secure, doesn’t it? The answer is no. Although it’s only a slight rise of 12.6%, Version B, without the image, proved to be more effecAve.”
People with hand up - please take a seat
Which Copy Increased Trial Sign-Ups? A B
Which Copy Increased Trial Sign-Ups?
Winner: Version A
“In this test, Version A increased sign-‐ups by 38% – a big rise. However, your gut feeling might iniAally be that version B is the bener design.”
Which Copy Increased Trial Sign-Ups?
Winner: Version A
“In this test, Version A increased sign-‐ups by 38% – a big rise. However, your gut feeling might iniAally be that version B is the bener design.”
People with hand up - please take a seat
Decision-making using data: A/B testing
• When should you should run an experiment (use A/B tesAng)? Need to decide which opAon is bener
• What are the alternaAves? Expert opinion / intuiAon Historical data
Why is experimentation better?
• Our intuiAon is not very good (people are complex)
• Historical data
(becomes outdated fast)
What do we get if using A/B testing?
• Reduced design discussions • Less dependent on expert opinions • Make sure all changes are actually (commercial)
improvements • AddiAonal safety net for new changes in the app
How complex is it?
if ( newFeatureEnabled == true ) { // new feature code
} else { // old reliable solu2on
}
+ add control from server-side
Today: {
Experiment A: ON Experiment B: ON
}
Server Control experiments
At any moment of Ame: {
Experiment A: ON Experiment B: OFF
}
It’s easy! Can I start experimenting now?
Yes, but first: • Pick a metric • Find users to include in an experiment
Metrics
Metrics are the main determinant of success!
1. Pick your own metric! 2. Diversify metrics!
Metrics
Metrics are the main determinant of success!
1. Pick your own metric! 2. Diversify metrics!
Metrics for Booking.com?
What is my metric?
Type of business Possible key metrics
Ecommerce: Booking.com, Amazon
items sold, conversion, loyalty
Content: Euronews, CNN visitors, loyalty, engagement, ad clicks
Social Networking visitors, engagement, ad clicks
Gaming: Angry birds purchases, subscripAons, ad clicks
Come up with new idea to improve your
app Run an
experiment to see which
variant is the most successful
Select the winner based on
metric
Remove non-‐winner from the
code
Experiment lifecycle
Example of an experiment A B
Tools for mobile that will allow you to start experimenting today?
Yes, just Google it! For example: • Airlock • Arise.io • AppAmize • Leanplum …
A/B testing in apps provides
• Final changes are real improvements • All changes are server-‐side controlled and can be
enabled \ disabled at any point of Ame
A/B testing in apps provides
• Final changes are real improvements • All changes are server-‐side controlled and can be
enabled\disabled at any point of Ame
Is it enough for our apps?
Booking apps development
• Use Experiments • Develop fast • Be stable • Release fast • Monitor how you are doing
Source: XKCD 303 http://imgs.xkcd.com/comics/compiling.png
Fast, iterative development
Android – slow, > 3 min to build on big projects
• Buck + exopackage • Booking DCL (internal soluAon) • others
iOS – usually not a problem
Trunk development
What & how: • Everything goes into master • Local branches up to a developer • Tests keep master stable
Why: • As a result saves Ame on merging
Making sure project builds and stable
• Hooks • CI • TesAng
Hooks
Master can always build!
git push
Developer
pass
fail
git repository
Hooks:
build & run test
CI
git push git repository
CI system Great job
Developer
pass
fail
Email && jabber developer about the problem
• all dev build • special builds
• TranslaAons • UI tests
Testing
• Dedicated tesAng team • Easy maintainable tests on core funcAonality + UI tests • Bugbash sessions • Internal releases • 3rd party tesAng
Release model
Train release model • Regular • Pre-‐planned schedule
Release model
Train release model • Regular • Pre-‐planned schedule
Nobody waits: you take this train or the next one
Author: Maurits Vink http://commons.wikimedia.org/wiki/File:VIRM6.jpg
Why train model?
• Everyone know when the next release will take place • Team can plan the scope to be done • Complexity of each release more or less the same
App roll out
Rolling out apps not as fast as web: Android: • ~1 day to all users
iOS : • Depends on Apple • Apple review process 1-‐5 days
App roll out
Rolling out apps not as fast as web: Android: • ~1 day to all users
iOS : • Depends on Apple • Apple review process 1-‐5 days
Hot fixes can’t be fast…
( Experiments can help! )
Safe app release
Android: • staged rollouts • alpha & beta tesAng iOS: • Enterprise signing program • TestFlight – beta tesAng for iOS 8
Staged rollout
Get thing done right for Android users • Make sure that everything goes according to plan • NEW! 'Halt' A Staged Rollout if found a problem • When sure -‐ release to 100%
iOS: • No support for staged rollouts
Crash & data monitoring
Monitor your crashes, errors and main metrics via: • CrashlyAcs, • Bugsense, • Google AnalyAcs, • Flurry AnalyAcs, • ….
Booking apps
ExperimenAng • Why to try • How • What do you achieve
Developing TesAng Releasing
References
1. Smartphone Controlled Paper Airplanehnps://www.kickstarter.com/projects/393053146/powerup-‐30-‐smartphone-‐controlled-‐paper-‐airplane
2. Jot: CapaciAve Touch Stylus by Adonit!hnps://www.kickstarter.com/projects/531383637/jot-‐capaciAve-‐touch-‐stylus/descripAon
3. 12 Surprising A/B Test Results to Stop You Making AssumpAonshnp://unbounce.com/a-‐b-‐tesAng/shocking-‐results/
4. 5 Landing Page A/B Tests (And Their Surprising Results) hnp://blog.hubspot.com/markeAng/5-‐landing-‐page-‐a/b-‐tests-‐and-‐their-‐surprising-‐results
5. Removing Social Sharing Bunons Increases Conversions. Yes, You Heard That Right!hnps://vwo.com/blog/removing-‐social-‐sharing-‐bunons-‐from-‐ecommerce-‐product-‐page-‐increase-‐conversions/
6. Source: XKCD 303 hnp://imgs.xkcd.com/comics/compiling.png 7. Foto van twee VIRM VI stellen te Amsterdam Bijlmer ArenA. Author: Maurits Vink
hnp://commons.wikimedia.org/wiki/File:VIRM6.jpg