[elite camp 2016] annemarie klaassen & tom van de berg - moving beyond testing for absolute...

SUBTITLE BELOW

Optimize for €€€HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

@AM_Klaassen / @tomvdberg1

How we got here…

@AM_Klaassen / @tomvdberg1

The quest for female speakers…

@AM_Klaassen / @tomvdberg1

Female speakers…

?

@AM_Klaassen / @tomvdberg1

Our lovely colleagues

What we do…

@AM_Klaassen / @tomvdberg1

Conversion rate optimization

Analytics Psychology

Lots of A/B-tests

@AM_Klaassen / @tomvdberg1

Our clients

@AM_Klaassen / @tomvdberg1

Adding

direct value

Learning

user behavior

@AM_Klaassen / @tomvdberg1

@AM_Klaassen / @tomvdberg1

@AM_Klaassen / @tomvdberg1

Van der Valk

Unique website for each hotel

1 centralized team

Paid by % of their turnover

@AM_Klaassen / @tomvdberg1

Optimization of Valk

Increase revenue through CRO

Reduce costs per experiment

“We’re not in the business of

science, we’re in the business

of making money”

3 Tactics

1.

Free test software

@AM_Klaassen / @tomvdberg1

Jorrin Quest

GTM testing

http://gtmtesting.com

@AM_Klaassen / @tomvdberg1

GTM testing

Easy setup

Free

Preview mode

2.

Change of statistics

@AM_Klaassen / @tomvdberg1

Focus on finding proof

@AM_Klaassen / @tomvdberg1

Example test: Order flow

@AM_Klaassen / @tomvdberg1

Example test: Set completion

A B

@AM_Klaassen / @tomvdberg1

Test result

@AM_Klaassen / @tomvdberg1

• No effect on conversion

Don’t implement

Re-test with higher volumes

Conclusion

@AM_Klaassen / @tomvdberg1

@AM_Klaassen / @tomvdberg1

What’s the alternative?

Frequentist

statistics

Bayesian

statistics

@AM_Klaassen / @tomvdberg1

Example test: Set completionA B

@AM_Klaassen / @tomvdberg1

Bayesian Test evaluation

@AM_Klaassen / @tomvdberg1

Focus on risk assessment

88,4%

A test result is the probability that B outperforms A:

ranging from 0% - 100%

@AM_Klaassen / @tomvdberg1

IMPLEMENT B PROBABILITY * EFFECT ON REVENU

Expected risk 11,6% - € 69.791

Expected uplift 88,4% € 213.530

Contribution € 180.733

* Based on 6 months and an average order value of € 175

Make a risk assessment

@AM_Klaassen / @tomvdberg1

Implement BB

@AM_Klaassen / @tomvdberg1

abtestguide.com/bayesian/

Roy Schieving

Annemarie Klaassen

@AM_Klaassen / @tomvdberg1

Conclusion Bayesian

Easier to understand and

communicate

Better suits the business

Don’t throw away good ideas (indicatively significant)

Higher implementation rate and

revenue

€ 0

€ 1.000.000

€ 2.000.000

€ 3.000.000

€ 4.000.000

€ 5.000.000

€ 6.000.000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Expected impact on revenue in 1 year

€ Frequentist € Bayesian

3.

Change of test method

@AM_Klaassen / @tomvdberg1

50%

50%

A/B-test

@AM_Klaassen / @tomvdberg1

Traffic allocation over time

A

B

Tra

ffic

Time

Profit or loss?

@AM_Klaassen / @tomvdberg1

What’s the alternative?

A/B-test Bandits

If you have multiple options:

which bandit

will pay-out the

most?

@AM_Klaassen / @tomvdberg1

Explore Exploit

?

Explore / Exploit dilemma

@AM_Klaassen / @tomvdberg1

Bandit

100%

@AM_Klaassen / @tomvdberg1

Bandit

80%

20%

10%

90%

@AM_Klaassen / @tomvdberg1

Traffic allocation over time

A

B

Tra

ffic

Time

More Profit!

@AM_Klaassen / @tomvdberg1

Limited time frame Automation

When use Bandits?

Earn over learn

How?

@AM_Klaassen / @tomvdberg1

Example 1

@AM_Klaassen / @tomvdberg1

Example 1

Older target group

Comparing price on

other sites

Behavior of others

@AM_Klaassen / @tomvdberg1

Example 1

Control No message

Variation B Copy default

Variaton C

Variation D

11,1%

10,6%

10,4%

10,5%

@AM_Klaassen / @tomvdberg1

Example 1 – Effect on revenue

A

B

Tra

ffic

Time

+ € 6.637

@AM_Klaassen / @tomvdberg1

Example 2

Control New control

Variation B Copy control

Variation C

Variation D

9,2%

8,9%

7,8%

8,4%

@AM_Klaassen / @tomvdberg1

Example 2 – Effect on revenue

A

B

Tra

ffic

Time

+ € 19.862

@AM_Klaassen / @tomvdberg1

Conclusion Bandits

Less regret during testing

No implementation costs

Optimizing per user segment

1. Free test software

2. Bayesian statistics

3. Bandit testing

revenue

Maximize

@AM_Klaassen / @tomvdberg1

“We’re not in the business of

science, we’re in the business

of making money”

TÄNAN VÄGA!

Bayesian calculator: abtestguide.com/bayesian/

Slide deck: ondi.me/elitecamp2016

@AM_Klaassen

annemarie@onlinedialogue.com

nl.linkedin.com/in/amklaassen

@tomvdberg1

tom@onlinedialogue.com

nl.linkedin.com/in/vandenbergtom