dynamic a/b testing with ab/cd

Dynamic A/B Testing with AB/CD

Frontend Conference ZurichSeptember 2. 2016

Memi Beltrame

A/B Tests

Buy Now! Buy for $1’000

1. Divide users in 2 groups

A B

A/B Tests


1. Divide users in 2 groups2. Each group gets a button

A B

A/B Tests


1. Divide users in 2 groups2. Each group gets a button3. See what works and what not

A B

Example with 1000 users

Buy Now! Buy for $1’000500 500



3% Conversion 12% Conversion

500 500



3% Conversion 12% Conversion15 Sales 60 Sales

500 500



3% Conversion 12% Conversion15 Sales 60 Sales$15’000 $60’000

500 500



3% Conversion 12% Conversion15 Sales 60 Sales$15’000 $60’000

500 500

A difference of $45’000

Finding out which button works

just cost you $45’000

Is there a way to bring down that number?

Dynamic A/B Tests

2 Rules


2 Rules


1. 10% get a button at random

2 Rules


1. 10% get a button at random2. 90% get the best button

Find the best button

By tracking 3 values for each one


1. Views



1. Views2. Clicks



1. Views2. Clicks3. Ratio of Clicks / Views


How are Options Tracked?Start of Test

How are Options Tracked?After first user

How are Options Tracked?After 1000 users

Resulting in $111’000 for the test



That’s $36’000 less.

A service to set up dynamic A/B tests that handles the math behind it.

AB/CD

http://abcd.lessamess.ch

After Setup

1. Request an option via an API

After Setup

1. Request an option via an API2. Store the option in a session

After Setup

1. Request an option via an API2. Store the option in a session3. Report back to AB/CD when

an option performs

What you getResults in real time

Thanks!

Memi Beltrame@bratwurstkomet

Design