dynamic a/b testing with ab/cd
Post on 14-Apr-2017
300 Views
Preview:
TRANSCRIPT
Dynamic A/B Testing with AB/CD
Frontend Conference ZurichSeptember 2. 2016
Memi Beltrame
A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups
A B
A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups2. Each group gets a button
A B
A/B Tests
Buy Now! Buy for $1’000
1. Divide users in 2 groups2. Each group gets a button3. See what works and what not
A B
Example with 1000 users
Buy Now! Buy for $1’000500 500
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion
500 500
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion15 Sales 60 Sales
500 500
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion15 Sales 60 Sales$15’000 $60’000
500 500
Example with 1000 users
Buy Now! Buy for $1’000
3% Conversion 12% Conversion15 Sales 60 Sales$15’000 $60’000
500 500
A difference of $45’000
Finding out which button works
just cost you $45’000
Is there a way to bring down that number?
Dynamic A/B Tests
2 Rules
Buy Now! Buy for $1’000
2 Rules
Buy Now! Buy for $1’000
1. 10% get a button at random
2 Rules
Buy Now! Buy for $1’000
1. 10% get a button at random2. 90% get the best button
Find the best button
By tracking 3 values for each one
Find the best button
1. Views
By tracking 3 values for each one
Find the best button
1. Views2. Clicks
By tracking 3 values for each one
Find the best button
1. Views2. Clicks3. Ratio of Clicks / Views
By tracking 3 values for each one
How are Options Tracked?Start of Test
How are Options Tracked?After first user
How are Options Tracked?After 1000 users
Resulting in $111’000 for the test
Finding out which button works
just cost you $9’000
Finding out which button works
just cost you $9’000
That’s $36’000 less.
AB/CD
A service to set up dynamic A/B tests that handles the math behind it.
AB/CD
http://abcd.lessamess.ch
Setup
After Setup
1. Request an option via an API
After Setup
1. Request an option via an API2. Store the option in a session
After Setup
1. Request an option via an API2. Store the option in a session3. Report back to AB/CD when
an option performs
API
What you getResults in real time
Thanks!
Memi Beltrame@bratwurstkomet
top related