metrics. driven. design. (developer conference hamburg 2012)
DESCRIPTION
Did you ever work on a feature where you had multiple options on how to display it to the user but you did not know which one would work best? Why speculate about what would work best if you can actually measure it? Welcome to the world of a/b testing: Implement multiple variations of the same thing, and let different users experience different variations. Measure which variant works best and continue with that one. At Wooga we extensively use a/b testing to polish features to perfection. This helped us becoming one of the biggest providers of social games on Facebook with about 40 million players every month. We would like to show you what a/b testing is good for, how you do it and what pitfalls you need to avoid.TRANSCRIPT
Metrics Driven Design.
@chiefofspace
@jrirei
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
2010 W
16
2010 W
19
2010 W
22
2010 W
25
2010 W
28
2010 W
31
2010 W
34
2010 W
37
2010 W
40
2010 W
43
2010 W
46
2010 W
49
2010 W
52
2011 W
03
2011 W
06
2011 W
09
2011 W
12
2011 W
15
2011 W
18
2011 W
21
2011 W
24
2011 W
27
2011 W
30
2011 W
33
2011 W
36
2011 W
39
2011 W
42
2011 W
45
2011 W
48
2011 W
51
2012 W
02
2012 W
05
2012 W
08
2012 W
11
2012 W
14
2012 W
17
2012 W
20
DAU Monster World
>> Retention
>> Virality
>> Monetization
Retention
Retention
New Users last 24 hours 20.000
Step 1 - Loading begins
Step 2 - Loading complete
Step - 3
Step - 4
Step - 5
Step - 6
Level 2 reached
100%
85%
70%
68%
66%
50%
48%
Step - 7
Step - 8
46%
44%
Tutorial Steps
Virality
Virality
Monetization
~ 2 %
~ 10 % ~ 3 %
~ 12 %
h/ps://mw.t.wooga.com/k/pgr/? i=0& language=de_DE& s=100000107880707& ts=1347085569& w_friends=238
Tracking Service
KaNa
DBs
…
>> Day 1 Retention
Day 1 Retention
New Users last 24 hours 20.000
Step 1 - Loading begins
Step 2 - Loading complete
Step - 3
Step - 4
Step - 5
Step - 6
Level 2 reached
100%
85%
70%
68%
66%
50%
48%
Step - 7
Step - 8
46%
44%
Tutorial Steps
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins
Step 2 - Loading complete
Step - 3
Step - 4
Step - 5
Step - 6
Level 2 reached
100%
85%
70%
68%
66%
50%
48%
Step - 7
Step - 8
46%
44%
Is it for me?
New Users last 24 hours 20.000
Step 1 - Loading begins
Step 2 - Loading complete
Step - 3
Step - 4
Step - 5
Step - 6
Level 2 reached
100%
85%
70%
68%
66%
50%
48%
Step - 7
Step - 8
46%
44%
Do I get it?
Tutorial Steps
Do I get it?
Do I get it?
On initial design solutions
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins
Step 2 - Loading complete
Step - 3
Step - 4
Step - 5
Step - 6
Level 2 reached
100%
85%
70%
68%
48%
50%
Step - 7
Step - 8
46%
44%
Further, even further
66%
0% 10% 20% 30% 40% 50% 60%
Reached lvl 3
Reached lvl 4
1 day ret
3 day ret
7 day ret
3 min 5 min
Growth Time
Beautification
Beautification
73,2 % 73,0 %
Plateau
Tutorial Steps
New Users last 24 hours 20.000
Step 1 - Loading begins
Step 2 - Loading complete
Step - 3
Step - 4
Step - 5
Step - 6
Level 2 reached
100%
85%
70%
68%
66%
50%
48%
Step - 7
Step - 8
50%
48%
What's next?
>> Day to Day Retention
0
50.000
100.000
150.000
200.000
250.000
'201
0-09
-13'
'2
010-
09-1
4'
'201
0-09
-15'
'2
010-
09-1
6'
'201
0-09
-17'
'2
010-
09-1
8'
'201
0-09
-19'
'2
010-
09-2
0'
'201
0-09
-21'
'2
010-
09-2
2'
'201
0-09
-23'
'2
010-
09-2
4'
'201
0-09
-25'
'2
010-
09-2
6'
'201
0-09
-27'
'2
010-
09-2
8'
'201
0-09
-29'
'2
010-
09-3
0'
'201
0-10
-01'
'2
010-
10-0
2'
'201
0-10
-03'
'2
010-
10-0
4'
'201
0-10
-05'
'2
010-
10-0
6'
'201
0-10
-07'
'2
010-
10-0
8'
'201
0-10
-09'
'2
010-
10-1
0'
'201
0-10
-11'
'2
010-
10-1
2'
'201
0-10
-13'
'2
010-
10-1
4'
'201
0-10
-15'
'2
010-
10-1
6'
'201
0-10
-17'
'2
010-
10-1
8'
'201
0-10
-19'
'2
010-
10-2
0'
Launch
2nd Toy
Post responses
>> Understanding User Behaviour
0
5.000
10.000
15.000
20.000
25.000
30.000 8/
30/2
011
9/6/
2011
9/
13/2
011
9/20
/201
1 9/
27/2
011
10/4
/201
1 10
/11/
2011
10
/18/
2011
10
/25/
2011
11
/1/2
011
11/8
/201
1 11
/15/
2011
11
/22/
2011
12
/28/
2011
1/
4/20
12
1/11
/201
2 1/
18/2
012
1/25
/201
2 2/
1/20
12
2/8/
2012
2/
15/2
012
2/22
/201
2 2/
29/2
012
3/7/
2012
3/
14/2
012
3/21
/201
2 3/
28/2
012
4/4/
2012
4/
11/2
012
4/18
/201
2
Users without missions
>> Next try
Mr. Beazzy
>> + Engaged
>> + Engaged >> + $
>> Virality
>> Monetization
Old
New
Magic Wand Shop
Old
New
+ 24 %
Magic Wand Shop
Magic Wand Shop
Magic Wand Shop
+ 23 %
Magic Wand Shop
Remember the user
Match user to test group
Match user to test group
Feature switch
Wait for data to become relevant
Compare results
0% 10% 20% 30% 40% 50% 60%
Reached lvl 3
Reached lvl 4
1 day ret
3 day ret
7 day ret
3 min 5 min
Compare results
Side Effects
Side Effects
>> Endless Love
Meet the bad guy: Robert the robot
Meet his new date: Roberta the robot
