dan siroker data testing and business business of software conference 2013
Post on 29-Oct-2014
290 Views
Preview:
DESCRIPTION
TRANSCRIPT
A/B Testing
Dan SirokerCo-Founder & CEO
If data can help win elections, what can it help you do for your business?
‘05‘06‘07‘08’09’10
‘05‘06‘07‘08’09’10
$0
$1
$2
$3
$4
$5
$6
$7
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4
Mill
ion
s
$0
$1
$2
$3
$4
$5
$6
$7
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4
Mill
ion
s
Dan
Pete Eric Elliot
Team: 4Customers: 644
Annual Run Rate: $0.1m
$0
$1
$2
$3
$4
$5
$6
$7
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4
Mill
ion
s
Dan
Pete Eric Elliot
Ricky
Jeff P
Team: 10 (+150%)Customers: 1,508 (+134%)
Annual Run Rate: $1.2m(+982%)
Ryan M
Mat
Meng Xiao
Julio
$0
$1
$2
$3
$4
$5
$6
$7
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4
Mill
ion
s
Dan
Pete Eric Elliot
Ricky
Jeff P
Ryan M
Mat
Meng Xiao
Julio
Sonesh
Mik
Jodie
Sandy
Brendan
Tommy
Dogan
Tyle
Arthu
Greg
Alan
Cara
Cristian
Lauren
Katy
Ryan L
Dae-Ho
Ian
Steve
Jeff Z
Amina
Byron
Shanann
Dan G
Sophie
Jon
Joanna
Jonathan
Travis
Ashley
Cyrus
Eddie
Team: 42 (+320%)Customers: 3,460 (+129%)
Annual Run Rate: $7.6m (+533%)
$0
$1
$2
$3
$4
$5
$6
$7
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4
Mill
ion
s
$0
$1
$2
$3
$4
$5
$6
$7
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4
Million
s
$0 $1 $2 $3 $4 $5 $6 $7 $8 $9
$10 $11 $12 $13 $14 $15 $16 $17 $18 $19
$20 $21 $22 $23 $24 $25 $26 $27 $28
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4 2013 Q1
Q2 Q3 Q4
Mill
ion
s
Actual Projection
$0 $1 $2 $3 $4 $5 $6 $7 $8 $9
$10 $11 $12 $13 $14 $15 $16 $17 $18 $19
$20 $21 $22 $23 $24 $25 $26 $27 $28
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4 2013 Q1
Q2 Q3 Q4
Mill
ion
s
Actual Projection
Cyrus
Eddie
Dan
Pete Eric Elliot
Ricky
Jeff P
Ryan M
Mat Meng XiaoJulio
Sonesh
Mik
Jodie
Sandy
Brendan
Tommy
Dogan
Tyle
Arthu
Greg
Alan
Cara
Cristian
Lauren
Katy
Ryan L
Dae-Ho
Ian
Steve
Jeff Z
Amina
Byron
Shanann
Dan G
Sophie
Jon
Joanna
Jonathan
Travis
Ashley
Team: 42
$0 $1 $2 $3 $4 $5 $6 $7 $8 $9
$10 $11 $12 $13 $14 $15 $16 $17 $18 $19
$20 $21 $22 $23 $24 $25 $26 $27 $28
2010 Q1
Q2 Q3 Q4 2011 Q1
Q2 Q3 Q4 2012 Q1
Q2 Q3 Q4 2013 Q1
Q2 Q3 Q4
Mill
ion
s
Actual Projection
Cyrus
Eddie
Dan
Pete Eric Elliot
Ricky
Jeff P
Ryan M
Mat Meng XiaoJulio
Sonesh
Mik
Jodie
Sandy
Brendan
Tommy
Dogan
Tyle
Arthu
Greg
Alan
Cara
Cristian
Lauren
Katy
Ryan L
Dae-Ho
Ian
Steve
Jeff Z
Amina
Byron
Shanann
Dan G
Sophie
Jon
Joanna
Jonathan
Travis
Ashley
Bo
Lisa
Miyo
Robin
James
David
Marius
Steve
Brian
Amelia
Luke
Adam
Adrian
Bradley
Brett
Jen
Ash
Richar
Lucas
Aaron
Gary
Etienne
Andrea
Neha
Chrix
Nick
Nick
Chris
Jamie
Helen
Tammie
Evan
Grace
Benson
Westly
Marco
Daniel
Kasra
Omari
Eric
Mustafa
Waylon
Fabian
Cate
Saum
Khattaab
Junan
Anshu
Bryce
Lacy
Natarajan
Katie
Andy
Ranjit
Grant
SG
Andy
Pedro
Danielle
Tai
Laura
Darwish
Amanda
Nikhil
Ryan
Hemant
Megan
Oren
Courtney
Kent
Ural
Toby
Oliver
Julia
Jaron
Jonathan
Adam
Asia
Pamela
Martjin
Ben
Becca
Michael
Christop
Dan
Eduar
Yonat
Chris
Chris
Rama
Team: 130 (+210%)
How?
Limited Life Experience
Limited Life Experience+
Overgeneralization
Limited Life Experience+
Overgeneralization=
Advice-Paul Buchheit
Lesson #1:
Define success.
Lesson #1:
Define success .
Lesson #1:
Define quantifiable success metrics.
Visitvisitors per day
Emailsign up
rate$
$ per recipient
Visit Email $sign up
rate $ per recipientvisitors per day
Media
Button
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Splash page experimentVariations:
Button: “Sign Up”
Button: “Learn More”
Button: “Join Us Now”
Button: “Sign Up Now”
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Variations:
Splash page experiment
Media: “Get Involved”
Media: “Family”
Media: “Change”
Media: “Barack’s Video”
Media: “Springfield Video”
Media: “Sam’s Video”
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Variations:
Splash page experiment
Splash page experiment
Splash page experiment
Splash page experiment
Email Subscriptions Volunteers Amount
Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Email Subscriptions Volunteers Amount
Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Email Subscriptions Volunteers Amount
Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Lesson #1:
Define quantifiable success metrics.
Lesson #2:
Less is more. Reduce choices.
vs.
+15% conversion rate
+10% $ per visitor
vs.
+20% submissions
vs.
+16.8% $ per visitor
vs.
+43.8% purchases
Lesson #2:
Less is more. Reduce choices.
Lesson #3:
Words matter. Focus on your call to action.
Donation button experiment
Variations
Donation button experiment
VariationsNever
Signed UpSigned Up,
Never DonatedPreviously Donated
0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%
vs.
+14.6% clicks
vs.+15.75% $ per visitor
Lesson #3:
Words matter. Focus on your call to action.
Lesson #4:
Seek the global maximum.
Source: http://blog.intercom.io/criticism-and-two-way-streets/
vs.
Source: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html
vs.
+0.5% retention+1.4% engagement
vs.
+600% engagement
vs.
+49% conversion rate from 240 A/B tests
Lesson #4:
Seek the global maximum.
Lesson #5:
Start today.
Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
5,000+ Happy Customers!
Lesson #5:
Start today.
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call to action.
4. Seek the global maximum.
5. Start today.
Lessons Learned
One more thing...
A/B Testing
Dan SirokerCo-Founder & CEO
If data can help win elections, what can it help you do for your business?
top related