too big to fail · too big to fail: a/b tests when you’re betting the bank andrea burbank...
TRANSCRIPT
![Page 1: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/1.jpg)
Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014
#qconsf
![Page 2: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/2.jpg)
March 19, 2013
![Page 3: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/3.jpg)
holed up for 6 months
![Page 4: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/4.jpg)
replace of the code 2 3
![Page 5: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/5.jpg)
new features can’t ship
![Page 6: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/6.jpg)
desperately want to release
![Page 7: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/7.jpg)
many things have changed
![Page 8: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/8.jpg)
How did we get here?
![Page 9: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/9.jpg)
What have we done so far?
![Page 10: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/10.jpg)
What should we do right now?
![Page 11: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/11.jpg)
<how we got here>
![Page 12: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/12.jpg)
![Page 13: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/13.jpg)
![Page 14: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/14.jpg)
Rewrite the website
![Page 15: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/15.jpg)
Rewrite the website
MAINTAINABLE
TESTABLE ENCAPSULATED
EXTENSIBLE COHESIVE
MODULAR
![Page 16: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/16.jpg)
http
://w
ww
.cne
t.com
/new
s/pi
nter
est-t
ests
-new
-look
-with
-big
ger-
pins
-re
stru
ctur
ed-n
avig
atio
n/
![Page 17: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/17.jpg)
moved all the buttons
![Page 18: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/18.jpg)
rounded all the corners
![Page 19: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/19.jpg)
replaced the categories menu
![Page 20: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/20.jpg)
made the pins bigger
![Page 21: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/21.jpg)
removed friends to follow
![Page 22: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/22.jpg)
added related pins to closeup
![Page 23: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/23.jpg)
got rid of the “add a pin” menu
![Page 24: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/24.jpg)
removed second-order attribution
![Page 25: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/25.jpg)
renamed the Repin button
![Page 26: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/26.jpg)
uncluttered the pin closeup
![Page 27: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/27.jpg)
and much more
![Page 28: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/28.jpg)
Why so much change at once?
bigger pins
new header
web platform
![Page 29: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/29.jpg)
“sometimes, don’t shoot yourself in the foot.”
![Page 30: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/30.jpg)
Advantages of incremental change
![Page 31: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/31.jpg)
confidence
![Page 32: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/32.jpg)
extensible learnings
![Page 33: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/33.jpg)
focus your resources
![Page 34: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/34.jpg)
Incremental or all at once?
?
![Page 35: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/35.jpg)
understand (and rebut) arguments against incremental change
![Page 36: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/36.jpg)
avoid our mistakes
![Page 37: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/37.jpg)
Cons of small changes #1: time
time
novelty effect
![Page 38: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/38.jpg)
big changes will be slower than you anticipated
rebuttal #1: time
![Page 39: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/39.jpg)
Cons of small changes #2: psychology
*shrug*
*shrug*
*shrug*
*shrug*
*shrug*
*shrug*
*shrug*
![Page 40: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/40.jpg)
it’s not “incremental change” – it’s faster learning
rebuttal #2: psychology
![Page 41: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/41.jpg)
Cons of small changes #3: local maxima
![Page 42: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/42.jpg)
Cons of small changes #3: local maxima
![Page 43: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/43.jpg)
Cons of small changes #3: local maxima what you think you did:
awesome win!
![Page 44: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/44.jpg)
Cons of small changes #3: local maxima what you actually did
oops!
![Page 45: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/45.jpg)
mountains and valleys are neighbors
rebuttal #3: local maxima
![Page 46: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/46.jpg)
Incremental vs. all at once? time
avoid false optimism: big changes take time too psychology
help your organization embrace fast learning local maxima
yes! but avoid the valley, and watch your step
![Page 47: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/47.jpg)
a big leap
![Page 48: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/48.jpg)
a small change
![Page 49: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/49.jpg)
a $10M change
![Page 50: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/50.jpg)
Making a big change all at once
new = better!
![Page 51: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/51.jpg)
Making a big change all at once
![Page 52: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/52.jpg)
Making a big change all at once
![Page 53: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/53.jpg)
Making a big change all at once
![Page 54: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/54.jpg)
Making a big change all at once
![Page 55: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/55.jpg)
Making a big change all at once
![Page 56: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/56.jpg)
Making a big change all at once
![Page 57: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/57.jpg)
Making a big change all at once
![Page 58: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/58.jpg)
Making a big change all at once
![Page 59: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/59.jpg)
Making a big change all at once
![Page 60: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/60.jpg)
Making a big change all at once
![Page 61: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/61.jpg)
Making a big change all at once
![Page 62: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/62.jpg)
Making a big change all at once
![Page 63: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/63.jpg)
Making a big change all at once
![Page 64: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/64.jpg)
Making a big change all at once
![Page 65: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/65.jpg)
Making a big change all at once
what we gained uncaptured gain
![Page 66: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/66.jpg)
Making a big change incrementally
![Page 67: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/67.jpg)
? ?
Making the big leap
![Page 68: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/68.jpg)
we always think it’s a huge leap forward
(but we’re not always right)
![Page 69: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/69.jpg)
</how we got here>
![Page 70: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/70.jpg)
<what to do right now>
![Page 71: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/71.jpg)
Introduction to
![Page 72: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/72.jpg)
![Page 73: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/73.jpg)
Website redesign
• Rewrite entire website • Modular design for faster development • Consume our API
• Complete visual redesign
![Page 74: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/74.jpg)
![Page 75: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/75.jpg)
![Page 76: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/76.jpg)
Why run an experiment?
We know we’ll ship – we can’t go back to the old design now! We’ve already
told the press about it!
^ NOT
![Page 77: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/77.jpg)
Why run an experiment?
Identify and fix bugs
Understand the effects of our changes
![Page 78: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/78.jpg)
Option #1: Just ship it • Release the new site. • Watch metrics trend to evaluate success.
date
release date
![Page 79: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/79.jpg)
Option #1: Just ship it
date
Lots of news articles!
• But: other factors may be responsible
![Page 80: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/80.jpg)
• But: novelty effect could skew results
Option #1: Just ship it
date
release date
![Page 81: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/81.jpg)
• But: effect size may be too small
Option #1: Just ship it
date
release date
![Page 82: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/82.jpg)
• But: anything you find is too late!
Option #1: Just ship it
date
release date
![Page 83: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/83.jpg)
Option #1: Just ship it
![Page 84: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/84.jpg)
Options for the big release
• Just ship it
• Allow users to opt in
LEA
RN
MO
RE
GO
FA
STE
R
![Page 85: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/85.jpg)
Option #2: Let users opt in • Let users opt in. Listen to their feedback. • Watch metrics trends too.
date
release date It’s great!
Broken on IE 6?
![Page 86: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/86.jpg)
Option #2: Let users opt in
![Page 87: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/87.jpg)
Option #2: Let users opt in
![Page 88: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/88.jpg)
Options for the big release
• Just ship it
• (Allow users to opt in)
• Run an opt-in experiment
LEA
RN
MO
RE
GO
FA
STE
R
![Page 89: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/89.jpg)
All users Control
Treatment outc
omes
Option #3: Run an opt-in experiment
![Page 90: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/90.jpg)
Option #3: Run an opt-in experiment • Allow a random group of users to opt in • Compare metrics to a control group
date
release date
opt-in group control group
![Page 91: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/91.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
a person in your experiment
![Page 92: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/92.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
a happy person in your experiment
![Page 93: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/93.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% control 30% treatment
![Page 94: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/94.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% 30% +/- 1% 30% 40% +/- 1%
![Page 95: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/95.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% 30% +/- 1% 30% 40% +/- 1%
![Page 96: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/96.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
a person who opted into the new
website
![Page 97: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/97.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% control 30% treatment
![Page 98: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/98.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
happy
opted in
both
![Page 99: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/99.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% control 30% treatment
![Page 100: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/100.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% control 30% treatment
![Page 101: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/101.jpg)
Challenges of opt-in: dilution
• Dilution decreases signal : noise ratio
30% 30% +/- 1% 30% 32% +/- 1%
![Page 102: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/102.jpg)
dilution of results = may not be able to measure
![Page 103: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/103.jpg)
Challenges of opt-in: selection bias
• Engaged = more likely to try new features Li
kelih
ood
of tr
ying
feat
ure
Level of a user’s engagement
![Page 104: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/104.jpg)
Challenges of opt-in: selection bias
• Users who opt in differ from those who don’t m
easu
re o
f use
r act
ivity
Date
Users who opted in
Control group
Date opt-in first available
![Page 105: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/105.jpg)
Challenges of opt-in: selection bias
• Users who opt in differ from those who don’t
I The new
site is awesome!
?
![Page 106: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/106.jpg)
Option #3: Run an opt-in experiment
Data is skewed by:
• Dilution
• Selection bias
![Page 107: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/107.jpg)
Options for the big release
• Just ship it
• (Allow users to opt in)
• Run an opt-in experiment
• Run a forced-in experiment
LEA
RN
MO
RE
GO
FA
STE
R
I
![Page 108: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/108.jpg)
Option #4: Run an forced-in experiment
• Force some users into the new design • Compare metrics to a control group
date
release date forced-in group
control group
![Page 109: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/109.jpg)
April 3, 2013
![Page 110: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/110.jpg)
opt in forced
% of users who repin
date
control enabled
![Page 111: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/111.jpg)
“It’s not an experiment if you know it’s going to work.”
– Jeff Bezos
![Page 112: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/112.jpg)
“My experiment will be hugely successful.”
– every feature owner ever
![Page 113: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/113.jpg)
even optimists need a contingency plan
![Page 114: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/114.jpg)
should we ship today?
how about now?
or now?
![Page 115: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/115.jpg)
April 10, 2013
![Page 116: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/116.jpg)
bad enough to block shipping?
![Page 117: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/117.jpg)
how bad is too bad?
![Page 118: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/118.jpg)
we need to fix this before we ship!
![Page 119: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/119.jpg)
opt in forced
date
control enabled
% of users who follow a board
![Page 120: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/120.jpg)
![Page 121: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/121.jpg)
Theories about board follows
![Page 122: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/122.jpg)
New design Old design
![Page 123: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/123.jpg)
“It doesn’t matter how beautiful your theory is … If it doesn’t agree with experiment, it’s wrong.”
– Richard Feynman
![Page 124: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/124.jpg)
Show likes and repins
![Page 125: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/125.jpg)
Show related boards
![Page 126: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/126.jpg)
Show board & user under closeup
![Page 127: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/127.jpg)
Red follow button on sidebar
![Page 128: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/128.jpg)
Board suggestion after repin
![Page 129: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/129.jpg)
opt in forced
date
% of users who follow a board
experiment on follows
roll out
![Page 130: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/130.jpg)
should we ship now?
![Page 131: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/131.jpg)
% of users who follow a user
opt in forced date
novelty effect
![Page 132: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/132.jpg)
should we ship now?
![Page 133: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/133.jpg)
a controlled experiment gives you
disaster avoidance
![Page 134: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/134.jpg)
a controlled experiment gives you
accurate forecasting
![Page 135: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/135.jpg)
a controlled experiment gives you
confidence
![Page 136: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/136.jpg)
Roll out with confidence opt in forced
round 2
near-complete rollout
traffi
c on
new
site
![Page 137: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/137.jpg)
AND THEY LIVED
Happily ever after
![Page 138: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/138.jpg)
Epilogue
![Page 139: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/139.jpg)
Better logging
where users click
![Page 140: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/140.jpg)
More experimentation start of web
redesign experiment
month
# of
exp
erim
ents
![Page 141: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/141.jpg)
Automated diagnostic tools
![Page 142: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/142.jpg)
Experiment process
hypotheses decision criteria
![Page 143: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/143.jpg)
Lessons from our grand experiment
![Page 144: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/144.jpg)
be prepared: your org will propose big changes
![Page 145: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/145.jpg)
help your company decide when to go big
![Page 146: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/146.jpg)
consider the tradeoffs: less time testing, but more confusion
![Page 147: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/147.jpg)
consider the tradeoffs: incremental change = fast learning
![Page 148: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/148.jpg)
help your company learn from big changes
![Page 149: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/149.jpg)
measure the big bets: find mountains, not valleys
![Page 150: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/150.jpg)
measure the big bets: run a controlled trial
![Page 151: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/151.jpg)
measure the big bets: avoid the opt-in trap
![Page 152: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/152.jpg)
help your company become more data-driven
![Page 153: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/153.jpg)
learn from your big bets: value of experiments & data
![Page 154: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/154.jpg)
help your company think
![Page 155: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/155.jpg)
Lessons from our grand experiment be prepared: your org will propose and ship big things that are hard to learn from
understand the tradeoffs of rejecting incremental change
run an experiment to learn, and to avoid the valleys
randomize all users and measure predefined success criteria
expect high cost and high stakes, but extract high benefit
![Page 156: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/156.jpg)
Thanks!
Questions?
![Page 157: Too big to fail · Too big to fail: A/B tests when you’re betting the bank Andrea Burbank (@arburbank) November 4, 2014 #qconsf](https://reader036.vdocuments.site/reader036/viewer/2022080209/5f78358a81db70725a7799b5/html5/thumbnails/157.jpg)
References • www.infoq.com/presentations/ab-testing-
pinterest • www.pinterest.com/burbsie11/ab-testing/ • about.pinterest.com/en/careers