strata new york 2012: continuous experimentation with continuous deployment
Post on 13-May-2015
595 Views
Preview:
DESCRIPTION
TRANSCRIPT
Continuous Experimentationwith Continuous
Deployment
Steve MardenfeldEtsy
Monday, November 5, 2012
Monday, November 5, 2012
World’s Handmade Market
Monday, November 5, 2012
Continuous Deployment
Monday, November 5, 2012
Monday, November 5, 2012
Monday, November 5, 2012
Metrics
Monday, November 5, 2012
Graph Factory
Monday, November 5, 2012
Oops
Monday, November 5, 2012
Oops
Monday, November 5, 2012
Good Enough?
Monday, November 5, 2012
What is Better?
Monday, November 5, 2012
Sometimes it’s Obvious
Monday, November 5, 2012
Most Times it’s Not
Monday, November 5, 2012
Most Times it’s Not
Cart Page Restyled
Monday, November 5, 2012
Most Times it’s Not
Etsy Mentioned on Oprah
Internet Goes Down in Brooklyn
Justin Beiber Sneezes
New Cat Meme
Cart Page Restyled
Monday, November 5, 2012
A/B Testing?Monday, November 5, 2012
Monday, November 5, 2012
Monday, November 5, 2012
Test Them Both
Monday, November 5, 2012
Statistics?
Monday, November 5, 2012
First Pass
HADOOP
Monday, November 5, 2012
First Pass
CASCADING
Monday, November 5, 2012
First Pass
R
Monday, November 5, 2012
First Pass
RHADOOP
CASCADING
Monday, November 5, 2012
Automate It
A/B Analyzer
Monday, November 5, 2012
A/B Analyzer
•Framework for looking at experiments
•Done for every experiment beforehand
Monday, November 5, 2012
A/B Analyzer
Monday, November 5, 2012
A/B Analyzer
Monday, November 5, 2012
Monday, November 5, 2012
3 Types of Metrics
•Co-occurence
•visits that did X (searched)
•average number of X (searches)
•Funnels
•Special Cases*
Monday, November 5, 2012
Co-Occurence
•Co-occurence of all actions with the variant
•i.e. - how many people saw this variant and then purchased
•i.e. - average number of purchases per variant
Monday, November 5, 2012
Funnels
BuySearch Listing Buy
Monday, November 5, 2012
Funnels
Buy
BuySearch Listing Buy
Search Listing BuyHome
Monday, November 5, 2012
Funnels
BuyBuy
BuySearch Listing Buy
Search Listing BuyHome
Monday, November 5, 2012
Special Cases
•Exiting pages
•Bounce rate
•Visit length
Monday, November 5, 2012
Pipeline
HDFS Events Visits MySql
A/B
PROCESSING
Client
Logs
DB
COLLECTION
Monday, November 5, 2012
Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012
Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
Idea
Monday, November 5, 2012
Changing Search
Monday, November 5, 2012
Related Searches?
Monday, November 5, 2012
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
Be Specific
•What does an improvement look like?
•More clicks?
•Less searches?
•Shorter visits?
•Longer visits?
Monday, November 5, 2012
Bet Driven Development
Monday, November 5, 2012
Is this even worth it?
Monday, November 5, 2012
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
Code It Up
Monday, November 5, 2012
Code
Monday, November 5, 2012
Turn it On
Monday, November 5, 2012
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
Realtime?
Monday, November 5, 2012
Wait
Monday, November 5, 2012
Process
•Idea
•Hypothesis
•Code
•Wait
•Analysis
Monday, November 5, 2012
Making Sense of ResultsMonday, November 5, 2012
Back To Related Searches
Monday, November 5, 2012
Back To Related Searches
Monday, November 5, 2012
Monday, November 5, 2012
Monday, November 5, 2012
Final Thoughts
•Bet driven development - be explicit about your changes
•Easy experimental setup and quick analysis makes for more experiments
•An experiment is successful if you learned something, not if you launched
Monday, November 5, 2012
Monday, November 5, 2012
http://www.slideshare.net/powerlexis/retro-slides-from-ibm
Monday, November 5, 2012
top related