4 steps toward scientific a/b testing

Download 4 Steps Toward Scientific A/B Testing

Post on 05-Sep-2014




0 download

Embed Size (px)


To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.


  • What is A/B Testing?
  • #ScienceOfTesting A/B testing is not... Validation of guesswork Consumer psychology gimmicks Meek Tweaking Images: Hubspot, Conversion Rate Experts
  • #ScienceOfTesting Its also not... A waste of time Impossible to get right Beyond the scope of your job
  • A/B Testing: Dened Conducting experiments to optimize your customer experience. What is A/B testing? OR
  • 4 Steps of Scientic A/B Testing
  • #ScienceOfTesting The 4 Steps of A/B Testing Step 1 Analyze data Step 2 Form a hypothesis Step 3 Construct an experiment Step 4 Interpret results
  • Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting
  • #ScienceOfTesting Use quantitative & qualitative data Quantitative data tells you where to test Qualitative data gives you an idea of what should be tested
  • #ScienceOfTesting Quantitative datasets Web trac Email marketing Order history CRM interactions Support tickets and more!
  • #ScienceOfTesting Run high-impact tests
  • Dont choose tests randomly Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/ how-to-use-data-to-choose-your-next-ab-test/
  • #ScienceOfTesting Qualitative data User testing Survey data Heat mapping Your sales & account teams
  • #ScienceOfTesting Parts of a hypothesis If [Variable], then [Result], because [Rationale]. The element that is modied Isolate one variable for an A/B test Call to action, visual media, forms
  • #ScienceOfTesting Parts of a hypothesis If [Variable], then [Result], because [Rationale]. The predicted outcome Use data to determine the size of eect More email sign-ups, clicks on a CTA
  • #ScienceOfTesting Parts of a hypothesis If [Variable], then [Result], because [Rationale]. Demonstrate your customer knowledge What assumption will be proven wrong if the experiment is a draw or loses?
  • #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the call-to-action is shorter, the conversion rate will increase. Strong Hypothesis If the call-to-action text is changed to Complete My Order, the conversion rates in the checkout will increase, because the copy is more specic and personalized.
  • #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the checkout funnel is shortened to fewer pages, the checkout completion rate will increase. Strong Hypothesis If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our trac drop out of the funnel by clicking on these links.
  • A/B Testing: Dened Every test has 3 parts DESIGNTECH CONTENT
  • #ScienceOfTesting Content: What are you saying? VS.
  • #ScienceOfTesting Design: How does it look? VS.
  • #ScienceOfTesting Tech: How does it work? VS.
  • The most eective tests often combine all 3 elements: content, design, tech #ScienceOfTesting
  • #ScienceOfTesting What are we looking for? How condent am I that the observed dierence from my experiment was not due to chance? 95% Statistical Signicance = 5% probability that the observed dierence was due to chance.
  • #ScienceOfTesting Condence intervals High statistical condence Lower risk of implementing a test that won by chance
  • #ScienceOfTesting Sample size calculator http://optimize.ly/StatCalculator
  • #ScienceOfTesting Once you reach signicance: Variation wins: Launch the variation or update your website. Original wins: Learn why hypothesis was incorrect. In either case: Think about what to test next.
  • Examples!
  • A/B Testing: Dened A simple test
  • A/B Testing: Dened Iterative testing on a core hypothesis A solid test
  • A/B Testing: Dened Cohort analysis + website changes + biz process changes A more complicated test A B
  • #ScienceOfTesting Step 1: Data collection
  • #ScienceOfTesting Step 2: Hypothesis If [Variable], then [Result], because [Rationale]. If prospects access to a free trial is gated by a conversation with a sales rep, well be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.
  • #ScienceOfTesting Step 3: Experiment Changes to heading text Custom elds in Salesforce.com Business process changes for sales reps Custom analysis in RJMetrics based on oine conversion event
  • #ScienceOfTesting Step 4: Results TBD A B
  • Arm Your Organization
  • Marketing Increase the impact of your tests by bringing more team members into the process #ScienceOfTesting Product Sales Engineering
  • Document your test results in a central repository. #ScienceOfTesting Heat maps Optimizely results Hypothesis What we learned Variations
  • #ScienceOfTesting Other tried and true tactics Build excitement by sharing your wins with the company Hold a competition for the biggest winning variation Votes on variations to see who has the highest accuracy of predicting winners
  • Thanks!