Designing & Measuring Experiments
@GoAgileCamp#GoAgileCamp
Real Experiment
Create a social news reader that delivers inspiring articles to you based on your likes and follows.
You’ll know if it’s successful based on daily usage and the viral coefficient.
Constraints
You have no funding but two great developers.
Vision
Common Mistake #1:Not having a Learning Goal
Real Experiment
Learning goal:Do customers want our VP?
Hypothesis:50% of people we interview will commit to paying $20 a month for the product.
Plan:Pitch it, see who says yes!
Common Mistake #2:Assuming the hypothesis
is a hypothesis
Research
• Who is our customer?• What are their pains?• What job needs to be done?• Is our customer segment too
broad?• How do we find them?• How can we solve this problem?• What form should this take?• How important is the design?• What’s the quickest hack?• What’s the minimum feature set?• How should we prioritize?
Generate Ideas & Identify Assumptions
Experiments
• How much will they pay?• How do we convince them to buy?• How much will it cost to sell?• Can we scale marketing?• Is this solution working?• Are people using it?• Which solution is better?• How should we optimize this?• What do people like / dislike?• Why do they do that?
Test Hypotheses
Real Experiment
Learning goal:Which value prop is better? Faster download time or obfuscated IP address.
Hypothesis:Using VP of Faster download time instead of obfuscated IP address will increase conversion rate on home page above 15% (our current benchmark).
Plan:Use optimizely to change VP on home page for 2000 visits.
Time box: 7 days
Result: 16%, Win!
14
15
16
17
18
Version A Version B
Common Mistake #3:Measuring, misreading, and miscalculating the metric
14
15
16
17
18
Version A Version B
14
15
16
17
18
Version A Version B
14
15
16
17
18
Version A Version B
Defining Good Experiments
!Experiment Herder Template* by TriKro LLC!(For S.M.A.R.T Experiments: Smart Measurable Achievable Relevant Timely)
Learning Goal!!What do we need to learn?!What is our riskiest assumption?!What is our one priority?
Hypothesis!!Is it falsifiable?!Is it specific?!Is it causal? e.g. If X then Y!Is it Relevant to the learning goal?
Metric!!Qualitative or quantitative?!Is it actionable?!Is it Measurable?
Fail Condition!!!!!If this happens,!our hypothesis is clearly false!
wtf,s!!!!!!If this happens, stop!!Experiment is broken, retro!
Experiment!!How will you collect the data?!Is it Specific?!Is it Achievable?!!Link to any supporting documents
Time Box!!Is the experiment Timely?!Can we get data faster?!Would less data be sufficient?!Is the retro in our calendar?
Start End Retro
Result!!What happened?!What data did we collect?!Anything unexpected?
Next Step!!Pivot or persevere?!Another experiment for this goal?!Do we need to clean up?
Designed by: TriKro LLC - Lean Coaching and Innovation Ecosystem Design!The Smart Experiment Template by TriKro LLC is licensed under!a Creative Commons Attribution-ShareAlike 4.0 International License
Designed for:
Designed by:
* Templates are bad,! make your own!
Common Mistake #1:Not having a learning goal
Common Mistake #2:Assuming the hypothesis
is a hypothesis
Common Mistake #3:Measuring, misreading, and miscalculating the metric
Common Mistake #4:Failure to define failure
Common Mistake #5:Actually…
people get this one right
Common Mistake #5:Too much time
Common Mistake #6:Regretfully not doing retros
Common Mistake #7:Ignoring results
Common Mistake #8:Not learning from learning
Common Mistake #9:Ignoring warning signs
Experiment/Research Templates*
1. Learning Goal2. Hypothesis/Assumptions3. Metric4. Fail Condition5. Plan6. Time Box7. Results8. Next Step9. wtf,s!
TheReal Startup Book
• 5 second tests• Comprehension• Conjoint Analysis• Data mining / market research
• Surveys* (closed)
• Smoke tests• Video• Landing page• Sales pitch
• Customer Discovery Interviews
• Contextual inquiry / ethnography
• Data mining• Focus groups*• Surveys* (open ended)
• Solution interview
• Contextual inquiry / ethnography
• Demo pitch• Concierge test / Consulting
• Competitor Usability
• Picnic in the Graveyard
• Paper prototypes• Clickable prototypes
This presentation by TriKro LLC is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License excepting those images where other copyright holders are noted.
Based on a work at http://GrasshopperHerder.com.
