agilecamp silicon valley 2015: experiment design

Designing & Measuring Experiments

@GoAgileCamp#GoAgileCamp

Tristan.Kromer@TriKro.com

WWW

@TriKro

Tristan Kromer

i@mNotaNinja.comGrasshopperHerder.com

Real Experiment

Create a social news reader that delivers inspiring articles to you based on your likes and follows.

You’ll know if it’s successful based on daily usage and the viral coefficient.

Constraints

You have no funding but two great developers.

Vision

Common Mistake #1:Not having a Learning Goal

Real Experiment

Learning goal:Do customers want our VP?

Hypothesis:50% of people we interview will commit to paying $20 a month for the product.

Plan:Pitch it, see who says yes!

Common Mistake #2:Assuming the hypothesis

is a hypothesis

Research

• Who is our customer?• What are their pains?• What job needs to be done?• Is our customer segment too

broad?• How do we find them?• How can we solve this problem?• What form should this take?• How important is the design?• What’s the quickest hack?• What’s the minimum feature set?• How should we prioritize?

Generate Ideas & Identify Assumptions

Experiments

• How much will they pay?• How do we convince them to buy?• How much will it cost to sell?• Can we scale marketing?• Is this solution working?• Are people using it?• Which solution is better?• How should we optimize this?• What do people like / dislike?• Why do they do that?

Test Hypotheses

Real Experiment

Learning goal:Which value prop is better? Faster download time or obfuscated IP address.

Hypothesis:Using VP of Faster download time instead of obfuscated IP address will increase conversion rate on home page above 15% (our current benchmark).

Plan:Use optimizely to change VP on home page for 2000 visits.

Time box: 7 days

Result: 16%, Win!

14

15

16

17

18

Version A Version B

Common Mistake #3:Measuring, misreading, and miscalculating the metric

14

15

16

17

18

Version A Version B

14

15

16

17

18

Version A Version B

14

15

16

17

18

Version A Version B

Defining Good Experiments

!Experiment Herder Template* by TriKro LLC!(For S.M.A.R.T Experiments: Smart Measurable Achievable Relevant Timely)

Learning Goal!!What do we need to learn?!What is our riskiest assumption?!What is our one priority?

Hypothesis!!Is it falsifiable?!Is it specific?!Is it causal? e.g. If X then Y!Is it Relevant to the learning goal?

Metric!!Qualitative or quantitative?!Is it actionable?!Is it Measurable?

Fail Condition!!!!!If this happens,!our hypothesis is clearly false!

wtf,s!!!!!!If this happens, stop!!Experiment is broken, retro!

Experiment!!How will you collect the data?!Is it Specific?!Is it Achievable?!!Link to any supporting documents

Time Box!!Is the experiment Timely?!Can we get data faster?!Would less data be sufficient?!Is the retro in our calendar?

Start End Retro

Result!!What happened?!What data did we collect?!Anything unexpected?

Next Step!!Pivot or persevere?!Another experiment for this goal?!Do we need to clean up?

Designed by: TriKro LLC - Lean Coaching and Innovation Ecosystem Design!The Smart Experiment Template by TriKro LLC is licensed under!a Creative Commons Attribution-ShareAlike 4.0 International License

Designed for:

Designed by:

* Templates are bad,! make your own!

Common Mistake #1:Not having a learning goal

Common Mistake #2:Assuming the hypothesis

is a hypothesis

Common Mistake #3:Measuring, misreading, and miscalculating the metric

Common Mistake #4:Failure to define failure

Common Mistake #5:Actually…

people get this one right

Common Mistake #5:Too much time

Common Mistake #6:Regretfully not doing retros

Common Mistake #7:Ignoring results

Common Mistake #8:Not learning from learning

Common Mistake #9:Ignoring warning signs

Experiment/Research Templates*

1. Learning Goal2. Hypothesis/Assumptions3. Metric4. Fail Condition5. Plan6. Time Box7. Results8. Next Step9. wtf,s!

TheReal Startup Book

• 5 second tests• Comprehension• Conjoint Analysis• Data mining / market research

• Surveys* (closed)

• Smoke tests• Video• Landing page• Sales pitch

• Customer Discovery Interviews

• Contextual inquiry / ethnography

• Data mining• Focus groups*• Surveys* (open ended)

• Solution interview

• Contextual inquiry / ethnography

• Demo pitch• Concierge test / Consulting

• Competitor Usability

• Picnic in the Graveyard

• Paper prototypes• Clickable prototypes

For more info and updates:http://trikro.com/downloads/playbook

Tristan.Kromer@TriKro.com

WWW

@TriKro

Tristan Kromer

i@mNotaNinja.comGrasshopperHerder.com

This presentation by TriKro LLC is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License excepting those images where other copyright holders are noted.

Based on a work at http://GrasshopperHerder.com.