hypothesis testing workshop (cezary.co)

HYPOTHESIS TESTING WORKSHOP

CEZARY PIETRZAK

Intel EdTech Accelerator Redwood City, California

August 4, 2015

@ckp

MY BACKGROUND

2000 - 2004

2005 - 2008

CEZARY & CO

2008 - 2012

2012

2013

2014

AGENDA WORKSHOP (1:30PM - 4:30PM)

Developing hypotheses (25min) Exercise: Create your own (60min)

Break (10min)

Testing hypotheses (15min)Exercise: Deploy a test (70min)

HYPOTHESIS STATEMENTS ARE DECLARATIVE

Definition States a fact or an argument

Ends with a full stop

Example I believe that…(X is true)

THEY BEGIN WITH THE CUSTOMER-PROBLEM

Customer-Problem Customer X has Problem X

(that is worth solving)

I believe that… High school students struggle with

basic data literacy

*Always phrase the problem from the customer’s perspective *Add emotional language whenever possible

AND THEN ADDRESS THE PROBLEM-SOLUTION

Problem-Solution Problem X can be solved

by Solution X

I believe that… Bringing real-life data into the classroom

will make math and science more engaging

*Only relevant after you’ve validated that a problem exists *Solution should be unique to alternatives

STRONG HYPOTHESES ARE DERIVED FROM DATA & INSIGHTS

Customer-Problem High school students struggle with

basic data literacy

Rationale 25M high school students are

falling behind in quantitative subjects

Problem-Solution Bringing real-life data into the classroom

will make math and science more engaging

Rationale The explosion of data today makes it

more accessible than ever before

AND ARE ALWAYS BASED ON ASSUMPTIONS

Customer-Problem High school students struggle with

basic data literacy

Assumption Parents and school districts consider

math and science a high priority

Problem-Solution Bringing real-life data into the classroom

will make math and science more engaging

Assumption Most students are genuinely interested

in math and science

*Focus on the riskiest assumption

MANY EARLY-STAGE STARTUPS TEST CORE HYPOTHESES WITH THE EXPERIMENT BOARD

BEFORE MOVING TO MORE GRANULAR TESTING

Customer Overarching persona

Specific segment Stage in lifecycle

Acquisition source Geography/location

Users vs. influencers vs. decision-makers

Problem / Challenge Master problem

Sub-problem Distribution/marketing channels

Buying process/sales funnel Willingness to pay/pricing

Messaging Loyalty/retention

Evangelism/virality

Solution Existing product/service

New product/service Specific feature/flow

Marketing tactic

*Keep one variable constant

YOU CAN SOURCE HYPOTHESES FROM MANY PLACES

Behavioral insights Background research Industry benchmarks

Company stage Growth goals

Investor requirements Previous hypotheses

Highest risk Biggest potential impact

Start high-level, then get specific

Questions to ask Is the hypothesis core to your value proposition?

Will validating the hypothesis significantly de-risk your business? For each hypothesis, have you validated the higher-level one above it?

BUT MUST PRIORITIZE THEM TO CREATE VALUE

LAUNCH EXPERIMENTS IF THEY MEET 3 CRITERIA

Speed “one-afternoon” rule simple over elaborate

Costcheap tools sweat equity shared resources

Accuracy right audience right approach

SET GOALS AND SUCCESS METRICS

Goals should be quantitative when possible Qualitative results can be measured

Examples A minimum of X respondents will agree

The majority of respondents will exhibit positive language User will convert at a 5% significance level

Overall retention will rise by X% 1 in 5 people will recommend a friend

FOLLOW THESE BEST PRACTICES

Baseline Better

Keep it simple Customers understand the detailed technology behind my product

Customers understand why my product is faster than competition

Break big hypotheses into smaller batches I can acquire users via paid spend

I can acquire users via social ads I can acquire users via paid content I can acquire users via trade shows

Add details where possible

I can acquire new users from Facebook efficiently

I can acquire new users from Facebook mobile ads at $1/click

Test one thing at a time (avoid: and/either/or)

Customers prefer product X over product Y or Z

Customers prefer product X over Y Customers prefer product X over Z

BREAKOUTS(60 minutes)

Define core customer-problem & problem-solution Compile & prioritize list of key hypotheses

Share work with group

bitly.com/hypothtesting

PART 2(15 minutes)

Testing hypotheses

START WITH QUALITATIVE TESTING TO GATHER FEEDBACK

In-person conversations / live demos Focus groups

Video conversations User testing / video recording

Public forums / blogs Surveys ($ or no $)

FOCUS THE CONVERSATION FOR BETTER RESULTS

Don’ts Do’sDon’t rely on friends and family Do speak with outsiders who lack a filterDon’t tell respondents you’re working on an idea

Do tell respondents that you’re seeking an opinion

Don’t ask leading questions Isn’t it terrible when…?

Do let them describe their environment and motivations

Don’t provide options unless necessaryIs your preference A, B or C? Do let people decide on the answer

Don’t ask hypotheticalsWhat would you do if…?

Do ask about past behaviorHow have you solved this problem in the past?

Don’t ask about intentWould you tells others about this product?

Do test their willingness to take action Can you share this product with a friend?

USE QUANTITATIVE TESTING TO CAPTURE KEY DATA AND IDENTIFY ACTION

Product analytics Heat maps

Social media ads Paid search

Landing pages Placeholders / waitlists

A/B testing

ENSURE THAT YOUR DATA IS RELIABLE

Questions to ask Who is your audience?

How did they come to your app? Are the results real or sampled?

Is the sample representative of your audience? How is significance determined?

EVALUATE RESULTS

Significance of sample Minimum 10 responses for qual, 30 for quant

Tangible outcome Signed letter of intent, verbal agreement

Strong body language Overall enthusiasm, leaning in

Correct metric hierarchy Later funnel metrics trumps early funnel

If data is ambivalent, revisit hypothesis

BREAKOUTS(70 minutes)

Define your experiment Choose tools

Start implementing

LEARN ABOUT MARKETING + GROWTH + MOBILE AT

CEZARY.CO@ckp