mastering analytics for optimisation success · abraham maslow, “the psychology of science”...

@michelejkiss #CJAM19

Mastering Analytics for Optimisation SuccessMichele Kiss, Analytics Demystified

@michelejkiss #CJAM19

Analytics is my jam

@michelejkiss #CJAM19

So why am I here?

@michelejkiss #CJAM19

@michelejkiss #CJAM19

Analytics can help...Guide Your Success Metrics

@michelejkiss #CJAM19

Analytics should help youdefine your experiment’s success metric

@michelejkiss #CJAM19

define your experiment’s success metric

@michelejkiss #CJAM19

@michelejkiss #CJAM19

• Purchase; and• Email Signup; and• Create an Account

A single success metric

✗

@michelejkiss #CJAM19

“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail."

Abraham Maslow, “The Psychology of Science” (1966)

@michelejkiss #CJAM19

Data can come from many sources…

@michelejkiss #CJAM19

Analytics can help you … Understand your benchmarks

@michelejkiss #CJAM19

“Our conversion rate is 2%”

@michelejkiss #CJAM19

@michelejkiss #CJAM19

The average human has one breast and one testicle

@michelejkiss #CJAM19

Our conversion rate

@michelejkiss #CJAM19

Uhhhhh, except…

@michelejkiss #CJAM19

Uhhhhh, except…

@michelejkiss #CJAM19

Uhhhhh, except…

@michelejkiss #CJAM19

What benchmark are you using?

Promo period

@michelejkiss #CJAM19

Better control the uncontrollable

THE POWER

THE ABSOLUTE POWER

Analytics can help you...

@michelejkiss #CJAM19

Statistical significance is not enough

“Another way to think of it is that statistical significance isn’t enough. Large sites … get thousands of conversions per hour, and can see significance from their testing efforts quite quickly.

But this doesn’t mean they should only run their tests for a few hours. Rather, they should run their tests long enough to capture a representative sample of their users over time.”

- Peter Borden, Sum All

@michelejkiss #CJAM19

Random assignment.Not random sampling.

Credit: Matt Gershoff

@michelejkiss #CJAM19

We need to understand our

Artwork © Ryan Kapp

seasonality

@michelejkiss #CJAM19

Daily Seasonality

@michelejkiss #CJAM19

Daily Seasonality

@michelejkiss #CJAM19

Weekly Seasonality

@michelejkiss #CJAM19

Weekly Seasonality

@michelejkiss #CJAM19

Yearly seasonality

@michelejkiss #CJAM19

Inform Test IdeasAnalytics can…

@michelejkiss #CJAM19

Reporting / Analysis

Opportunity

Test runsResults

Further analysis

@michelejkiss #CJAM19

@michelejkiss #CJAM19

Funnel Drop-Off Points

@michelejkiss #CJAM19

Tricky form fields

@michelejkiss #CJAM19

Product Interest

@michelejkiss #CJAM19

Prioritise Test Ideas Analytics can help you …

@michelejkiss #CJAM19

Prioritising requests

Level of Effort v. Estimated Return

@michelejkiss #CJAM19

Revenue Forecastmkiss.me/forecast-impact

@michelejkiss #CJAM19

Revenue Forecast

@michelejkiss #CJAM19

Insert our current data

@michelejkiss #CJAM19

Start layering our assumptionsWe assume that some people who didn’t view images would have viewed a video, if we had them.

@michelejkiss #CJAM19

Conversion AssumptionsWe assume conversion to cart will improve, due to “added persuasion” from videos

@michelejkiss #CJAM19

Conversion AssumptionsWe’re not going to assume an increase from cart to purchase, or in AOV (but you could)

@michelejkiss #CJAM19

Voila!

@michelejkiss #CJAM19

mkiss.me/forecast-impact

@michelejkiss #CJAM19

Work with your Product Managers .

@michelejkiss #CJAM19

Track Forecast vs. Actual(Keep people honest!)

@michelejkiss #CJAM19

Lend Credibility To Your TestingAnalytics can…

@michelejkiss #CJAM19

Ever doubted your testing tool?

@michelejkiss #CJAM19

A/A TestsControl Control

vs

@michelejkiss #CJAM19

95% Confidence

95 times you’ll see the same winner

5 times you’ll see different results

@michelejkiss #CJAM19

95% Confidence

95 times you’ll see the same winner

5 times you’ll see different results

So 5% of the time, an A/A test may show a “winner”!

@michelejkiss #CJAM19

A/A Tests

A vs. A:

60% lift with

99.94% confidence

@michelejkiss #CJAM19

A/A testsDigging in…

Influx of paid media

@michelejkiss #CJAM19

Cross-variation pollutionAre users falling in to multiple test variations?

@michelejkiss #CJAM19

Cross-variation pollutionDeployed on a new brand:

@michelejkiss #CJAM19

Cross-variation pollutionDeployed on a new brand:

@michelejkiss #CJAM19

New Signup FlowStep 1 Step 2 Step 3 Signup

Complete

Step 1 Step 11 SignupComplete

@michelejkiss #CJAM19

The Problem?Step 1

Step 1 Step 11 SignupComplete

Step 2 Step 3 SignupComplete

@michelejkiss #CJAM19

The Problem?Step 1 Step 2 Step 3 Signup

Complete

Step 1 Step 11 SignupComplete

@michelejkiss #CJAM19

You don’t need a broken testing platform for this to matter…

@michelejkiss #CJAM19

Overlapping Test Groups

Home Page

Conversion

PricingPage

@michelejkiss #CJAM19

Dive deeper in to resultsAnalytics can help you...

@michelejkiss #CJAM19

Don’t forget your success metric!

This isn’t permission to judge your test by “anything positive you can find”!

@michelejkiss #CJAM19

Landing page redesignControl

vs

RedesignCheck

1 Smith Street, San Jose Check

Check

Check

Check

Performed

-8% worse

@michelejkiss #CJAM19

Landing page redesignControl

vs

RedesignCheck

1 Smith Street, San Jose Check

Check

Check

Check

PerformedSimilarly

@michelejkiss #CJAM19

Landing page redesignControl

vs

RedesignCheck

1 Smith Street, San Jose Check

Check

Check

CheckPerformedSimilarly

@michelejkiss #CJAM19

Landing page redesignControl

vs

RedesignCheck

1 Smith Street, San Jose Check

Check

Check

Check

@michelejkiss #CJAM19

Landing page redesignControl

vs

RedesignCheck

1 Smith Street, San Jose Check

Check

Check

1 Smith Street, San Jose Check

@michelejkiss #CJAM19

Skewed test

Control

Variation 1

Variation 2

Variation 3

Variation 4

@michelejkiss #CJAM19

{example} Skewed testWhy didn’t our offline data show

the same winner?

@michelejkiss #CJAM19

Bad Robot!

Control

Variation 1

Variation 2

Variation 3

Variation 4

✗

@michelejkiss #CJAM19

Skewed testNot the only time! Months later:

huge spike in traffic to control

@michelejkiss #CJAM19

Signup Flow TestStep 1 Step 2 Step 3 Signup

Complete

Steps1-X

SignupComplete

Step 4 Step 5 Step X

Performed same or

-20% worse

@michelejkiss #CJAM19

Signup Flow Test

@michelejkiss #CJAM19

Testing Past the Online ConversionAnalytics can help you

@michelejkiss #CJAM19

Trial to Paid

@michelejkiss #CJAM19

Trial PeriodControl

~~~~~~~~~~~~30-DAY TRIAL!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Variation

~~~~~~~~~~~~14-DAY TRIAL!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

No diff in trial starts

+4% paid conv.

@michelejkiss #CJAM19

eCommerce

@michelejkiss #CJAM19

eCommerce

@michelejkiss #CJAM19

Go forth and analyse!Accurate benchmarks Find opportunitiesEstimate the impactRun reliable testsDig in to your resultsTest past online

@michelejkiss #CJAM19

Questions?@MicheleJKiss

michelejkissjoin.measure.chat

michele@analyticsdemystified.com