mastering analytics for optimisation success · abraham maslow, “the psychology of science”...

@michelejkiss #CJAM19

Mastering Analytics for Optimisation SuccessMichele Kiss, Analytics Demystified


Analytics is my jam


So why am I here?



Analytics can help...Guide Your Success Metrics


Analytics should help youdefine your experiment’s success metric


define your experiment’s success metric



• Purchase; and• Email Signup; and• Create an Account

A single success metric

✗


“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail."

Abraham Maslow, “The Psychology of Science” (1966)


Data can come from many sources…


Analytics can help you … Understand your benchmarks


“Our conversion rate is 2%”


The average human has one breast and one testicle


Our conversion rate


Uhhhhh, except…


What benchmark are you using?

Promo period


Better control the uncontrollable

THE POWER

THE ABSOLUTE POWER

Analytics can help you...


Statistical significance is not enough

“Another way to think of it is that statistical significance isn’t enough. Large sites … get thousands of conversions per hour, and can see significance from their testing efforts quite quickly.

But this doesn’t mean they should only run their tests for a few hours. Rather, they should run their tests long enough to capture a representative sample of their users over time.”

- Peter Borden, Sum All


Random assignment.Not random sampling.

Credit: Matt Gershoff


We need to understand our

Artwork © Ryan Kapp

seasonality


Daily Seasonality


Weekly Seasonality


Yearly seasonality


Inform Test IdeasAnalytics can…


Reporting / Analysis

Opportunity

Test runsResults

Further analysis


Funnel Drop-Off Points


Tricky form fields


Product Interest


Prioritise Test Ideas Analytics can help you …


Prioritising requests

Level of Effort v. Estimated Return


Revenue Forecastmkiss.me/forecast-impact


Revenue Forecast


Insert our current data


Start layering our assumptionsWe assume that some people who didn’t view images would have viewed a video, if we had them.


Conversion AssumptionsWe assume conversion to cart will improve, due to “added persuasion” from videos


Conversion AssumptionsWe’re not going to assume an increase from cart to purchase, or in AOV (but you could)


Voila!


mkiss.me/forecast-impact


Work with your Product Managers .


Track Forecast vs. Actual(Keep people honest!)


Lend Credibility To Your TestingAnalytics can…


Ever doubted your testing tool?


A/A TestsControl Control

vs


95% Confidence

95 times you’ll see the same winner

5 times you’ll see different results


95% Confidence

95 times you’ll see the same winner

5 times you’ll see different results

So 5% of the time, an A/A test may show a “winner”!


A/A Tests

A vs. A:

60% lift with

99.94% confidence


A/A testsDigging in…

Influx of paid media


Cross-variation pollutionAre users falling in to multiple test variations?


Cross-variation pollutionDeployed on a new brand:


New Signup FlowStep 1 Step 2 Step 3 Signup

Complete

Step 1 Step 11 SignupComplete


The Problem?Step 1




The Problem?Step 1 Step 2 Step 3 Signup

Complete



You don’t need a broken testing platform for this to matter…


Overlapping Test Groups

Home Page

Conversion

PricingPage


Dive deeper in to resultsAnalytics can help you...


Don’t forget your success metric!

This isn’t permission to judge your test by “anything positive you can find”!


Landing page redesignControl

vs

RedesignCheck

1 Smith Street, San Jose Check

Check

Check

Check

Performed

-8% worse



vs

RedesignCheck


Check

Check

Check

PerformedSimilarly



vs

RedesignCheck


Check

Check

CheckPerformedSimilarly



vs

RedesignCheck


Check

Check

Check



vs

RedesignCheck


Check

Check



Skewed test

Control

Variation 1

Variation 2

Variation 3

Variation 4


{example} Skewed testWhy didn’t our offline data show

the same winner?


Bad Robot!

Control

Variation 1

Variation 2

Variation 3

Variation 4

✗


Skewed testNot the only time! Months later:

huge spike in traffic to control


Signup Flow TestStep 1 Step 2 Step 3 Signup

Complete

Steps1-X

SignupComplete

Step 4 Step 5 Step X

Performed same or

-20% worse


Signup Flow Test


Testing Past the Online ConversionAnalytics can help you


Trial to Paid


Trial PeriodControl

~~~~~~~~~~~~30-DAY TRIAL!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Variation

~~~~~~~~~~~~14-DAY TRIAL!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

No diff in trial starts

+4% paid conv.


eCommerce


Go forth and analyse!Accurate benchmarks Find opportunitiesEstimate the impactRun reliable testsDig in to your resultsTest past online


Questions?@MicheleJKiss

michelejkissjoin.measure.chat

[email protected]

mastering analytics for optimisation success · abraham maslow, “the psychology of science”...

Documents