The Data Never LiesHow to Get the Truth from Marketing Data

If you torture the data long enough, it will confess.”

- Ronald Coase

“

You, the marketer

The poor,poor data

Client Case: Men’s Skincare Product

• Holiday Gift Promotion

• Bootstrapped (Small Budget)

• Sell Primarily Online

• Ran Facebook Ads, No Sales

What’s the problem?

• Strategy

• Tactics

• Creative

Types of Problems You Can Have

or

Marketing Problems Statistics Problems

• Not Enough Data

• Bad Experiment Design

• Correlation vs. Causation

These are “real” problems.Things that we should test.

We’re wrong.But, we don’t know we’re wrong.

Resources:

Today’s Full Slide Deck

• On SlideShare

Open Source Statistics Textbook

After Today’s Talk

Subscribe at nickrosener.co/breakfast

Extra Goodies for Subscribers

Download and Use With Clients:

Excel Files

• Simulations and Calculations

Marketing Experiment Report Template

“Real” Marketing Problems

Strategy

Tactics

Creative

Things Like:

• Segmentation

• Targeting

• Positioning

• Product

“Real” Marketing Problems

Strategy

Tactics

Creative

Things Like:

• Social Media

Platform Choice

• Ad Format

“Real” Marketing Problems

Strategy

Tactics

Creative

Things Like:

• Copy

• Images

• Video

The Stats ProblemsWhat we’re going to cover

Estimation(basically, measurement)

Experiments(how to test stuff)

and

Up next:

Model the Process

Impressions

Profit

Visits to Landing Page

Purchases

Impressions:

Clickthrough Rate:

Cost / Click: $1

Conversion Rate:

Retail Price: $100/item

Gross Profit Margin: 50%

Gross Profit: $50/item

Model the Process

Impressions

Profit

Visits to Landing Page

Purchases

Impressions: Not Important*

Clickthrough Rate: Not Important*

Cost / Click: $1

Conversion Rate:

Retail Price: $100/item

Gross Profit Margin: 50%

Gross Profit: $50/item

Model the Process

Impressions

Profit

Visits to Landing Page

Purchases

Impressions: Not Important*

Clickthrough Rate: Not Important*

Cost / Click: $1

Conversion Rate:

Retail Price: $100/item

Gross Profit Margin: 50%

Gross Profit: $50/item

At breakeven, total ad cost = total gross profit

Model the Process

Impressions

Profit

Visits to Landing Page

Purchases

Impressions: Not Important*

Clickthrough Rate: Not Important*

Cost / Click: $1

Conversion Rate: 2% @ breakeven

Retail Price: $100/item

Gross Profit Margin: 50%

Gross Profit: $50/item

At breakeven, total ad cost = total gross profit

Now, New Questions

How do we find out if the adhas a conversion rate that is above breakeven?

(vs. a conversion rate of zero)

and

How much money do we needto spend to find out?

Estimate “The Truth”

Estimating a Proportion

Lots of thingsin marketing

are proportions.

Conversion Rates

Engagement Rates

Clickthrough Rates

Open Rates

How it Works

Your Data (Sample)

Estimating a Proportion

To estimate a conversion rate:

1. Set up an ad

2. Let it run

3. Count the # of purchases attributed to the ad

4. Divide by total # of clicks

Estimating a Proportion

Smaller Sample=

Higher Probabilityof Being Wrong

(and not knowing it)

=

More Affordable

Larger Sample

=

More ConfidenceThat You’re Right

=

More Expensive

>

Spending $5 on an Ad

90.4% of the time, our estimate of conversion is 0%while the true conversion rate is 2%.

Estimated Conversion %

Probability

0% 20% 40% 60% 80%

90.4% 9.2% <1% 0% 0%

100%

0%

Watch Your Stopping Point

$100

Bigger Budget = More Reliable

$500

Nick’s Budget Rule of Thumb

In our example, that’s $250 - $500+ @ $1/click and a 2% conversion rate.

Spend enough where you would expect to getat least 5 – 10+ “events.”

The Stats ProblemsWhat we’re going to cover

Estimation(basically, measurement)

Experiments(how to test stuff)

and

Let’s switch gears:

Study Population

Why Do Experiments?

Using “When to Poston Social Media” Studies

Here’s why:

Your Community

= , IMHO.

≠

Correlation

Why Do Experiments?

Using “When to Poston Social Media” Studies

Here’s why:

Causation

= , IMHO.

≠

Retrospective

Why Do Experiments?

Using “When to Poston Social Media” Studies

Here’s why:

Experiment

= , IMHO.

≠

Why Experiments > Retrospective

Meet John,a Social Media Manager

When are my customersmost engaged?

I’ll analyze my posts,and see when I got

the highest engagement!

Why Experiments > Retrospective

Looks like my customers prefer

to engage at midday.

This may be true.But there could be

another explanation.

Why Experiments > Retrospective

Wait. I also eat baconand drink coffee at

10:00 every day.

Hungry andTired

Bacon-InspiredBrilliance

Post-CoffeeCrash Remember:

Correlation ≠ Causation

Typical Problems with Retrospectives

Hidden Third Factor

ObservableThing 1

ObservableThing 2

HiddenThing

Causes Also Causes

Correlated,but not causal

Stats Concept:Lurking Variable

Typical Problems with Retrospectives

“Fishing” for Insights

What

Stats Concept:Spurious Correlation

Disclaimer:This is also a problem for poorly-designed

experiments.

Post Afternoon

Simple Randomized Experiment Design

Post Early

Randomizeinto Groups

Write Social Media Posts

Post Midday

MeasureMeasure

Measure

Sample Size:At least 5 – 10 events

per group.

What We Went OverWe covered a ton.

1. “Real” Marketing Problems vs. Statistics Problems

2. Model the Process

3. Estimate a Proportion

4. Bigger Sample = More Confident

5. At Least 5 – 10+ “Events” Per Group

6. Experiments > Retrospective, When Possible

Resources:Slides, Excel Files, Report Templateshttp://nickrosener.co/breakfast

Contact:@nickrosenernick@nickrosener.co

?

Questions