adma video testing and measurement

Australian Direct Marketing Association

Testing

Australian Direct Marketing Association

What is testing?

Testing is a marketing experiment in a live environment:

•  Observing what customers actually do rather than what they say they are going to do

•  Monitoring how results vary as various components of a campaign are altered.

•  Using the results of testing to:

–  predict the outcome of future activity.

–  Improve our marketing performance

Australian Direct Marketing Association

Australian Direct Marketing Association

3 approaches

?

1.  Test on a small scale first then roll-out

3. Same approach to all then analyse results among different segments to spot variations

2. Do testing on a portion of a live campaign

Test

Test

Roll-out

Live campaign

Live campaign

Australian Direct Marketing Association

Summary of testing options

Message components •  Product •  Proposition •  Offer •  Creative •  Call to action

Delivery components •  Targeting & segmentation •  Communication channels •  Format •  Timing

Australian Direct Marketing Association

Test the important things…..

Australian Direct Marketing Association

Testing case study

•  Two simple objectives –  Improve response rates –  Increase amount donated

•  Understanding donor segments –  Relationship to disease –  Value

With permission from Salmat

Australian Direct Marketing Association

•  Relationship to disease –  Have the disease –  Parent of someone with the disease –  Relative / friend of someone with the disease –  No relationship to the disease

With permission from Salmat

Test different propositions for each segment

Australian Direct Marketing Association

•  Value –  Variable donations boxes based on last donation, increased in

increments of 20%

With permission from Salmat

Personalisation case study

Australian Direct Marketing Association

With permission from Salmat

Testing case study

Australian Direct Marketing Association

Example: Landing Page Test Options

Page components 1. Headline 2. Secondary Headline 3 Body Copy 4 Social Proof 5/6 Call to Action 7  Navigation 8  Image 9  Above/below the Fold

Australian Direct Marketing Association

Australian Direct Marketing Association

Guidelines for accurate testing

• Ensure you always have a “control” – So you have a benchmark for comparison

• Have a big enough sample to test with – To produce reliable results

• Ensure your test sample is not skewed at all (compared to control) – Use a randomly-selected sample

• Decide how you are going to measure performance: –The measurement chosen must prove/disprove your hypothesis

Australian Direct Marketing Association

Australian Direct Marketing Association

The importance of a control

Australian Direct Marketing Association

May July June

Response rate

Standard offer

New offer A

New offer B

Here there is no control: - A separate offer has been run in each month - Offer A appears to have out- performed the current offer - Offer B appears to have performed worse = Offer A appears to win

Offer test

Australian Direct Marketing Association

The importance of a control

Australian Direct Marketing Association

May July June

Response rate

Standard offer

New offer A

New offer B

Introduction of control: - The current offer has been run in each month as a benchmark - Offer A did not perform as well as the current offer - Offer B performed better than the current offer = Offer B is the real winner

Offer test

Australian Direct Marketing Association

The “significance” of a test result

Control New Offer A

Response rate

1.7%

1.4%

The test out-performed the control by 0.3% Should you adopt New Offer A – or could it be just a chance result? We need to know whether these results are likely to be repeated if we ran the test again

Australian Direct Marketing Association

Having a Big Enough Sample

• In direct marketing, your sample is typically a subset of your customer base or contact list

•  In digital, your sample is typically the volume of unique visitors or unique impressions accumulated over a period of time

•  You need a quantity of sample that reflects your desire to detect even small differences between your test and control groups

Australian Direct Marketing Association

50%$ 60%$ 70%$ 80%$ 90%$ 95%$ 99%$

Req

uire

d S

ampl

e Q

uant

ity

Ability to Detect Statistically Significant Differences

Australian Direct Marketing Association

Not Just Statistical Significance

Do a sense-check when interpreting results: • What was the competition doing when this test was running? • Just because this worked in one location does it mean it will work in

another? • The offer was successful in Summer – would it still work in Winter? •  Were there any other abnormal factors in the marketplace which

might have affected the response?

Australian Direct Marketing Association

Australian Direct Marketing Association

Not Just Statistical Significance

Did we make money?

When we balance the costs of a campaign against the incremental volume of sales – was it worth it?

What should we test next? e.g. offer A performed better than offer B, what will beat offer A?

Australian Direct Marketing Association

Measurement

Australian Direct Marketing Association

Tracking the customer journey

Awareness Interest Action Desire

A buyer typically goes through several stages before they buy

…which means we should have measures to track performance for each stage

Reach Influence Closure

Australian Direct Marketing Association

Example: eCommerce

Awareness

Interest

Action

Desire

Daily  Unique  Visitors  

Product  Page  Views  

Completes  a  Registra:on  Form  

Purchase  

Australian Direct Marketing Association

Tracking issues

How do you match response to source when:

–  Your campaign runs in several channels simultaneously

–  Your product is sold through a third party sales channel

–  Your product has a long consideration period

Australian Direct Marketing Association

Common tracking options

Unique campaign and comms channel specific phone numbers

Outbound Channel Response Channel

Offline Broadcast

Mail

Display Media

Call-Centre

In-Store

Website

Unique redemption codes

Specific Search Call to Action

Australian Direct Marketing Association

Indirect sales – a missing link

Clicks/visits

Sales

Calls

Known

Known

Unknown

Browse

Enquire

Buy

Australian Direct Marketing Association

Long sales cycles

Sales

Time

What does this response curve look like for your product?

•  Track enquiries & sales over a long period

•  Establish a pattern

•  Create a rule of thumb

Australian Direct Marketing Association

Rule of Thumb Approach

•  Can be used for indirect sales as well as an ‘early read’ for long campaign cycles

•  Typical approach:

1.  Establish a ratio for website visits or calls to reseller enquiries/sales

2.  Establish a pre-campaign baseline for calls and website visits

3.  Measure the uplift in calls/visits during and following the promotion

4.  Extrapolate to sales using typical ratio