Download - ADMA Video Testing and Measurement
Australian Direct Marketing Association
Testing
Australian Direct Marketing Association
What is testing?
Testing is a marketing experiment in a live environment:
• Observing what customers actually do rather than what they say they are going to do
• Monitoring how results vary as various components of a campaign are altered.
• Using the results of testing to:
– predict the outcome of future activity.
– Improve our marketing performance
Australian Direct Marketing Association
Australian Direct Marketing Association
3 approaches
?
1. Test on a small scale first then roll-out
3. Same approach to all then analyse results among different segments to spot variations
2. Do testing on a portion of a live campaign
Test
Test
Roll-out
Live campaign
Live campaign
Australian Direct Marketing Association
Summary of testing options
Message components • Product • Proposition • Offer • Creative • Call to action
Delivery components • Targeting & segmentation • Communication channels • Format • Timing
Australian Direct Marketing Association
Test the important things…..
Australian Direct Marketing Association
Testing case study
• Two simple objectives – Improve response rates – Increase amount donated
• Understanding donor segments – Relationship to disease – Value
With permission from Salmat
Australian Direct Marketing Association
• Relationship to disease – Have the disease – Parent of someone with the disease – Relative / friend of someone with the disease – No relationship to the disease
With permission from Salmat
Test different propositions for each segment
Australian Direct Marketing Association
• Value – Variable donations boxes based on last donation, increased in
increments of 20%
With permission from Salmat
Personalisation case study
Australian Direct Marketing Association
With permission from Salmat
Testing case study
Australian Direct Marketing Association
Example: Landing Page Test Options
Page components 1. Headline 2. Secondary Headline 3 Body Copy 4 Social Proof 5/6 Call to Action 7 Navigation 8 Image 9 Above/below the Fold
Australian Direct Marketing Association
Australian Direct Marketing Association
Guidelines for accurate testing
• Ensure you always have a “control” – So you have a benchmark for comparison
• Have a big enough sample to test with – To produce reliable results
• Ensure your test sample is not skewed at all (compared to control) – Use a randomly-selected sample
• Decide how you are going to measure performance: –The measurement chosen must prove/disprove your hypothesis
Australian Direct Marketing Association
Australian Direct Marketing Association
The importance of a control
Australian Direct Marketing Association
May July June
Response rate
Standard offer
New offer A
New offer B
Here there is no control: - A separate offer has been run in each month - Offer A appears to have out- performed the current offer - Offer B appears to have performed worse = Offer A appears to win
Offer test
Australian Direct Marketing Association
The importance of a control
Australian Direct Marketing Association
May July June
Response rate
Standard offer
New offer A
New offer B
Introduction of control: - The current offer has been run in each month as a benchmark - Offer A did not perform as well as the current offer - Offer B performed better than the current offer = Offer B is the real winner
Offer test
Australian Direct Marketing Association
The “significance” of a test result
Control New Offer A
Response rate
1.7%
1.4%
The test out-performed the control by 0.3% Should you adopt New Offer A – or could it be just a chance result? We need to know whether these results are likely to be repeated if we ran the test again
Australian Direct Marketing Association
Having a Big Enough Sample
• In direct marketing, your sample is typically a subset of your customer base or contact list
• In digital, your sample is typically the volume of unique visitors or unique impressions accumulated over a period of time
• You need a quantity of sample that reflects your desire to detect even small differences between your test and control groups
Australian Direct Marketing Association
50%$ 60%$ 70%$ 80%$ 90%$ 95%$ 99%$
Req
uire
d S
ampl
e Q
uant
ity
Ability to Detect Statistically Significant Differences
Australian Direct Marketing Association
Not Just Statistical Significance
Do a sense-check when interpreting results: • What was the competition doing when this test was running? • Just because this worked in one location does it mean it will work in
another? • The offer was successful in Summer – would it still work in Winter? • Were there any other abnormal factors in the marketplace which
might have affected the response?
Australian Direct Marketing Association
Australian Direct Marketing Association
Not Just Statistical Significance
Did we make money?
When we balance the costs of a campaign against the incremental volume of sales – was it worth it?
What should we test next? e.g. offer A performed better than offer B, what will beat offer A?
Australian Direct Marketing Association
Measurement
Australian Direct Marketing Association
Tracking the customer journey
Awareness Interest Action Desire
A buyer typically goes through several stages before they buy
…which means we should have measures to track performance for each stage
Reach Influence Closure
Australian Direct Marketing Association
Example: eCommerce
Awareness
Interest
Action
Desire
Daily Unique Visitors
Product Page Views
Completes a Registra:on Form
Purchase
Australian Direct Marketing Association
Tracking issues
How do you match response to source when:
– Your campaign runs in several channels simultaneously
– Your product is sold through a third party sales channel
– Your product has a long consideration period
Australian Direct Marketing Association
Common tracking options
Unique campaign and comms channel specific phone numbers
Outbound Channel Response Channel
Offline Broadcast
Display Media
Call-Centre
In-Store
Website
Unique redemption codes
Specific Search Call to Action
Australian Direct Marketing Association
Indirect sales – a missing link
Clicks/visits
Sales
Calls
Known
Known
Unknown
Browse
Enquire
Buy
Australian Direct Marketing Association
Long sales cycles
Sales
Time
What does this response curve look like for your product?
• Track enquiries & sales over a long period
• Establish a pattern
• Create a rule of thumb
Australian Direct Marketing Association
Rule of Thumb Approach
• Can be used for indirect sales as well as an ‘early read’ for long campaign cycles
• Typical approach:
1. Establish a ratio for website visits or calls to reseller enquiries/sales
2. Establish a pre-campaign baseline for calls and website visits
3. Measure the uplift in calls/visits during and following the promotion
4. Extrapolate to sales using typical ratio