direct mail testing - · pdf filedirect mail testing: ... (claude hopkins, my life in...

1

Direct Mail Testing:Innovations and Insightsfor Challenging Markets

• Richard Varey Vi P id t C M k ti

2

• Richard Varey, Vice President, Consumer MarketingFinancial Times

• Chris Law, Senior Director, Marketing and AnalysisTHD

• Gordon Bell, PresidentLucidView

Tuesday, October 16, 2007

Scientific DM Testing – Opportunity

• Perhaps one time in fifty a guess may be right. But fiftytimes in fifty an actual test tells you what to do and avoid.

(Claude Hopkins, My Life in Advertising, 1927)

• The more you test, the more profitable your direct mail

3

will become. Innovations, provided you test them, canwork wonders.

(David Ogilvy, Ogilvy on Advertising, 1983)

• “Test everything” has been a rallying cry in the marketingand advertising industry throughout the 20th century.

(Ledolter & Swersey, Testing 1-2-3, 2007)

Scientific DM Testing – Challenge• Improving upon success

The secret of all effective originality in advertising is not thecreation of new and tricky words and pictures, but one of puttingfamiliar words and pictures into new relationships. (Leo Burnett)

• Constant change in the marketplaceIt is not the strongest of the species that survives nor the most

4

It is not the strongest of the species that survives, nor the mostintelligent that survives. It is the one that is the most adaptable tochange. (Charles Darwin)

• Tight budgetsThe man who stops testing to save money is like the manwho stops the clock to save time.

• RiskNo one tests the depth of a river with both feet. (African proverb)

Scientific Testing = structured frameworkbeneath each campaign…

5

… But in the marketplace it can get messy

6

Strategic Testing – approach

1. Robust statistical design underlying each test2. Clear, consistent execution3 Fast tests � short term metrics

7

3. Fast tests � short-term metrics4. Strategic cycle of tests

• Screen � Refine � Adapt

Strategic Testing – test designs

• Split-run tests• Scientific multivariable tests

Key decisions

8

– Key decisions• Number of test elements• Levels for each element (optimal = 2)• Effects of interest: main effects and/or interactions• Number of recipes (test cells)• Statistical complexity

THD: Approach to DM testing

• Philosophy: Testing is paramount to a successfuland ever-growing file.

• Live by the adage: Test, test more, then test again• Become intimately involved in your client’s data

9

Become intimately involved in your client s data– Gain a crystal clear understanding of all that has occurred

in the past and use that learning to develop future testingstrategies.

• Everything is imminently testable: Creative, Offer,Message, Ask, Audience, etc..

• Apply same vigilance and methodical approach inall facets of the program

Case study #1: Non-profit fundraising

• Test was for the American Lung Association– Holiday name sticker renewal campaign– 3.1 million pieces mailed in the campaign

10

• One 15-element test within this campaign– Mailed in 16 different recipes– 500,000 mailed within this test

• Challenge: improve upon on-going creativetests

Test ElementsTest Elements (-) Control (+) New Idea

A Address label on envelope Window Mailing label on closed faceB Envelope graphic and teaser Control (label graphic and teaser) No art, new teaserC "Urgent" stamp on envelope No stamp "Urgent" stamp on OED Copy theme Control copy "Urgent" copy themeE Campaign name Generic, unbranded appeal Holiday Drive campaign nameF Add deadline No deadline "Please respond by" date

11

F Add deadline No deadline Please respond by dateG Gift ask Control ask array One gift askH Accentuate gift ask Control reply slip Highlight gift askJ Check boxes on reply slip Control (check boxes on front) Move check boxes to backK Backer color Control (black) Red and blackL Label design Control label design New label designM Monograms on labels Labels with monograms No monogramsN Additional labels Control (6 panel form) Add more labelsO Lift note None Add lift noteP Reply envelope CRE ("place stamp here") BRE (postage paid)

Creatives – control package (recipe 2)

12

Creatives – control package (recipe 2)

13

Recipe #8 outer envelope (A+, B+, C+, L-)

(� versus control OE)

A+

14

Recipe #8 outer envelope (A+, B+, C+, L-)

(� versus control OE)

15

B+

Recipe #8 outer envelope (A+, B+, C+, L-)

(� versus control OE)

C+

16

Recipe #6 outer envelope (A-, B+, C-, L+)

(� versus control OE)

17

A-L+

Recipe #3: reply slip and copy (D+, E-, F+, G+, H+, J+)

G+

J+

H+

18

D+

F+

Test Design

Addr

ess

labe

l on

enve

lope

Enve

lope

gra

phic

and

teas

er

"Urg

ent"

stam

p on

env

elop

e

Cop

y th

eme

Cam

paig

n na

me

Add

dead

line

Gift

ask

Acce

ntua

te g

ift a

sk

Che

ck b

oxes

on

repl

y sl

ip

Back

er c

olor

Labe

l des

ign

Mon

ogra

ms

on la

bels

Addi

tiona

l lab

els

Lift

note

Rep

ly e

nvel

ope

(disguised data)

Recipe A B C D E F G H J K L M N O P Volume Responserate

AvgGift

19

rate Gift1 + - + + - - + - + + - - + + - 31250 1.24 2.7%2 - - - - - - - - - - - - - - - 31250 1.00 0.0%3 - - - + - + + + + - - + - + + 31250 0.91 3.9%4 - + - + + - + - - + - + + - + 31250 0.99 9.6%5 - - + - + - + + + - + - + - + 31250 1.09 7.0%6 + - - + + - - + - + + - - + + 31250 0.96 3.2%7 - + - - + + - + + + - - + + - 31250 1.19 3.4%8 + + + + + + + + - - - - - - - 31250 1.09 1.8%9 + + - + - + - - + - + - + - + 31250 0.93 4.5%10 + - + - - + - + - + - + + - + 31250 1.12 5.6%11 + - - - + + + - + + + + - - - 31250 1.00 4.1%12 - + + - - + + - - + + - - + + 31250 0.80 8.9%13 - + + + - - - + + + + + - - - 31250 0.95 5.2%14 - - + + + + - - - - + + + + - 31250 1.09 6.3%15 + + + - + - - - + - - + - + + 31250 0.91 4.9%16 + + - - - - + + - - + + + + - 31250 1.09 6.3%

500,000 1.02 4.8%

ALA test results: 15 main effects (and interaction)ALA holiday test: Response Rate

optimal = 30.1% lift

--

-

++

+

A: OE address label (new)H: Accentuate gift ask (yes)

B: OE graphic/teaser (control)L: Label design (control)

P: Reply envelope (CRE)N: Additional labels (yes)

LM t i t ti ( t )

20

-

-

-

+

++

++

+

O: Lift noteD: Copy theme

J: Check boxes on reply slipG: Gift ask

F: Add deadlineK: Backer color

C: OE urgent stamp (yes)M: Label monograms (yes)

E: Campaign name (yes)A: OE address label (new)

Significant effects(beyond line)

LM two-way interaction (response rate)

L-: Control L+: New DesignLabel Design

M-: Include monograms M+: No monograms

ALA test results: Net RevenueALA holiday test: Net/M

--

+-

+-

M: Label monograms (yes)

L: Label design (control)

H: Accentuate gift ask (yes)

B: OE graphic/teaser (control)

E: Campaign name (yes)

P: Reply envelope (CRE)

21

-++

+-+-

++

J: Check boxes on reply slip

O: Lift note

N: Additional labels

K: Backer color

F: Add deadline

A: OE address label

D: Copy theme

C: OE urgent stamp (yes)

G: Gift ask (one)

Response Rate versus Net Revenue

ALA holiday test: Response Rateoptimal = 30.1% lift

+

++

--

-

A: OE address label (new)H: Accentuate gift ask (yes)

B: OE graphic/teaser (control)L: Label design (control)

P: Reply envelope (CRE)N: Additional labels (yes)

ALA holiday test: Net/M (in same order as response)

++

--

-+

22

+

++

++

+

-

-

-O: Lift note

D: Copy themeJ: Check boxes on reply slip

G: Gift askF: Add deadlineK: Backer color

C: OE urgent stamp (yes)M: Label monograms (yes)

E: Campaign name (yes)A: OE address label (new)

Significant effects(beyond line)

+-

-+

-+

+

-+

Test Design – pros and cons

Pros• Tested 16 out of 32,768 possible combinations

– Yet could predict outcome for all other combos• Quicker to market – confidence in results allows for larger

roll-outsEach main variable tested to over half of test volume or 250 000 donors

23

– Each main variable tested to over half of test volume or 250,000 donors• In contrast, split-run testing:

– Limits the number of variables tested or else requires sample size> 4.7 million or 15 drops for equal confidence

– Cannot analyze interactionsCons• Greater cost and effort than split-run testing• Complex statistical test design requires guidance

THD: Agency Insights

• Don’t be afraid of the water– It wasn’t as bad as it looks!

• Production staff needs to be relied upon

24

• Production staff needs to be relied uponmuch more than current process

• Need a capable vendor network– Data processing, printing, & lettershop

Financial Times:campaign test strategy

• The Financial Times (FT) is a daily newspaper offering businessnews and analysis to a global audience, with a third of itscirculation in the US. For subscription growth, direct mail is stillthe primary DTP source.

• Direct mail strategy relies on database marketing (modeling

25

• Direct mail strategy relies on database marketing (modeling,optimization) and continual testing, historically on a split-run basis.

• The prime control package has been a voucher or statement ofbenefits format for 5 years, with a “hard” offer of $49 for 26 weeks(cash with order or bill-me).

• A second “soft” control, a different package with a 4 week risk-freetrial offer, is used selectively on some lists.

• In this period we increased mail volume, and improved efficiency,but to quicken the process, decided to try testing with LucidView.

Case study #2: Subscriptions DM test

• The Financial Times• “Hard offer” direct mail package• 1 creative test + 1 price test grid

26

• 1 creative test + 1 price test grid• Challenge: allow adequate sample size with

relatively low response rates• Challenge: how to improve already-strong

control

Creative test – elements

Test Elements (-) Control (+) New Idea

A Live stamp Meter StampB Teaser on envelope (front and back) No teaser Add teasers (front and back)C Picture of newspaper No Yes, add graphic of newspaperD Benefits copy Control New benefits copy

27

E Reply slip layout Control New layoutF Reply slip creative Control (Prof Rate Offer) New (green Savings Certificate)G "Reply by" date Control Hype deadlineH Call-to-action Control (no URL) Add URLJ Request e-mail address No YesK Brochure Control (mini-newspaper) New brochureL Letter from Robin No Yes, add letter

Creative test – design & recipesLi

ve s

tam

pTe

aser

s on

env

elop

ePi

ctur

e of

new

spap

erBe

nefit

s co

pyR

eply

slip

layo

utR

eply

slip

cre

ativ

e"R

eply

by"

dat

eC

all-t

o-ac

tion

Req

uest

e-m

ail a

ddre

ssBr

ochu

reLe

tter f

rom

Rob

in

Recipe A B C D E F G H J K L1 – – – – – – – – – – –2 + + + – + + – + – – –3 + – + + – + – – – + +4 – – – + + + – + + – +5 + + – + + – + – – – +

Package A C D ERecipe 1(control) Recipe 2 Recipe 3 Recipe 4

A: Live stamp Meter Stamp Stamp Meter

B: Teasers onenvelope No teaser Add teasers

(front and back) No teaser No teaser

C: Picture ofnewspaper No Yes, add graphic

of newspaperYes, add graphic

of newspaper No

D: Benefits copy Control Control New benefits copy New benefits copy

28

6 + – – – + + + – + + –7 + – + – – – + + + – +8 – + + – + – – – + + +9 – – + + + – + + – + –10 – + – – – + + + – + +11 – + + + – + + – + – –12 + + – + – – – + + + –

E: Reply sliplayout Control New layout Control New layout

F: Reply slipcreative

Control(Prof Rate Offer)

New (greenSavings Certificate)

New (greenSavings Certificate)

New (greenSavings Certificate)

G: "Reply by"date Control Control Control Control

H: Call-to-action Control(no URL) Add URL Control

(no URL) Add URL

J: Requeste-mail address No No No Yes, request

e-mail address

K: Brochure Control(mini-newspaper)

Control(mini-newspaper) New brochure Control

(mini-newspaper)L: Letter fromRobin No No Yes, add letter Yes, add letter

Creatives – control package (recipe 1)

29

Recipe #4 (C-, D+, E+, F+, G-, H+, J+) and #7 (C+, D-, E-, F-, G+, H+, J+)

30

FT creative test: 11 main effectsFT hard offer Creative Test: Gross response rate

+

+

+

-

-

C: Picture of newspaper

E: Reply slip layout

D: Benefits copy

B: Teasers on envelope (none)

F: Reply slip creative (control) 11.3% drop in response versus the control

9.8% drop

8.4% lift

Positive, butnot significant� t it

31

-

-

+

-

-

+

+

J: Request e-mail address

L: Letter from Robin

K: Brochure

G: Reply by date

A: Live stamp

H: Call-to-action

C: Picture of newspaper

Significant effects(beyond line)

� opportunityfor “refining”

Price test – elements

Test Elements (-) Control (+) New Idea

A Price $49 for 26 weeks $99 for 12 monthsB Longer term (+25%) No Yes (32 weeks / 15 months)C Shi i & h dli f N Y $4 95

32

C Shipping & handling fee No Yes, $4.95D Streamline price box

(3 lines vs 5)No Yes, redesign price box and

reply slip

Price test – design & recipesPr

ice

Long

er te

rm (+

25%

)

S&H

(del

iver

y) fe

e

Stre

amlin

e pr

ice

box

Recipe A B C D1 – – – –2 + – – +

A P R SRecipe 1(control) Recipe 2 Recipe 3 Recipe 4

A: Price $49 $99 $49 $99

B: Longer term(+25%)

No($49 for 26 weeks)

No($99 for 12 months)

Yes($49 for 32 weeks)

Yes($99 for 15 months)

C: S&H (delivery)fee No No No No

D: Streamlineprice box No Yes, redesign price

box and reply slipYes, redesign pricebox and reply slip No

33

3 – + – +4 + + – –5 – – + +6 + – + –7 – + + –8 + + + +

price box box and reply slip box and reply slip

T U V W

Recipe 5 Recipe 6 Recipe 7 Recipe 8

A: Price $49 $99 $49 $99

B: Longer term(+25%)

No($49 for 26 weeks)

No($99 for 12 months)

Yes($49 for 32 weeks)

Yes($99 for 15 months)

C: S&H (delivery)fee Yes, $4.95 Yes, $4.95 Yes, $4.95 Yes, $4.95

D: Streamlineprice box

Yes, redesign pricebox and reply slip No No Yes, redesign price

box and reply slip

Creatives – control (recipe 1) vs. “all plus” recipe 8

34

FT price test: main effects & interactionsFT hard offer Price Test: Gross response rate

optimal = 32.7% lift

-

-

+

BC interaction

C: Shipping &handling fee (no)

B: Longer term(yes, +25%)

18.9% increase in response vs the control

17.2% drop

13.8% further increase in response if...BC Interaction

35

+

+

-

-

AB interaction

AC interaction

A: Price

D: Streamline pricebox

B-: Current(26 wk / 12 mo)

B+: Longer term (+25%)

Subscription Term

C-: No S&H fee C+: Add $4.95 S&H fee

Overall Test Results

• 5 significant main effects + 1 interaction• 41% overall increase in net response

• Creative test: 13% increase in net response (call-to-action: add URL)+ 7% increase in % payup

• Price test: 28% increase in net response

36

• Price test: 28% increase in net response

• If these 15 elements were tested with split-run techniques…• Not one effect would have been statistically significant• Interactions would be impossible to see• Only 3 or 4 elements could be tested with equal confidence

The Financial Times:Insights & advice

• Faster deeper learning: we raised the number of test panels nearly3-fold to 19, but tested 2,064 combinations. LucidView gave scopeto analyze main effects, comparative effects and multipleinteractions with confidence.

• Initial investment (consultancy & production) pays off on roll out

37

• Initial investment (consultancy & production) pays off on roll-out.• Get brainstorming participation from a wide group, not just the

direct marketers, and think bold. Use LucidView’s flexibility totest a wide range of designs. Scale the experiment to your budget.

• Expect a few variables to make a positive difference, a fewnegative and most none.

• Continue: next steps to confirm and refine tests (Fall 07).• If you are not a statistical genius, consult an expert, ideally with the

relevant industry experience.

Direct Mail Testing:Innovations and Insightsfor Challenging Markets

• Richard Varey Vi P id t C M k ti

38

• Richard Varey, Vice President, Consumer Marketing,Financial Times

• Chris Law, Senior Director, Marketing and AnalysisTHD

• Gordon Bell, PresidentLucidView

Tuesday, October 16, 2007