Multivariable Testing in Direct MailEfficient tests with limited resources

Gordon H. Bell

President

LucidView

Analytics & Testing: 2 sources of insight

Testing– Front-end planning– Prove cause-and-effect– Ask new questions – Design the test to give the specific

data you want

Analytics– Back-end analysis– Look for correlations– Answer questions, but cannot ask any– General data

Testing has always been important

• “Almost any question can be answered, cheaply, quickly and finally, by a test campaign.”

– Claude Hopkins, Scientific Advertising, 1923

• “Testing is still the best way to find true breakthroughs.”– Stone & Jacobs, Successful Direct Marketing Methods, 2008

Testing has always been important

• “Almost any question can be answered, cheaply, quickly and finally, by a test campaign.”

Many questions

^multivariable

is now more efficient

What is Multivariable Testing?

• Many different test designs and statistical techniques developed to test variables more efficiently

• Change many variables at once – in an organized way – so you can separate out the impact of each– Number of test “recipes” (test versions)– Each with a unique combination of all elements– Analyzing all recipes together to quantify effects

• Test design = collection of related test recipes

4 Key Benefits

Benefit Scientific Multivariable Testing Split-run Testing

Efficiency 2 - 35 variables in one test 1 variable = 1 test

Speed and Sample size

Constant sample size (no matter how many variables)

Increasing sample size

Depth of insights

Accurate, robust, and comparative main effects and interactions

One main effect (conditional, with no interactions)

Flexibility Wide range of test designs One choice

Getting Started

Important steps include how to:

1. Define the scope

2. Select a good design

3. Determine the right sample size

4. Create multivariable test recipes

5. Analyze and interpret results 

#1 Define the Scope

Where is multivariable testing valuable and manageable?

• High potential return-on-investment• Clear metrics• Fast results• Flexibility to create many versions of the

mailing• Insights can be applied long-term

Financial Times: Price Test(from the DMA07 Conference)

• Focus on price testing• Measure response rate and revenue• Test in one direct mail campaign• All variables lasered in black ink

(easy and inexpensive to change)

Test Elements (-) Control (+) New Idea

A Price $49 for 26 weeks $99 for 12 monthsB Longer term (+25%) No Yes (32 weeks / 15 months)C Shipping & handling fee No Yes, $4.95D Streamline price box

(3 lines vs 5)No Yes, redesign price box and

reply slip

FT “control” mail package

#2 Select a Good Test Design

Many choices of multivariable test designs• Which “handful” of combinations should be tested to

clearly see results?• Want to have

– Few test recipes (versions of the mailing), but…– Lots of data for better insights– Clear, actionable results– A test design that the marketing team can

understand and believe

Pric

e

Lon

ger

term

(+

25%

)

S&

H (

deliv

ery)

fee

Str

eam

line

pric

e bo

x

Recipe A B C D AB AC BC Mailed

1 – – – – + + + 25,000

2 + – – + – – + 25,000

3 – + – + – + – 25,000

4 + + – – + – – 25,000

5 – – + + + – – 25,000

6 + – + – – + – 25,000

7 – + + – – – + 25,000

8 + + + + + + + 25,000

Financial Times:test design and recipes

Pric

e

Lon

ger

term

(+

25%

)

S&

H (

del

iver

y) fe

e

Str

eam

line

pric

e bo

xRecipe A B C D Mailed

1 – – – – 25,000

2 + – – + 25,000

3 – + – + 25,000

4 + + – – 25,000

5 – – + + 25,000

6 + – + – 25,000

7 – + + – 25,000

8 + + + + 25,000

Financial Times:test design and recipes

A: PriceB: Longer term

(+25%)C: S&H fee

D: Streamline price box

Recipe 1(control)

$49 No

($49 for 26 weeks)No No

Recipe 2 $99 No

($99 for 12 months)No

Yes, redesign price box

Recipe 3 $49 Yes

($49 for 32 weeks)No

Yes, redesign price box

Recipe 4 $99 Yes

($99 for 15 months)No No

Recipe 5 $49 No

($49 for 26 weeks)Yes, $4.95

Yes, redesign price box

Recipe 6 $99 No

($99 for 12 months)Yes, $4.95 No

Recipe 7 $49 Yes

($49 for 32 weeks)Yes, $4.95 No

Recipe 8 $99 Yes

($99 for 15 months)Yes, $4.95

Yes, redesign price box

Pric

e

Lon

ger

term

(+

25%

)

S&

H (

del

iver

y) fe

e

Str

eam

line

pric

e bo

xRecipe A B C D Mailed

1 – – – – 25,000

2 + – – + 25,000

3 – + – + 25,000

4 + + – – 25,000

5 – – + + 25,000

6 + – + – 25,000

7 – + + – 25,000

8 + + + + 25,000

#3 Find the Right Sample Size

The best sample size depends upon– Response rate (or variation in sales)– How small a change you want to see

Lower response rate, or the need to see a smaller change, requires a larger sample size.

Sample Size (N) example

• For Financial Times…– Say we usually have a 1.5% response rate (R)– And we want to see if any test elements affect

response by 10% or more (0.15%)– Then…

31.38 ● R ● (1-R)

(smallest change)2N =

31.38 ● 1.50% ● 98.5%

(0.15%)2N = = 206,062

#4 Create the multivariable recipes

• The science determines which “handful” of combinations should be tested

• The art of testing is making sure…– Elements remain bold

(in the difference between “–” and “+” levels)

– Execution is clear and consistent

Creatives: control (recipe 1) and “all plus” recipe 8

A+

B+

C+

D+

#5 Analyze and Interpret Results

• A well-designed test is relatively easy to analyze– Analysis of variance, regression, or…

pencil and calculator

(with some more in-depth analysis)

• Three main steps in the analysis are:1. Analyze main effects (each column in the design)

2. Analyze interactions

3. Consider alternative solutions and additional metrics

FT price test: main effects & interactions

FT Price Test: Gross response rateoptimal = 32.7% lift

+

-

-

-

-

+

+

0.000% 0.010% 0.020% 0.030% 0.040% 0.050% 0.060% 0.070%

AB interaction

AC interaction

A: Price

D: Streamline price box

BC interaction

C: Shipping &handling fee (no)

B: Longer term(yes, +25%)

Significant effects(beyond line)

18.9% increase in response versus the control

17.2% drop

13.8% further increase in response if...

BC Interaction

B-: Current(26 wk / 12 mo)

B+: Longer term (+25%)

Subscription Term

C-: No S&H fee C+: Add $4.95 S&H fee

Term and S&H do notaffect response rate…

… unless both are setat the best combination

Financial Times: summary

Multivariable testing offered a way to• Test easy, high-impact changes• Learn more with a small sample size• Increase insights

– More accurate main effects– Clear interactions

• In contrast, with split-run testing– We would have needed 3x the sample for equal confidence– At this sample size, not one effect would be significant– Interactions would be impossible to see

Hearst Launch of theFood Network Magazine(presented at the DMA09 conference)

New magazine for a new market =

Need for speed • Learn fast ramp-up quickly• Test a lot of ideas to see what works• Increase confidence in test results for a

rapid roll-out• Optimize price/offer along with creatives

Food Network Magazine Creative Test(13 Creative Elements)

A Window on OE H Reply By dateB Change logo on OE and form J Add chef photos on formC Return address on OE K Change descriptionD Form layout L BrochureE Change "Charter Discount" M ChitF Audience Development copy N Lift noteG Savings presentation

Changes to the outer envelope

Additional inserts

Elements on the Envelope

B: Logo C: Return Address A: Window

B: Change logoE: Change “Charter Discount”F: Audience Development copyG: Savings presentation

H: Reply-By date

J: Add chef photos

K: Change description

D: form layout

Win

dow

on

OE

Cha

nge

logo

on

OE

and

form

Ret

urn

addr

ess

on O

E

For

m la

yout

Cha

nge

"Cha

rter

Dis

coun

t"

Aud

ienc

e D

evel

opm

ent c

opy

Sav

ings

pre

sent

atio

n

Rep

ly B

y da

te

Add

che

f pho

tos

on fo

rm

Cha

nge

desc

riptio

n

Bro

chur

e

Chi

t

Lift

not

e

Recipe A B C D E F G H J K L M N MailedNet

response1 – – – – – – – – – – – – – 25,000 0%2 + + – – – – + + – – + + + 25,000 -21%3 – – – + – + + + + – – + – 25,000 -8%4 + + + – + – – – + – – + – 25,000 -6%5 + + + + + + + + – – – – – 25,000 -11%6 – + – + + – + – – + – + + 25,000 -25%7 – – + – + – + + + – + – + 25,000 -20%8 + – + – – + – + – + – + + 25,000 -10%9 + – + + – – + – + + – – + 25,000 -11%

10 + – – + + – – + – + + – – 25,000 -42%11 – + – – + + – + + + – – + 25,000 -20%12 + + – + – + – – + – + – + 25,000 -10%13 – + + + – – – + + + + + – 25,000 -31%14 – – + + + + – – – – + + + 25,000 -26%15 + – – – + + + – + + + + – 25,000 -37%16 – + + – – + + – – + + – – 25,000 -38%

400,000

16-recipe Creative Test

Design

Test Elements (on the 1-page form)

Control (recipe #1) Recipe #11

B+

E+

F+

J+

K+

H+

FNM: Net Response RateNet Response Rate

optimal = 9.75% lift

0.00% 0.25% 0.50% 0.75% 1.00% 1.25%

F-: Audience Development copy

B-: Change logo on OE and form

C+: Return address on OE

M-: Chit

H-: Reply By date

D-: Form layout

A+: No window on OE (closed-faced)

G-: Savings presentation (control)

N+: Lift note (yes)

J+: Add chef photos on form (yes)

E-: Change Charter Discount (no)

K-: Change description (control)

L-: Brochure (yes) Response drops 16.8% when brochure is removed

Test element (optimal setting)

"Research Discount" hurts 7.3%

New description reduces response 14.3%

3.8% lift

+2.5%

-3.1%

3.5% lift

Significant effectsbeyond line

Multivariable "refining" tests(July 09 test results)

8.8% increase in NRR17.3% increase in Profit

Multivariable "screening" tests(Mar09 tests - Jul09 confirmation)

8.4% increase in NRR17.6% increase in Profit

Original control(March 2009)

Food Network Magazine Results

35% jump in profitover 2 campaigns

(versus 2 years with A/B splits)

(sample size of 600,000 vs. 4.7 MM for split-run testing)

Whirlpool Contact Strategy Test(presented at DM Days – New York, 2007)

Test Elements

A TimingB Early mail dropC Expiration mail dropD Late mail dropE Outer envelopeF Lift noteG Phone Call in Week 1

Focus = Annual renewal of the extended service contract

Contact Strategy Test

Week: 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7

Control

A+

B+

B+

C+

C+

D+

D+

E+E+

E+E+

G+

X

X

Expiration Date

Tim

ing

Ear

ly m

ail d

rop

Exp

irat

ion

mai

l dro

p

Lat

e m

ail d

rop

Out

er

enve

lope

Lift

not

e

Pho

ne

Recipe A B C D E F G Response

1 - - - - - - -2 + - - - + - +3 - + - - + + -4 + + - - - + +5 - - + - + + +6 + - + - - + -7 - + + - - - +8 + + + - + - -9 - - - + - + +10 + - - + + + -11 - + - + + - +12 + + - + - - -13 - - + + + - -14 + - + + - - +15 - + + + - + -16 + + + + + + +

WhirlpoolTest Design

Week 1customers

Week 2customers

Week 3customers

Week 4customer

Test ResultsEffects: Total Response Rate

-4.67%

+3.00%

-2.68%

-2.43%

+2.22%

+1.92%

+1.61%

+1.41%

-0.83%

+0.79%

+0.63%

+0.61%

+0.27%

-0.11%

+0.02%

0.0% 0.5% 1.0% 1.5% 2.0% 2.5%

AEACAF

E: Outer envelopeC: Expiration mail drop

ADAG

F: Lift noteB: Early mail drop (yes)

ABDBD

AB (add early drop)G: Phone call (yes)

D: Late mail drop (yes)A: Timing (spread out)

(effects in percentage points on bottom; as % of control on end)

Significant effects(beyond line)

Interaction Effect

AB Interaction (total response rate)

45%

46%

47%

48%

49%

A-: control (9,4,1,-4) A+: condensed (6,3,1,-3)

A: Timing of contacts

B-: No early mail drop B+: Yes (+14 or +9)

On average,the early mail drop(B) has no impact

But it does help ifcontacts are keptspread out

Whirlpool Test: summary

• Response rate increased 7.81% (7.3% predicted)

• 8.3% increase in revenue (worth $ millions)• 6.76% increase in annual profit

• With split-run tests, would have…– Needed 2½ years of testing (vs. 4 months)– Completely missed the interaction– Seen nothing significant in 4 months

(due to 3x greater error)

“Testing is still the best wayto find true breakthroughs.”

– Bob Stone and Ron Jacobs, Successful Direct Marketing Methods, 2008

“ Like trading in your bicycle for a helicopter, multivariable testing gives you the speed, power, and flexibility to get farther, faster.”

– Gordon Bell, Marketing Day, 23 March 2010