wood for trees 1wood for trees 1 outstanding marketing services provider 2011 (database marketing...
TRANSCRIPT
wood for trees 1wood for treeswww.woodfortrees.net
1Outstanding Marketing Services Provider 2011 (Database Marketing awards); Supplier of the Year 2012 (IoF SIG Awards); Innovation on Data Quality 2012 (Database Marketing Awards)
Experimental Design in the Commercial
SectorWood for Trees
March 2015
wood for trees 2wood for treeswww.woodfortrees.net
2
Background – Capital One
Founded by Richard Fairbank and Nigel Morris 1988
Massive Growth - now 8th Largest Bank in US
Revenue US$ 24 Billion
Net Income US$ 4 Billion
40,000 Employees
USP = Information base strategy
100,000s test per year
"Using scientific testing on a massive scale, we gather huge amounts of information to help us tailor products and services to the individual consumer, rather than simply offering one product to broad socioeconomic groups. We don't believe that 'one size fits all'.
wood for trees 3wood for treeswww.woodfortrees.net
3
Background – Traditional Test Methods
UK employed traditional A / B Testing
Success in some cases, but issues too:
Large number of tests
Relatively slow process
Documentation
Lack of consistency
No consideration for impact of tests on different groups of supporters
Feedback from Stakeholders around the
business was not great …
wood for trees 4wood for treeswww.woodfortrees.net
4
Background – Stakeholder Feedback
How should I combine multiple tests?
We Keep Retesting the same things ….
I can’t remember or find out what we’ve test before
Is this really working?
There seems to be different ways of analysing the tests
The tests are too large for us to do many of them
The test do not take into account the different types of supporter
wood for trees 5wood for treeswww.woodfortrees.net
5
Solution – Introduce a New Framework
Statistics Team would take
responsibility for testing as a
whole
Roll out of a New “Testing
Framework”
Based on DOE
Agreement from Senior
Management
Workshop for all Marketing
Teams
Specialist support
All Tests required formal
approval
Analysis Team Marketing Stakeholders
wood for trees 6wood for treeswww.woodfortrees.net
6
New Framework – Stakeholders Responsibility
Required to complete a
Form for each test
Hypothesis – objective
Desired Outcome
Power of test
Contact information – for
results
Approval
Documented and stored by
Analysis Team
wood for trees 7wood for treeswww.woodfortrees.net
7
New Framework – Analyst Responsibility
Move away from existing “OFAT” approach
Considered inefficient (sample size) and
ineffective (interactions not tested)
Tools for planning (sample size and power)
Design
Analysis
Required more analysis (and resource) post
campaign
But in theory analysis could be shared between
business lines and statisticians
wood for trees 8wood for treeswww.woodfortrees.net
8
New Creative
Normal Creative
Freebie 10,000 10,000
No Freebie 10,000 10,000
DOE – Benefits (Sample Size)
Freebie (20,000)
No Freebie(20,000)
VS
New Creative(20,000)
Normal(20,000)
VS
Assume that we want to test 2 factors: Freebie and New Creative
Standard sample size calculation suggest that 20,000 is needed in each cell
for OFAT Test
For DOE we would only need half the size!
OFAT DOE
wood for trees 9wood for treeswww.woodfortrees.net
9
DOE – Benefits (Interaction)
Model allows for
Interaction between
factors to be
examined
In this hypothetical
example the biggest
increase in response
rate is found by
combining Freebie and
the New CreativeCreative A vs B
Resp
onse
Rate
Freebie
No Freebie
Testing OFAT basically ignores the fact that different elements may work
together
A B
wood for trees 10wood for treeswww.woodfortrees.net
10
DOE – Benefits (Covariate Analysis)
The other key question is does the effect of the envelope size and free gift
vary for different groups of supporters?
This can be
analysed in a
similar way
as to the
interaction
between
variables
VS
wood for trees 11wood for treeswww.woodfortrees.net
11
DOE – Analysis
Finally, this could be analysed formally using a form of the General Linear
Model:Response Rate It is possible to analyse some
simpler designs “by hand”
However the approach taken at
Capital One was to adopt the
more statistical approach using
appropriate software
wood for trees 12wood for treeswww.woodfortrees.net
12
DOE – Considerations
It requires a certain level of statistical knowledge and the right software
Even then it is a very deep subject: how many factors can be tested, what
level of interactions are needed, fractional factorial design and d-optimal
design …
… Plus it can be hard for stakeholders to appreciate and is sensitive to
execution issues
wood for trees 13wood for treeswww.woodfortrees.net
13
Summary
What went well?
Senior Management buy in
Standardised process – documentation, consistency
Workshops
What can cause problems?
DOE complex and not understood by all
Sensitive to operational issues
Other Take-outs
DOE is the most powerful way to ‘learn’ ?!
Internationally recognised (six sigma)
More success stories outside of manufacturing and agriculture – such as web site design (quitting smoking)