iir october 19, 2009 - mktg, inc · iir october 19, 2009 “unusual survey results have caused ......
TRANSCRIPT
The State of Online Research Panels
IIROctober 19, 2009
“Unusual Survey Results Have Caused Alarm”
• Ron Gailey, working for WAMU, presented (at the IIR, November 2008) the results of 29 research studies, a total of 40,000 interviews, conducted from 2006 to 2007.
• Problem: demand for financial products decreased over time; a phenomenon not supported by experience in the market.
Copyright Mktg, Inc. 2009. All rights reserved.
• Inconsistency in his sample left Ron struggling to understand the results of two years of tracking work.
• Business decisions were clearly at risk.
• There was no external measure of consistency for Ron to refer to.
Copyright Mktg, Inc. 2009. All rights reserved.
“In every study examined…
…people with more panel tenure gave lower demand.”
Ron Gailey, 2008
Copyright Mktg, Inc. 2009. All rights reserved.
Panel tenure is progressive; it changes through time.
It represents an inconsistency.
Copyright Mktg, Inc. 2009. All rights reserved.
Copyright Mktg, Inc. 2009. All rights reserved.
0%10%20%30%40%50%60%70%80%90%
100%M25
‐Russia
M79
‐Russia
M10
0‐Ru
ssia
M10
9‐Ru
ssia
M50
‐China
M99
‐China
M12
3‐Ch
ina
M24
‐Japan
M10
4‐Japan
M12
5‐Japan
M44
‐S. Korea
M11
1‐S.Ko
rea
M12
6‐S. Korea
M43
‐Australia
M12
2‐Au
stralia
M47
‐Hon
g Ko
ng
M12
4‐Ho
ng Kon
g
M48
‐Taiwan
M52
‐Thailand
M12
1‐New
Zealand
M67
‐US
M76
‐ US
M95
‐ US
M96
‐ US
M10
1 ‐ U
S
M12
7 ‐ U
S
M12
9 ‐US
AGING
Aging 0 Months Aging 6 months Aging 12 months Aging 18 months Aging 24 months
A Global Effort
• We began with 17 American Panels as a preliminary sample. Mktg, Inc. released the analysis in January 2009, at CASRO.
• We are now collecting data in 35 countries and 140 panels.
• We call the project the “Grand Mean™”
Copyright Mktg, Inc. 2009. All rights reserved.
The “Grand Mean Project”
“The Grand Mean” is the average value of available panel data for a country or region.
For example, a Grand Mean can be maintained for respondent tenure measured across panels within a country.
Copyright Mktg, Inc. 2009. All rights reserved.
So we began……… Selected demographic quotas (age, income, gender, ethnicity) were used to simulate census.
Median length was 15 minutes.
Questions covered: Technology and the media, Participation in market research, Buyer Behavior, Values and Lifestyle, Demographics, Questionnaire Satisfaction.
Copyright Mktg, Inc. 2009. All rights reserved.
BUYING BEHAVIOR…..
…….it’s at the core of Market Research.
Copyright Mktg, Inc. 2009. All rights reserved.
Copyright Mktg, Inc. 2009. All rights reserved.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Buyer Behavior
Buyer Behavior Segments Domestic/Coupons Buyer Behavior Segments Credit/Environment
Buyer Behavior Segments Price Sensitive Shoppers Buyer Behavior Segments Broad Range/Credit
Copyright Mktg, Inc. 2009. All rights reserved.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%M83
‐UK
M85
‐ UK
M68
‐Spain
M73
‐Spain
M69
‐Germany
M74
‐Germany
M86
‐Germany
M75
‐ France
M80
‐France
M87
‐ France
M98
‐ Brazil
M10
5 ‐Brazil
M10
7 ‐Swed
en
M11
4 ‐Swed
en
M99
‐China
M12
3‐Ch
ina
M10
4‐Japan
M12
5‐Japan
M12
8‐Japan
M11
1‐ S. korea
M12
6‐ S. korea
M12
2‐Au
stralia
M12
4‐Ho
ng Kon
g
M79
‐Russia
M10
0‐Ru
ssia
M13
3‐Ru
ssia
M10
9‐Ru
ssia
M67
‐US
M76
‐US
M95
‐ US
M96
‐ US
M10
1 ‐ U
S
M12
7‐ U
S
M12
9 ‐ U
S
Automotive ‐ Buyer Behavior
Automotive Replacement Automotive Brand over Price Automotive Quality/Shopping
Copyright Mktg, Inc. 2009. All rights reserved.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%M83
‐ UK
M85
‐ UK
M68
‐ Spain
M73
‐ Spain
M69
‐ Germ
M74
‐ Germ
M86
‐ Germ
M75
‐ France
M80
‐ France
M87
‐ France
M71
‐ Ita
lyM84
‐Italy
M89
‐ Ita
lyM99
‐China
M12
3‐Ch
ina
M10
4‐Japan
M12
5‐Japan
M12
8‐Japan
M11
1‐Skorea
M12
6‐Skorea
M12
2‐Au
stralia
M12
4‐Ho
ng Kon
gM79
‐Russia
M13
3‐Ru
ssia
M10
0‐Ru
ssia
M10
9‐Ru
ssia
M10
7 ‐ Swed
enM11
4 ‐ Swed
enM67
‐US
M76
‐US
M95
‐ US
M96
‐ US
M10
1 ‐ U
SM12
7‐ U
SM12
9 ‐ U
S
Clothing ‐Buying Behavior
Clothing New Clothes/On‐line Clothing Credit Clothing Brands/Decision Maker
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%M83
‐ UK
M85
‐ UK
M68
‐ Spain
M73
‐ Spain
M69
‐ Germany
M74
‐ Germany
M86
‐ Germany
M75
‐France
M80
‐France
M87
‐France
M72
‐Canada
M97
‐Canada
M10
2‐Canada
M11
0‐Canada
M10
7‐Sw
eden
M11
4‐Sw
eden
M99
‐China
M12
3‐Ch
ina
M10
4‐Japan
M12
5‐Japan
M11
1‐Skorea
M12
6‐Skorea
M12
2‐Au
stralia
M12
4‐Ho
ng Kon
gM79
‐Russia
M10
0‐Ru
ssia
M10
9‐Ru
ssia
M67
‐US
M76
‐US
M95
‐ US
M96
‐ US
M10
1 ‐ U
SM12
7‐ U
SM12
9 ‐ U
S
Entertainment and Travel ‐Buying behavior
Entertainment & Travel Personal Travel
Entertainment & Travel Buy Tickets
Entertainment & Travel Frequent Fliers
Entertainment & Travel Non‐Travel/Non‐Entertainment
Copyright Mktg, Inc. 2009. All rights reserved.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Sociographic
Sociographic Segments Opinionated/Not Computer
Sociographic Segments Happy with Life/Not Computer
Sociographic Segments High Computer/Stays Informed
Copyright Mktg, Inc. 2009. All rights reserved.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Media
Media Segments Internet Media Segments Stay Informed
Media Segments Enjoys Politics Media Segments Concerned
Copyright Mktg, Inc. 2009. All rights reserved.
What is a researcher to do?
Uncontrolled variables can drive inconsistency:• Respondent tenure• Professional respondents• Speeders• Consistency errors• Satisficing• Shifting sample sources
Copyright Mktg, Inc. 2009. All rights reserved.
Complex weighting schemes mask the problem and are not likely the answer…there is no census to fall back on. Especially when the problem is in purchasing intent.
Copyright Mktg, Inc. 2009. All rights reserved.
Optimization
We can minimize the risk of inconsistency by using optimization models.
Copyright Mktg, Inc. 2009. All rights reserved.
Optimization Profile The third panel is determinate.
This is a three panel solution. We want to try to minimize the number of panels and minimize the distance from the Grand Mean.
0.00
0.01
0.02
0.03
0.04
0.050.06
0.07
RM
S Er
ror
0%20%
40%60%
80%100%0% 20% 40% 60% 80% 100%
Percent of M8
Percent of M17
Copyright Mktg, Inc. 2009. All rights reserved.
We can optimize to the Grand Mean.In this example we show the expected standard error from the
Grand Mean based on the average of all random choices (8.31%).
Based on equal weighting of three panels selected by optimization to the Grand Mean (2.36%)
… and the same three panels blended in proportions to optimize to the Grand Mean (0.40%).
Panels Optimum AverageExpected
(1SE)Inherent (1SE)
M8 24% 33%M17 26% 33%M12 50% 34%
Root Mean Square Error 0.40% 2.36% 8.31% 2.45%
Copyright Mktg, Inc. 2009. All rights reserved.
CONSISTENCY
We must know if the data shifts we see are real or changes in the sample.
Copyright Mktg, Inc. 2009. All rights reserved.
Consistent Track™ An Audit Program
• The objective is to capture the variability in online sample sources through time.
• It is a natural off shoot of the Grand Mean™ project.• The first wave of data collection for each research partner is potentially
the first wave of a consistency analysis.• A series of repeat studies, when analyzed according to standard quality
control metrics, provides a measure of panel variability through time. • The cost is minimal as each panel need complete only one sequence of
waves at regular time intervals.• Interim reports are provided at every wave and can be used as a reference
for all tracking studies being completed at that time.• The combination of multiple consistency analysis within a market provides
data similar to the Grand Mean Project only in greater volume and sequentially through time.
Copyright Mktg, Inc. 2009. All rights reserved.
The results of the average consistency of the sample‐sets compared to the overall error bound for the various metrics.
Fifteen metrics were analyzed. The ‘lighter’ line/circle defines the expected error. The ‘darker’ line/circle is the
measured variability of the data.
Test Panel Consistency Summary: Average Deviation Vs. Overall Average Values
0.0%0.5%1.0%1.5%2.0%2.5%3.0%
Buyer Behavior SegmentsSociographic Segments
Media Segments
Age
Income
Education
MaritalBelonging to > 4 PanelsSurveys Almost Every day
>30 surveys/Month
Failure to Follow Instructions
Inconsistent with Opinion onStandard of Living
Inconsistent of Opinion on Brandover Price
Speeders
Straight-Liners
Expected Error Distance
Copyright Mktg, Inc. 2009. All rights reserved.
Buyer Behavior Segment Distribution Profile
0%
20%
40%
60%
80%
100%
120%
OverallAverage
12/2007 5/2008 5/2009 US GrandMean
Res
pond
ent P
erce
nt
Broad Range/Credit Price Sensitive ShoppersCredit/Environment Domestic/Coupons
Chi-Square Test of Buyer Behavior Segments
0%
20%
40%
60%
80%
100%
120%
12/2007 5/2008 5/2009Like
lihoo
d be
ing
Equa
l to
Ref
eren
ce
Compared to Overall Average Compared to US Grand Mean
Distance of Buyer Behavior Segments from the Reference
0%
1%
2%
3%
4%
5%
12/2007 5/2008 5/2009
Roo
t-Mea
n-Sq
uare
Dis
tanc
es
Compared to Average Values Compared to US Grand MeanPrecision (2 Standard Errors)
BUYER BEHAVIOR
Copyright Mktg, Inc. 2009. All rights reserved.
A safety net for online tracking data
• Sample variability through time is a threat to data interpretation.
• The risk of incorrect business decisions can be tempered by consistent sample.
• In order to know if you have consistent sample you need to measure it.
• Consistent Track™ is here and achievable.• The Grand Mean™ project provides us with a new metric to anchor our data.
• Optimization gives us precision
Copyright Mktg, Inc. 2009. All rights reserved.
THANK YOU!Steven Gittelman, Ph D.
President200 Carleton Ave., East Islip, New York
[email protected] – 631‐277‐7000Cell – 631‐466‐6604
Copyright Mktg, Inc. 2009. All rights reserved.