iir october 19, 2009 - mktg, inc · iir october 19, 2009 “unusual survey results have caused ......

The State of Online Research Panels

IIROctober 19, 2009

“Unusual Survey Results Have Caused Alarm”

• Ron Gailey, working for WAMU, presented (at the IIR, November 2008) the results of 29 research studies, a total of 40,000 interviews, conducted from 2006 to 2007.

• Problem: demand for financial products decreased over time; a phenomenon not supported by experience in the market.

Copyright Mktg, Inc. 2009. All rights reserved.

• Inconsistency in his sample left Ron struggling to understand the results of two years of tracking work.

• Business decisions were clearly at risk.

• There was no external measure of consistency for Ron to refer to.


“In every study examined…

…people with more panel tenure gave lower demand.”

Ron Gailey, 2008


Panel tenure is progressive; it changes through time.

It represents an inconsistency.



0%10%20%30%40%50%60%70%80%90%

100%M25

‐Russia

M79

‐Russia

M10

0‐Ru

ssia

M10

9‐Ru

ssia

M50

‐China

M99

‐China

M12

3‐Ch

ina

M24

‐Japan

M10

4‐Japan

M12

5‐Japan

M44

‐S. Korea

M11

1‐S.Ko

rea

M12

6‐S. Korea

M43

‐Australia

M12

2‐Au

stralia

M47

‐Hon

g Ko

ng

M12

4‐Ho

ng Kon

g

M48

‐Taiwan

M52

‐Thailand

M12

1‐New

Zealand

M67

‐US

M76

‐ US

M95

‐ US

M96

‐ US

M10

1 ‐ U

S

M12

7 ‐ U

S

M12

9 ‐US

AGING

Aging 0 Months Aging 6 months Aging 12 months Aging 18 months Aging 24 months

A Global Effort

• We began with 17 American Panels as a preliminary sample. Mktg, Inc. released the analysis in January 2009, at CASRO.

• We are now collecting data in 35 countries and 140 panels.

• We call the project the “Grand Mean™”


The “Grand Mean Project”

“The Grand Mean” is the average value of available panel data for a country or region.

For example, a Grand Mean can be maintained for respondent tenure measured across panels within a country.


So we began……… Selected demographic quotas (age, income, gender, ethnicity) were used to simulate census.

Median length was 15 minutes.

Questions covered: Technology and the media, Participation in market research, Buyer Behavior, Values and Lifestyle, Demographics, Questionnaire Satisfaction.


BUYING BEHAVIOR…..

…….it’s at the core of Market Research.



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Buyer Behavior

Buyer Behavior Segments Domestic/Coupons Buyer Behavior Segments Credit/Environment

Buyer Behavior Segments Price Sensitive Shoppers Buyer Behavior Segments Broad Range/Credit


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%M83

‐UK

M85

‐ UK

M68

‐Spain

M73

‐Spain

M69

‐Germany

M74

‐Germany

M86

‐Germany

M75

‐ France

M80

‐France

M87

‐ France

M98

‐ Brazil

M10

5 ‐Brazil

M10

7 ‐Swed

en

M11

4 ‐Swed

en

M99

‐China

M12

3‐Ch

ina

M10

4‐Japan

M12

5‐Japan

M12

8‐Japan

M11

1‐ S. korea

M12

6‐ S. korea

M12

2‐Au

stralia

M12

4‐Ho

ng Kon

g

M79

‐Russia

M10

0‐Ru

ssia

M13

3‐Ru

ssia

M10

9‐Ru

ssia

M67

‐US

M76

‐US

M95

‐ US

M96

‐ US

M10

1 ‐ U

S

M12

7‐ U

S

M12

9 ‐ U

S

Automotive ‐ Buyer Behavior

Automotive Replacement Automotive Brand over Price Automotive Quality/Shopping


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%M83

‐ UK

M85

‐ UK

M68

‐ Spain

M73

‐ Spain

M69

‐ Germ

M74

‐ Germ

M86

‐ Germ

M75

‐ France

M80

‐ France

M87

‐ France

M71

‐ Ita

lyM84

‐Italy

M89

‐ Ita

lyM99

‐China

M12

3‐Ch

ina

M10

4‐Japan

M12

5‐Japan

M12

8‐Japan

M11

1‐Skorea

M12

6‐Skorea

M12

2‐Au

stralia

M12

4‐Ho

ng Kon

gM79

‐Russia

M13

3‐Ru

ssia

M10

0‐Ru

ssia

M10

9‐Ru

ssia

M10

7 ‐ Swed

enM11

4 ‐ Swed

enM67

‐US

M76

‐US

M95

‐ US

M96

‐ US

M10

1 ‐ U

SM12

7‐ U

SM12

9 ‐ U

S

Clothing ‐Buying Behavior

Clothing New Clothes/On‐line Clothing Credit Clothing Brands/Decision Maker

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%M83

‐ UK

M85

‐ UK

M68

‐ Spain

M73

‐ Spain

M69

‐ Germany

M74

‐ Germany

M86

‐ Germany

M75

‐France

M80

‐France

M87

‐France

M72

‐Canada

M97

‐Canada

M10

2‐Canada

M11

0‐Canada

M10

7‐Sw

eden

M11

4‐Sw

eden

M99

‐China

M12

3‐Ch

ina

M10

4‐Japan

M12

5‐Japan

M11

1‐Skorea

M12

6‐Skorea

M12

2‐Au

stralia

M12

4‐Ho

ng Kon

gM79

‐Russia

M10

0‐Ru

ssia

M10

9‐Ru

ssia

M67

‐US

M76

‐US

M95

‐ US

M96

‐ US

M10

1 ‐ U

SM12

7‐ U

SM12

9 ‐ U

S

Entertainment and Travel ‐Buying behavior

Entertainment & Travel Personal Travel

Entertainment & Travel Buy Tickets

Entertainment & Travel Frequent Fliers

Entertainment & Travel Non‐Travel/Non‐Entertainment


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Sociographic

Sociographic Segments Opinionated/Not Computer

Sociographic Segments Happy with Life/Not Computer

Sociographic Segments High Computer/Stays Informed


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Media

Media Segments Internet Media Segments Stay Informed

Media Segments Enjoys Politics Media Segments Concerned


What is a researcher to do?

Uncontrolled variables can drive inconsistency:• Respondent tenure• Professional respondents• Speeders• Consistency errors• Satisficing• Shifting sample sources


Complex weighting schemes mask the problem and are not likely the answer…there is no census to fall back on. Especially when the problem is in purchasing intent.


Optimization

We can minimize the risk of inconsistency by using optimization models.


Optimization Profile The third panel is determinate.

This is a three panel solution. We want to try to minimize the number of panels and minimize the distance from the Grand Mean.

0.00

0.01

0.02

0.03

0.04

0.050.06

0.07

RM

S Er

ror

0%20%

40%60%

80%100%0% 20% 40% 60% 80% 100%

Percent of M8

Percent of M17


We can optimize to the Grand Mean.In this example we show the expected standard error from the

Grand Mean based on the average of all random choices (8.31%).

Based on equal weighting of three panels selected by optimization to the Grand Mean (2.36%)

… and the same three panels blended in proportions to optimize to the Grand Mean (0.40%).

Panels Optimum AverageExpected

(1SE)Inherent (1SE)

M8 24% 33%M17 26% 33%M12 50% 34%

Root Mean Square Error 0.40% 2.36% 8.31% 2.45%


CONSISTENCY

We must know if the data shifts we see are real or changes in the sample.


Consistent Track™ An Audit Program

• The objective is to capture the variability in online sample sources through time.

• It is a natural off shoot of the Grand Mean™ project.• The first wave of data collection for each research partner is potentially

the first wave of a consistency analysis.• A series of repeat studies, when analyzed according to standard quality

control metrics, provides a measure of panel variability through time. • The cost is minimal as each panel need complete only one sequence of

waves at regular time intervals.• Interim reports are provided at every wave and can be used as a reference

for all tracking studies being completed at that time.• The combination of multiple consistency analysis within a market provides

data similar to the Grand Mean Project only in greater volume and sequentially through time.


The results of the average consistency of the sample‐sets compared to the overall error bound for the various metrics.

Fifteen metrics were analyzed. The ‘lighter’ line/circle defines the expected error. The ‘darker’ line/circle is the

measured variability of the data.

Test Panel Consistency Summary: Average Deviation Vs. Overall Average Values

0.0%0.5%1.0%1.5%2.0%2.5%3.0%

Buyer Behavior SegmentsSociographic Segments

Media Segments

Age

Income

Education

MaritalBelonging to > 4 PanelsSurveys Almost Every day

>30 surveys/Month

Failure to Follow Instructions

Inconsistent with Opinion onStandard of Living

Inconsistent of Opinion on Brandover Price

Speeders

Straight-Liners

Expected Error Distance


Buyer Behavior Segment Distribution Profile

0%

20%

40%

60%

80%

100%

120%

OverallAverage

12/2007 5/2008 5/2009 US GrandMean

Res

pond

ent P

erce

nt

Broad Range/Credit Price Sensitive ShoppersCredit/Environment Domestic/Coupons

Chi-Square Test of Buyer Behavior Segments

0%

20%

40%

60%

80%

100%

120%

12/2007 5/2008 5/2009Like

lihoo

d be

ing

Equa

l to

Ref

eren

ce

Compared to Overall Average Compared to US Grand Mean

Distance of Buyer Behavior Segments from the Reference

0%

1%

2%

3%

4%

5%

12/2007 5/2008 5/2009

Roo

t-Mea

n-Sq

uare

Dis

tanc

es

Compared to Average Values Compared to US Grand MeanPrecision (2 Standard Errors)

BUYER BEHAVIOR


A safety net for online tracking data

• Sample variability through time is a threat to data interpretation.

• The risk of incorrect business decisions can be tempered by consistent sample.

• In order to know if you have consistent sample you need to measure it.

• Consistent Track™ is here and achievable.• The Grand Mean™ project provides us with a new metric to anchor our data.

• Optimization gives us precision


THANK YOU!Steven Gittelman, Ph D.

President200 Carleton Ave., East Islip, New York

[email protected] – 631‐277‐7000Cell – 631‐466‐6604


iir october 19, 2009 - mktg, inc · iir october 19, 2009 “unusual survey results have caused ......

Documents