by steven gittelman, ph.d. and elaine trimarchi

Global Patterns in Panel Research

BySteven Gittelman, Ph.D.

and Elaine Trimarchi

Objectives

Compare Global and American panel patterns.

Demonstrate trends in panel evolution.

Illuminate how various problem respondents, through their impact on purchasing data, drive evolutionary changes.

Clarify the issues that now confront the American Panels.

Arrive at workable solutions‐‐‐blending methodologies.

I can see clearly now! Compared survey results from 12 US Consumer Panels, 1 panel in

each of twenty‐five global markets. 400 completes per source. June 2008 ‐ February 2009.

We are grateful to our research partners for providing sample for the following global (non US) markets.

‐ ‐ 17‐ global panels‐Argentina, Brazil, Czech Republic, Denmark, Finland, France, Germany, Italy, Norway, Poland, Portugal, Russia, Spain, Sweden, Switzerland, UK, Ukraine

Clear Voice Research‐Australia, Canada

‐China, Japan, South Korea, Singapore, Hong Kong, Taiwan

Methods

Selected demographic quotas (age, income, gender, ethnicity) were used to simulate census.

Median length was 15 minutes.

Questions covered: Technology and the media, Participation in market research, Buyer Behavior, Values and lifestyle, Demographics, Questionnaire Satisfaction.

Respondent Types Professional Respondents fall into four categories:

• (1) Self report taking on‐line Surveys “practically every day”.

• (2) Self report (open ended) taking over 30 online surveys “in the past month”.

• (3) Multiple panel membership > 5 panels.

• (4) Respondent panel tenure.

Inconsistency: Brand vs. Price, Price vs. Brand, Happy with standard of living vs. unhappy with standard of living.

Failure to follow instructions: Instructed to enter a predetermined answer.

Speeders: Lowest 10% of survey lengths.

Percent Respondents Doing More than 30 Surveys/Month

Red = US Panels Green = International Panels

0%

5%

10%

15%

20%

25%

30%

35%

40%

Singap

ore

Portugal

Hong K

ong

Switzerl

and

Finlan

d

Czech

Rep

Ukraine

Russia

Brazil

Argen

tina

ChinaS. K

orea

Norway

Denmark

Poland

Taiw

anUS11US10

Sweden

Fran

ceSpainIta

lyGerm

any

Canad

aUS16 UK

Australi

aJa

pan

US12US9US6

US17US14US13US7US18US8

Perc

ent 3

0+ s

urve

ys p

er m

onth

US11=RiverUS10= Social Network

Percent Respondents Doing Surveys Every Day

RED = US Panels Green = International Panels

0%

10%

20%

30%

40%

50%

60%

Singapore

Portugal

Hong K

ong

Switzerl

and

Finland

Czech

Rep

Ukraine

Russia

Brazil

Argen

tina

China

S. Korea

Norway

Denmar

kPola

ndTaiw

anUS11US10

Sweden

France

Spain Italy

German

yCan

ada

US16 UK Aus

tralia

Japan

US12 US9US6

US17US14US13 US7US18 US8

Perc

ent D

oing

Sur

veys

Dai

ly

Percent Respondents Enrolled in > 4 Panels

0%

10%

20%

30%

40%

50%

60%

70%

Singap

ore

Portugal

Hong K

ong

Switzerl

and

Finlan

d

Czech

Rep

Ukraine

Russia

Brazil

Argen

tina

ChinaS. K

orea

Norway

Denmark

Poland

Taiw

anUS11US10

Sweden

Fran

ceSpainIta

lyGerm

any

Canad

aUS16 UK

Australi

aJa

pan

US12US9US6


Enro

lled

in o

ver 5

Pan

els

or M

ore/

mon

th

Percent Respondents Who had an Inconsistent Brand Over Price Response

0%

2%

4%

6%

8%

10%

12%

14%

16%

Singap

ore

Portugal

Hong K

ong

Switzerl

and

Finlan

d

Czech

Rep

Ukraine

Russia

Brazil

Argen

tina

ChinaS. K

orea

Norway

Denmark

Poland

Taiw

anUS11US10

Sweden

Fran

ceSpainIta

lyGerm

any

Canad

aUS16 UK

Australi

aJa

pan

US12US9US6


Perc

ent I

ncon

sist

ent o

f Bra

nd o

ver P

rice

Percent Respondents Who Failed to Follow Instructions by Panel

0%

5%

10%

15%

20%

25%

30%

35%

Singap

ore

Portugal

Hong K

ong

Switzerl

and

Finlan

d

Czech

Rep

Ukraine

Russia

Brazil

Argen

tina

ChinaS. K

orea

Norway

Denmark

Poland

Taiw

anUS11US10

Sweden

Fran

ceSpainIta

lyGerm

any

Canad

aUS16 UK

Australi

aJa

pan

US12US9US6


Perc

ent W

ho F

aile

d to

Fol

low

Inst

ruct

ions

Percent Respondents Who are Speeders by Panel

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

Singap

ore

Portugal

Hong K

ong

Switzerl

and

Finlan

d

Czech

Rep

Ukraine

Russia

Brazil

Argen

tina

ChinaS. K

orea

Norway

Denmark

Poland

Taiw

anUS11US10

Sweden

Fran

ceSpainIta

lyGerm

any

Canad

aUS16 UK

Australi

aJa

pan

US12US9US6


Perc

ent S

peed

ers

1.3

2.5

6.0 6.04.8

6.3

4.5

1.2 1.1

5.2 5.25.8

2.23.4

6.4

4.4

8.0

0123456789

M3 M4 M5 M6 M7 M8 M9M10M11M12M13M14M15M16M17M18

All Pan

els

Panels

Num

ber o

f Pan

els

Average Panel Membership by Panel in the U.S.

Impact of Max Panel Age in the U.S. on Sociologic and Buyer Segmentations

0%

5%

10%

15%

20%

25%

30%

35%

40%

0 Months 6 Months 12 Months 18 Months 24 Months

Max Age on Panel

Dev

iatio

n (S

quar

e R

oot o

f the

Su

m o

f Squ

are

Dev

iatio

ns)

Socio Deviation Buyer Deviation

Max Age on Panel by Panel in the U.S.

0%

20%

40%

60%

80%

100%

M6 M7 M8 M9 M10 M11 M12 M13 M14 M16 M17 M18 GrandTotal

Panel

% o

f Res

pond

ents

0 Months 6 Months 12 Months 18 Months 24 Months

Social/Psychographic Variation Social opinions and behavior can be expected to drive

purchasing behavior or at least provide a basis for segmenting the market. Consistency of these measurement may likewise be critical.

Variables Groups• Internet Use• Taking Surveys• Having a Passport• Social Characteristics

Measures:• Driving Variables

Global variation from grand mean of standardized sociographic segments.

-60

-40

-20

0

20

40

60

Social

Netw

ork

Contri

bute O

n Line

Instant

Messa

ging Blog

Share

Pictur

es

Downloa

d Vide

o

On-line C

alend

ar

Games

Onli

ne

Uncon

ventio

nal

Enjoy R

isks

Time o

ver M

oney Mag

Stay Info

rmed

Compu

ters m

akes

Easier

Radio

Speak

Mind

Read S

unday

Newsp

aper

Lower

Std to

Cons

erve

Happy

w/f S

td of L

iving

Good w

ork/l

ife B

alanc

e

Asked

for A

dvice

Resea

rchM

odes

Read N

ewpap

er

Enjoy P

olitic

s

Too M

uch C

once

rn on

Env.

Alcoho

l off T

V

Childr

en A

ds of

f TV TV

No Com

puter

Global

Warming

Passpo

rtSt

anda

rd E

rror

s

High Computer/Stays Informed (40%) Happy with Life/Not Computer (29%) Opinionated/Not Computer (31%)

Global Average Sociographic Segmentation Distribution

0%

20%

40%

60%

80%

100%

USCan

adaFran

ceGerm

any

Italy

Spain UK Argen

tina

Australi

aBraz

ilChina

Czech

RepDen

markFinlan

dHong Kong

Japan

Norway

PolandPortu

galRuss

iaS. K

oreaSingap

oreSwed

enSwitz

erland

TaiwanUkra

ine

% o

f Res

pond

ents

in S

egm

ent

High Computer/Stays Informed Happy with Life/Not Computer Opinionated/Not Computer

US Sociographic segment distribution by panel and phone.

0%

20%

40%

60%

80%

100%

US1*US2*US3*US4*US5* US6

US7US8US9US10US11US12US13US14US16US17US18

US Pho

ne*

% o

f Res

pond

ents

in S

egm

ent

High Computer/Stays Informed Happy with Life/Not Computer Opinionated/Not Computer

* EM Algorithm for Missing Data & Logit Model for Segmentation

Social Network

Variation in Buyer Behavior

Measuring buyer behavior is the objective of most marketing research. And therefore, consistency of those measurement are critical.

Variables• Number of High Tech Items Purchased. • Internet Purchase behavior• Purchasing Opinions

Measures:• Clusters (Segments) • Driving Variables

Buyer Behavior Segment Profiles

-50

-40

-30

-20

-10

0

10

20

30

40

50

High Tech Purch

ases

Download

Music

Lastes

t Elec

tronic

s

Interne

t Rad

io

Purcha

se O

nline

Techn

ology

Brand ov

er Pric

e

Video Gam

es

Online B

ankin

gTrav

elApp

roval

Quality

Takes

Trips

Impro

ve Home

Passp

ort

Price o

ver B

rand

Shop Around

EMailCred

it

Enviro

nment

Freque

nt Flie

rDom

estic

Use Coupon

sCou

ponsSmoke

Hrs. O

n-line

Informati

onSt

anda

rd E

rror

s

Broad Range/Credit (34%) Price Sensitive Shoppers (18%) Credit/Environment (27%) Domestic/Coupons (21%)

Distribution of Buyer Behavior Segments by Countries.

0%

20%

40%

60%

80%

100%

USCanad

aFran

ceGerm

any

Italy

Spain UK Arg

entina

Australia

Brazil

ChinaCze

ch R

epDenmark

Finland

Hong Kong

Japa

nNorw

ayPolandPortu

galRussia

S. Kore

aSingap

oreSwed

enSwitz

erland

Taiwan

Ukraine

Perc

ent o

f Pan

el in

Seg

men

ts

Broad Range/Credit Price Sensitive Shoppers Credit/Environment Domestic/Coupons

US and Global Distribution of Buyer Behavior among Panels

0%

20%

40%

60%

80%

100%

US6US7US8US9US10US11US12US13US14US16US17US18 UKSpain

Germany

FranceIta

lyJapan

Total US Panels

Perc

enta

ge o

f Res

pond

ents

Broad Range/Credit Price Sensitive Shoppers Credit/Environment Domestic/Coupons

Social Network

-10-8-6-4-202468

10

M11 M3 M4 M10 M2 M16 M15 M1 M9 M12 M7 M13 M5 M14 M6 M17 M8 M18

Stan

dard

Err

ors

Conventional Purlchasers On-liners Low Card OL Banking

97%99%

River SocialNetwork

PointSystem

UK Access Panels

Statistical Panel Profiles Against Buyer Segments

+/-14.8%

+/-10.8%

+/-12.9% +/-14.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Broad Range/ Credit Price SensitiveShoppers

Credit/ Environment Domestic/ Coupons

Perc

ent o

f Res

pond

ents

Buyer Behavior Segments

Expected Range of Values for a Random 3 Panel Sample Showing 1.281 Standard Errors (20% of being beyond this range) in the

U.S.

+/- Coefficient of variation

+/-20.8%+/-15% +/-17.1%

0%

10%

20%

30%

40%

50%

60%

High Computer/ StaysInformed

Happy with Life/ NotComputer

Opinionated/ Not Computer

Perc

ent o

f Res

pond

ents

Sociographic Segments


U.S.

+/- Coefficient of Variation

+/- 22.1%

+/- 18.6% +/- 15.8% +/- 16.4%

0%

10%

20%

30%

40%

50%

60%

Internet Stay Informed Enjoys Politics Concerned

Perc

ent o

f Res

pond

ents

Media Segments


U.S.

+/- Coefficient of Variation

Optimization Profile

0.00

0.01

0.02

0.03

0.04

0.050.06

0.07

RM

S Er

ror

0%20%

40%60%

80%100%0% 20% 40% 60% 80% 100%

Percent of M8

Percent of M17

OptimizationsPanels Optimum Average

Expected (1 SE) Inherent (1 SE)

M8 24% 33%M17 26% 33%M12 50% 34%

Root Mean Square Error 0.40% 2.36% 8.31% 2.45%

Panels Optimum AverageExpected (1 SE) Inherent (1 SE)

M8 0% 33%M13 91% 33%M16 9% 34%


Panels Optimum AverageExpected (1 SE) Inherent (1 SE)

M10 8% 33%M13 66% 33%M16 27% 34%


Summary Panel ageing in the U.S. has led to degradation.

Professional Respondents and other problem respondent types appear to greatly affect the reliability of panel research results.

Sample sources around the world are beginning an ageing cycle. There is still time to document and stabilize the situation.

Reliability and consistency of samples can be improved by combinations of panels (Blending Methodologies).

Optimization models improve blending methods, data is needed within each market to create the baselines so that these models can be employed.

Thank youSteven Gittelman, Ph.D.

and Elaine Trimarchi 200 Carleton Avenue

East Islip, New York 117301‐631‐277‐7000

by steven gittelman, ph.d. and elaine trimarchi

Documents