esomar panel research 2006 barcelona the effects of panel recruitment and management on research...

ESOMAR Panel Research 2006 Barcelona

The Effects of Panel Recruitment and Management on Research

Results

Dutch Online Panel StudyNOPVO 2006

An initiative ofPieter Willems - Millward Brown

Robert van Ossenbruggen - ProCression Ted Vonk - Onderzoekpaleis

with the support of the MOA

ESOMAR Panel Research 2006 – Barcelona - 2

COPYRIGHT

This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006).

It is allowed to publish the outcomes of this study if you refer to:

• The name of the study (NOPVO 2006)

• The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown)

Contact: [email protected]


Agenda

• Introduction

• Different approaches, different results?

• Does the response rate affect the results?

• Does the type of respondent explain the differences between panels?

• Conclusions and recommendations


Introduction


Some facts and figures about The Netherlands

Demographics

• Population: 16,3 million

• 18-65 years: 10,5 million

• Internet penetration: 82%

• Broadband penetration: 63% (of all households, not internet users)

Market Research

• Total market in 2005: € 278 million ($ 346 – 12th largest market)

• Online data collection: 25% of quantitative revenue

• Over 25 commercial online access panels

• Majority of panels exist for longer than 5 years


The Dutch Online Panel Study (NOPVO)

Why?1. To provide an answer to quality concerns

2. To provide input for guidelines for online panel innovation

3. A unique situation: large number of panels available

Research questions - NOPVO1. How do panels work? Panel overview

2. Who is the online panel member?

3. How do panellists respond?

4. To what extent is opinion and marketing data influenced by the choice of panel?

No objective: to qualify good or bad panels


Two parts of NOPVO

1. Inventory of all online panels in the Netherlands1. Answers to the 25 questions of ESOMAR

2. Overview available at www.moaweb.nl

2. Empirical study• One survey

• Every panel draws sample of 1000 respondents

• Same fieldwork period of 7 days (no reminders)

• Omnibus questionnaire to avoid selective response

19 panels participated:

Representing over 1.600.000 panellists


Extensive data available

Per panel:-Type of incentive-Age of Panel-Invitation policy -etc. Per panel member:

-Duration of panel membership-Method of recruitment-Number of invitations-Individual response rate-Socio demos, etc.

Per respondent:-Response (time)-Membership of other panels- Data from the questionnaire


Some facts about participating panels

• Almost all panels give incentives for cooperation• Usage of different types of incentives• Average € 1,20 for 10 minutes

• Most panels have a restriction policy for inviting panel members to complete a survey

• 1.650.000 million panel members• Smallest panel:15.000• Largest panel: 230.000

• Several panels do not have full details available about interview history of panel members


Observations about the samples• Composition of samples to duration of panel membership differs a

lot per panel• Panels with mainly respondents recruited during last 6 months• Panels with mainly panel members recruited during set up of panel

• Recruitment methods vary substantially across panels • Most panels use a mixture of recruitment methods• Some panels use one single method (online, by telephone)

Recruitment method of panel members in gross sample

16%

15%

22%

6%

18%

1%

1%

5%

10%

5%

Buying addresses

Self registration

Via links/banners

Snowballing

By telephone

From existing panels

Advertising in print media

Via website surveys

Via traditional research

Other


Different approaches, different results?


Differences in results between panels limited

• There will always be some fluctuations because of sampling (sampling error)

• Most of the differences between panels are about what one could expect because of confidence intervals

• There are huge differences between extreme scores• Extreme scores lead to different conclusions • Change of panel could lead to change in trends

Substantial differences between minimum and maximum scores

53%

12%

60%

21%

62%

19%

69%

35%

82%

31%

77%

50%

Paid job for over 15hours per week

Travels with publictransport toworkplace

Awareness Corona Surfing on theInternet: 10+ hours

per week

min total max


Substantial differences caused by a few panels with extreme scores

How to read the graph

•Largest differences (N=19)

•Minimum and maximum excluded (N=17)

•2 minima and maxima excluded (N=15)

Reduced differences by excluding extreme panels

0%

20%

40%

60%

80%

100%

n=19 n=17 n=15 n=19 n=17 n=15 n=19 n=17 n=15

Paid job for over 15 hoursper week

Travels with publictransport to workplace

Awareness Corona

min total max

28% 15% 9% 19% 10% 9% 17% 9% 7%

Extreme score are spread over all panels•Lowest number of extreme scores: 1%

•Highest number of extreme scores 12%


Does the response rate affect the

results?


NOPVO response overall: 50%

• Level of response is a result of several factors

• Panel age, method of recruitment, incentives, panel care, etc

• It is hard to determine the effects of a single factor

• NOPVO response overall: 50%• 19.000 panel members invited….9.462 completed the questionnaire…

• Substantial differences in response per panel

Response per panel

18 19

36 39 39 39 40 4149 54 55 56 60 60 62 66 68 70

77

50%


High/low response explained(1)• Panels with a low response rate are panels that do not remove non-

responding panellists from their panel database• Panels invited panel members who did not complete a single questionnaire in

over a year despite many invitations• Number of panel members is an important communication topic to the market

• New panel members respond better than old panel members

• Effect of incentives• Certainty about incentive is important• Positive effect of incentive stops at 1 euro (10 minute questionnaire)

Response per type of incentive

70%

62%

58%

49%

39%

29%

Cash

No incentive

Value points

Charity and lotteries

Value points and lotteries

Lotteries

response %Response and value of incentive

59%

57%

37%

42%

> 1 euro

1 euro

< 1 euro

No (fixed) incentive

response %


High/low response explained(2)

• Method of recruitment: • Recruitment by invitation (by telephone, traditional research) gives

higher response than self selection (self registration, links/banners)

Response per recruitment method

58%

45%

By invitation

Self selection

respons %


Outcomes do not depend on response rate

Number of …Panels with

response <40%

Panels with response 40-60%

Panels with response>60%

Words in open ended question 24,7 25,7 25,9

Awareness beer brands (spont.) 6,1 6,1 6,3

Awareness beer brands (aided) 20,1 19,8 20,0

Glasses beer per week 11,1 10,9 10,4

Advertisements seen 5,8 5,8 5,8

Awareness TV brands (aided) 13,8 13,5 13,8

No different means on survey questions between low/high responding panels


No response effects explained

• Non response stages

• During recruitment

• Removal of bad/non responding panel members• For a specific survey

• Non-response effects only occur if there are differences between respondents and non-respondents• There is no distortion because of missing categories (they are missing

in all panels)

• Members of online panels are a homogeneous group• They are willing to participate in surveys• They are heavy internet users

• Non-response effects take place while recruiting panels• double opt-in• explicit consent to participate in research


Representative power of panels is more limited than assumed

• Missing groups in panels• Ethnic minorities/immigrants are missing almost completely

(10% of population, 35% in major cities)

• They are missing in all data collection methods

• Structural deviations in NOPVO study • Too many heavy Internet users

• Too many voters (90% instead of 79%)

• Voting behaviour compared with last elections

> Number of Christian Democrat voters (16% instead of 29%)

> Too many Socialist Party voters (14% instead of 6%)

• People who go to church are underrepresented

• Large part of the population is not represented in the panels


Response rate: panel strategy

• A high response rate does NOT indicate high sample quality

• Response rate is a panel strategy component, driven by rules of economics• Panel setup and maintenance determine response rate

• Value of response rate is evaluated against recruitment costs

• Response rate can be influenced by• Excluding hard to reach groups

• Selection of good i.e responding panellists

• The response problems of traditional research are not solved. Online research is dealing with it more efficient.


Panel overlap diminishes panel management effects • 62% of respondents are a member of multiple panels

• Average number of memberships of all respondents: 2,73 panels

• Panel overlap varies between 29% and 88% per panel

• Highest single overlap between two panels: 42%

With this level of overlap a restriction policy for inviting panellists does not make much sense

Number of panel memberships1 panel

38%

2 panels15%

3 panels14%

4 panels10%

6-10 panels13%

Over 10 panels

3%

5 panels7%


Highest overlap with addresses from a broker

• 81% of panellists recruited with bought addresses are members of multiple panels

• Recruitment by invitation (telephone, traditional research) smallest overlap

• Average number of panels varies between 2,0 recruitment by telephone and 4,3 by bought addresses

Number of panels per recruitment method

19 26 33 4053

63

38

4346

46 35

3429

39

2820 14 18

9 616

Buyingaddresses

Via links/banners

Via websitesurveys

Selfregistration

Viatraditionalresearch

By telephone Total

%

1 panel 2-5 panels over 6 panels


20%-80% rule is valid for online panel research

• Multiple panel members respond better and faster

• Limited group of panellists complete substantial part of all questionnaires

Respondents, completed questionnaires and panel memberships

78%

20%

6%

12%

5%

13%

4%

12%

2%4%

25%

1%8%

9%

900.000 Panellists25.000.000 completed

questionnaires

11 panels of more

6-10 panels

5 panels

4 panels

3 panels

2 panels

1 panel


Does the type of respondent explain differences between

panels?


“It's the perfect job for women who want to stay at home with their children, or anyone who needs extra money they would like to earn from home.”

“By joining this site, you are joining a community of professional survey-takers with information on the best, paid survey sites in the UK, links to UK survey news, bulletin boards and advice on the business of online paid surveys.”

(source: www.thesurveypro.com)


Nice to meet you: the Professional Respondent

• Member of many panels

• Lots of questionnaires completed

• Operationalization NOPVO:‘index professionalism’=

1. # panels membership (Z) +

2. # questionnaires completed last 4 weeks (Z)

• Selection for analysis: top 10%


Nice to meet you: the Loyal Respondent

• The ‘loyal’ respondent: takes time to complete questionnaire and puts effort in open ended questions

• Operationalization NOPVO: ‘index loyalty’=

1. duration of interview (Z) +

2. # words open ended questions (Z)



Nice to meet you: the Inattentive Respondent

• The inattentive respondent: opposite of loyal respondent

• Thus: speeds through questionnaire and minimum input open ended questions

• Operationalization NOPVO:‘Index inattentiveness’= index loyalty * -1



Present typology analogous to earlier studies

1 Smith, R. & Hofma Brown, H., (2005). Assessing the Quality of Data from Online Panels: Moving Forward with Confidence. White paper, Harris Interactive.

2 Downes-Le guin, T. (2005). Satisficing Behavior in Online Panelists. White paper, Doxus.

also relevant in this discussion:

Knapton, K. (2006). Catch Me If You Can! (How to Keep Cheaters Out of Your Next Online Survey). Presentation at The Market Research Event. Los Angeles, October 23, 2006.

NOPVO Similar to… Harris1 Doxus2

Professional Hyperactives

Loyal Optimizers

Inattentive Inattentives Satisficers

Fraudulents Gamers


Respondent type & characteristics

• Professional respondent• internet frequency • ‘fresh’ panels • education • unfit for work • satisfaction income • democracy • perceived health

• Loyal respondent• education

• Inattentive respondent• older respondents


35%56% 47%

17%

23%

14%

17%

19%

33%

9%

30%

31%

14%

45%

17%

23%

Total professional loyal inattentive

Traditional datacollection

Adresses purchased

Through links/banners

Other

Recruitment: loyals from traditional data collection methods

inattentive:Self select

loyal: By invitation

professional: Self selection

recruitment panellist (data available of 12 panels)


4% 1% 3%

54%

39%

69%

34%

17%

22%

9%

30%

26%

39%

17%

33%

5%

Total professional loyal inattentive

Nice to do

Incentive

Other intrinsic drivers*

Other

Motivation: incentive or intrinsic

inattentive:incentive

driven

loyal: intrinsic

motivations

professional: nice to do

Reasons for participating in internet surveys. Which one is the most important to you?

*Such as:

• Curiosity

• I like to give my opinion

• I like to see results

• I like to be part of an online

community


7% 8%15%

16% 18%

9%

22%

22% 21%

18%

21%

43% 42%

50%

34%

12% 11%21%

8%

2%Totaal professioneel loyaal gemakzuchtig

Strongly

Normal

Some

A bit

not interested

Political interest: large differences

inattentive: weak

politicalinterest

loyal: strong

politicalinterest

professional: no

effect

To what extent are you interested in politics?


Add familiarity: differences between types up to 15%...• Largest effect for inattentive respondents

• Professional respondent just as atypical as loyal respondents

(zero =

total)

-12%

-8%

-4%

0%

4%

8%

12%

Ach

mea

Cit

roën

Do

ve

Fo

rtis

Maz

da

Niv

ea

OH

RA

ON

VZ

T-M

ob

ile

Vo

daf

on

e

Vo

lksw

agen

professional loyal inattentive


Brand familiarity: differences between types up to 20%...

-15%

-10%

-5%

0%

5%

10%

15%

Aiw

a

Aka

i

Ban

g &

Olu

fsen

Bek

o

Fin

lux

Gru

nd

ig

Hit

atch

i

JVC

Len

co LG

Lo

ewe

Met

z

NE

C

Ori

on

Pan

aso

nic

Ph

ilip

s

Pio

nee

r

San

yo

Sam

sun

g

Sh

arp

So

ny

Th

om

son

To

shib

a

Ves

tel

Vie

wso

nic

professional loyal inattentive


Inattentive respondents: the easy way out

1%2%

14%

6%

1% 1%

12%

7%

0%1%

12%

4%

7%6%

21%

12%

clicking through('straightlining')

'don't want to say' (1+) 'don't know' (1+) 6 or more (of 10) neutralscores on 5-p scale

total professional loyal inattentive


Main concern: the inattentive respondent

• Q: Is the concern for the professional respondent justified?

• A: Partly, because this type of respondent is atypical, but the effects are not large

• Q: Is the loyal respondent also the ideal respondent?

• A: This is material for further investigation• Serious effort (by definition)• High data quality• But: atypical… now who really is atypical? The loyal or the not so loyal?

• Q: Is the inattentive respondent a threat to online panel research?

• A: Absolutely: atypical group with large effects responsible for noise in data


Respondent behaviour varies between panels

relatively fewinattentiverespondents

relatively manyinattentiverespondents

0% 20% 40% 60% 80% 100%

S

R

Q

P

O

N

M

L

K

J

I

H

G

F

E

D

C

B

A

total

inattentive

professional

loyal

other


Conclusions

Conclusions

• In general: the differences in outcomes between panels are limited despite differences in panel management and panel maintenance

• However, there are differences• The differences are not a result of differences in response or sample

composition. • Respondent types, defined by questionnaire experience or effort,

display more or less atypical and suspect response behaviour• Differences between panels can substantially be explained by

differences in distribution of these respondent types

• Panels comprise a specific group of respondents that differ on relevant criteria from the national population. The representative power of online panels is more limited than assumed so far

• Response percentage does not indicate sample or panel quality. The response rate is an indication of the level of efficiency of the panel provider.

• Online panels have a substantial overlap in terms of panellists. This reduces the impact of a panel management strategy.


Recommendations

• Monitor suspect respondents and remove the really bad ones

• Deploy other, new recruitment methods to get the missing part of the population in the panels

• Use online panels for the right kind of research goalsuse panels for

• continuous research or trend measurement

• concept testing

but not for

• incidence rating


Practice makes perfect

Thank You

esomar panel research 2006 barcelona the effects of panel recruitment and management on research...

Documents

panel panels

panel slide

esomar panel research

largest panel

panel members smallest

age of panel

panel overview

panel care