Ensuring Sample Quality A Two-Tiered Approach

Lisa Wilding-BrownVP, Global Panel and Sampling Operations, uSamp

Peter GoldCEO, VeraQuest

Today’s Topics

• Sample Quality Landscape

• The Quality Continuum

• 5 Tips for Optimizing Sample Quality

• Research on Research: A Case Study

State of the Industry

• Online market research has matured and stabilized

• Host of quality-focused consortiums and sample validation products available

• Data quality is an evolving and dynamic topic

• Our work is never done!

Vigilance Required

Spectrum Challenge

There is no silver bullet

Damage Control

How well do you know your sample?

• Demographic, behavioral and attitudinal data together provide a complete picture of source quality

• Analyzing benchmark data against known population characteristics helps to identify skews

Source Testing At Work

Source 9

Source 8

Source 7

Source 6

Source 5

Source 4

Source 3

Source 2

Source 1

+/- 5%

+/- 6-10%

+/- 11-20%

+/- 20%+

Yes

Yes

Yes

Yes

No

No

Maybe

Maybe

Maybe

What’s your first impression?

• Leveraging available tools helps verify respondent identity at the point of registration:

– Email and IP address verification

– Geo-IP look-up

– Digital fingerprinting

– Proxy server detection

How good is your balancing act?

• Using demographically balanced sample helps achieve more representative results

• Stratifying sample frames by activity levels, tenure and source helps to minimize bias

Are you getting consistent performance?

• Monitoring quality behaviors throughout lifetime, not just at registration point, helps to maintain consistency

• Utilizing Outlier / Black Swan algorithms helps to lessen the data impact of highly improbable characteristics or events

The Black Swan

Caucasian CEO of a fortune 500 company…

…suffering from alopecia…

…and a super rare skin condition…

…currently living in New York…

…No! San Francisco…

…who drives a cherry red Lincoln…

…and owns a show dog…

…who won Best in Show at Westminster.

How can researchers help?

• Using thoughtful, disguised screening ensures the intended audience is reached

• Inclusion of Red-Herring questions weeds out over-zealous and inattentive respondents

• Maintaining an open feedback loop with sample suppliers helps manage potential quality offenders

All Hands On Deck!

Research on ResearchA case study in identifying fraudulent

or inattentive respondents

Artisan Bread Study

Objective

Assess brand awareness among national population for west coast artisan bread brand relative to other artisan brands in same region.

Brands

- Brand X (Client Brand) - Tribeca Oven - Maple Leaf - California Goldminer - Cuisine de France - Chabaso- Ace Bakery - Ecce Panis

Artisan Bread Study Percent Straight-Liners

97%

Straight-Liners

3%

Artisan Bread Study Percent Aware of 5+ Artisan Bread Brands

89%

Aware of 5+

Brands11%

Most Recent Case Study

Design:

All respondents were asked brand awareness for ten or twelve brands in three categories

• Shampoo • Juice • Chips

• Cell 1: yes/no grid - 10 fictitious brands and 0 real brands • Cell 2: pick list - 10 fictitious brands and 0 real brands• Cell 3: yes/no grid 10 fictitious brands and 2 real brands• Cell 4: pick list - 10 fictitious brands and 2 real brands

The Questions We Set Out to Answer

1. Will lists of fictitious brands help us to ID fraudulent responders?

2. Are pick lists preferable to yes/no grids?

3. Does it make sense to include at least two real brands in the brand list?

Number of Respondents Claiming Awareness of Fictitious Brands

0 1 2 3 4 5 6 7 8 9 10

767

104 61 47 27 25 15 13 7 3 32

3+ Brands = 15%

4+ Brands = 11%

Shampoo

0 1 2 3 4 5 6 7 8 9 10

610

182

98 73 43 36 19 6 11 2 21

3+ Brands = 19%

4+ Brands = 13%

Juice

Number of Respondents Claiming Awareness of Fictitious Brands

0 1 2 3 4 5 6 7 8 9 10

684

177

90 50 29 22 10 9 5 5 20

3+ Brands = 14%

4+ Brands = 9%

Chips

Number of Respondents Claiming Awareness of Fictitious Brands

Fictitious Brands Correlation, by Category

Shampoo & Juice Shampoo & Chips Juice & Chips

0.763 0.776 0.843

R2

Question 1: Will lists of fictitious brands help us to ID fraudulent responders? Answer: Yes. Or at least we think so.

Using Fictitious Brand Names to Identify Fraudulent Responders

Percent of Respondents Claiming Awareness of Fictitious Brands

Yes/No Grid Pick List Fictitious Brands Aware Shampoo Juice Chips Shampoo Juice Chips

10 4% 3% 3% 1% 1% 1%9+ 5% 3% 4% 1% 2% 1%8+ 6% 5% 4% 1% 1% 1%7+ 7% 5% 6% 2% 2% 1%6+ 8% 8% 7% 4% 2% 1%5+ 11% 13% 10% 6% 4% 3%4+ 14% 17% 13% 8% 8% 5%3+ 19% 23% 18% 12% 15% 9%2+ 23% 31% 25% 18% 25% 18%1+ 32% 48% 39% 28% 41% 37%0+ 100% 100% 100% 100% 100% 100%

Yes/No vs. Pick List

Percent of Respondents Aware of at Least One of the Real Brands

97%

Yes/No Grid Respondents Aware of at Least One Real Brand

97%

Pick List Respondents Aware of at Least One Real Brand

Yes/No vs. Pick List

Using Fictitious Brand Names to Identify Fraudulent Responders

Question 2:

Are pick lists preferable to yes/no grids for detecting fraudulent respondents?

Answer: Probably.

We believe yes/no grids may actually exacerbate fraudulent behavior.

Percent of Respondents Claiming Awareness of Fictitious Brands

10 Fictitious/No Real Brands 10 Fictitious/2 Real Brands Fictitious Brands Aware Shampoo Juice Chips Shampoo Juice Chips

10 4% 2% 2% 2% 2% 1%9+ 4% 3% 3% 2% 2% 1%8+ 5% 4% 4% 3% 2% 2%7+ 6% 4% 5% 4% 3% 2%6+ 8% 6% 6% 5% 5% 3%5+ 11% 9% 8% 6% 8% 5%4+ 14% 15% 11% 9% 10% 8%3+ 18% 22% 15% 13% 17% 12%2+ 25% 30% 23% 17% 26% 21%1+ 33% 43% 38% 28% 47% 38%0+ 100% 100% 100% 100% 100% 99%

10 Fictitious/No Real Brands vs. 10 Fictitious Brands/2 Real Brands

Using Fictitious Brand Names to Identify Fraudulent Responders

Question 3:

Does it make sense to include at least two real brands in the brand list?

Answer: Yes.

The absence of real brands from which to choose likely causes respondents to erroneously select a fictitious brand.

In Summary

Tips for Optimizing Sample Quality:

Source Testing

Registration Verification

Sample Balancing

Respondent Monitoring

Research Design Best Practices

Research on Research Case Study:

Red Herring questions are effective

Pick lists are preferable to yes/no grid designs

Inclusion of 2 or more real brands is optimal if using fictitious brand lists

Thank you!

Lisa Wilding-BrownVP, Global Panel and Sampling Operations, uSamplisa@usamp.com

Peter GoldCEO, VeraQuest

peter.gold@veraquestresearch.com