whither survey response rates - tns canadatnscanada.ca/conferences/mria/tnscf-whither survey...
TRANSCRIPT
© 2006 MRIA Response Rate Committee
Whither Survey Response Rates
Do They Still Matter?
Gary Halpenny, TNS Canadian FactsDon Ambrose, Consumer Contact
June, 2006
© 2006 MRIA Response Rate Committee
Background
© 2006 MRIA Response Rate Committee
Telephone surveys have been under attack recently on the grounds that:
“Results are no longer accurate nor representative”
Low response rates are cited as the the reason
© 2006 MRIA Response Rate Committee
However: a growing body of research begs to differ
A number of investigative projects in the U.S. have shown:
for most commercial and public opinion applications a 30% response rate produces essentially the same results as a 50% response rate
© 2006 MRIA Response Rate Committee
Some of the research literature:In 1997, two identical surveys, one at 61% response rate and the other at 36%, produced no meaningful differences
This project was replicated in 2003 with 51% and 27% response rates and with similar results
Researchers concluded “carefully conducted polls with relatively low response rates still yield representative samples and accurate data”(Keeter et al, Pew Research)
© 2006 MRIA Response Rate Committee
The reality today is that few commercial telephone surveys even approach the 30% level
The demand for faster turnaround means most telephone response rates are now in the 10% to 20% range
Quick 1 or 2-day polls can yield even lower rates
© 2006 MRIA Response Rate Committee
The Critical Issue!!
Can response rates at these levels still produce accurate and meaningful data?
Clearly more research was needed
© 2006 MRIA Response Rate Committee
MRIA’s Research Project
© 2006 MRIA Response Rate Committee
The Plan
In 2005, the MRIA Response Rate Committeesponsored research to investigate whether response rates as low as 10% can still produce reliable and useful data.
Five Canadian research companies who regularly conduct national omnibus surveys volunteered to combine efforts.
© 2006 MRIA Response Rate Committee
Stage 1:
Using an identical 5 minute question set, each company completed approx. 250 interviews on a single wave of its Omnibus in January, 2006.
1,238 completed interviews in total4 days in field9% aggregate response rate
© 2006 MRIA Response Rate Committee
Stage 2:
Using the same 5 minute question set, each company completed a second sample of approx. 250 interviews over Jan/Feb, 2006.
1,273 completed interviews in total4 to 5 weeks in fieldFirst refusals recontacted31% aggregate response rate
© 2006 MRIA Response Rate Committee
National RDD, age 18+
Weighted to Census for:Age
Gender
Province
Community Size
Both samples were:
© 2006 MRIA Response Rate Committee
Fieldwork undertaken by:
Ipsos-Reid
Maritz
Opinion Search
Synovate
TNS-Canadian Facts
© 2006 MRIA Response Rate Committee
Record of Call Comparison
Higher contact ratio (lower proportion of busy/no answer)
Completion/refusal ratio increases from .26 to .77
Means that fewer good telephone numbers required to yield same number of interviews.
Table on next slide indicates that additional call attempts yield three main benefits
© 2006 MRIA Response Rate Committee
79
1,273
105
1,238
Disqualified
Completed Interviews
1,35231.1%
1,3439.1%
Cooperative Respondents
7801,647
569
5,8434,8262,820
Busy/No AnswerRefusedOther Non-responding
4,348100%
14,832100%
Valid numbers attempted31% RR9% RRDisposition of last attempt
© 2006 MRIA Response Rate Committee
Key Findings
© 2006 MRIA Response Rate Committee
Both Studies Yield Identical Results* for:Incidence of food items used in past 6 mos.List of items bought in last 12 monthsAppliances in HouseholdPrint media readership – not title specificIncidence of Travel outside CanadaPersonal Access to the InternetCell phone ownership and carrier used
*Differences considered significant at the 90% level
© 2006 MRIA Response Rate Committee
9% RR 31%RR Sig. Diff % % Eggs 97 96 N Cold Cereals 86 86 N Cheese (Not Processed) 69 71 N Honey 67 66 N Frozen Pizza 55 55 N
Food Items Used in Past 6 MonthsResults Identical
© 2006 MRIA Response Rate Committee
Items Bought In The Last 12 MonthsSame result regardless of whether category incidence is high, medium or low
9% RR 31%RR Sig. Diff % %
Men’s Or Women’s Clothing 93 93 N Sunscreen/Suntan Lotion 54 54 N Paint or Stain 52 51 N Camping Equipment 23 23 N Car Polish/Wax 21 20 N Traveler’s Cheques 8 8 N
© 2006 MRIA Response Rate Committee
Appliances in HouseholdSimilar findings for both commonplace and more esoteric items
9% RR 31%RR Sig. Diff
% %
Microwave Oven 95 95 N Automatic Dishwasher 63 61 N Gas BBQ 59 57 N Security System 34 37 N Espresso/Cappuccino Maker 14 13 N
© 2006 MRIA Response Rate Committee
Print Media ReadershipSimilar estimates of generic print media consumption
9% RR 31%RR Sig. Diff % %
Read A Daily Newspaper -Yesterday 60 60 N - Past Week 84 84 N
Last Time Read A Magazine - Yesterday 38 40 N - Past Week 72 72 N
© 2006 MRIA Response Rate Committee
Traveled Outside Canada In Past 12 MonthsParallel results for both business and personal travel behaviour
9% RR 31%RR Sig. Diff % % For Personal 32 33 N For Business 8 9 N
© 2006 MRIA Response Rate Committee
Personal Access To The InternetPenetration levels virtually identical
9% RR 31%RR Sig. Diff % %
Any Access 76 76 N
At Home 70 70 N At Work 45 46 N
© 2006 MRIA Response Rate Committee
Cell PhonesNo differences in either ownership incidence or carrier share
9% RR 31%RR Sig. Diff % % %
Has a Cell Phone 58 58 N
Cellular Provider * Bell 29 28 N Telus 25 27 N Rogers 25 25 N Fido 6 6 N Other 11 11 N * Base Total Cell phone owners
© 2006 MRIA Response Rate Committee
Credit Card Ownership and Usage
Higher response rate yields higher incidence of credit card ownership
Among card owners, high RR yields a higher incidence of owning American Express and a lower incidence of MasterCard
Posit that the higher RR captures a more upscale, harder-to-find group of people but not proven in the demos
Equally as likely to be a statistical anomaly
No differences in card used most often
Differences are found here.
© 2006 MRIA Response Rate Committee
Credit Cards Owned 9% RR 31%RR Sig. Diff % % Has Any Credit Cards 78 82 +5
Specific Cards Owned* Visa 67 70 N MasterCard 52 48 -4 American Express 13 18 +5 Diners 1 1 N Any Department Store 45 46 N Any Gasoline Company 15 14 N
Avg. # of Cards Owned* 2.4 2.5 N * Base: Total Credit Card owners
© 2006 MRIA Response Rate Committee
Credit Card Used Most OftenClaimed usage level unaffected by higher response rate
9% RR 31%RR Sig. Diff Base = Owners of Credit Cards % % Visa 49 52 N MasterCard 30 29 N American Express 3 4 N Any Department Store Card 3 3 N Any Gasoline Co, Card 1 1 N
© 2006 MRIA Response Rate Committee
12 Attitudinal Statements Measured
Mean scores the same on 11 attributes out of 12
Difference on the statement related to shopping was statistically significant but would not have changed the interpretation
© 2006 MRIA Response Rate Committee
Attitudinal Statements 9% RR 31%RR Sig. Diff # #
I like to try new and different products 6.0 6.0 N I am willing to pay extra to save time 5.4 5.4 N I lead a fairly busy social life 5.9 6.0 N A person’s career should be their 1st priority 4.9 4.9 N TV is a primary source of entertainment 5.7 5.7 N I have more self-confidence than most people my age 6.9 6.9 N I keep up-to-date with changes in style 5.4 5.3 N I am careful of what I eat 7.2 7.2 N I go out with friends a great deal of the time 5.1 5.2 N To me shopping is a chore rather than a pleasure 6.1 5.9 -0.2 I prefer to postpone a purchase rather than buy on credit 6.6 6.7 N
© 2006 MRIA Response Rate Committee
Conclusions
Previous findings are corroborated – “carefully conducted polls with relatively low response rates still yield representative samples and accurate data”
Important that all other aspects of good survey design also must be present:
The set of telephone numbers is a randomly drawn, representative sample of the universeRespondent selection at HH level is as random as possibleThe data are weighted appropriately
© 2006 MRIA Response Rate Committee
Conclusions (Cont’d)High response rates are still achievable for studies where this is an important design criterion
Fast field turnaround and high response rates are incompatibleAvailable time to complete the fieldwork is the main factorMore focus on the sample management process is required e.g. call scheduling, elapsed time between attempts, etc.
© 2006 MRIA Response Rate Committee
Consider a repeat of this test this fallCan the overall findings be replicated?Are the few data differences found real or merely random data anomalies
Modify the question set somewhatReplace the attitudinal questions with questions related to public policy
Where Next?
© 2006 MRIA Response Rate Committee
Contributors To This Project
Design and AnalysisDon Ambrose - Consumer ContactGary Halpenny - TNS Canadian FactsIvor Thompson - Maritz
Data Merging/TabConsumer Contact
TranslationJL Translation
Census DataSM Research
FieldworkIpsos-ReidMaritzOpinion SearchSynovateTNS Canadian Facts
© 2006 MRIA Response Rate Committee
MRIA Response Rate Committee
Patricia Simmie* CMRP Ipsos Camelford GrahamDon Ambrose CMRP Consumer ContactGary Halpenny CMRP TNS Canadian FactsPeter Chan CMRP TNS Canadian FactsIvor Thompson CMRP Maritz ResearchJohn Smart CMRP SM Research TechnologiesDerrick Gray BBM Canada
*Chair