implementation (fielding) strategiesfaculty.nps.edu/rdfricke/survey_short_course_docs... · source:...
TRANSCRIPT
1
Implementation (Fielding) Strategies!
Professor Ron Fricker!Naval Postgraduate School!
Monterey, California!
3/26/13
Reading:!Dillman et al. chapter 7!
Goals for this Lecture!
• Learn about proven strategies for improving survey response rates !– Based on Don A. Dillman’s “Tailored Design
Method”!– Focus will be on mail and web surveys, but
principles generalize to other modes!• Basic principle: repeated (appropriate)
contact with respondents will improve response rates!– Without, expect a decrease of 20-40 percent!!
3/26/13 2
Improving Survey Response Rates: Multiple Contacts Important!
• Good response rates not just a function of survey instrument and question design!
• Implementation procedures have a much greater influence on response rates!
• Experimental research shows that multiple contacts with respondents is the primary factor in improving response rates!– Pre-survey letter(s), survey cover letter, follow-up
reminders, thank you cards, etc.!
3/26/13 3
“A system of five compatible contacts…”!
1. A brief pre-notice letter !– All electronic: Use e-mail!
2. A questionnaire mailing that includes a cover letter!– All electronic: E-mail with URL link!
3. A thank you postcard !– All electronic: Use e-mail!
4. A replacement questionnaire sent to nonrespondents 2-4 weeks after mailing!
– All electronic: Repeated requests with URL more rapidly!– Almost all electronic: Follow up with paper option!
5. A final contact by a different mode of delivery 2-4 weeks after previous mailing!
– Almost all electronic: Switch modes, don’t wait 2-4 weeks!3/26/13 4
Improving Survey Response Rates: Token Incentives Can Help !
• Second most important factor is the use of token (financial) incentives!– Only needs to be a token, such as include a two
dollar bill in a mail survey!– Can also be a token gift, such as a pen or button!
• Research shows that sending such tokens with no strings attached gives recipients a sense of obligation to complete the survey!
• Unfortunately, financial incentives generally not allowed in federal government surveys!
3/26/13 5
Incentive Effects!
3/26/13 6
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Incentive Effects!
3/26/13 7
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Improving Survey Response Rates: Other Techniques!
• Other techniques have a more modest effect!– Professional appearance of correspondence and
survey instrument!– Personalization of correspondence!– Introduction letter from person of authority!– Stamped return envelopes!– Etc.!
• Should not focus on any one technique!– Implement as many as relevant / appropriate to
maximize response rates!
3/26/13 8
Mail Surveys!
• Mail may seem “old school” but is often the only (affordable) option for general populations!– E-mail/web often only applies to organizations that
maintain e-mail address listings!• Still tends to achieve better response rates
than using electronic modes!– Access partly explains this, but also people are
simply less likely to respond to e-mail !– Partly due to spam; also because electronic
modes can look less “serious”!
3/26/13 9
Dillman’s “Five Needed Elements for Achieving High Response Rates”!
1. Respondent friendly questionnaire!2. Four contacts by first class mail, with an
additional “special” contact!3. Return envelopes with real first class stamps!4. Personalization of correspondence!5. Token prepaid financial incentive!
Ø Clearly, these are for a mail surveys!– Basic principles still apply to other modes!– Can be more challenging to translate to /
implement in other modes!3/26/13 10
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2004.
“Four contacts by first class mail…”!
• A brief pre-notice letter sent a few days prior to the survey!
• The survey with a cover letter!• A thank you postcard sent a few days to a week after
the questionnaire!– Both expresses appreciation for responding as well as a
request that, if not, will do so soon!• A replacement survey to nonrespondents 2-4 weeks
after original survey!• A final contact – by telephone, FedEx, priority US
mail, etc. – 2-4 weeks after previous mailing!
3/26/13 11
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2004.
Sample Pre-notice Letter!
• Sent a few days prior to the survey!– Not too far in advance or recipients will forget /
not connect it to survey itself!• Should specifically note:!
– Will receive an important survey in a couple of days!
– Participation / response is greatly appreciated!• Better if sent by a person of authority using
recognizable letterhead and/or logo!– If so, note in letter that the survey will be from
______ organization on behalf of the person!
3/26/13 12
13
Sample Pre-notice Letter!
3/26/13
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Survey Cover Letter!
• Introduction should indicate:!– Who is conducting the survey and why!– (Short) explanation of why survey is important!– Topics to be covered in the survey!
• Administrative matters!– State whether responses are confidential or not!– Any IRB stipulations!– Perhaps indicate how long the survey should take!– Keep it to one page in length if at all possible!
• Again, if possible, put on letterhead and have signed by someone considered an authority by respondents!– Dillman calls them “trust-inducing elements”!
3/26/13 14
3/26/13
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Sample Survey Cover Letter
15
16
Another Sample Survey Cover Letter!
3/26/13
Anonymous Survey Return Cards!
• Generally, paper surveys contain a code so that we can know who has responded!
• For surveys about very sensitive topics, where anonymity is crucial, include a postcard to be mailed back separately!– See Dillman for
paragraph to include in cover letter!
3/26/13 17
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Postcard Thank You / Reminder!
• Generally, people either respond immediately to a survey or not at all (to the initial mailing)!
• Postcard designed to jog memories of those who intended but forgot to do the survey!– It’s not for converting those that chose not to do it!– Postcard format deliberately supports this purpose!
• Sent to all potential respondents and arrives shortly after the original survey!– Re-conveys a sense of importance for the survey!– Has contact information for getting a replacement!
3/26/13 18
Postcard Thank You / Reminder!
3/26/13 19
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Cover Letter with First Replacement Questionnaire!
• Fielding strategy changes here with a focus on only nonrespondents!
• Send out once survey returns start to tail off!• Cover letter has a tone of insistence!
– First paragraph in particular specifically states that the person’s survey has not yet been received!
– It’s an attempt to convert those who seem to have decided not to respond!
• Must be careful to neither under- or over-sell!• For mail surveys, always include a
replacement questionnaire!3/26/13 20
21
Cover Letter with First Replacement Questionnaire!
3/26/13
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Final Contact Letter (Priority Mail)!
• Continued focus on nonrespondents with an increased insistence!– Increased insistence achieved not by tone of
letter, but by repeated try and priority mailing!– Since previous approaches failed, change the
stimulus with new priority packaging!• Tone of letter about the same as previous!
– Focus on importance of survey and the possibility that these individuals have something important to contribute!
3/26/13 22
23
Final Contact Letter (Priority Mail)!
3/26/13
Source: Mail and Internet Surveys: The Tailored Design Method, 2nd edition, by Don A. Dillman et al., 2004.
Web Surveys!
• Many of the same guidelines apply:!– Personalize contacts whenever possible!– If possible, send a token of appreciation!
• Same as mail if using a mail contact mode!• More difficult if using all-electronic approach!
– Use multiple contacts and vary the message!• Remember, if only using web response mode,
be sure all respondents have access!– If using e-mail for contact, must have a complete
list of addresses for the population !
3/26/13 24
If Using E-mail as Contact Mode!
• Keep e-mail contacts short an to the point!– I.e., follow the norms of e-mail!
• Short is more appropirate than long!• Don’t use all capitals, etc.!
• Carefully select the subject line!– This is the only thing that potential respondents
see unless they open the e-mail!• Carefully and strategically time all contacts
with the population!– E-mail interaction much faster than mail…!
3/26/13 25
3/26/13 26
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
3/26/13 27
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
3/26/13 28
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Other Important Guidelines!
• Establish a procedure for dealing with bounced e-mails!
• Consider contacting respondents by another mode when necessary!– Similarly, consider the necessity of having an
alternate response mode!• Make sure e-mails will not be flagged as
spam!• Make obvious connections between the
opening screen and the contact mode!
3/26/13 29
Additional Guidelines for Large Web Surveys!
• Provide clear instructions for how to access the survey!– Particularly important for less web savvy
populations!• Establish and staff a hotline to respond to
inquiries!• Know and respect the limitations of web
servers!
3/26/13 30
A Bit on Interviewer-Assisted Modes!
• Multiple contacts approach still critical!– If possible, conduct pre-notification by some other
mode (e.g., mail) !– Respondents more likely to cooperate with
interviewers if the first contact is not a “cold call”!• Convincing people to take survey now rests
with interviewers, not prepared materials!– Good interviewers and good training critical!– Should also have one or more individuals
dedicated to converting nonrespondents!
3/26/13 31
A Bit on Interviewer-Assisted Modes!
• Instead of focusing on supporting materials for convincing potential respondents to take self-administered survey, must focus on interviewer training and logistics:!– How to conduct an appropriate interview!– How to handle and convert skeptics!
• To include some supporting materials!– How to overcome barriers to the interview
(voicemail, gated communities, etc.)!– How to conduct nonrespondent follow-ups!
3/26/13 32
Mixed-mode Approaches: An Example!
3/26/13 33
Source: Mail and Internet Surveys: The Tailored Design Method, 3rd edition, by Don A. Dillman et al., 2009.
Summary: Doing Fielding Well Takes Attention to Detail and Flexibility!
• Attention to detail:!– Require complete sampling frame with necessary
contact information (addresses, e-mail addresses)!• If personalizing materials, must have complete
and accurate names (and ranks)!– Must ensure materials are written well and are
appropriate for respondents!– Must track respondents and non-respondents!
• Flexibility: Must be prepared to lengthen or curtail fielding period, adjust (e-)mailing dates of materials, etc.!
3/26/13 34 34
My Take: Achieving High Response Rates Means Getting It All Right!
• Potential respondents pick up on small details!– You’re asking them to spend valuable time – you have to
make it look like it’s worth their time!– For me, all it takes is one thing being “off” and I may judge
survey not worth my time !
• With sensitive information, you’re also asking respondents to trust you to with their data!– Why should they trust you if they see sloppiness in the
survey or the fielding materials???!• During fielding, flexibility important!
– Have to be able to adjust as the fielding unfolds!– Have back-up plan(s)!
3/26/13 35 35
Remember: There are Lots of Ways One Can Fail to Complete a Survey !
3/26/13 36 36
Receive!
Open!
Start!
Finish!
Return!Kulka, Holt, Carter, and Dowd (1991)!
Breakdowns in!Implementation:!
What We Have Covered!
• Learned a number of proven strategies for improving survey response rates !– Based on Don A. Dillman’s “Tailored Design
Method”!– Focused on mail and web surveys, but principles
generalize to other modes!• Basic principle: repeated (appropriate)
contact with respondents will improve response rates!– Without, expect a decrease of 20-40 percent!!
37 3/26/13