d esigning s urveys september 18, 2012 1:00 – 2:15 pm

DESIGNING SURVEYSSeptember 18, 2012 1:00 – 2:15 pm

WORKSHOP OBJECTIVES By the end of this session you will:

Know the major strengths and weaknesses of alternative methods for measuring key public health variables using survey questionnaires

Be able to write clear, specific survey questionsBe able to independently design a

questionnaire for a small-scale survey

2

PREPARING FOR A SURVEY Defining the survey purpose

Identifying survey recipients

Defining key concepts

Designing the survey instrument

Putting it all together (structure)

3

WHY USE A SURVEY? Target large amount of people Use to describe, compare or explain Can cover activities and behavior, knowledge,

attitudes, preferences Specific objectives, standardized and highly

structured questions Used to collect quantitative data – information

that can be counted or measured

4

STRENGTHS

Reach respondents in widely dispersed locations Can be relatively low cost in time and money Relatively easy to get information from people quickly Standardized questions Analysis can be straight-forward and responses pre-

coded Low pressure for respondents

5

LIMITATIONS

Low response rate and consequent bias and confidence in results

Seeks information only by asking, can we trust what people say? e.g. issues with over-reporting

Question wording can have major effect on answers

BIASES THAT MUST BE UNDERSTOOD

The selection of questions reflect the biases and perspectives of the researchersThe questions being posed often include

inherent biases Answers to survey questions reflect the

perspective of the respondentMultiple respondents may answer differently

The selected sample may not necessarily reflect the general populationDifficulty in getting “accurate” responses from

different populations

6

DEFINING THE SURVEY PURPOSE

7

Think about the overall purpose for

your questionnaire

and what data you

want to get out of it

DEV

ELO

P T

HE

PU

RP

OS

E

Write a couple of sentences about the purpose of the survey- what is the general thing you want to find out?

Write down ideas about the kinds of questions you could ask

Brainstorm the data you could collect related to the purpose you have chosen

Write down the users for the questionnaire

Group your ideas into three or four key areas to investigate and write down what these are

Why will this be important to the person who wants the questionnaire written?

8

THINGS TO THINK ABOUT: Kind of information: What do you want to

know? Is the information already available? What do you NEED to know vs. “would like to

know” Who will complete the questionnaire - when?

(think about readability, length, interest in the topic, etc)

9

IDENTIFYING SURVEY RECIPIENTS

10

WHO DO YOU WANT TO FIND OUT INFORMATION ABOUT?This is the target population.

WHO WILL YOU GIVE THE QUESTIONNAIRE TO?

This is called the sample.

SAMPLING

Population Good Sample

Poor Sample

12

COMBINE WITH PREVIOUS

WHAT IS A RESPONSE RATE?The proportion of people who respond:

Example: If you distribute 50 questionnaires and you get 25 questionnaires back, your response rate is 50%.

# that answered = response rate# you contacted

13

High response rate promotes confidence in results.

Lower response rate increases the likelihood of biased results.

WHICH TYPE OF SURVEY SHOULD I CHOOSE?

14

5 ALTERNATIVE INTERVIEW METHODS

Interview type Advantages Disadvantages

Personal interview

•Stronger commitment to completing the interview.•Greater confidence in finding the right recipient•Longer interviews tolerated

•More costly•Greater potential for creating a non-representative sample

Telephone surveys

•Quicker turn-around •Wide availability of telephone numbers•Availability of CATI to address complex questions•Role skilled interviewers to elicit more complete answers

• Negative perception of surveyors as “telemarketers”•Challenges of getting past “gatekeepers” (e.g., administrative assistants, caller ID)

Mail surveys •Relative expense per response•Ability to use only names and address to reach respondent•Can use photos•Answer at respondent’s leisure•Viewed as less intrusive

•Survey takes longer•Response rates lower•Respondents are more likely to have “something to say” •Less ability to correct “misunderstanding” of questions

15

5 ALTERNATIVE INTERVIEW METHODS(CONTINUED)

Interview type Advantages Disadvantages

Email surveys •Inexpensive with quick turnaround•Simple•Allows attachments

•Cannot handle complex logic•Requires having email addresses•Perceptions about viruses and spam•Potential missed messages due to “junk” mail filters•Difficult to manage sample due to ease of forwarding•Loss of anonymity•Response bias/inability to generalize

Web surveys •High speed in getting responses•Low cost•Use of email to generate responses•Flexibility in creating complex logic•More responsive to ‘sensitive’ questions•Greater ability to use open-ended questions•Respondent inputs data

•Increase survey “quitting”•Restricting response to the “sample” •Even greater difficulty in generalizing results

16

WHICH TYPE OF SURVEY SHOULD I CHOOSE?

“It depends”… uponWhat you want to

know – how complex or sensitive the information is

Who the respondents are – their characteristics and which type of survey will be most appropriate

Your timelineAvailable resources 17

WILL USING A SURVEY BE CULTURALLY APPROPRIATE?

Literacy level Tradition of reading, writing Setting Not best choice for people with oral tradition Translation (demands more than just literal

translation) How cultural traits affect response How to sequence the questions Pretesting the questionnaire when it may be

viewed as intrusive Computer access and use if an electronic

survey

18

IDENTIFYING CHARACTERISTICS OF THE RESPONDENTS IS IMPORTANT BECAUSE:

Their educational backgrounds determine the most effective wording

Their ages and writing abilities will affect the type of question you use

Their potential interest in the topic affects what you need to do to ensure they respond

Physical and ability levels may affect how the questionnaire is administered

19

DEFINING KEY CONCEPTS20

DEFINE THE VARIABLES TO BE COLLECTED Write a detailed list of

the information to be collected and the concepts to be measured in the study. Are you trying to identify:

AttitudesNeedsBehaviorDemographicsSome combination

of these concepts Translate these

concepts into variables that can be measured.

Define the role of each variable in the statistical analysis:PredictorConfounderOutcome

21

DESIGNING A SURVEY INSTRUMENTCreating and aggregating information about a large number of respondents

22

STUDY PROTOCOL AND QUESTIONNAIRE

Definition of objectives

List of variables to collect

Analysis plan

Study protocol

Design questions

Control and reading

Pilot test

Questionnaire

questions must relate to objectives

collect appropriate information for analyses

23

QUESTIONNAIRE DESIGN

Design individual questions:

Adopt questions used in other questionnaires Adapt questions used in other questionnaires

(search the literature) Develop your own questions (may want to use

focus groups or key informant interviews)

24

Good questionnaires are difficult to construct; bad questionnaires are difficult to analyze.

REVIEW THE LITERATURE

Review current literature to identify related surveys and data collection instruments that have measured concepts similar to those related to your study’s aims.

Saves development time and allows for comparison with other studies if used appropriately.

Proceed with caution if using only a subset of an existing questionnaire as this may change the meaning of the scores. Contact the authors of the questionnaire to determine if a smaller version of the instrument exists that has also been validated.

25

QUESTIONNAIRE NEEDS TO BE ADAPTED TO STUDY POPULATION

Know the respondents languageeducationoccupation

ethnic groupsensitive issues

26

5 MINUTE BREAK

QUESTIONS

1. Type of information2. Type of questions3. Types of response4. Common wording problems5. Problems with response6. Validity7. Reliability

28

TYPE OF INFORMATION

Knowledge - what people know

29

What is the recommended interval between eye checks for patients with uncomplicated diabetes?

o 6 monthso 1 yearo 2 yearso Not sure

TYPE OF INFORMATION

Knowledge - what people know Opinions, attitudes, beliefs, values - what

people think about an issue

30

What do you think are the major issues affecting general practice physicians at the moment?

______________________________________________________________________________________________________

TYPE OF INFORMATION


people think about an issue Behavior - what people do

31

Have you developed a care plan for any of your patients?

o Yeso No

TYPE OF INFORMATION


people think about an issue Behavior - what people do Attributes – who people are (characteristics)

32

When did you graduate from college? _______

QUESTIONS


33

34

OPEN AND CLOSED QUESTIONS (FROM OPPENHEIM, 1992)

Strength Limitation

OPEN Freedom & spontaneity of answer

Time-consuming

Opportunity to probe Coding more problematic

Useful for testing hypothesis about ideas or awareness

More effort from respondents

CLOSED Requires little time Loss of spontaneous responses

No extended writing Bias in answer categories

Low costs Sometimes too crude

Easy to process May irritate respondents

Make group comparisons easy

Useful for testing specific hypothesis

CLOSED-ENDED QUESTIONS

Typically used in quantitative studies. Assumption is researcher has knowledge to

pre-specify response categories. Data can be pre-coded and therefore in a

form amenable for use with statistical packages (e.g., SPSS, SAS) – data capture therefore easier.

More difficult to design but simplifies analysis.

Used in studies involving large samples. Limited range of response options.

35

BRANCHING QUESTIONS

Are used to direct respondents to answer the right questions as well as questions in the proper sequence

‘Skips’ in telephone questionnaires more easily managed if these are computer-assisted

Consider including ‘not applicable’ category:In the past week, how often have you used MEDLINE:o Not at allo At least onceo More than onceo I do not have access to MEDLINE 36

SCREENING OR FILTER QUESTIONS

Are used to ensure respondents in the study are those that meet the pre-determined criteria of the target population.

“Tonight we are talking with individuals who are 18 years of age or older and have 50 percent or more of the responsibility for healthcare decisions in your household. Are you that person?”

__ Yes __ No

37

QUESTIONS


38

RESPONSE – WORDING THE ANSWER

Wording the response is as important as clear wording in the question. Make the answer options clear, logical, comparable and mutually exclusive.

39

RESPONSES Closed ended questions are usually followed

by a set of responses Limited choices of responses, lack of

consistencyin what a yes/no, agree/disagree response

meansDo you have trouble climbing stairs?

Attitudes and behaviors lie on a continuumTo what extent do you experience difficulty

whenclimbing stairs in your house?o Noneo A littleo Quite a bito A loto I do not have stairs in my house

40

RESPONSES

Choose type of scale:Nominal

41

Are you:o Maleo Female

What is your marital status?o Singleo Marriedo Widowedo Divorcedo Separatedo Other, specify __________

RESPONSES

Choose type of scale:NominalOrdinal

42

What is your current stage of cancer?o Localizedo Regionalo Metastatic

What is the highest level of education you have reached?

o Did not complete high school

o High school diploma/GED

o Some collegeo Bachelor’s degreeo Post-graduate

education

RESPONSES

o NumericalHow severe was your pain after you were stung?

(please circle) 1 2 3 4 5 6 7

Not painful at all Very painful

o AnalogueHow severe is your pain (put the tick on the line)

0 1043

Choose type of scale:NominalOrdinalContinuous

HOW MANY TIMES HAVE YOU PARTICIPATED IN THE HEALTH FAIR?

0−11−33−67−12More than 12 times

0 times (never)1−3 times4−6 times 7- 9 times10 or more times

Poor spacing and logic

Better

Vertical order of answer choice options

44

EXHAUSTIVE AND MUTUALLY EXCLUSIVE RESPONSE OPTIONS

From which one of these sources did you first learn about the health clinic?

o A friendo A neighboro A relativeo An advertisemento At school

What is your age ? o 16-20o 20-25o 25-30o 35-40o Over 40?

45

• Mutually exclusive: In the above example, I might have first learned about the health clinic through a friend who is also a neighbor – which would I check? Also, the choices include both sources and locations.

• Exhaustive: make sure the options include all possible, important answers. Use ‘other’ fields.

INCLUDE BOTH POSITIVE AND NEGATIVE SIDES IN THE QUESTION STEM AND ALL POSSIBLE ANSWER OPTIONS

Poor: Do you agree that our planning committee needs to meet once per month?

o Agreeo Disagree

Better: Do you agree or disagree that our planning committee needs to meet once per month?

o Agreeo Disagreeo No opinion

Another option: How often do you think our planning committee should meet?

o More than once a montho Once a montho Less than once a month

46

47

RATING SCALES

FIVE TYPES OF ORDINAL RATING SCALES

EndorsementDefinitely true, true, don’t know, false,

definitely false Frequency

Always, very often, fairly often, sometimes, almost never, never

IntensityNone, very mild, mild, moderate, severe

InfluenceBig problem, moderate problem, small problem,

very small problem, no problem Comparison

Much more than others, somewhat more than others, about the same as others, somewhat less than others, much less than others 48

PROVIDE BALANCED SCALES WITH CATEGORIES THAT ARE EVENLY SPACED CONCEPTUALLY

Overall, how would you rate your experience as a nurse?o Excellento Very goodo Goodo Fairo Poor

Overall, how would you rate your experience as a nurse?o Very good o Goodo Fairo Pooro Very poor

Poor Better

49

Balance the scale with an equal number of positive and negative options.

MAKE SURE THE RESPONSE SCALE MATCHES THE QUESTION

To what extent do you think the health leadership project helped you develop your leadership skills?o Excellento Very goodo Goodo Fairo Poor

To what extent do you think the health leadership project helped you develop your leadership skills?o Very great extent o Great extent o Some extent o Little extento Very little extent

Poor Better

50

KEEP THE SCALE CONSISTENT

o Not worth my timeo Very little interesto Somewhat

interestedo Very interested

o Not at all interestedo Very little interesto Somewhat

interestedo Very interested

Poor Better

51

The left column includes two concepts – “worth” and “interest level.”

ALWAYS INCLUDE LABELS FOR YOUR NUMBERED SCALE CATEGORIES TO AVOID MISINTERPRETATION

Incomplete labels:To what extent have you increased your triage skills as a result of this program? A great deal Not at all

1 2 3 4

Fully labeled: To what extent have you increased your triage skills as a result of this program?

A great deal Somewhat Not much Not at all

1 2 3 4

52

LIKERT SCALE

Rate agreement with a series of statements.

To what extent do you agree or disagree with each of the following statements…

o Strongly Agreeo Agreeo Neithero Disagreeo Strongly disagree

53

Which of the following issues are barriers to Tibetans’ overall tuberculosis (TB) care?

Language Immigration Misconceptions about TB Lack of insurance Patient’s work schedule

CHECKLIST

54

Which of the following issues are barriers to Tibetans’ overall tuberculosis (TB) care?

Language Yes No Don´t know

Immigration Yes No Don´t know

Misconceptions about TB Yes No Don´t know

Lack of insurance Yes No Don´t know

Patient’s work schedule Yes No Don´t know

QUESTIONS


55

QUESTION WORDING1. Clarity/specificity

“Have you seen a doctor during the past year?”

“In the last 12 months, have you visited a

general practitioner?” Yes No

“How long has it been since you last visited a general practitioner?

o within the last month,o between 1 and 12 months agoo more than 12 months ago

56

2. SimplicityAvoid complex or technical phrasing.Use complete but short sentences (avoiding

double negatives and redundancy between question stem and response options)

Avoid “deceptively short” questions with high respondent burden (complex tables, rank ordering, mental calculations, “check all that apply”)

QUESTION WORDING

57

QUESTION WORDING

3. NeutralityAvoid loaded questions, arguable prefatory

statements, and judgmental words/phrases“Do you prefer being examined by a doctor of your own sex?”

Would you rather be examined by a:

o Male doctoro Female doctor o Either/doesn’t matter

58

QUESTION WORDING4. Sensitivity

Encourage ‘unorthodox’ and “socially undesirable" responses (depersonalization):

“Everybody does it…”“How many times…”“Many experts say…”

Discourage over reporting of ‘socially desirable’ responses“Did you happen to…”

“Many people have trouble remembering…”

Allow for either agreement or disagreement in question stem

Funneling questions to introduce sensitive topics

59

QUESTION WORDING5. One topic per question

Avoid ‘double-barreled’ questions“Do African-Americans and Latinos suffer from discrimination”

6. Specify an appropriate time frame

Typical/usual versus maximal/minimal Depends on salience of topic

7. Improving recall Aided recall (memory cues, prompts) Bounded recall (time window) Records or diaries

60

QUESTION WORDING

8. Consider question polarity and sequence (survey as conversation)

Avoid ‘yea-saying’ patterns and carryover effects, but maintain consistency when possible

9. Check for technical accuracy

61

QUESTION ORDER Decide on the order of items/questions

easy difficultgeneral particularfactual abstractmost salient least salient

Questions should be ordered so as to seem logical to the respondent

Group questions

Be aware of ordering effects/position bias

Don’t put the most important items last

62

QUESTION ORDER

To avoid wasting time, qualify early To boost response rate, put innocuous

questions first, personal questions last Start with questions that are easy to answer

and relevant to the stated purpose of the survey

To boost response rate, put demographic questions last

To increase accuracy, keep similar questions together

63

QUESTION WORDING – THINGS TO AVOID

Abbreviations Alternative meanings Negatives, words with negative prefixes Inappropriate categories Asking same question twice Jargon Leading questions Typographical errors Threatening questions Being condescending

64

THE GOAL OF WRITING A GOOD QUESTION…

is to develop a question that every potential respondent will:

1. interpret in the same way,

2. be able to respond to accurately, and

3. be willing to answer.

65

QUESTIONS

1. Type of information2. Type of questions3. Common wording problems4. Types of response5. Problems with response6. Validity7. Reliability

66

PROBLEMS WITH RESPONSE Fatigue/boredom/disinterest

agree with everything just say ‘don’t know’always choose first response ‘randomly’ respond without considering the

question (Christmas treeing) Social desirability

instruct that it is ok not to know something Aversion to extreme ends of the scale Ceiling effects Halo effects Framing effects

67

PROBLEMS WITH RESPONSE

Minimizing fatigue/boredomKeep task simple

e.g. easier to recall more recent eventsKeep words short and easy to understand

Maintain motivation of participantsEnsure task is relevant

Ask people to justify their responses Position Bias

May be more likely to endorse first or last response option

Preceding questions may influence responses to questions that follow

68

AVOID POSITION BIAS

“How important are flexible hours in evaluating job alternatives?”

“What factors are important in evaluating job alternatives?”

“What factors are important in evaluating job alternatives?”

“How important are flexible hours in evaluating job alternatives?”

Position Bias No Position Bias

69

QUESTIONS


70

VALIDITY

Does the question measure what you claim it measures?

Problems with self-reportPeople don’t know the causes of their own

behaviorPeople’s memories are inaccuratePeople are not good at predicting their

future behaviorPeople are not telling you what they know

because ofSocial desirability biasDemand characteristicsResponse sets

71

QUESTIONS

1. Type of information2. Type of questions3. Common wording problems4. Types of response5. Problems with response6. Validity7. Reliability

72

RELIABILITY

Results are reproducible or consistent with similar groups of respondents, over time and when other people administer the questionnaire

Questions measure consistentlyTest-retest Inter-rater Internal consistency

73

FOR EACH QUESTION GIVEN, IDENTIFY WHICH TYPE OF QUESTION IT IS, IDENTIFY ANY ISSUES WITH THE QUESTION, AND EXPLAIN HOW YOU MIGHT IMPROVE THE QUESTION AND WHY.

Identify issues with questions and discuss improvements

Question One

Is there normally an adult at home during the day?’

Identify which type of question it is

Identify any issues with the question

Explain how you might improve the question and why.

75

Question Two

How much money would you be prepared to spend on a burglar alarm?’

$500 $1000 $1500

LO: Identify issues with questions and discuss improvements




76

Question Three

How much television did you watch last year?

LO: Identify issues with questions and discuss improvements




77

Question Four

Do you agree that trains and buses and taxis are kept clean enough?




78

Question Five

Do you think nurses are:

very kind fairly kind not very kind?




79

PUTTING IT ALL TOGETHERStructure of a questionnaire

STRUCTURE OF A QUESTIONNAIRE 1. Identification2. Introduction3. Instructions on how to answer4. Questions5. Conclusion

81

ANONYMITY AND CONFIDENTIALITY Anonymous means that NO ONE can identify who

provided the information Difficult to assure if there is a need to follow-up

with non-respondents or when the survey is administered online (internet or intranet)- Don’t promise anonymity!

Confidentiality means that you are able to identify the person, but you guarantee that the information will not be identified with the person All aspects of data collection, analysis and reporting When reporting and communicating, ensure that no

names or other identifying information is used

82

ALWAYS PILOT TEST THE QUESTIONNAIRE

By experts- your colleagues, others who are knowledgeable about writing questions or your program

By potential respondents- or people as similar to respondents as possibleDo they understand the questions? The

instructions?Do questions mean the same thing to all?Do questions elicit the information you

want?How long does it take?

83

SUMMARY

A well designed survey instrument: collects appropriate data to answer your research

question minimizes potential sources of bias increases the validity of the replies will much more likely be completed

84

CONTACT INFORMATION

Kelleigh Trepanier

KSU Center (West Entrance)Burruss.kennesaw.edu

[email protected]

85

THANK YOU

RESOURCES

www.aapor.org“Design and use of questionnaires: a review of best

practice applicable to surveys of health service staff and patients”, Health Technology Assessment, 2001. Vol.5, No. 31.

De Vaus, D.A. (1990) Surveys in Social Research. Allen and Unwin, London.

Dillman, D., Smyth, J., Christian, LM. 2009. Internet, Mail and Mixed-Mode Surveys: The tailored design method. John Wiley and Sons.

Fink, A. (Ed.)(1995) The Survey Kit. Sage, London.Foddy W (1993). Constructing questions for

interviews and questionnaires: Theory and practice in social research. Cambridge University Press, Melbourne.

Fowler, Floyd J. (2002) Survey Research Methods. Sage, London.

87

RESOURCES

Hoinville, G and Jowell, R. (1982) Survey Research Practice. Heinemann, London.

Moser, C. and Kalton, G. (2001) Survey Methods in Social Investigation. Ashgate, Aldershot.

Oppenheim AN (1992). Questionnaire design, interviewing and attitude measurement. Pinter Publishers, London

Scheuren, F. What is a survey. American Statistical Association. Free download at http://www.whatisasurvey.info/

Schuman H, Presser S (1996). Questions and answers in attitude surveys: experiments on question form, wording, and context. Sage Publications, San Diego.

Streiner DL, Norman GF (1995). Health Measurement Scales: a practical guide to their development and use, Oxford University Press, Oxford, 1995.

88

LARGE SCALE SURVEYS Converse, Philip E., Jean D. Dotson, Wendy J.

Hoag, and William H. McGee III (eds.), American Social Attitudes Data Sourcebook 1947-1978 (Cambridge, MA: Harvard University Press, 1980).

Gallup, Alec, George Gallup, The Gallup Poll Cumulative Index: Public Opinion1935-1997 (Wilmington, DE: Scholarly Resources, Inc., 1999).

Hastings, Philip K. and Jessie C. Southwick (eds.), Survey Data for Trend Analysis: An Index to Repeated Questions in U.S. National Surveys Held by the Roper Public Opinion Research Center (Roper Public Opinion Research Center, 1974).

89

Martin, Elizabeth, Diana McDuffee, and Stanley Presser, Sourcebook of Harris National Surveys: Repeated Questions 1963-1976 (Chapel Hill: Institute for Research in Social Science, University of North Carolina Press, 1981).

Miller, Warren E., Arthur H. Miller, and Edward J. Schneider, American National Election Studies Data Sourcebook 1951-1978 (Cambridge, MA: Harvard University Press, 1980).

National Opinion Research Center, General Social Surveys 1972-2008: Cumulative Code Book (Chicago: NORC, 2009).

Robinson, John P., Robert Athanasiou, and Kendra B. Head, Measures of Occupational Attitudes and Occupational Characteristics (Ann Arbor, MI: Institute for Social Research, 1969).

90

Robinson, John P., Phillip R. Shaver, and Lawrence S. Wrightsman, Measures of Political Attitudes (San Diego, CA: Academic Press, 1999).

Robinson, John P., and Phillip R. Shaver, Measures of Social-Psychological Attitudes (Ann Arbor, MI: Institute for Social Research, 1980, rev. ed.).

Converse, Jean M., and Stanley Presser, Survey Questions: Handcrafting the Standardized Questionnaire (Thousand Oaks, CA Sage Publications, 1986).

Inter-university Consortium for Political and Social Research (ICPSR) www.icpsr.umich.edu

91

TOP TEN STRATEGIES TO MAXIMIZERESPONSE RATES

1. More interesting better than less interesting2. Certified or priority delivery better than regular mail3. Incentive (especially money) better than no

incentive4. Short better than long5. Incentive with questionnaire better than on return6. Pre-contact better than no pre-contact7. Colored ink better than standard ink8. User friendly better than standard questionnaire9. Follow-up better than no follow up10. Request explanation for not completing better than

no requestEdwards, Roberts, Clarke et al BMJ 2002; 324: 1183

92

d esigning s urveys september 18, 2012 1:00 – 2:15 pm

Documents