the do’s and don’ts of survey design
TRANSCRIPT
The Do’s and Don’ts of Survey Design
Tips, tricks, and techniques for making your survey research
more effective
What do surveys measure? Surveys measure characteristics of people
Behaviors Attitudes Beliefs Opinions Demographics Sociographics
What do surveys measure? Surveys measure characteristics of people
BehaviorsActions that are directly observable by others.
What do surveys measure? Surveys measure attributes of people:
BehaviorsActions that are directly observable by others.
DemographicsPhysical characteristics that are directly observable and directly verifiable.
What do surveys measure? Surveys measure attributes of people:
BehaviorsActions that are directly observable by others.
DemographicsPhysical characteristics that are directly observable and directly verifiable.
SociographicsSocial characteristics that are directly observable but not directly verifiable (i.e., requires confirmation).
What do surveys measure? Surveys measure mindsets of people:
Attitudes, Opinions, Beliefs
Surveys measure preferences of people: Wants and needs
Surveys measure constructs of people: Satisfaction, Confidence, Persistence, etc.
Types of survey questions Open response Partially open response Closed response Semantic differential scales Agreement and rating scales Ranking scales Checklists
Open responseWhat was your entry status when you started your first semester at UNF? ____________________________
Partially open response
What was your entry status when you started your first semester at UNF?
1. RECENT HIGH SCHOOL GRADUATE2. TRANSFERRED FROM COMMUNITY
COLLEGE OR OTHER INSTITUTION3. OTHER________________________
ClosedIf you could do it all over again, would you choose to come to UNF?
1. YES, I WOULD CHOOSE UNF2. NO, I WOULD CHOOSE ANOTHER
UNIVERSITY3. NO, I WOULD NOT GO TO
COLLEGE
Semantic differential On a scale of 1 to 5, please indicate the quality level of your instructors by checking one of the boxes below:
11 The Best11 11 11 11 The Worst
Agreement scale The quality of my instructors is excellent.
1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree
Rating scale Overall, how would you rate the quality of your instructors?
1. Very Good2. Good3. Fair4. Poor5. Very Poor
Ranking scale Write the letter of each instructor in the numbered spaces below in order of their relative quality: A. Instructor Jones 1. __________B. Instructor Smith 2. __________C. Instructor Brown 3. __________D. Instructor Lane 4. __________E. Instructor White 5. __________
Checklist In which activities did you participate? (Check all that apply):
STUDENT GOVERNMENT POLITICAL ACTIVITIES INTRAMURAL SPORTS INTERCOLLEGIATE ATHLETICS COMMUNITY SERVICE
Why people answer surveys Theory of Social Exchange Perceived importance Costs and rewards Established trust Anonymity Peer pressure
Common wording problems Bias
Objectionable questions Unequal comparisons Bias in more than one direction Unbalanced categories Emotionally charged words Threats to self-esteem Personalization
Common wording problems Objectionable questions
How old are you? ___________ (WRONG)
Please indicate your age range: (RIGHT)a. 21 and underb. 22-35c. 36-49d. 50 and over
Common wording problems Unequal comparisons
Which of the following poses the greater threat to peace in the Middle East? (WRONG)
a. Palestinian attacks against Israeli citizensb. Israeli attacks against Palestinian militantsc. Both a and b threaten peace
Which of the following must cease for there to be peace in the Middle East? (RIGHT)
a. Palestinian attacks against Israeli citizensb. Israeli attacks against Palestinian militantsc. Both a and b must cease
Common wording problems Bias in more than one direction
Are you against same sex marriage and in favor of a constitutional amendment to ban it? (WRONG)
a. Yesb. Noc. Unsure
What is your view on same sex marriage? (RIGHT)a. I think marriage is a matter of personal choiceb. I’m against it but don’t want a constitutional amendmentc. I want a constitutional amendment banning it
Common wording problems Unbalanced categories
Please indicate your age range: (WRONG)a. 18-20b. 21-30c. 31-50d. 51 and over
Please indicate your age range: (RIGHT)a. 21 and underb. 22-35c. 36-49d. 50 and over
Common wording problems Emotionally charged words
Was the FDC negligent by ignoring the warnings about Vioxx during testing and approving it for sale? (WRONG)
a. Yesb. Noc. Unsure
If the FDC knew that Vioxx caused serious side effects during testing, what should it have done? (RIGHT)
a. Ban it from ever being sold.b. Require more testing before approving itc. Unsure
Common wording problems Threats to self-esteem
How often have you driven drunk? (WRONG)a. Neverb. Oncec. Two or more times
Have you ever driven a vehicle after you have had too much to drink? (RIGHT)
a. Neverb. Yes, on one occasionc. Yes, on more than one occasion
Common wording problems Personalization
Dear Mr. Jones: How would you rate the level of service that you received from Salesman Mr. Smith? (WRONG)
a. Excellentb. Goodc. Faird. Poor
Dear Valued Customer: How would you rate the level of service that you received from your salesperson? (RIGHT)
a. Excellentb. Goodc. Faird. Poor
The problem with scales of agreement
Artificial position statements are made to fit these formats.
“Agree/Disagree” statements require respondents to take a position “For” or “Against” an issue.
“Agreement-disagreement” is not a semantic differential.
There is no clear-cut way to interpret one’s agreement or disagreement with a statement when the proper response choices are unrelated to agreement.Too many items using agreement scales are built upon “face validity” alone. The underlying assumption is that agreeing with an item indicates a person has a specific opinion about that item.
Agree-Disagree is not a semantic differential scale! Agreeing with a statement worded positively may
have a different meaning than disagreeing with the same statement worded negatively.
True semantic differential items:
Hot Cold Weak Strong Very Good Very Poor Frequently NeverVery Helpful Not Helpful at all
1. There should be designated parking areas for students.
SA Strongly agree A Agree N Neutral D DisagreeSD Strongly disagree
What was the student thinking when he chose “Agree?”
“I shouldn’t have to waste time looking for parking spaces!”
parking areas for students.
SA Strongly agree A Agree N Neutral D DisagreeSD Strongly disagree
What was the student thinking when he chose “Disagree?”
not
1. There should be designated
“I should have a place to park just like faculty have!”
You can improve survey items by using a more direct statement of outcome and a more direct scale of measurement.
Here is a common exampletaken from the ISQ survey:
Professor clearly explains complex concepts and ideas.
Strongly Agree Agree Neutral Disagree Strongly Disagree No response
How would you describe your professor’s explanations of complex concepts and ideas?
1. Always clear and concise2. Mostly clear and concise3. Sometimes clear and concise & other times obscure and vague4. Mostly obscurely and vague5. Always obscure and vague
And now….a better way to ask this question:
The problem with scales of satisfaction:
To classify data as "satisfaction information," survey items are often “shoehorned” into a Satisfaction-Dissatisfaction format. These items use the same scale format for expediency sake only, and not because the scale fits the meaning of the statement.
Satisfaction surveys are good at identifying general problem areas, but do not lend themselves to specific solutions.
All of the statements are typically couched in positive terms thus increasing the likelihood of a positive response bias.
In a vague attempt to not be overly positive, many items overuse the words “Adequate” and “Reasonable.”
Keys to making your surveys better and more useful
Use a scale covering the whole range of responses.
Use a scale or response list that matches the statement
Identify a potential problem first, and then use survey data to define its scope and suggest possible solutions.
Use a survey only for its intended purpose.
Don’t be afraid of criticism. Make your items balanced.
Refine your surveys each time you use them.
Don’t forget to include free response input.
The Do’s and Don’ts of Survey Design
The End