Developing Questionnaires and Questionnaire Items

Keeping it Simple, Clear and Organized

Objectives

• Reviewing the implications of error– Random sampling error– Measurement error

• Questionnaire items that introduce error• Organizing the questionnaire to improve

administration of the survey• Review of survey modes and types of

questions

Error Components in Parameters

22

22

222

σ

σ

μ

iaserror or b

systematic

rrorsampling e

random s

errors

errorx

Addressing Error

• Random Sampling Error:– Increase sample size– Stratified sampling

• Systematic Error– Composite measures– Interviewer training– Questionnaires that improve response rates and

minimize respondent confusion.

Measurement of Meeting Expectations

• Previous form:Exceeding expectationsMeeting expectationsAlmost meeting expectationsFailing to meet expectations

• Alternative:Meeting expectationsFailing to meet expectations

What would be the number of respondents from a branch necessary to determine within +/-10% (at a 95% confidence level) if a branch was significantly different on its proportion of respondents “stating” it was exceeding expectations?

2

2

e

qpzn

Measurement of Meeting Expectations

What would happen if choices were increased?

Far exceeding expectationsExceeding expectationsJust coming short of exceeding expectationsBarely meeting expectationsFalling just short of meeting expectationsFailing to meet expectationsNo apparent attempt made to ever meet

expectations

Example from 5-point scale:• What is the 95% confidence interval for the population mean if the sample mean is 1.47, the sample standard deviation is 1.08, and the sample size was 36.

• Error or Acceptable error is +/-.35

82.1 ,12.1

35.047.1or )3608.1(96.147.1

95

nszx

Coding of Responses to Items and Standard Deviation

• Service at my FCS branch is…Getting better Getting poorer

A) 1 2 3

B) 1 2 3 4

C) 1 2 3 4 5 6 7

Increasing Sensitivity

• By increasing the possible number of responses, you may increase the variability in responses and standard deviation.

• Gain “sensitivity,” • But increase sample size requirements.

Questionnaire Design

• Determine the most important constructs or variables to be measured.

• What survey mode is to be used (telephone, mall interview, self-administered)?

• Respondent issues: Sensitivity, need to disguise questions, interest in the subject matter.

“Backward” Influence

• What type of analysis does the client understand?– Determines the coding of variables.

• Coding affects the nature of the question, and time required for respondent.– Determines position in the questionnaire.

• Position in questionnaire affects participation.

Functions of Well-Planned Questionnaires

Translate research objectives into specific questions

Standardized questions and response categories

Foster cooperation and keep respondents motivated

Serve as permanent records of the research

Speed up the process of data analysis

Contain information for reliability assessment

Questionnaire Items

• Open-ended questions pose some problem and ask the respondent to answer in his or her own words.

• Closed-ended give the respondent specific, limited alternative responses, and ask respondent to choose one closest to his or her viewpoint.– Simple dichotomy– Determinant choice (multiple choice)– Frequency determination (how often do you...)– Checklist, allows respondent to check multiple responses.

Likert Questionnaire Items

• Two components:– Strongly worded statement, “My experience at

this hotel was one of my best ever.”– Scaled response, ranging from “strongly agree” to

“strongly disagree.”

• Easy to create• Easy for the respondent

Disguised Items: Materialism• “I admire people who own expensive homes, cars,

and clothes.• “Some of the most important achievements in life

include acquiring material possessions.• “I don’t place much emphasis on the amount of

material objects people own as a sign of success.*• The things I own say a lot about how well I’m doing in

life.• I don’t pay much attention to the material objects

other people own.** Reverse coded

The Five “Shoulds” of Question WordingThe Question Should...

Be focused on a single issue or topicBe brief, unambiguousBe interpreted the same way by all respondentsUse the respondents’ core vocabularyBe a grammatically simple sentence (if possible)

Leading Questions

• Suggest or imply certain answers.• "Many people are using dry cleaning less because of

improved wash-and-wear clothes. How do you feel wash-and-wear clothes have affected your use of dry cleaning facilities in the past four years?”

• “Drunken drivers continue to claim thousands of motorists’ lives each year. In the last year, has your drinking in restaurants increased or decreased?

Loaded Questions

• Suggests socially desirable answers or is emotionally charged.

• "In light of today's crisis with education in our public schools, how do you feel about the Republican Party's plan to destroy The Department of Education?

• “We are happy when you like programs on Channel 7. We are sad when you dislike programs on Channel 7. Write us and let us know what you think of our programming.”

Double-Barreled Question

• May induce bias (confusion) because it covers two issues at once.

• "Please indicate your agreement with the following statement: 'Wholesalers and retailers are responsible for the high cost of meat'.“

• “Chuck E. Cheese’s service and food quality has been some of the best our family has experienced.”

The “Should Nots” of Question Wording:All Increase Measurement Error

1. Assume criteria that are not obvious.“Which grocery store has best specials?”

2. Be beyond the respondent’s ability or experience.Asking teenagers questions on household purchasing.

3. Use a specific example to represent a general case.Including an example with a question on a broader issue.

4. Ask the respondent to recall specifics when only generalities will be remembered

Asking respondents to recall a specific price paid for a product.

Additional “Should-Nots”

5. Requiring the respondent to guess a generalization.The question is a generalization, usually gets an affirmative

response.

6. Ask for details that cannot be related“How many brands did you consider prior to buying…?”

7. Use words that overstate the condition“Do you believe a tuition increase will prevent the exodus of faculty

from the University…”

8. Have ambiguous wording“On your last visit to the Northwest Arkansas Mall…

Cover Letters and Questionnaire Introductions

Identify the Surveyor/Sponsor of the Survey

Describe the Survey’s Purpose

Indicate How Prospective Respondents were Selected

Request the Prospective Respondent’s Participation in the

Survey/Provide incentive

Screen the Prospective Respondent

Typical Question Sequence

Screening Questions

Warm-ups, open-ended, easy to answer

Most important variables

Transitions (statements & skips)

Complicated, sensitive & difficult-to-answer

questions

Classification & demographic questions

Screening Questions

• “Are you the person that does the majority of the grocery shopping in your home?”

• “May I speak to the person who watched television last night?”

Difficult or a Warm-Up?

• “At what supermarkets do you regularly shop?• “List the supermarkets where you regularly

shop.• “I’m going to read a list of supermarkets in the

area, please identify if it is a store where you regularly shop.

Organization Approaches

Funnel Approach—A wide to narrow approach, from general to specific questions.

Work Approach—An easy-to-difficult approach, begin with brief sections and move to sections which take more time.

Sections Approach—Keep the respondent on the same subject within a section.

Importance—Move from most important sections to the least important.

Questionnaires are a result of a combination of approaches.

Sections

• The questionnaire should be organized in sections.

• Each section should contain related questions.

• Use the same type of questionnaire items within a section.

Funnel Approach

• Applies to the organization within sections of the survey.

• Begin with a general question, “warm-up” respondent.

• Finish with a more specific question.

“Work” Approach

• Applies to organizations of sections within the questionnaire.

• Begin with shorter sections

• Finish with longer, more tedious sections

Importance

• Not all sections of the survey are of equal importance to the survey’s objectives.

• Do not “bury” the more important sections.

• If sensitive items are important, do not precede them with sections that are not critical to the survey’s objectives.

Sensitive Items

• Response rate decline due to respondent distrust and concern over confidentiality.

• Income• Age• Education

• What is your household’s annual income to the nearest thousand?– Too direct on a highly sensitive item.

• What percent of your household’s income is spent at grocery stores?– Combined with shopping information, could provide a

measure of income, but assumes respondent understands percents, and can calculate this readily (?)

• Is your household income above or below $50,000?– Easy to answer, and can be followed up with “Above

$100,000; Above 90,000; Above 70,000”

Layout of the Printed Survey Form

• Paper and photocopying costs, in light of other survey costs, are small.

• Don't crowd items on a page to make the survey appear short -- the opposite effect happens with the respondent.

• Keep ample empty spaces in the questionnaire booklet, it help both the interviewer and the respondent in a self-administered survey.

• Use vertical answer formats for individual questions.

• Number questions to facilitate the interviewer.

Pre-testing the Interview/Questionnaire

• Small subgroup of representative respondents.

• Identifies wording of questions and responses that may create problems.

• Improves interpretation of questions, reduces respondent source of error

Pre-testing the Interview/Questionnaire

• Can the questionnaire format be followed by the interviewer?– Does the questionnaire flow naturally and

conversationally?• Are the questions clear and easy to understand--can

the respondent answer the questions clearly?• Which alternative forms of questions work best?• Can the questionnaire be coded for easy data entry?