Questionnaire Design

What is a Questionnaire?

A set of Questions designed to generate the information needed to accomplish the research objectives

Ensures standardization and comparability of the data across interviews – everyone is asked the same questions

Increases speed and accuracy of recording

Facilitates data processing

Allows the researcher to collect the relevant information necessary to address the management decision problem

Purposes of the Questionnaire

Improper design can lead to?

Incomplete information

Inaccurate data

Higher costs

Criteria to consider

Does it provide the necessary information

Does it consider the respondent

Does it meet editing, coding and data processing requirements

Designing the Questionnaire

Determine survey objectives: Plan what to measure.

Decide on format. E.g. personal interview, telephone, self. Formulate questions to obtain the needed information Decide on the wording of questions Decide on the order and layout of the questionnaire Using a sample, test the questionnaire for omissions and

ambiguity Correct the problems (pretest again, if necessary)

Logical Steps to Develop a Good Questionnaire

1. Content - What should be asked?

2. Wording - How should each question be phrased?

3. Sequence - In what order should the questions be presented?

4. Layout - What layout will best serve the research objectives?

The Major Decisions in Questionnaire Design

The most difficult step is specifying exactly what information is to be collected from each respondent

1. Are the questions relevant. Do they pertain to the research problem

2. Are the questions accurate. Do they accurately depict the attitudes, behaviors, etc. intended to investigate

3. Do respondents have the necessary information?

• Qualify respondents

4. Do respondents understand and interpret the question correctly

5. Will respondents give the information?

Content: Questions must meet 5 requirements

Format: How much freedom do we give respondents in answering questions

Open-ended questions– What do you look for most in a job?

– Is there anything else you would like to add about the product

How should the questions be asked?

Open Ended Questions: key advantages

Wide range of responses and information can be obtainedAnswers based on respondent’s not researcher’s frame of reference – consumer’s terms (helpful in marketing campaigns) Lack of influence. Don't channel respondents thinking Can help interpret closed-ended questions - why Particularly useful as introduction to survey or topic When it’s important to measure the salience of an issue When too many possible responses to be listed or unknown

Ability and/or willingness of respondent to answer

Interviewer’s ability to record answers quickly or summarize accurately & probe effectively

Interviewer’s attitude influences response

Time consuming (interview sessions, tabulation, classification, assignment, validation)

Difficulty in coding

Require respondents to be articulate

Respondents may miss important points

Non-response

Open-ended questions: Key disadvantages

What do you look for most in a job?____ Work that pays well

____ Work that gives a sense of accomplishment

____ Work where you make most decisions by yourself

____ Work that is steady with little chance of being laid off.

Ease of understanding Requires less effort on part of interviewer and respondent Ease of tabulation & analysis Less error prone Less interviewer biasLess time consuming Answers directly comparable from respondent to respondent

Closed-ended questions (Fixed-alternative responses)

Advantages

Middle/Neutral categories often selected inappropriately (ignorance, safety) Less opportunity for self-expression or subtle qualifications Less involving for respondents Order of response categories can have major impact on results

Closed-ended questions (Fixed-alternative questions)

Disadvantages

Key tradeoff

Want to get respondent to address issues our research is concerned with (Forced response) and at same time give respondent opportunity to honestly opt out of question (i.e., Don’t Know, No Answer, Neither Agree nor Disagree) so as not to dilute data collected

Dichotomous QuestionsShould the Alberta Government give consumers an energy rebate?

1. Agree

2. Disagree

Advantages

Easy to administer and tabulate

Disadvantages

Prone to large amounts of error since polarized reposnses prevent gaining information on the range of variation

Fail to communicate any intensity of feeling

Multiple Choice QuestionsAre all possible alternatives included?

Too many alternatives

Position Bias

Scaled Response Questions Closed ended questions where the response choices are designed to capture an intensity of feeling

Easy to code and more powerful statistical tools

Main problem: Respondent misunderstanding

Complexity: use simple, direct, conversational language

leading questions -- that suggest or imply certain answers

loaded questions -- suggest social desirability, or are emotionally charged. Have you purchased a high quality Sony TV this year

Ambiguity and vagueness: Words such as “often”, “occasionally”, “usually”, “regularly”, “frequently”, “many”, should be used with caution. If these words have to be used, their meaning should be explained properly.

Which province is bigger Manitoba or Alberta? – would your answer be based on population or area?

Question WordingCan have major impact on how respondent interprets questionAll respondents should interpret in the same way

Things to avoid

long-worded questions

double-barreled items. Questions that refer to two or more issues within the same question. Where respondent may agree with only 1 part of multipart statement.

Do you think Nike offers better pricing and variety than other brands

Making implicit assumptions

Jargon

More things to avoid

More things to avoid

burdensome questions - that may tax the respondent’s memory

How many tubes of toothpaste have you purchased in the last 3 months

Have you purchased toothpaste in the past week

Embarrassing, sensitive, or threatening questions:

Have you charged more on your credit card than you should

Use third person do you think most people…..

Surveys are more than a collection of unambiguous questions

How questions are specified and put together will influence the respondents’ willingness to participate & the responses they provide

Sequence & Layout DecisionsInitial stages Screening or qualifying Questions: Have you been to the cinema this month? Need to gain & maintain respondent’s cooperation Make questionnaire simple for interviewer to administeropening questions should be interesting, simple, and easy to answer.

which theatre did you go to last?

Transition Questions

What aspects of the theatre did you like best

Questions directly related to research objectives which require more effort and get respondent thinking about topic

Difficult and Complicated Questions

The following 10 questions relate to the characteristics of theatres

Respondent now committed and can see an end in sight

Classifying and Demographic Questions

What is your average annual household income

May not be answered (sensitive or threatening) but most questions have been answered

General questions should be asked before more specific ones

Demographic questions should come at the end.

Use multiple questions instead of one

Similar questions together

– consistent mindset for respondents

Develop a logical flow

Use transitions between sections (E.g. In this section we ask

questions about X)

Distinguish between instructions, questions and responses

CAPS or BOLD or Underline versus lower case or

unformatted

Other Considerations

INSTRUCTIONS LAYOUT A:

Do you agree, disagree or have no opinion that this company has:

• A good vacation policy - disagree/not sure/agree.

• Good management feedback - disagree/not sure/agree.

• Good medical insurance - disagree/not sure/agree.

INSTRUCTIONS LAYOUT B:

Does this company have:

___  Disagree Not Sure Agree    

• A good vacation policy   1       2       3   

• Good management feedback   1       2    3   

• Good medical insurance     1      2  3   

Distinguishing question and responses:

–Filter and pivot questions should be used as necessary.

– A FILTER question is one that screens out respondents who are not qualified to answer a second question.

–A PIVOT question is a type of filter question that is used to determine what version of a second question to ask.

If the questionnaire deals with several topics, complete questions on a single topic before moving on to a new topic

If topics are related, ask questions on related topics before asking questions about unrelated topics

If you ask questions about behaviors over some time period, follow chronological order backward in time

When changing topics, use some transitional phrase

Developing a logical flow

The layout and physical attractiveness of a questionnaire are important aspects Questionnaires should be designed to appear as

short as possible Questionnaires should not appear overcrowdedLeave lots of space for open ended questions Questionnaires in booklet form are often

recommended

QUESTIONNAIRE LAYOUT

OPENINGProvide first name (at least)

Provide name of company doing research

Provide reason for survey and topic

State that no selling will be involved and no personal data other than for statistical purposes

Tell respondent approximate time to complete

Reinforce that respondent’s time is appreciated

Invite to participateAND CLOSING

Thank for time

Ask if they had a positive experience and remind them that their opinions count

Pretesting and Correcting ProblemsPurpose of pretest: To ensure that the questionnaire meets the expectations in terms of the information that will be obtainedIs question necessary – does it serve a purpose, will info be usedMissing important variables – does it provide the info neededMatch questions to objectivesPretest Specific Questions For

• Variation• Meaning• Task difficulty• Respondent interest and attention• Ambiguous, ill-defined, loaded, double-barreled questions

Pretest the Questionnaire• Flow of the questionnaire• Skip patterns•Length

Respondent Interest and Attention

Step 1: Specify what information will be sought

Step 2: Determine type of questionnaire and method of administration

Step 3: Determine the content of individual questions

Step 4: Determine form of response for each question

Step 5: Determine wording for each question

Step 6: Determine sequence of questions

Step 7: Determine physical characteristics of questionnaire

Step 8: Revise steps 1-7, revise if necessary

Step 9: Pretest questionnaire, revise if necessary

Step 10: implement

Questionnaire Design Flow Chart

A QUESTIONNAIRE IS ONLY AS GOOD AS THE QUESTIONS

IT ASKS