questionnaire design van bennekom, chapter 4. identifying the questions to ask what are concerns and...

28
Questionnaire Design Van Bennekom, Chapter 4

Upload: aurora-blizard

Post on 16-Dec-2015

219 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Questionnaire Design

Van Bennekom, Chapter 4

Page 2: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask• What are concerns and interests from your

– Upstream shareholders• E.g., monitor service quality to avoid future problems• Different managers might have different issues

– Immediate recipients• The direct target, truly reflect their concerns (otherwise, low response rate or

wrong answers)• Personal interview: interview representatives • Focus groups: small group interviews• Related literatures, documentations, white papers, reports, etc.

– Downstream customers• Might be the indirect cause, their influence to the immediate recipients

– Follow your statement of purpose with clear definition of scopes and goals.

Page 3: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Personal Interviews– Defining a set of interview questions that helps

elicit the survey purpose– Semi-structured questionnaire (creating your

questions during the interview)– Questionnaire: add more questions, alter the

sequences of questions, change wording,

Page 4: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Personal Interviews– How to take the notes

• Using tape recorder (please check the legal issues)• Start with the statement of the interview• The best notes are quotes from the interviewee.

– How many interviewers should we interview?• 1 to many

– Who should conduct the interviews• Someone with the interview training• Person who will design the questionnaire, conduct the

survey

Page 5: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Personal Interviews– How should the interview be sampled

• Purposive sample (most representative, hard and critical

– Where should these interviews be conducted• Convenient for interviewees• Face-to-face is the best

– How long should these interviews last?• 30min-one hour

– When should we stop• When you stop hearing new things

Page 6: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Personal interview– Pros• Deep in context• Explore unexpected paths• Paints the initial picture

– Cons• How structured a questionnaire?• How many interviewers• By phone or in person• length

Page 7: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Focus group– To have a directed discussion with a small group of

people– The dynamic interaction with groups can bring

some issues or ideas that would have been missed or ignored during personal interview.

– Time efficient– How many people should be in a focus group• 7-15

Page 8: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Focus group– Who should be invited

• Purposive sample, attendee covers different fields

– Who should not be invited• Limit the number of observers

– What is the role of the moderator?• Help to bring out all sides• Encourage members to discuss/present their ideas• Control an overbearing opinion leader

– What type of questionnaire should be used?• Same as personal interview: semi-structured

Page 9: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Focus group– Where should the focus group be held and how should

the discussion be captured.• A comfortable meeting room• Audio taping

– How many focus groups should be conducted?• Until you did not hear new information.

– How long should it last• Two hours, morning is better than afternoon, evening is also

good• Serving a light snack is always good.

Page 10: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Identifying the questions to ask

• Focus group– Pros

• Deep in context• Explore unexpected paths• Interaction among participants

– Cons• Planning & execution critical• Good moderator critical• Have one central theme• How many to invite• Whom to invite• Time consuming textual analysis

Page 11: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Documentary Data

• Existing documents• White papers• Related literature

Page 12: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Analyzing data

• Doing content analysis of the textual notes– Boiling down the words into categories, supplemented

with a few well-chosen statements– Identify patterns: group similar comments to reveal

patterns– Some free-form comments can address the “why” for a

question– Literal transcript of every word is not necessary, only

capture key points and key focuses. – Connect words to interviewee’s background– A distillation process

Page 13: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Drafting the questionnaire• It is an iterative process: • Step 1: Organizing your essential findings from the interviews, group them

into logic groups• Step 2: Consider three basic types of questions (e.g., demographic, specific,

overall)• Step 3: Select question formats: scale type (limit the number of different

scale types), open-end • Step 4: Draft some questions• Step 5: Take a break• Step 6: Repeat steps 3-5.• Step 7: seek other input (look for external reviewers to go through your

questionnaire, ask domain experts• Step 8: Pilot test your draft questionnaire with a representative sample

(critical stage)

Page 14: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Elements of a questionnaire

• A questionnaire contains several components– An introduction– Instructions– Initial questions– Sections and their headings– Summary– A thank you

Page 15: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Elements of a questionnaire

• Introduction– Personally addressed to the respondent– To explain the purpose of the survey – Who should complete the survey– The expected time to complete the survey– Emphasize anonymity, if appropriate– What to do with the completed questionnaire– When to do the survey (now!)– Definition of certain terminology

– Example: Van Bennekom, p75

Page 16: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Elements of a questionnaire

• Instructions– How to enter answers– Explain the scales

– Example: Van Bennekom, p78

Page 17: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Elements of a questionnaire

• Initial questions– Pay attention to the initial questions (engage

them, subject matter, motivation to move ahead, easy to answer)

– Never open with demographic questions in the beginning

Page 18: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Elements of a questionnaire

• Section headings– Should be short– Group 50 questions into 3 section vs. put all 50

questions in one section.

Page 19: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Elements of a questionnaire

• Summary and Thank you

Page 20: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Sequencing questions

• From general to specific– Initial questions should be general

• Demographic questions– Do not ask questions which data you do not need

• Question interaction– Rating a person, rather than a system– Connection of different questions

• Rotating questions

• Branching: direct respondents to the right sections

Page 21: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Question formats

• Unstructured– Open-ended, free-form– Limited the number of open-ended questions

(respondent burden)– Post them after certain structured questions about

key theme, or ask them to clarify or expand on previous answers

Page 22: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Question formats

• Structured (see page 85)– multiple choice or categorical, – ordinal scales, – interval-rating scales, – ratio scales

Page 23: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Question formats

• Structured: multiple choice or categorical, – For frequency distribution (e.g., job title, company size)– Kinds:

• Multiple choice, multiple response• Multiple choice, single response• Binary choice• Adjective checklist (page 90)

– Design issues:• Range of choices in response sets (include other category)• Distinctions among choices• Number of choices (max 6-8)• Sequencing of choices

Page 24: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Question formats

• Structured: Ordinal scales (page 93)– Basic ordinal scale: age range, or salary range– Forced-ranking scale: ask respondent to rank some

items– Paired comparison– Design issues• Number of items• Clarify of instructions on what the rank means

Page 25: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Question formats• Structured: Interval-rating scales (page 95)

– Such as: level of agreement– Verbal scales: scale in text– Numeric scales: scale in number– Likert-type scale: strongly disagree-disagree-neither agree nor disagree-agree-strongly agree– Many others (page 98-100)– Design issues

• Advantages of grouping by type• Which step first, developing scale or writing questions?• Importance of anchor choice• Scale direction (1 is best or 5 is best?)• Interval equality• Anchor balancing (extremely dissatisfied, vs. very satisfied)• Even (1-7) vs. odd (1-6)-numbered scales• Number of points on the scale: 10 points are too much.• Presenting the scale low to high or high to low• Use of multiple scale types creates confusion• Avoiding response ruts• Dispersion of results

Page 26: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Question formats

• Structured: Ratio scales (page 108)– Fractionation scale: open-ended scale, no upper

limit, has better precision.

Page 27: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

General guidelines

• Goal– Maximizing the information we receive– Minimizing response burden

• Do not ask meaningless questions• Keep the instrument focused• A survey is not a test (instruction should be clear)• Be concerned about the visual and layout of your

questionnaire• Keep wording and grammar simple• Be creative in question design

Page 28: Questionnaire Design Van Bennekom, Chapter 4. Identifying the questions to ask What are concerns and interests from your – Upstream shareholders E.g.,

Pilot testing

• Don’s skip it• It is not complicated• It should be conducted as one-on-one or personal

interview on purposive sampled targets

• The purposes are– To know whether questions are well designed, difficult to

answer, or have proper response choices– Engage the respondent?, too long? Clarity of instructions,

find awkward wording