QUESTIONNAIRE DESIGN

&EXPERIMENTATION

BYAMIT KUMARRAGUNANTHAN

2

Face to face interview

• Advantages– Direct interaction– Clarity and display of exhibits– Better quality and quantity of data– Higher response rate– No sequence bias– Identifying respondents– Unstructured

• Disadvantages– High cost– Longer time– Interviewer bias– Anonymity not maintained– Interviewer cheating– Time bias exists– Field control needed

3

Person Administered Surveys II

2. Indirect, non- face-to-face Interview:• The interviewer and the interviewee do not see but

talk direct to each other.– Telephone Interview

4

Telephone Interview

• Advantages– Faster Results– Inexpensive– Better geographical

coverage– Irresistibility– Reaching hard-to-reach

people– Timing: early or late OK– Privacy and better control– Coincidental data:

immediate feedback.

• Disadvantages– No exhibits– Long interview not

possible– Inability to make judgment– Answering machines and

caller identification device– Sampling problem

• Obsolete directory: poor sampling frame

5

Self-administered surveys

Mail

• Advantages– Wide geographical coverage– Providing thoughtful answers– Ability to ask sensitive questions– No interviewer bias– Inexpensive– Better control– Anonymity– Clarity

Survey• Disadvantages

– Mailing list problem– Unidentifiable respondent– Questionnaire exposure– Data limitation– No interviewer assistance

• no exhibits– Assumed literacy– Poor response rate– Longer time

Survey through Internet has similar advantages and problems

6

A questionnaire [also called research instrument]

• Data collection instrument used for gathering data;

• A formalized schedule of an assembly of a carefully formulated questions;

• Six important functions– Converts research objectives

into specific questions– Standardizes the questions– Keeps respondents motivated

to complete the research– Serve as a permanent record– Speed-up the process of data

analysis– Reliability and validity

purposes

7

Questionnaire Development Process

Steps in a Questionnaire Development Process

Pre-design activities

Design activities

Post-design activities

DetermineSurvey

Objectives

Decide DataCollectionMethods

QuestionDevelopment

QuestionEvaluation

by Researcherand by Client

Pretest theQuestionna

ire

Revise,Finalize,

andDuplicate

Gather DataUsing theQuestion-

naire

Tabulate andAnalyze Dataand Finalize

Report

8

Basic Question Formats

Various Question Formats

BasicOpen-ended

Questions

ProbingQuestions

ClarifyingQuestions

Open-endedQuestions

DichotomousQuestions

Single-coded

Multiple-coded

Multiple-ChoiceResponses

Closed-endedQuestions

LabeledQuestions

UnlabeledQuestions

ScaledQuestions

BasicQuestion Formats

9

Basic Open Ended Questions.

Q. What do you particularly like about Lipton Tea?______________________________________________________________________________

Q. Why are you unwilling to buy a cellular phone when it is available in the market? ________________________________________________________________________

Probing Questions

Q. What do you particularly like about Johnson & Johnson baby oil? __________________________________________________________________________

PROBE: Anything else?

OPEN-ENDED QUESTIONS: Questions to which respondents give their responses freely, according to their own will.

What should you ask?

• The questions asked are a function of previous decisions

• The questions asked are a function of future decisions (such as statistical analysis)

Key criteria

• Questionnaire relevancy– No unnecessary information is collected and only

information needed to solve the problem is obtained. Be specific about your data needs; tie each question to an objective

• Questionnaire accuracy– Information is both reliable and valid

Phrasing Questions

• Open ended response versus fixed alternative questions

“?”• Decision criteria: type of research; time;

method of delivery; budget; concerns regarding researcher bias

Avoid

• Leading questions• Overly complex questions • Use of jargon• Loaded questions (can use a counterbiasing

statement)• Ambiguity• Double barreled questions• Making assumptions

Order?

• Order bias results from an alternative answer’s position in a set of answers or from the sequencing of questions– Funneling technique: general to specific helps

understand the frame of reference first• Anchoring effect: the first concept measured

tends to become a comparison point from which subsequent evaluations are made

• Ranking, sorting, rating or choice?• How many categories or response positions?• Balanced or unbalanced?• Forced choice or nonforced choice?• Single measure or index?

Decisions

16

Advantages and disadvantages of Open-ended Questions

Advantages· Since they do not restrict the

respondent’s response, the widest scope of response can be attained.

· Most appropriate where the range of possible responses is broad, or cannot be predetermined.

· Less subject to interviewer bias. • Responses may often be used as

direct quotes to bring realism and life to the written report.

Disadvantages· Inappropriate for self-administered

questionnaire since people tend to write more briefly than they speak.

· The interviewer may only record a summary of the responses given by an interview and fail to capture the the interviewer’s own ideas.

· It is difficult to categorize and summarize the diverse responses of different respondents.

· May annoy a respondent and prompt him/her to terminate the interview, or ignore the mail questionnaire.

17

Dichotomous Questions With No Neutral Response

_________________________________________________Q. Do you have a cellular phone?

Yes ...................................... 1 without neutral No ...................................... 2 response

Dichotomous Questions With Neutral Response

Q. Is it likely that you will purchase a cellular phone in the nextsix months?

Yes ...................................... 1 with neutral No ...................................... 2 response

Not Sure .............................. 3

CLOSED-ENDED QUESTIONS: Questions to which respondents are required to answer from set of alternative responses provided by the researcher. Could be dichotomous or multiple choice.

18

Single- and multi-coded multiple choice questions

_____________________________________________________________________Q. On an average, how much do you spend on newspapers, books and magazines in a

month? (Please check one from the following responses.)

Less than $15 ................................... 1 Single-coded Between $16 & $30 ......................…. 2 question Between $31 & $45 .......................… 3 Between $46 & $60 .....................….. 4 $60 or more .................................….. 5

Q. Which of the following household appliances does your household have?(Please check as many responses that are applicable to you.)

TV …………1 VCR …… … 5LCD ………….2 Microwave ….…… 6 Multi-codedPC …………3 Cellular phone ……….7 questionFax …………4 Others ……….8

Specify ________________________________________________________________________________________________________________

19

Advantages and Disadvantages of Closed-ended Questions

ADVANTAGES· All respondents reply on a

standard response set. This ensures comparability of responses, facilitates coding, tabulating and interpreting the data.

· Easier to administer and most suited for self-administered questionnaire.

· If used in interviews, less skilled interviewer may be engaged to do the job.

DISADVANTAGES· Preparing the list of

responses is time-consuming.

· If the list of responses is long, the respondents may be confused.

· If the list of responses is not comprehensive, responses may often fail to represent the respondent’s point of views.

20

Scaling questions

Scaling questions are special types of closed-ended questions. They include, among others, the following categories of questions.

Behavioral/Attitudinal questions Buying-intent questions Agree-Disagree questions Preference questions Ranking questions Semantic differential questions Constant-sum questions

The questions can be labeled or unlabeled

21

Labeled and Unlabeled Scaling Questions

Type ofScaled

Questions

Examples Advantages Disadvantages

Unlabeledscaled-responsequestion

“On a scale of 1 to 7,how would you ratethe IBM Thinkpad onease of operation?

1. Allows a respondent toexpress the degree ofhis/her intensity offeelings.2. Easy to administer andcode.

Respondentsmay not relate tothe scale well.

Labeledscaled-responsequestion

“Do you disagreestrongly, disagree,agree, or agree stronglywith the statement,‘IBM laptops are abetter value thanCompaq laptops’?”

1. Allows a respondent toexpress the degree ofhis/her intensity offeelings.2. Easy to administer andcode.3. Respondents can relateto the scale.

Scale may be“forced” oroverly detailed.

22

Considerations in choosing a question format

• Nature of the property being measured– Subjective Vs objective

• Previous research studies– Need for comparison with past studies

• Data Collection Mode– Telephone/face-to-face-interview/mail

• Scale level desired– Statistical analysis

• Ability of the respondents

23

Phrasing and Sequencing of Questions

PHRASING• Focus on a single issue or

topic• Ask precise questions

using respondent’s core vocabulary

• Avoid – use of vague words– asking leading or loaded

questions– estimation questions– double barreled questions– presumptuous questions

SEQUENCING • Start with simple opening

questions• Place

– broad-based questions first; – more specific and narrow

questions and difficult, sensitive, embarrassing questions should come later(Funnel approach)

– Classification questions last.• Transition from one topic to

another should be smooth

24

Questionnaire Layout

• Provide sufficient spaces• Use prominent print for instructions• Use filtered questions• Do not slit the same question over two

pages• Number the questions• Layout should facilitate editing and coding

25

Comparative Evaluation of Various survey Methods.

________________________________________________________________________ Face-to-face Interview

Criteria In-home/ Mall- Telephone MailIn-office intercept Interview Survey

Flexibility of data collection High High Moderate LowDiversity of questions High High Low ModerateSample control Potentially Moderate Moderate to Low

high highControl of data collection Moderate to High Moderate Low environment highResponse rate High High Moderate LowShow of exhibits Yes Yes No NoQuantity of data High Moderate Low HighAnonymity of the respondents Absent Absent Moderate HighAccess to sensitive information Low Low High HighSpeed Moderate Fast Fast SlowCost High Moderate Moderate LowPotential interviewer bias High Moderate to Moderate Low

Copyright 2004 David J. Lilja 26

Design of Experiments

• Goals• Terminology• Full factorial designs

– m-factor ANOVA

• Fractional factorial designs• Multi-factorial designs

Copyright 2004 David J. Lilja 27

General Procedure

• Determine upper/lower bounds for parameters• Simulate configurations to find response• Compute effects of each parameter for each

configuration• Rank the parameters for each benchmark based on

effects• Average the ranks across benchmarks• Focus on top-ranked parameters for subsequent analysis