QUESTIONNAIRES

© LOUIS COHEN, LAWRENCE MANION, AND KEITH MORRISON

STRUCTURE OF THE CHAPTER• Ethical issues• Approaching the planning of a questionnaire• Types of questionnaire items• Asking sensitive questions• Avoiding pitfalls in question writing• Sequencing questions• Questionnaires containing few verbal items• The layout of the questionnaire• Covering letters/sheets and follow-up letters• Piloting the questionnaire• Practical considerations in questionnaire design• Administering questionnaires• Processing questionnaire data

ETHICAL ISSUES• Intrusion• Informed consent• Rights to withdraw at any stage or not to

complete particular items• Beneficence• Non-maleficence• Confidentiality, anonymity and non-traceability • Threat or sensitivity • Avoidance of bias • Validity and reliability in the questionnaire• Reactions of the respondent

APPROACHING THE PLANNING OF A QUESTIONNAIRE

• Stage One: Decide the purposes/objectives/ research questions;

• Stage Two: Decide the population and sample• Stage Three: Itemize the topics/constructs/

concepts;• Stage Four: Decide the kinds of measures or

responses needed;• Stage Five: Write the questionnaire items;• Stage Six: Check that each research question

has been covered;• Stage Seven: Pilot the questionnaire and refine;• Stage Eight: Administer the questionnaire.

OPERATIONALIZING A QUESTIONNAIRE

• Clarify the questionnaire’s general purposes and then translate them into a specific, concrete aim or set of aims.

• Identify and itemize subsidiary topics that relate to its central purpose.

• Formulate specific information requirements relating to each issue.

• Plan with the data analysis in mind.

STRUCTURED, SEMI-STRUCTURED AND UNSTRUCTURED QUESTIONNAIRES

• The larger the size of the sample, the more structured, closed and numerical the questionnaire may have to be.

• The smaller the size of the sample, the less structured, more open and word-based the questionnaire may be.

• Structured questions take a lot of time to set up but then a short time to process and analyze.

• Open questions take a shorter time to set up but a longer time to process and analyze.

TYPES OF QUESTION

• Open to closed• Choose the metric (scale of data):

– Nominal– Ordinal– Interval– Ratio

• Do not assume that respondents have the information/knowledge/views

TYPES OF QUESTION

DICHOTOMOUS

MULTIPLE CHOICE

RATING SCALES

OPEN-ENDED

YES/NO

ONE/MANY RESPONSES

ODD/EVEN NUMBERS

FREE RESPONSE

TYPES OF QUESTION

RANKING

RATIO DATA

CONSTANT SUM

RATIO DATA

1st/2nd. ETC.

MANY RESPONSES

DISTRIBUTING MARKS

MARKS OUT OF TEN

DICHOTOMOUS QUESTIONS

• Good for clear answers;• Yes/no questions are often better rephrased as

‘to what extent’ or ‘how much’ types of question.

MULTIPLE CHOICE

• Need for a pilot to gather exhaustive categories of response;

• Do not allow for range of response;• If more than one response permitted then each

choice is a separate variable.

LIKERT SCALES• Useful for measuring degrees of intensity of feeling;• No assumption of equal intervals;• No assumptions of matched intensity of feeling;• No way of knowing if respondents are telling the truth;• No way of knowing if there should be other categories

or items;• Halo effect;• Allows for different scaling and mid-points, e.g.:

(a) strongly disagree – neither agree nor disagree – strong agree;(b) not at all – a very great deal;

• Central tendency;• Ordinal data.

SEMANTIC DIFFERENTIAL SCALES

• A word and its semantic opposite, e.g.:– Approachable . . . unapproachable– Generous . . . Mean– Friendly . . . hostile

• Same concerns as for Likert scales.

OPEN-ENDED QUESTIONS

• Enable authentic responses;• More time-consuming and difficult to

analyze/process.

RANKING SCALES

• Enables comparisons to be made by respondents across items;

• Enables sensitivity of response to be addressed;• Can be ‘strong on reality’ of decision making;• Too many items to rank may result in unrealistic

ranking (people may not have strong enough opinions to be able to rank)

• Too many decisions to be made;• Ordinal data.

RATIO DATA: MANY RESPONSES

• Avoids forcing responses into categories;• Allows for very great accuracy (e.g. ‘how old are

you?’)• Ratio data: mean, standard deviation, median

CONSTANT SUM

• Divide a fixed number of points between a range of items;

• Yields priorities, comparative highs and lows and equality of choice quickly and easily – in the respondents’ own terms;

• Requires participants to make comparative judgements and choices across items;

• May be too difficult if there are too many items across which to spread marks;

• People may make computational errors in distributing marks;

• Ordinal data.

RATIO DATA: MARKS OUT OF TEN

• Enables proportions/ratios to be calculated;• Enables high level statistics to be computed,

e.g. regression, factor analysis, structural equation modelling.

ASKING SENSITIVE QUESTIONS • Use open rather than closed questions about

socially undesirable behaviour.• Use long rather than short questions about socially

undesirable behaviour.• Use familiar words when asking about socially

undesirable behaviour.• Use data from informants. • Deliberately load questions so that overstatements

of socially desirable behaviour and understatements of socially undesirable behaviour are reduced.

• With regard to socially undesirable behaviour, firstly ask whether the respondent has engaged in that behaviour previously, and then move to asking about current behaviour.

ASKING SENSITIVE QUESTIONS • Locate sensitive topics within a discussion of other

less sensitive matters.

• Use alternative ways of asking standard questions.

• Ask respondents to keep diaries.

• At the end of an interview ask respondents their views on the sensitivity of the topics that have been discussed.

• Find ways of validating the data.

• As questions become more threatening and sensitive, expect greater bias and unreliability.

‘RULES’ FOR QUESTIONNAIRE DESIGN (1)

• Ensure that each issue is explored in more than one question;

• Decide the most appropriate kind of question and the kind of scale;

• Plan with the kind of analysis in mind;• Avoid leading questions: ‘Do you prefer abstract,

academic-type courses, or down-to-earth, practical courses that have some benefit in day-to-day work?’

• Ensure that the question stem does not frame the answer: ‘The tourism industry is successful because . . . .’.

‘RULES’ FOR QUESTIONNAIRE DESIGN (2)

• Avoid highbrow questions: ‘What particular aspects of the current positivistic/interpretive debate would you like to see reflected in the course of developmental psychology?’

• Avoid negatives and double negatives: ‘How far do you agree that without a Consumer Association the public cannot discuss consumer matters?’

• Avoid complex questions: ‘Would you prefer a short award-bearing course with part-day release and one evening per week attendance, or a longer, non-award-bearing course with full-day release, or the whole day designed on part-day release without evening attendance?’

• Avoid too many open-ended questions on self-completion questionnaires

‘RULES’ FOR QUESTIONNAIRE DESIGN (3)

• Try to convert dichotomous questions into rating scales: ‘Do you. . .’ / ‘Are you. . .’ become ‘How far . . .?’/ ‘How much . . .?’

• Provide anchor statements for rating scales;• Have a minimum five-point rating scale if you

opt for uneven numbers;• Decide whether to have odd or even

numbered rating scales;• Avoid extremes in rating scales ( e.g. ‘always’,

‘never’)• Avoid too many open-ended questions in

large surveys;

‘RULES’ FOR QUESTIONNAIRE DESIGN (4)

• Only ask one thing at a time: Avoid: ‘how much do you think Business courses should be available to all students, or should they only be available to higher ability students?’

• Avoid ambiguity: ‘Are your parents in employment?’ ‘Have you done your homework this week?’ Do you do your homework regularly?’ ‘How many staff are there in your institution?’ How many computers does your institution have?’

• Avoid threatening/irritating questions: ‘How frequently do you drink alcohol each week?’

‘RULES’ FOR QUESTIONNAIRE DESIGN (5)

• Clarify the kind of response sought in an open-ended question: ‘Please indicate the most important factors that reduce staff participation in decision making.’

• Start with simple factual questions and then move to more sensitive questions;

• Provide instructions for how to complete and return the questionnaire;

• Have a cover sheet which explains the purposes of the questionnaire and which sets out ethical issues.

‘RULES’ FOR QUESTIONNAIRE DESIGN (6)

• Ensure categories are mutually exclusive and comprehensive (covering all possible choices of response);

• Avoid pressurising/biasing by association: e.g. ‘do you agree with your school principal that boys are more troublesome than girls?’

• Have translations verified/use back translation;

• Keep it as short, simple and clear as possible;• Pilot the questionnaire.

SEQUENCING QUESTIONS

• Take care with order effects – Earlier responses affect later responses– Early tone/mood-setting affects later moods in

completing questionnaires• Primacy effect

– Items high in a list tend to be chosen more than items lower in a list

– Respondents choose the first reasonable answer from a list, even though a later response statement might be more fitting

• Avoid placing sensitive questions at the start– Embed them later questions

• Move from objective facts to subjective views

QUESTIONNAIRES CONTAINING FEW VERBAL ITEMS

• A questionnaire might:– include visual information and ask participants

to respond to this (e.g. pictures, cartoons, diagrams)

– might include some projective visual techniques (e.g. draw a picture or diagram)

– join two related pictures with a line, write the words or what someone is saying or thinking in a ‘bubble’ picture

THE LAYOUT OF THE QUESTIONNAIRE • It must look easy, attractive and interesting. • Keep it as uncomplicated as possible. • Clarity of wording. • Simplicity of design. • Simple, short and clear instructions for completion.• Avoid placing instructions at the bottom of a page.• Break down long lists of numbered items into separate

sections, each item in the section starting with the number ‘1’.• Make it clear if respondents are exempted from completing

certain questions or sections (filter), and where they go next if they are exempted.

• Include a preliminary statement of anonymity/confidentiality.• Place response categories to the immediate right of the text.

THE COVERING LETTER SHOULD . . .

• Provide a title to the research;• Introduce the researcher and contact details;• Indicate the purposes of the research;• Indicate the importance and benefits of the

research;• indicate why the respondent has been selected for

receipt of the questionnaire;• Indicate any professional backing, endorsement,

or sponsorship of, or permission for, the research;• Set out how, where and by what date to return the

questionnaire;• Indicate what to do if questions or uncertainties

arise;

THE COVERING LETTER SHOULD . . .

• Indicate any incentives for completing the questionnaire;

• Provide assurances of confidentiality, anonymity and non-traceability;

• Indicate how the results will and will not be disseminated, and to whom;

• Thank respondents in advance for their co-operation.

• To check clarity of items/layout/sections/presentation/ instructions;

• To gain feedback on appearance;• To eliminate ambiguities/uncertainty/poor wording;• To check readability;• To gain feedback on question type (suitability/feasibility/

format (e.g. open/closed/multiple choice);• To gain feedback on appropriateness of question stems;• To generate categories for responses in multiple choices;• To generate items for further exploration/discussion;• To gain feedback on response categories;• To gain feedback on length/timing (when to conduct the

data collection as well as how long each takes to complete (e.g. each interview/questionnaire))/coverage/ease of completion;

PURPOSES OF PILOTING

• To identify redundant items/questions (those with little discriminability);

• To identify irrelevant questions;• To identify non‑responses;• To identify how motivating/non‑engaging/threatening/

intrusive/offensive items may be;• To identify sensitive topics and problems in conducting

interviews;• To test for inter‑rater reliability;• To minimise counter‑transference;• To gain feedback on leading questions;• To identify items which are too easy/difficult/complex/

remote from experience;• To identify commonly misunderstood or non‑completed

items.

PURPOSES OF PILOTING

PRACTICAL CONSIDERATIONS IN QUESTIONNAIRE DESIGN

• Operationalize the purposes of the questionnaire.• Be prepared to have a pre-pilot to generate items for a

pilot questionnaire, and then be ready to modify the pilot questionnaire for the final version.

• If the pilot includes many items, and the intention is to reduce the number of items through statistical analysis or feedback, then be prepared to have a second round of piloting, after the first pilot has been modified.

• Decide on the most appropriate type of question – dichotomous, multiple choice, rank orderings, rating scales, constant sum, ratio, closed, open.

• Ensure that every issue has been explored exhaustively and comprehensively; decide on the content and explore it in depth and breadth.

• Use several items to measure a specific attribute, concept or issue.

PRACTICAL CONSIDERATIONS IN QUESTIONNAIRE DESIGN

• Ask more closed than open questions for ease of analysis (particularly in a large sample).

• Balance comprehensiveness and exhaustive coverage of issues with the demotivating factor of having respondents complete several pages of a questionnaire.

• Ask only one thing at a time in a question. Use single sentences per item wherever possible.

• Keep response categories simple.• Avoid jargon.• Keep statements in the present tense wherever possible.• Strive to be unambiguous and clear in the wording.• Be simple, clear and brief wherever possible.• Clarify the kinds of responses required in open questions.

PRACTICAL CONSIDERATIONS IN QUESTIONNAIRE DESIGN

• Balance brevity with politeness, e.g. replace a blunt phrase like ‘marital status’ with a gentler ‘please indicate whether you are married, living with a partner, or single....’.

• Ensure a balance of questions which ask for facts and opinion.• Avoid leading questions.• Try to avoid threatening questions.• Do not assume that respondents know the answers, or have

information to answer the questions, or will always tell the truth (wittingly or not)..

• Avoid making the questions too hard.• Balance the number of negative questions with the number of

positive questions.• Consider the readability levels of the questionnaire and the

reading and writing abilities of the respondents.• Put sensitive questions later in the questionnaire.• Intersperse sensitive questions with non-sensitive questions.

PRACTICAL CONSIDERATIONS IN QUESTIONNAIRE DESIGN

• Be very clear on the layout of the questionnaire so that it is unambiguous and attractive.

• Avoid, where possible, splitting an item over more than one page,.

• Ensure that the respondent knows how to enter a reply to each question.

• Pilot the questionnaire.• With the data analysis in mind.• Decide how to avoid falsification of responses.• Be satisfied if you receive a 50 per cent response to the

questionnaire• Decide what you will do with missing data.• Include a covering letter.• If the questionnaire is going to be administered by someone

other than the researcher, ensure that instructions for administration are provided and that they are clear.

METHODS OF ADMINISTERING QUESTIONNAIRES

• Post• Self-administration in presence of researcher• Self-administration without researcher present• In situ completion (e.g. workplace/ home)• Face-to-face interview• Telephone• Internet

PROCESSING QUESTIONNAIRE DATA

• Coding• Data reduction techniques• Editing and cleaning data