1

Survey Research

PS 1006 Lecture 1

Sam Cromie

2

Research Methods

• Observational research– Unstructured– Structured– Participatory

• Experiments– Group design

• Within-subject• Between subject

– Single-subject design

• Surveys• Qualitative research

– Interviews– Focus groups– Document analysis– Etc.

• Case studies

Survey research

Writing good questions

Data Collection Methods

Research Designs

SamplingMethods

Essential Reading:

Research Methods in Psychology (5th or 6th edition)

Shaughnessy, Zechmeister & Zechmeister

Surveys: Advantage and Disadvantages

• Positives– Easy to sample a

large number of people

– Context likely to retains ecological validity

– Relatively easy to administer

• Negatives– Danger of biased

surveys– People may not

answer truthfully– No true IVs - Cannot

infer causality

5

Types of question

• Open-ended - ‘How do you feel today?’

– Allows wide range or responses - preferable when researcher has little idea of the type or response to expect

– Respondents decide how much information to give

– Lack of structure can make scoring of the responses difficult

6

Types of question

• Closed questions - ‘On a scale of 1-5 how happy do you feel today?’

– A fixed set of responses is provided– Respondent must pick one– Easy to score– May miss responses that some participants

wants to give since the scope for answers is limited

7

Types of question

• Mixed questions– A fixed set of responses is provided– An open-ended choice is given to cover any

alternatives that may have been missed– Still has a limited scope

How do you feel today?A. HappyB. SadC. AngryD. RelaxedE. Other

(specify)_________________________________

Response scales

• Categorical questions – non-directional optionsHow are you happy today?

• Likert Scales - Labeled points on scaleI am Happy today

• PercentagesHow happy are you on a percentage from 1-100

• Continuous scalesPlease mark the line below to express you level of happiness:

Strongly Agree

Agree Disagree Strongly Disagree

Happy Unhappy

Happy Angry Sad Tired

Categorical Questions

• Often used for demographic information:– Job type, ethnic background, etc.

• Dichotomous questions – only two options: Yes/no, Male Female

– No indication of response strength

• Can be used as quasi IVs in data analysis

• Can be analysed with other categorical questions using Chi Squares

10

Likert Scales

• Labeled points on scale• Several possibilities

– Strongly agree, Agree, Disagree, Strongly disagree

– Scale from 1-5 with ends (poles) labeled

– Useful to try and define parts of the scale (anchoring)

• How many values should be put on a scale?– 7±2 options - above this

difficult to use the full range

• When normally distributed can be analysed parametrically

11

Percentages

• Gives participants a wider range of possible responses

• But resposes tend to cluster around commonly used percentages – 10, 20, 25, 50, 75, 90

Continuous scalesHappy Unhappy

• Line is normally 10cm long• Score = distance in mm from start of line

to mark• Get a fuller use of the scale than

percentages• Allows more sensitive measurement than

Likert• More challenging to code

Writing good questions

• A survey lives and dies by its questions– Unclear questions confuse

• Vocabulary should be clear and simple

– Questions should be specific• Should government do something about health

care?• Should government pass a law on health care?• Should government pass a law on health care that

would guarantee health coverage for all?

Writing good questions

• Include conditional information before the key idea in the question– If you were the victim of a violent crime, would

you consider buying a gun?

• Consider varying the polarity of the questions– Questions should be phrased both negatively

and positively• Some people just like to say No (or Yes)!

• Leading questions - suggesting the ‘right’ answer– People often recognize the importance of gun control in

the prevention of crime. What is your opinion of gun control?

• Loaded questions - emotionally charged– Are you in favor of laws that that allow dangerous

convicted felons out of prison early?

• Double-barreled questions - packing too much in – Do you favor life sentences for repeat offenders and

stricter sentences for criminals who used guns?

Avoid when writing Qs

Basic sampling terms

• Population - all members of defined group– Census - ask entire population– Survey - ask representative sample

• Sampling frame - specific list of all the members of the population - not always accurate – e.g., register of currently enrolled students - what

about students who are abroad or taking year out?

• Element - a member of the population

17

Basic sampling terms

• Representativeness - – dictates how generalisable the results are. Clearly,

biased samples = bad.

• Selection bias – – procedures used to select sample which result in non-

representative sample

18

Sampling methods

• Probability sampling - each element has same probability of being included– Simple random sampling– Stratified random sampling - population

divided into strata and random samples are drawn from each e.g, make sure ethnic balance reflects population

• Most sampling is not really random

Sampling methods

• Nonprobability sampling - each element has different probability of being chosen– Accidental sampling - selection on basis of

availability and willingness to respond e.g., television polls. People may participate:

• For money• Because they are interested in topic• Because they have a particular view point

– Purposive sampling - elements included on the basis of their special characteristics

Potential sampling problems

• Flawed surveys often reflect poor samples– Sampling bias can enter in two places

• At selection of sampling frame– Use of phone directories for general population

– List of subscribers to a magazine

• At selection of a sample– Use of mail survey

– Calls made to college students on Friday and Saturday nights

• Any study in which subjects self-select is potentially problematic

21

Survey research designs

• Cross-sectional designs – view of population at one time period only

Time

Example

• Suppose we conduct a survey of the number of left-handers in the population

• Of inds below 60, left-handers approx = 10%• Of Inds above 60, left-handers are much less than 10%

– Conclusion: left-handers die younger– Is this valid

• Need to ensure that timing does not produce a major confound

Successive independent samples

Time

• Can be used to assess change over time– Must ensure that samples are comparable

• Any change detected should be due to time and not sample differences

Longitudinal studies

Time

• Follow a single group of individuals over time– Requires a lot of time and effort– Subject drop out a problem

• Sometimes mortality• Most often people drop out for some reason• Must ensure that there is not a systematic reason for

drop out

Data collection

• There are several ways to conduct surveys– Through the post– In person– Over the phone– Via internet

• Each of these methods has its own advantages and disadvantages

Postal surveys

• Advantages– Minimizes effort needed to

collect data– Good for personal or

embarrassing topics• Respondent’s replies are

anonymous

• May cut down on the amount of socially desirable responding

• Disadvantages – May not be filled out be

intended participant– Respondents cannot ask

for clarification– No follow-up questions can

be asked– No control over the order in

which completed– Big potential for response

bias

Self-selection - postal surveys• Respondents in postal surveys always self-

select• Controlling response bias in postal surveys

– Try to ensure a high return rate• Over 50% is considered very good

• Response bias is less severe with return rates over 50%

– Practical steps to improve response rate• Include a SAE

• Personalize the introduction (at least with name)

• Do follow-up mailings with new copies of the survey to those who do not respond

Personal interviews

• Advantages– High response rate (often over

80%, though less in high-crime areas)

– Can control order of responding

– Control over context of responding

– Ensure the correct participant answers question

– Respondent can ask for clarification

– Interviewer can ask follow-up questions

• Disadvantages– There is a potential for

interviewer bias• May suggest the desired

response (non-verbally)

– People may give socially desirable responses

• They may not want to express their true beliefs to the interviewer

– Difficult to ask about embarrassing topics

• No anonymity for the respondent

Telephone surveys

• Advantages– Better than personal interviews in high-crime

areas• People do answer the phone!

– More anonymity than for personal interviews– Can control order of responding– Respondent can ask for clarification– Interviewer can ask follow-up questions

Telephone surveys - disadvantages

• People may not want to be bothered over the phone– Particularly now that telemarketers often use surveys

as a sales pitch

• No control over surroundings– Respondent may be doing a number of things at once

• There is the potential for interviewer bias– May suggest the desired response through non-

semantic content of dialogue

• People may provide socially acceptable responses– They may not want to express their true beliefs to the

interviewer

Internet surveys

• More and more widely used

• Data can be imported directly into analysis package

• Quick and efficient• Cuts down on paper use• Excellent for international

surveys• Can use audio-visual

content, etc.

• Lack of control of context in which it is filled out

• Identity of participant• Respondents cannot ask

for clarification• No follow-up questions

can be asked• No control over the order

in which completed• Biased against the less

computer literate• Inappropriate for many

populations