Non-Experimental designs: Surveys & Correlational

Psych 231: Research Methods in Psychology

Exam 2 results

Mean = 75.2 Median = 78 Max = 98 Min = 47

Most common errors

Between vs. within designs Independent vs. dependent vars Scales of measurement

Confounds vs. extraneous variables

Main effects vs. interactions

Non-Experimental designs

Sometimes you just can’t perform a fully controlled experiment Because of the issue of interest Limited resources (not enough subjects, observations are too

costly, etc). • Surveys

• Correlational studies

• Quasi-Experiments

• Developmental designs

• Small-N designs

This does NOT imply that they are bad designs Just remember the advantages and disadvantages of each

Stages of survey research

Stage 1) Identify the focus of the study and select your research method

Stage 2) Determining the research schedule and budget

Stage 3) Establishing an information base Stage 4) Identify the sampling frame Stage 5) Determining the sample method and

sampling size Review Probability and Non-Probability methods

• Voluntary response method Importance of sample size

Stages of survey research cont.

Stage 6) Designing the survey instrument Question construction: How the questions are

written is very important• Clearly identify the research objectives

• Do your questions really target those research objectives (think Internal and External Validity)?

• Take care wording of the questions• Keep it simple, don’t ask two things at once, avoid

loaded or biased questions, etc.• How should questions be answered (question type)?

Good and poor questions

GoodPoorWas the FDC negligent by ignoring the warnings about Vioxx during testingand approving it for sale?

a) Yesb) Noc) Unsure

Do you favor eliminating the wasteful excess in the public school budget?

a) Yesb) Noc) Unsure

If the FDC knew that Vioxx caused serious side effects during testing,what should it have done?

a)Ban it from ever being soldb)Require more testing before approving itc)Unsure

Do you favor reducing the public school budget?

a)Yesb)Noc)Unsure

Problem: emotionally charged words

Problem: emotionally charged words

Good and poor questions

GoodPoorShould senior citizens be given moremoney for recreation centers and food assistance programs?

a)Yesb)Noc)Unsure

Should senior citizens be given more money for recreation centers?

a)Yesb)Noc)Unsure

Should senior citizens be given more money for food assistance programs?

a)Yesb)Noc)Unsure

Problem: asks two different questions

Problem: asks two different questions

Good and poor questions

GoodPoorAre you against same sex marriage and in favor of a constitutional amendment to ban it?

a) Yesb) Noc) Unsure

What is your view on same sex marriage?

a) I think marriage is a matter of personal choice

b) I’m against it but don’t want a constitutional amendment

c) I want a constitutional amendment banning it

Problem: Biased inmore than one direction

Problem: Biased inmore than one direction

Problem: Asks two questions

Problem: Asks two questions

Survey Questions

Question types Open-ended (fill in the blank, short answer)

• Can get a lot of information, but• Coding is time intensive and potentially

ambiguous

Close-ended (pick best answer, pick all that apply)• Easier to code• Same response alternatives for everyone• Take care with your labels

• Decide what kind of scale

• Decide number/label of response alternatives

What is the best thing about ISU? (choose one)

1. Location 2. Academics 3. Dorm food 4. People who sell

things between Milner and the Bone

What is the best thing about ISU?

Decide what kind of rating scales• Rating: e.g., Likert scale

Survey Questions: Close-ended

PSY 231 is an important course in the major. 1 2 3 4 5 Strongly Agree Neutral Disagree Strongly Agree Disagree

• Semantic differential: Rate how you feel about PSY 231 on these dimensions

Important _____: _____: _____: _____: _____: Unimportant

Boring _____: _____: _____: _____: _____: Interesting

• Nonverbal scale for children:

Point to the face that shows how you feel about the toy.

Decide number/label of response alternatives• Use odd number (mid point and equal # of responses above

and below the mid point)• Questions should be uni-dimensional (each concerned with

only one thing)• Labels should be clear

Survey Questions: Close-ended

Stages of survey research cont.

Stage 7) Pre-testing the survey instrument Fix what doesn’t seem to be working

Stage 8) Selecting and training interviewers For telephone and in-person surveys Need to avoid interviewer bias

Stage 9) Implementing the survey Stage 10) Coding and entering the data Stage 11) Analyzing the data and preparing a

final report

Non-Experimental designs

Sometimes you just can’t perform a fully controlled experiment Because of the issue of interest Limited resources (not enough subjects, observations are too

costly, etc). • Surveys

• Correlational

• Quasi-Experiments

• Developmental designs

• Small-N designs

This does NOT imply that they are bad designs Just remember the advantages and disadvantages of each

Correlational designs

Looking for a co-occurrence relationship between two (or more) variables We call this relationship a correlation.

3 properties: form, direction, strength

Form

Non-linearLinear

Direction

Positive

• X & Y vary in the same direction

Y

X

Negative

• X & Y vary in opposite directions

Y

X

Strength

r = 1.0“perfect positive corr.”

r = -1.0“perfect negative corr.”

r = 0.0“no relationship”

-1.0 0.0 +1.0

The farther from zero, the stronger the relationship

Correlational designs

Looking for a co-occurrence relationship between two (or more) variables Used for

• Descriptive research• do behaviors co-occur?

• Predictive research• is one behavior predictive of another?

• Reliability and Validity• Does your measure correlate with others (and itself)?

• Evaluating theories• Look for co-occurrence posited by the theory.

Correlational designs

Looking for a co-occurrence relationship between two (or more) variables Example 1: Suppose that you notice that the

more you study for an exam, the better your score typically is

Explanatory variables (Predictor variables) Response variables (Outcome variables)

For our example, which variable is explanatory and which is response? And why?

It depends on your theory of the causal relationship between the variables

At a descriptive level this suggests that there is a relationship between study time and test performance.

Scatterplot

Hours study

X

Exam perf.

Y

6 6

1 2

5 6

3 4

3 2

Y

X

1

2

3

4

5

6

1 2 3 4 5 6

For this example, we have a linear relationship, it is positive, and fairly strong

Scatterplot

Y

X

1

2

3

4

5

6

1 2 3 4 5 6

Response (outcome) variable

Explanatory (predictor) variable

For descriptive case, it doesn’t matter which variable goes where

Correlational analysis

For predictive cases, put the response variable on the Y axis

Regression analysis

Correlational designs

Advantages: Doesn’t require manipulation of variable

• Sometimes the variables of interest can’t be manipulated Allows for simple observations of variables in

naturalistic settings (increasing external validity) Can look at a lot of variables at once

Example 2: The Freshman 15 (CBS story)• Is it true that the average freshman gains 15 pounds?• Recent research says ‘no’ – closer to 2.5 – 3 lbs• Looked at lots of variables, sex, smoking, drinking, etc.• Also compared to similar aged, non college students

Disadvantages: Don’t make casual claims

• Third variable problem

• Temporal precedence

• Coincidence (random co-occurence)

Correlational designs

Correlational results are often misinterpreted

Misunderstood Correlational designs

Example 3: Suppose that you notice that kids who sit in the front of class typically get higher grades. This suggests that there is a relationship between

where you sit in class and grades.

Daily Gazzett

Children who sit in the back of the classroom receive lower grades than those who sit in the front.

Possibly implied: “[All] Children who sit in the back of the classroom [always] receive worse grades than [each and every child] who sits in the front.”

Better: “Researchers X and Y found that children who sat in the back of the classroom were more likely to receive lower grades than those who sat in the front.”

Example from Owen Emlen (2006)

Non-Experimental designs

Sometimes you just can’t perform a fully controlled experiment Because of the issue of interest Limited resources (not enough subjects, observations are too

costly, etc). • Surveys

• Correlational

• Quasi-Experiments

• Developmental designs

• Small-N designs

This does NOT imply that they are bad designs Just remember the advantages and disadvantages of each

Quasi-experiments

What are they? Almost “true” experiments, but with an inherent

confounding variable

General types1) An event occurs that the experimenter doesn’t

manipulate• Something not under the experimenter’s control

• (e.g., flashbulb memories for traumatic events)

2) Interested in subject variables– high vs. low IQ, males vs. females

3) Time is used as a variable

Quasi-experiments

Advantages Allows applied research when experiments not

possible Threats to internal validity can be assessed

(sometimes) Disadvantages

Threats to internal validity may exist Designs are more complex than traditional

experiments Statistical analysis can be difficult

• Most statistical analyses assume randomness

Quasi-experiments

Program evaluation– Research on programs that is implemented to achieve

some positive effect on a group of individuals.– e.g., does abstinence from sex program work in schools

– Steps in program evaluation– Needs assessment - is there a problem?– Program theory assessment - does program address the

needs?– Process evaluation - does it reach the target population? Is it

being run correctly?– Outcome evaluation - are the intended outcomes being

realized?– Efficiency assessment- was it “worth” it? The the benefits

worth the costs?

Quasi-experiments

Nonequivalent control group designs with pretest and posttest (most common)

(think back to the second control lecture)

participants

Experimentalgroup

Controlgroup

Measure

Measure

Non-Random Assignment

Independent Variable

Dependent Variable

Measure

Measure

Dependent Variable

– But remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity)

Quasi-experiments

Interrupted & Non-interrupted time series designs Observe a single group multiple times prior to and after a

treatmentObs Obs Obs Obs Treatment Obs Obs Obs Obs

• Look for an instantaneous, permanent change

• Interrupted – when treatment was not introduced by researcher, for example some historical event

Variations of basic time series design• Addition of a nonequivalent no-treatment control group time series

O O O T O O O & O O O _ O O O

• Interrupted time series with removed treatment• If treatment effect is reversible