Quasi-experimental Quasi-experimental Designs & SurveyDesigns & Survey

ResearchResearch

Hedrick, Bickman & Rog (1993) “a Hedrick, Bickman & Rog (1993) “a quasi-experimental design is not the quasi-experimental design is not the method of choice, but rather a fallback method of choice, but rather a fallback strategy”strategy”

Cannot infer cause and effect, Cannot infer cause and effect, BUTBUT well well designed quasi-experiments enable designed quasi-experiments enable you to demonstrate that rival you to demonstrate that rival interpretations are rendered unlikelyinterpretations are rendered unlikely

Quasi-experimentsQuasi-experiments

Non-equivalent Group DesignsNon-equivalent Group Designs

Posttest only nonequivalent control group Posttest only nonequivalent control group design (aka static group comparison)design (aka static group comparison)

X O X O (treatment grp)(treatment grp) O O (nonequivalent control grp)(nonequivalent control grp)

X=treatmentX=treatment O=measurement/observation O=measurement/observation

Because there is no random assignment to Because there is no random assignment to groups, confounding variables may explain groups, confounding variables may explain any difference observedany difference observed

Pre-test/Post-test Non-equivalent Pre-test/Post-test Non-equivalent Control GroupControl Group

O X O treatment group

O O non-equivalent control group

Pre Post

Example: Research Methods Example: Research Methods and Reasoning Abilityand Reasoning Ability

0

1

2

3

4

5

Pre Post

Mea

n R

easo

nin

g S

core

Develop Methods

Intervention: critical thinking seminar

Research Methods students receive the intervention (i.e., participate in the critical thinking seminar)

Developmental Psychology students are used as a nonequivalent control group

(i.e., do not attend the seminar)

Interrupted Time Series DesignsInterrupted Time Series Designs

Extension of the simple one group pre Extension of the simple one group pre

and post designand post design

Participants are pre-tested a number of Participants are pre-tested a number of

times and then post-tested a number of times and then post-tested a number of

times after being exposed to the times after being exposed to the

treatment interventiontreatment intervention

OO11 O O2 2 OO33 O O44 X O X O5 5 O O66 O O77 O O88

Useful when: Useful when: cannot randomize participantscannot randomize participants

it is possible to obtain a series of it is possible to obtain a series of

assessments of the DV before and after assessments of the DV before and after

treatmenttreatment

Interrupted Time Series DesignsInterrupted Time Series Designs

Example: Interrupted Time Series Example: Interrupted Time Series DesignsDesigns

Intervention: Course to change students’ study habits, implemented during the summer (after semester 4).

DV: semester GPA0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 4 5 6 7 8

Mea

n G

PA

X

Baseline

Post-tx

How Many Measurements Are How Many Measurements Are Needed for a Time Series Design?Needed for a Time Series Design?

Depends on:Depends on: amount of random fluctuation (noise) that amount of random fluctuation (noise) that

may occur in the outcome being measuredmay occur in the outcome being measured

how much of an impact the intervention is how much of an impact the intervention is

expected to haveexpected to have

Multiple Time Series DesignsMultiple Time Series Designs Add a comparison group to the simple interrupted time Add a comparison group to the simple interrupted time

series design:series design:

OO11 O O2 2 OO33 O O44 X O X O5 5 O O66 O O77 O O88

------------------------------------------------------------------------------------------------------------------------------------

OO11 O O2 2 OO33 O O44 O O5 5 O O66 O O77 O O88

Campbell & Stanley consider this an excellent quasi-Campbell & Stanley consider this an excellent quasi-experimental designexperimental design

Survey Research Survey Research

SurveysSurveys

Items are composed of 2 parts: stem & Items are composed of 2 parts: stem & response optionsresponse optionsThe university should not raise tuition under any The university should not raise tuition under any

circumstances circumstances ((stemstem)) Agree Agree DisagreeDisagree ((response optionsresponse options))

Question WordingQuestion Wording Important consideration when writing Important consideration when writing

questions:questions: SimplicitySimplicity Avoid jargon and technical termsAvoid jargon and technical terms Avoid abbreviationsAvoid abbreviations Be specificBe specific Avoid double-barreled questions -Avoid double-barreled questions -

Responses to QuestionsResponses to Questions Open-endedOpen-ended

Closed-endedClosed-ended

“…closed questions should be used where alternative replies are known, are limited in number, and are clear cut. Open ended questions are used where the issue is complex, where relevant dimensions are not known, and where process is being explored.”

Stacey (1969)

Open vs. Closed QuestionsOpen vs. Closed Questions

Rating ScalesRating Scales

Likert Scales Likert Scales Semantic DifferentialSemantic Differential Graphic Rating ScaleGraphic Rating Scale Non-verbal scaleNon-verbal scale

Question PlacementQuestion Placement

Very importantVery important

Confusing questionnaire = Confusing questionnaire = demoralizing to respondentsdemoralizing to respondents

Can affect people’s responses to Can affect people’s responses to themthem

Layout of QuestionnaireLayout of Questionnaire

Use closed-ended questionsUse closed-ended questions

Use a consistent item formatUse a consistent item format

Don’t crowd questions togetherDon’t crowd questions together

Don’t split questions or response options between Don’t split questions or response options between pagespages

It is most likely to be completed if easy on the eye, It is most likely to be completed if easy on the eye, relevant, logical and as short as possible.relevant, logical and as short as possible.

Response BiasesResponse Biases Social desirabilitySocial desirability Acquiescence response - tendency either to Acquiescence response - tendency either to

agree/disagree with statementsagree/disagree with statements Yea-SayingYea-Saying Nay-SayingNay-Saying

CChecklisthecklist for for SSurvey urvey UUsese1.1. Do the items provide a representative sample Do the items provide a representative sample

of the behaviour under study?of the behaviour under study?

2.2. Do the items ask for irrelevant information?Do the items ask for irrelevant information?

3.3. Does the question have the same meaning for Does the question have the same meaning for

researcher and respondent?researcher and respondent?

4.4. Is the question worded clearly?Is the question worded clearly? Are the words Are the words

too difficult?too difficult?

5.5. Does the question concern a sensitive issue?Does the question concern a sensitive issue?

CChecklisthecklist for for SSurvey urvey UUsese

6.6. Does the question suggest a socially Does the question suggest a socially desirable answer?desirable answer?

7.7. Open vs closed questions?Open vs closed questions?

8.8. How many questions are necessary to How many questions are necessary to obtain the required information?obtain the required information?

9.9. How might the questions be arranged to How might the questions be arranged to encourage the respondent’s encourage the respondent’s participation?participation?