experimental design. more threats to internal validity… instrumentation or measurement procedure...
Post on 19-Dec-2015
218 views
TRANSCRIPT
Experimental Design
More threats to internal validity…
• Instrumentation or measurement procedure– If your way of measuring something changes over
time, it could alter how your outcome is measured– E.g. if how I ask the question “did you have sex”
varies, responses may vary over time– E.g. If grading papers, and my view of what a
“good answer” changes as I read answers, this could change how I grade later versus earlier responses
Instrumentation?
• Avoid this by piloting your measure first and then sticking with it
• If you must change your measure then collect new data
• Keep track of changes in questions
Testing
• When testing or surveying people at baseline influences their responses– E.g. giving someone a survey about sex could also
operate as a way of teaching them about sex– Giving someone a math test could also give them
a chance to practice their math skills
Testing
• Avoid testing by– Having a control group so that if you have a testing
effect you can at least control for it– Consider carefully the ways a survey or test may
be a learning experience for respondents
Regression to the mean
• Extreme scores will be less extreme when tested again– Very low scores will be less low– Very high scores will be less high– Why?
• If you have more extreme scores in one condition– This could make it look like your intervention works
or doesn’t work when they are less extreme at the second measurement
Avoiding regression to the mean
• If you have extreme scores use stratification or block randomization to make sure groups are equally balanced for scores
• Remember that simple randomization won’t always fix this, especially with a small sample
Placebo and demand characteristics
• Placebo– When they think they are getting better, they feel
better!– We control for this using blinding—participants
are blind to the group they are in• But what about the researcher?– Researchers may influence results by
communicating expectations– So where possible we use double-blind
Double blind
• When both the participant and the researcher are unaware of the treatment the participant receives
• This can be very difficult to achieve in psychology
• Why?
Confounding
• When a third variable accounts for the influence of your IV on your DV
• Many types of threats to internal validity end up functioning as confounds– Placebo– History– Selection…etc.
• The point is that a confound is an unmeasured influence that is actually responsible for the effect
External validity?
• So for whom and under what circumstances is this treatment actually effective?
External validity
• To whom and under what conditions can results be generalized?
• A question of great practical and theoretical significance
• If your intervention only works under very specific conditions, is it really useful?
External validity--example
A university clinic uses an intervention totreat depressed patients. --only patients diagnosed with depression alone--using graduate students who see only a few
patients each week--each patient gets a 3 hour battery of tests plus
an indepth diagnostic interview--graduate students who get weekly supervision to
make sure they are maintaining the treatment approach--treatment is free--it takes place in a quiet clinic on an attractive
university campus
Selection bias
• The whole sample is biased – Not in a way that makes intervention different
from control– But in a way that makes them all different from
the likely population• E.g. most people with depression don’t ONLY
have depression
Testing
• Testing won’t make the intervention and control groups different
• It may make their experiences different from those of people who get the treatment later– E.g. a 3 hour battery of tests and interviews may
itself be therapeutic under some circumstances
Reactive effects of experimental arrangements
• Most therapists don’t only see a few patients a week
• Most therapists don’t get weekly supervision to make sure they are maintaining protocol
• Most patients don’t get therapy in lovely quiet offices on pretty university campuses
How to avoid problems with external validity?
• Difficult• The higher your internal validity—the more
you control alllllll the factors that could muddy or influence your outcome– The lower your external validity will be
Building external validity?
• Takes time• You may have to “redo” the intervention
several times, changing and varying and measuring the circumstances
• Start neat and tidy– Then slowly add in and measure real-world
messiness• This can be costly and time consuming.
“pre experimental designs”
• Also called pilot studies• Generally low in internal validity• But a good place to start – Cheaper – Quicker
• You want to know you have something before you go to the trouble and expense of a full blown randomized experiment
Quasi-experimental designs
• Use experimental and control groups• Do not use random assignment– Why?
• May use “matching.”– Matching on qualities of interest
Pilot studies
• Pretest-posttest– No control group– Just measure if your intervention scores change
from baseline to post-test• E.g. if I treated my depressed people with my
intervention and just measured their improvement
• Why would I do this?
Pilot studies
• Post test only• Ok for measuring an outcome– E.g. the SAT could be considered a posttest only
design– Gives your achievement scores– Gives no sense at all of what your achievement
might have been before, how it changed, and what caused the change
Pilot Studies
• Static group design– If I simply examined the outcomes of two different
treatments– I don’t control selection– I don’t measure baseline
• Again, this can be a useful first step
Equivalent time sample design
• More succinctly known as single subject design
• May have one participant• Design is:– Baseline (no treatment)– Treatment– No treatment– treatment
Single subject designs
• Quite common in behavioral research• E.g. treatment for OCD– Baseline-picking behavior– Treatment—withhold “reinforcement”—behavior
goes away– Withdraw treatment (return to reinforcement)• Behavior returns
– Treatment—again withhold reinforcement—behavior goes away
Single subject designs
• Are actually a very powerful experimental technique
• Commonly used in behavior analysis and treatment
• A good way to establish causality