experimental design making your experiment more valid and more efficient
TRANSCRIPT
Experimental Design
Making your experiment more valid and more efficient
So, the basic experimental procedure in social science is:
• Gathering of a subject pool and then random assignment to conditions– Note: the sample is not random, but once you
have a subject pool you flip a coin, etc. to determine which subjects receive which condition
• Depending upon the experimental design, a measure of the dependent variable may be taken from the groups prior to the manipulation
• Subjects are exposed to the ‘treatment’ assigned to their group– May be some level of the factor or a zero-level (control)
treatment– For example, the ‘’experimental group” may play a video
game with lots of violence while the control group plays a non-violent game
• Both groups are then measured on the dependent variable of interest– In this case, some measure of aggressiveness
• The data are then analyzed to test the research hypothesis
Basic design(After-only, control group design)
R X O
R O
R=randomization, X=manipulation, O=observation
What’s the effect?
• The difference between group scores on the dependent variable is assumed to be due to the manipulation
• But what if the groups were different at the start?– Randomization should have taken care of that– However, pure chance can sometimes lead to
differences at the outset• So, measure them before the manipulation as well as
after
Before-after, control group design
R XO1 O2
R O2O1
R=randomization, X=manipulation, O=observation
Before-after, control group design
• Now there’s a new problem:– Measuring the subjects and then measuring them
again later may cause them to respond differently to the second measure than they otherwise would have (true for both experimental and control subjects)
– Measuring the experimental subjects before exposing them to the treatment may change their response to the manipulation
Problems with the basic design
• The difference between control and experimental conditions includes more than violence– Experimenters may have control group play a
nonviolent video game• Many other variables may affect the outcome– For example, gender• Measure and include in statistical analyses• Use factorial design
So:
• There will always be some problems with any procedure
• The goal is to minimize the threats to validity as best your circumstances (including budget) allow– Especially internal validity
How do we do that?
• Experimental design• Statistical analyses• Careful application of procedures
Experimental design
• Two facets– Choices in identifying the independent and
dependent variables to be measured and/or manipulated
– Choices in developing the procedures for manipulation, measurement, environment of testing, etc.
Choosing the variables
• Theory determines variables manipulated and measured
• Experimental designs where the research manipulates more than one independent variable are known as factorial designs– Mediating concepts often are included
Choosing the measures
• The measures are chosen to best represent the concepts in the original theory– Measurement validity
• Must work in an experimental setting– Lengthy questionnaires unlikely to work– Physical measures more easily applied in a
laboratory setting
Factorial designs
• Factorial designs are useful because they:– Are efficient
• With two independent variables we can test three hypotheses instead of just one– Don’t need three times as many subjects
• However, there is a practical limit to independent variables– As the number rises, the required number of subjects increases quickly (5
per cell)
– Help us avoid coming to invalid conclusions• They allow us to control for potentially confounding variables and
interactions– We can test the impact of third variables
• Factorial designs can be more difficult to administer
An example
• We may want to test the following hypothesis:– “Exposure to Dancing with the Stars will lead to
greater liking for the contestants”• We might, however, expect that men and
women will be affected differently by the show
• So, we might add gender as another independent variable in the design
Factorial design
Women
See Dancing with the Stars
Don’t see Dancing with the Stars
All groups are measured on liking for contestants
Men
What we can test
• “Main effect” of exposure to Dancing with the Stars
• “Main effect” of gender• “Interaction effect” of exposure by gender• Remember: the dependent variable is liking
for contestants
Results(% who like contestants)
Didn't see DWTS Saw DWTS0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
MenWomen
Conclusions
• Main effect of gender– Yes: Gender affects liking for celebs
• Main effect of exposure to DWTS– Yes: Exposure increases liking
• Interaction effect of exposure by gender:– Yes: Exposure has a clear positive effect for
women, little or even negative effect for men
Procedural choices
• How many times will you measure the subjects?– You can account for chance fluctuation by
increasing the number of times the groups are measured• You increase the likelihood of testing reactivity, though
R O1 O2X O3 O4
R O1 O2 O3 O4
Additional design concerns
• How many times/how long will the manipulation be presented?– Testing a single exposure to DWTS may not be
valid• Viewers may watch many shows during a season
R O1 X1 O2 X2 O3
R O1 O2 O3
Removal of the treatment
R O1X O2 -X O3
R O1 O2 O3
• How many levels of the independent variable will be presented?– May want to test high v. medium v. low levels of
violence, etc.
R O X(high) O
R O X(low) O
R O X(medium) O
R O O
Factorial nomenclature
• The number of treatments on each factor are crossed– 2X3 factorial design has 2 treatment levels on one
factor and 3 treatment levels on another factor– If we tested the effects of soft, medium and loud
music and gender upon ease in a social situation, we would have a 2X3 factorial study• Factor 1: music volume 3 levels: soft, medium and loud• Factor 2: gender 2 levels: male, female• Dependent variable: ease in a social situation
Additional design concerns
• Artificiality of the testing environment– Labs allow for the greatest control over third
variables• Volume, distance from screen, prevent other people from
walking by, etc.
– Lab experience can sometimes be made less artificial• Some commercial-testing services provide the subjects
with a simulated living room for exposure• Loss of control over third variables weighed against
increase in generalizability (external validity)
Statistical analyses
• If you measure potential confounds you can use statistical procedures to remove their impact on the dependent variable
Careful application of the design
• Unplanned variations among treatments can easily occur when conducting experiments in the real world– The person you wanted to greet everyone and
provide instructions may have to leave for an emergency
– Subjects may talk to each other outside the lab before they have all completed their part of the experiment
– And so on
Remember:
• Experiments are the most effective quantitative research method for testing hypothesized causal relationships
• Experiments emphasize internal validity but are often weak with regard to external validity
• Many different experimental designs exist, each having a set of advantages and disadvantages
• Your goal is to provide the most valid test of theory your circumstances allow you