urbdp 591 a lecture 8: experimental and quasi-experimental design objectives basic design elements...
TRANSCRIPT
URBDP 591 A Lecture 8: Experimental and Quasi-Experimental Design
Objectives• Basic Design Elements• Experimental Designs• Comparing Experimental Design Example• Quasi-Experimental Designs• The Nature of Good Design
Research ApproachesGeneral Purpose Explore relationships between variables Describe variables
General Approach Experimental Non-Experimental Descriptive
SpecificApproach
RandomizedExperimental
Quasi-Experimental
Comparative Associational Descriptive
Specific Purpose DetermineCause
ExamineCause
Compare Groups Find Associations Summarize Data
Type of HypothesisDifference
Relate variables Descriptive
Statistics Difference Inferential Statisticse.g. ANOVA
AssociationalInferentialStatistics(eg. Correlation,Multiregression
Descriptive StatisticsEg, Histograms,Means etc.
Basic Design Elements
Time: A causal relationship implies that some time has elapsed between the occurrence of the cause and the consequent effect.
Program(s) or Treatment(s). The presumed cause may be treatment under the explicit control of the researcher or the occurrence of some natural or human induced event not explicitly controlled. In design notation we depict it with the symbol "X". When multiple programs or treatments are being studied, we can use subscripts such as "X1" or "X2".
Observation(s) or Measure(s). Measurements are typically depicted in design notation with the symbol "O” if the same measurement or observation is taken at every point in time in a design. If different measures are given at different times, we can use subscripts such as ”O1" or ”O2".
Groups or Individuals. Typically, there will be one or more program and comparison groups. In design notation, each group is indicated on a separate line. Assignment to the group is indicated by "R" to indicate the group was randomly assigned, "N" was nonrandomly assigned and a "C" that a cutoff score or a measurement was used.
Types of Designs
A. Post-only design
B. Pre and post design
C. Multiple Levels of single IV
D. Multiple Experimental and Control Groups
E. Multiple IVs (Factorial design)
Experimental Designs
• Subjects randomly assigned to experimental\control groups
• Introduction of IV in experimental condition
• Measurement of DV (single or multiple instances)
Post-Only Design
Group 1 R X OGroup 2 R O
Randomized IV DV
Post-Only Design
• Subjects randomly assigned to experimental/ control groups
• Preliminary measurement of DV before before treatment (check of random assgn)
• Introduction of IV in experimental condition
• Measurement of DV (single or multiple instances)
Pre and Post Design
Group 1 R O1 X O2Group 2 R O1 O2
Randomized IV DV
Pre- and Post- Design
DV
• Subjects randomly assigned to experimental\control groups
• Introduction of multiple levels of IV in experimental condition
• Measurement of DV across different conditions
Multiple Levels of Single IV
Group 1 R X1 OGroup 2 R X2 OGroup 3 R X3 OGroup 4 R O
Randomized IV DV
Multiple Levels Design
• Subjects randomly assigned to experimental\control groups
• Preliminary measurement of DV in one exp\control pair
• Introduction of IV in both experimental conditions
• Measurement of DV (assess effects of pretest)
Multiple Experimental and Control Groups (Solomon Four-Group design)
Group 1 R O1 X O2Group 2 R O1 O2Group 3 R X O2Group 4 R O2
Randomized IV DV
Multiple Levels Design
DV
• Subjects randomly assigned to experimental\control groups
• Introduction of multiple levels of IVs in experimental condition
• Measurement of DV across different conditions (cells)
Multiple IVs (Factorial Design)
Group 1 R X1 Y1 OGroup 2 R X2 Y2 OGroup 3 R X1 Y2 OGroup 4 R X2 Y1 O
Randomized IV DV
Multiple IVs Design
IV
Requirements for randomized experimental design • has at least two groups • has two distributions (measures), each with an average and variation • assess treatment effect = statistical difference between the groups
Different statistical tests
t-test and one-way Analysis of Variance (ANOVA)
The t-test
Two group posttest only randomized experimental design :T-test or One way ANOVA
Pretest-posttest randomized experimental design : ANCOVA
Pretest-posttest randomized experimental design with interaction: ANCOVA
Quasi-Experimental Design
• Matching instead of randomization is used. For example, someone studying the effects of growth management on residential density will try to find a similar metropolitan areas similar to the experimental metropolitan area. That other town is not technically a control group, but a comparison group, and this matching strategy is sometimes called nonequivalent group design.
• Time series analysis is involved. A time series is perhaps the most common type of longitudinal (over time) research found in public policy. A time series can be interrupted or noninterrupted. Both types examine changes in the dependent variable over time, with only an interrupted time series involving before and after measurement.
Non-equivalent Comparison Group
• Post Only, Pre-Post, Multiple Treatments
• No random assignment into exp\con
• “Create” comparison groups– Selection criteria or eligibility protocol– Partial out confounding variance – (statistical control)
Time Series
• Multiple observations before and after treatment or intervention is introduced
• Examine changes in data trends (slope and intercept)
• Investigate effects of both onset and offset of interventions
0
2
4
6
8
10
12
14
16
Intervention
Time Series
Group 1 N R O1 O2 X O3Group 1 C O1 O2 O3
Regression Discontinuity
• Separate sample based on some criterion (pre-test)
• One group administered treatment, other is control group
• Examine trends in both groups; hypothesize equivalent
Pre-test Cut-off Point
TreatmentNo Treatment
PostTestScores
Regression DiscontinuityGroup 1 C O1 O2 X O3Group 1 C O1 O2 O3
Advantages of Experiments1. Isolation of the experimental variable
2. Allows for (relatively) easy replication
3. Establishing causality
4. Control - A true experiment offers the ultimate in control
5. Longitudinal analysis - The experiment offers the opportunity to study change over time.
Disadvantages of Experiments1. Artificial environment
2. Experimenter effect - The experimenter's expectations can affect the results of the experiment.
3. Lack of control - Placing subjects in a laboratory may alter the very behavior the experimenter is trying to study.
4. Sample size - The larger the group, the more difficult it is to control extraneous variables.
Summary: Experiment
• Aim to measure the effect of IV on DV
• Involve experimenter’s intervention
• Should control for experimental error (confounding variables)
• Comparing the DV produced by at least two levels (conditions) of IV
The Nature of Good Design
1.Theory-Grounded. Good research strategies reflect the theories which are being investigated. Where specific theoretical expectations can be hypothesized these are incorporated into the design to improve discriminant validity and demonstrates the predictive power of the theory.
2.Situational. Good research designs reflect the settings of the investigation. This was illustrated above where a particular need of teachers and administrators was explicitly addressed in the design strategy.
3.Feasible. Good designs can be implemented. The sequence and timing of events are carefully thought out. Potential problems in measurement, adherence to assignment, database construction and the like, are anticipated.
4.Redundant. Good research designs have some flexibility built into them. Often, this flexibility results from duplication of essential design features. For example, multiple replications of a treatment help to insure that failure to implement the treatment in one setting will not invalidate the entire study.
5.Efficient. Good designs strike a balance between redundancy and the tendency to overdesign. Where it is reasonable, other, less costly, strategies for ruling out potential threats to validity are utilized.