emr 6550: experimental and quasi- experimental designs dr. chris l. s. coryn kristin a. hobson fall...
TRANSCRIPT
![Page 1: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/1.jpg)
EMR 6550:Experimental and Quasi-
Experimental DesignsDr. Chris L. S. Coryn
Kristin A. HobsonFall 2013
![Page 2: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/2.jpg)
Agenda
• Regression discontinuity designs
![Page 3: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/3.jpg)
Questions to Consider
• Throughout today’s discussion, consider the following questions1.What characteristics of regression discontinuity
designs make them more amenable to causal interpretation versus those discussed previously (i.e., quasi-experimental designs with and without pretests and control groups and interrupted time-series)?
2.What are the associated limitations or drawbacks of regression discontinuity designs, if any?
3.How could you apply regression discontinuity designs to your own work?
![Page 4: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/4.jpg)
Note
• The principles of regression discontinuity can be confusing, so…–…ASK QUESTIONS!!!
![Page 5: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/5.jpg)
Basic Structure of Regression Discontinuity Designs
![Page 6: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/6.jpg)
Basic Structure
OA C X O2
OA C O2
![Page 7: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/7.jpg)
![Page 8: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/8.jpg)
![Page 9: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/9.jpg)
![Page 10: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/10.jpg)
![Page 11: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/11.jpg)
Theory of Regression Discontinuity
![Page 12: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/12.jpg)
Some Assumptions
• Most randomized experiments compare posttest means
• Regression discontinuity designs compare regression lines for treatment and control groups
• Rather than making the assumption that pretest means are equivalent for both groups, regression discontinuity looks for a change between the functional form of the regression line (e.g. slope or intercept) for the treatment group and control
![Page 13: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/13.jpg)
Some Assumptions
• In (most) other designs for quasi-experiments, the selection process is never fully known (selection is determined by a large system of variables beyond the researcher’s control)
• Regression discontinuity designs– No unknowns in the assignment process– Selection process is completely known and perfectly
measured (even though there will be some error associated with the assignment variable)
– Assignment variables only measure how participants got into conditions, and when assignment is based only on that score, error is effectively zero
![Page 14: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/14.jpg)
Some Assumptions
• Additionally, most experiments and quasi-experiments try to equate treatment and control
• Regression discontinuity designs – Explicitly acknowledge—and, in fact, base
assignment on—pre-existing differences between treatment and control groups
– Units are assigned to conditions based on a cutoff score (i.e., cut score) on an assignment variable
– The assignment variable must occur prior to treatment• Units on one side of the cut score are assigned to one
condition, and units on the other side to another condition
![Page 15: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/15.jpg)
Implementing Regression Discontinuity Designs
![Page 16: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/16.jpg)
Implementation
• Like randomized experiments, regression discontinuity designs yield unbiased estimates of treatment effects (as long as assumptions are met)
• Assignment to treatment must be based only on the cutoff score
• The assignment variable cannot be caused by the treatment, and must be continuous– Dichotomous variables should not be used as an
assignment variable, because this makes it impossible to estimate a regression function
• The assignment variable is often a pretest score, but it can be almost anything
![Page 17: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/17.jpg)
Implementation
• The cut score should be near the mean of the assignment variable, because extreme values can cause problems– Modeling the regression function can become more
difficult and/or error-prone in smaller samples– Statistical power depends on sample size, and also
“prefers” samples with equal or nearly equal sizes
• Researchers can use a composite variable for assignment, to include the effect of multiple influences
• Avoiding selection bias requires that– Assignment to conditions is strictly controlled– All units could have received treatment
![Page 18: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/18.jpg)
Forms of Effects
![Page 19: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/19.jpg)
Types of Effects
• First, note that “discontinuity” means– A treatment effect (if present) causes an
upward or downward displacement in the regression function
– A discontinuity can be a change in either the intercept or the slope
– The discontinuity should occur at exactly the cut score
![Page 20: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/20.jpg)
Threats to Validity
![Page 21: EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013](https://reader035.vdocuments.site/reader035/viewer/2022062318/551bf90b550346ad4f8b4814/html5/thumbnails/21.jpg)
Validity Threats
• Few internal validity threats are plausible with a well planned and correctly executed regression discontinuity design
• A plausible threat would have to cause a discontinuity in the regression line that corresponds precisely with the cut score– Except in rare cases, selection, history, and maturation
threats are drastically reduced– However, attrition can be a major concern, especially if
attrition rates are correlated with the assignment variable
• Therefore, validity concerns mainly focus on statistical conclusion validity– In particular, the regression lines must be good models of
functional form (e.g., nonlinear functions, interaction terms)