causal inference from multiple studies, critical assessment of assumptions presented by: bethany...
TRANSCRIPT
![Page 1: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/1.jpg)
CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS
Presented by:
Bethany Harmon, Jenny Jackson, and Jill Pawlowski
H 615
November 22, 2013
![Page 2: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/2.jpg)
Overview● Generalized Causal Inference: Methods for Multiple
Studies (Ch. 13)○ Multistudy programs of research○ Summaries of multiple studies
○ Narrative reviews ○ Quantitative reviews
● Critical Assessment of Assumptions (Ch. 14)● Reflection Activity (10 minutes)
![Page 3: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/3.jpg)
Choosing a psychotherapist• Imagine for a moment you are experiencing a difficult emotional time in
your life…
• Because you are an informed, intelligent person, you do some reading on psychotherapy and discover that many different approaches are available.
• These assorted styles of psychotherapy, although they stem from different theories and employ different techniques, all share the same basic goal:
to help you change your life in ways that make you a happier,
more productive, and effective person
http://www.edmondschools.net/Portals/3/docs/Terri_McGill/READ-TherapistChoice.pdf
![Page 4: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/4.jpg)
Choosing a psychotherapist• Which type of therapy should you choose?
• What you would really like to know is: • Does psychotherapy really work? • If it does, which method works best?
• While many comparison studies have been done, most of them:• support the method used by the psychologists conducting the study• were rather small • are spread over a wide range of books and journals
• This makes a fully informed judgment extremely difficult.
What should you do?
![Page 5: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/5.jpg)
You have 2 options
![Page 6: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/6.jpg)
Generalized causal inference: methods for multiple studies
• Single studies typically lack large, heterogeneous samples across persons, settings, times, treatments, and outcome measures; rarely use diverse methodologies
• Multiple studies usually have greater variation in all of the above = allows for better tests of generalized causal inference
http://library.downstate.edu/EBM2/2100.htm
![Page 7: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/7.jpg)
Generalizing from multiple studies
• How is this accomplished?• Multistudy programs of research• Summaries of multiple studies
![Page 8: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/8.jpg)
Multistudy programs of research
• Phased models• Same topic, same researcher/lab• Research proceeds in phases, from basic research discovery to
application in real world settings
• Directed programs of experiments• One researcher/one lab over time, or multiple researchers/multiple
labs at same time• Aim is to systematically investigate over many experiments the
explanatory variables (moderators, mediators, constructs) that may account for an effect, gradually refining generalizations
![Page 9: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/9.jpg)
Multistudy programs of research
• Phased models• Example: phases of cancer research
• Pre-phase: basic research on the effects of a treatment on cancer cells in test tubes and on animals
• Phase I: review existing basic and applied research to determine testable hypothesis about new cancer agent
• Phase II: method development to ensure accurate and valid research procedures and technology are available to study the agent
• Phase III: intervention efficacy trials• Phase IV: intervention effectiveness studies• Phase V: field studies aimed at entire communities to determine public
health impact
![Page 10: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/10.jpg)
Multistudy programs of research
• Phased models• Excel at exploring generalized causal inference• Rely on purposive sampling of persons, settings, times, outcomes, and
treatments• Address the 5 principles of causal explanation:
• Surface similarity: using human cancer cell lines to create comparable tumors on animals
• Ruling out irrelevancies: deliberately making patient characteristics diverse
• Making discriminations: identifying which kind of cancers are most/least affected
• Interpolation/extrapolation: varying drug doses to see how toxicity and clinical response may vary
• Causal explanation: developing models of how a drug acts on human cancer cells so that the potency of that action can be increased
![Page 11: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/11.jpg)
Multistudy programs of research
Advantages
• Control over aspects of generalized causal inference from study to study
• Pursuit of precisely the questions that need answering at any given time
Disadvantages
![Page 12: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/12.jpg)
Summaries of multiple studies• Same topic, different researchers• Allows more precise estimates of effects than could be had from
single studies• Better exploration of how causal relationships might change over
variation in study features• Helps clarify nature of relationship, boundaries, behavior within those
boundaries, and explanations• Wider knowledge base often results in more
credibility than claims based on single studies
![Page 13: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/13.jpg)
Narrative reviews of experiments
• Describe existing literature using narrative descriptions without attempting quantitative syntheses of study results
• Reviewer relies on statistical significance levels to draw conclusions about intervention effectiveness
• Summary of votes suggests whether treatment works • Also allows examination of potential moderators of the generalizability
of intervention effects
![Page 14: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/14.jpg)
Narrative reviews combining experimental and nonexperimental research
• Field experiments plus evidence from surveys, animal studies, and basic laboratory studies
• Purpose is to match evidence across studies to a pattern that would be consistent with and explain an effect and clarify its generalizability• Useful when direct experimental manipulation of the treatment would be
unethical or impossible with humans
• Results can be controversial
![Page 15: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/15.jpg)
Narrative reviews
Advantages
• Hypothesis generation• Thick description of a literature• Theory development with
qualitative categories and relationships among variables
Disadvantages
• Difficult to keep straight all of the relationships between effects and potential moderators
• Traditionally rely on box score summaries of results from significance tests of outcomes
• Imprecise descriptions of study results• Even more complex when trying to
examine relationships among outcomes and potential moderators
• Overwhelming!
![Page 16: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/16.jpg)
Quantitative reviews of existing research
• Glass (1976) coined the term meta-analysis • AKA “Mega-Silliness”, “Blossoming Nose Blemish”, and “Classic of Social Science”
• http://garfield.library.upenn.edu/classics1983/A1983QF87200001.pdf
• Meta-analysis converts each study’s outcomes to a common effect size metric, such that different outcomes have the same means and standard deviations.
• The common effect size metric can then be more readily averaged across studies
![Page 17: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/17.jpg)
Essential steps of meta-analysis
• Problem identification and literature review• Aim: develop clear research question and framework of
inclusion criteria for studies in the meta-analysis
• Studies included in review must address the question of interest and be relevant to the treatments, units, settings, measures, and times outlined in the guiding problem formation
• May also have methodological criteria
![Page 18: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/18.jpg)
Essential steps of meta-analysis
• Coding of studies• Meta-analyses use a common coding scheme to quantify study
characteristics and results
• Codes should reflect the researcher’s hypotheses• For example, in a review of interventions for childhood obesity, codes might be
developed based on whether the intervention included…• Behavioral, educational, environmental, diet, and/or physical activity features• Parental or family involvement• Outcome measures (e.g. BMI, change in diet or physical activity behaviors)
• Codes often include characteristics of the study report, participants, intervention, intervention process, and methodology
![Page 19: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/19.jpg)
Essential steps of meta-analysis
• Computing effect sizes• Various study outcomes must be converted to a common metric before
outcomes can be meaningfully compared over studies
• Two most appropriate effect size measures for meta-analyses of experiments:
• Standardized mean difference statistic (d) – for continuous outcome measures
di = Xti – Xc
i where Xti = mean of treatment group in ith study
si Xci = mean of comparison group in ith study
si = pooled standard deviation of the two groups
• Odds ratio (o) – for dichotomous outcome measures
oi = AD
BC
![Page 20: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/20.jpg)
Essential steps of meta-analysis
• Analyzing meta-analytic data• Theoretically, meta-analytic effect sizes are analyzed just like any other
social or behavioral data, using both descriptive and inferential statistics with univariate and multivariate techniques
• Unusual statistical features• Desirability of weighting effect size estimates by a function of study sample size• Use of tests for homogeneity of effect sizes• Hierarchical nature of meta-analytic data• Dependency of effect sizes within studies• Presence of publication bias
![Page 21: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/21.jpg)
Essential steps of meta-analysis
• Interpreting and presenting results• Generally, interpreting and presenting meta-analytic results poses few
special problems
• Keep in mind: most meta-analytic data are correlational• We never randomly assign studies to the categories analyzed• Quasi-experimental design features to rule out threats to validity are rarely used
in meta-analysis• Meta-analysts record data observationally• Exception is for reviews of randomized experiments of the same intervention
![Page 22: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/22.jpg)
Meta-analysis and the 5 principles of generalized causal inference
• Surface similarity: assess the match between research operations and the prototypical features of the targets of generalization
• Multiple studies typically represent many more constructs than any one study
• Greater chances of finding studies with operations that reflect constructs that may be of particular policy or research interest
• Reviews are superior to single studies in their potential to better represent prototypical attributes
![Page 23: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/23.jpg)
Meta-analysis and the 5 principles of generalized causal inference
• Ruling out irrelevancies: consider deviations from the prototype that might not change the causal conclusion
• Must show that a given causal relationship holds over irrelevant features present in multiple studies
• Some meta-analyses aim to decrease, not increase heterogeneity of irrelevancies
• Others welcome heterogeneity to the extent that it resembles that found in the settings of desired application
![Page 24: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/24.jpg)
Meta-analysis and the 5 principles of generalized causal inference
• Making discriminations: demonstrate that an inference holds only for the construct as specified, not for some alternative or respecified construct
• Aim is to make discriminations about the parts of the target persons, settings, measures, treatments, and time for which a cause and effect relationship will not hold
• Meta-analysis helps to clarify discourse about important constructs that have and have not been studied, and to clarify the range and boundaries of causal relationships
Relevance and irrelevance are always subject to revision over time
![Page 25: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/25.jpg)
Meta-analysis and the 5 principles of generalized causal inference
• Interpolation and extrapolation: specify the range of persons, settings, treatments, outcomes, and times over which the cause and effect relationship holds more or less strongly or not at all
• Requires exploration of the existing range of instances to discover how effect sizes vary with position along the range
• The extremes are likely to be further apart and the intermediate values more numerous than in a single study
• Response surface modeling works better with meta-analysis than with single studies
![Page 26: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/26.jpg)
Meta-analysis and the 5 principles of generalized causal inference
• Causal explanation: developing and testing explanatory theories about the target of generalization1. Easier to break down persons, settings, treatments,
and outcomes into their component parts in order to identify casually relevant components
2. Multiple regression can be used to identify redundancy among variables that moderate outcomes
3. Full explanation requires analysis of the micromediating causal processes that take place after a cause has been varied and before and effect has occurred
![Page 27: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/27.jpg)
Conclusion• “Our enthusiasm for the methodology must be tempered by realistic
understandings of its limitations.” (Shadish et al, 2002; pg. 446)
• Flaws of meta-analysis are less problematic than those of narrative reviews• Be no less or more critical of meta-analysis than of any other scientific method
![Page 28: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/28.jpg)
Quasi-experimentation: Ruling out Threats
• Relay heavily on researcher judgments especially on plausibility
• Improve inferences by adding design elements
![Page 29: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/29.jpg)
Quasi-experiments: Pattern matching • Description
• Intended to rule out specific identified alternative causal pathways
• Problems• Specificity of theory• Chance
• More likely with more complex predictions• Statistical tests for fit over mean differences
• not as well developed
![Page 30: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/30.jpg)
Excuses Against RCT• Problem: Quasi designs undermine likelihood of
conducting RCT
• RCT: Must be convinced of inferential-benefits over quasi before incurring costs
• Role of disciplinary culture
• “Each study calls for the strongest possible design, not the design of least resistance.” (pg. 487)
![Page 31: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/31.jpg)
RCT Objection: Successful Implementation
• Comparison not between well don RCT and Quasi but between imperfect RCT and well done Quasi
• Counter:• RCT’s may have better counterfactual even with treatment
degradation• Even with attrition, RCT give better effect size estimates
• ** “Careful study and judgment to decide” (pg. 489)
![Page 32: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/32.jpg)
RCT Objection: Strong Theory and Standardized Treatment Implementation
• Intervention-as-implemented
• Loss of statistical power • Large samples• Reduce influence of extraneous variables• Examine implementation quality
• Understanding of effectiveness in real world setttings
![Page 33: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/33.jpg)
RCT Objection: Unfavorable Trade Offs?
• Critiques• Conservative Standards • Descriptive over explanatory • Belated program evaluation
• Counters• Based on artificial dichotomies, correctible problems, and
oversimplifications
![Page 34: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/34.jpg)
RCT Objection: Invalid Model of Research Utilization
• Critique• Use of naïve rational choice model
• Counters• Causal relationships explained with information from multiple
sources• Contested rather than unanimous outcomes • Short-term instrumental use with minor variants on existing
practices • Research for conceptual changes
![Page 35: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/35.jpg)
RCT Objection: Experiment and Policy Conditions Differ
• Experiment outcomes affected by:• Enthusiasm • Implementation differs form real-world setting• Random-assignment
• Counters• Lack of generalizability is an empirical question• Representation of real world conditions is a tradeoff • Planning can improve similarities in populations • True of any methodology• Disruptions in implementation is true in locally invented novel programs
• Major Problem: Effects of policy can only be seen by studding the entire event
![Page 36: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/36.jpg)
RCT Objection: Imposing Treatment Vs. Local Solutions
• Critique:• Locally generated solutions produce better results
• Counters:• Experiments were needed to find out that imposed treatment did
not work • Null effects could be due to methodological weaknesses
• Distinguish between political-economic currency and intervention effects
![Page 37: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/37.jpg)
Causal Generation Theory • Randomization allows for causation with few assumptions
• Follow 5-principles of generalized causal inferences when randomization is not an option• Researchers rely to heavily on weak purposive alternatives
• Causal generalization is more complicated than establishing causal relationships
![Page 38: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/38.jpg)
Non-experimental Alternatives • Intensive causal studies
• Theory-based Evaluations
• Weaker quasi-experiments
• Statistical controls
![Page 39: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/39.jpg)
Review of Key Terms• How do we know what is true?
• Correspondence theory• Coherence theory • Pragmatism
![Page 40: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/40.jpg)
Review of Key Terms• Quine-Duhem thesis as a reaction to logical positivism• An epistemology is a philosophy of the justifications for
knowledge claims.• Naïve realism• Epistemological realism• Epistemological relativism
![Page 41: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/41.jpg)
Shadish, Cook & Campbell on “Facts”
• Critical multiplism• “Stubbornly replicable” observations by independent critics• Building multiple theories into observations
• Dialectic process• Example of psychotherapy
• Use and misuse of causal knowledge
![Page 42: CAUSAL INFERENCE FROM MULTIPLE STUDIES, CRITICAL ASSESSMENT OF ASSUMPTIONS Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November](https://reader034.vdocuments.site/reader034/viewer/2022051401/56649ebd5503460f94bc6893/html5/thumbnails/42.jpg)
Reflection• What is the role of theory in the construction of
knowledge?• How will you choose your research question?• What do you do when you end up with non-significant
results?• If you have significant results, how can you be confident in
your findings?• How will you approach issues around use and misuse of
your findings?