RMF – Meta-analysis workshop (Marsh, O’Mara, Malmberg) 1

NCRM Research Methods Festival

University of Oxford

Dept of Education, University of Oxford

Traditionally, education researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue:

1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation.

2. Systematic review -- systematic, qualitative review of published research in a particular field

3. Secondary Data Analyses -- using large (typically public) databases

4. Re-analyses of published studies -- often in ways critical of the original study.

2

Systematic synthesis of various studies on a particular research question

Do boys or girls have higher self-concepts? Collect all studies relevant to a topic

Find all published journal articles on the topic An effect size (the ‘dependent variable’) is calculated for each

outcome Determine the size/direction of gender difference for each study

“Content analysis” code characteristics of the study; age, setting, ethnicity, self-concept domain (math, physical, social), etc.

Effect sizes with similar features are grouped together and compared; tests moderator variables

Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc.

3

Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative)

Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative)

Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative)

4

Compared to traditional literature reviews:

(1) there is a definite methodology employed in the research analysis; and

(2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis.

Therefore less biased and more replicable

5

Increased power: increases the chance of detecting a true treatment effect

Improved precision: with more information than a single study, the treatment effect estimate is improved

When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies

Easy to interpret summary statistics (useful if communicating findings to a non-academic audience)

6

The essence of good science is replicable and generalisable results.

Do we get the same answer to important research questions when we run the study again?

The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question.

Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency?

7

Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines

Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it

Relied upon by policymakers

Important that we understand the method, whether we conduct or consume meta-analytic research

Should be one of the topics covered in all introductory research methodology courses

8

There exists a critical mass of comparable studies designed to address a common research question.

Data are presented in a form that allows the meta-analyst to compute an effect size for each study.

Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study.

9

10

The number of meta-analyses is increasing at a rapid rate.

11

12

Psychology: Citations

Psychology: Articles

Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis . Psychological Bulletin, 110, 26-46. Times Cited: 471

Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis . Child Development, 56, 1479-1498. Times Cited: 570

Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis . Psychological Bulletin, 89, 47-62. Times Cited: 426

Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review . Personnel Psychology, 44, 703-742 Times Cited: 387

Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis . Psychological Bulletin, 104, 53-69. Times Cited: 316

Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis . Psychological Bulletin, 97, 251-273. Times Cited: 263.

13

De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment . Child Development, 68, 571-591. Times Cited: 340

Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development: The truth about false belief . Child Development, 72, 655-684. Times Cited: 276

Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups . Review of Educational Research, 64, 1-35. Times Cited: 235

Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990 . Health Education Research, 7, 403-430. Times Cited: 207

Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer-Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, 525-544. Times Cited: 198.

14

Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research . Journal of Consumer Research, 15, 325-343. Times Cited: 515

Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings . Organizational Behavior and Human Decision Processes, 36, 16-78. Times Cited: 401

Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings . IEEE Transactions On Engineering Management, 29, 28-4. Times Cited: 269.

Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7, 385-425. Times Cited: 203.

Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis . Journal Of Marketing Research, 22, 103-118. Times Cited: 189.

15

Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1-12. Times Cited:2008

Boushey Cj, Beresford Saa, Omenn Gs, Et . Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, 1049-1057. Times Cited: 2,128

Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, 899-909. Times Cited:1,591

Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, 1-29. Times Cited: 1,422

16

Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant.Results of 375 controlled evaluations of psychotherapy

and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy.

On the average, the typical therapy client is better off than 75% of untreated individuals.

Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral).

17ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)17

18

Need to have explicit inclusion and exclusion criteria

The broader the research domain, the more detailed they tend to become

Refine criteria as you interact with the literature

Components of a detailed criteria distinguishing features research respondents key variables research methods cultural and linguistic range time frame publication types

19

Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC)

Examine the reference lists of included studies to find other relevant studies

If including unpublished data, email researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International

20

Random selection of papers coded by both coders

Meet to compare code sheets

Where there is discrepancy, discuss to reach agreement

Amend code materials/definitions in code book if necessary

May need to do several rounds of piloting, each time using different papers

21

__ Study ID

_ _ Year of publication

__ Publication type (1-5)

__ Geographical region (1-7)

_ _ _ _ Total sample size

_ _ _ Total number of males

_ _ _ Total number of females

22

Publication type (1-5)1.Journal article2.Book/book chapter3.Thesis or doctoral dissertation4.Technical report5.Conference paper

1

99

2

1

87

41

46

Code SheetCode Book/manual

ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)

The effect size makes meta-analysis possibleIt is the “dependent variable”It standardizes findings across studies such that they

can be directly compared

Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but mustbe comparable across studies (generally requires

standardization)represent the magnitude and direction of the relationship

of interestbe independent of sample size

23

Means and standard deviations

Correlations

P-values

F-statistics

d

t-statistics

“other” test statistics

Almost all test statistics can be transformed into an standardized effect size “d”

ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)24

Lipsey & Wilson (2001) present many formulae for calculating effect sizes from different information

All effect sizes in a single meta-analysis must be in a common metric, typically based on the “natural” metric given research in the area. E.g.: Standardized mean difference Odds-ratio Correlation coefficient

25

26

Each study is one line in the data base Effect size Duration Sample sizes Reliability of

the instrumentVariance of the effect size

There are various ways of analysing meta-analytic data

Three main methods based on different statistical assumptions: Fixed effects models Random effects models Multilevel models

These will be discussed in the afternoon workshop

27

Meta-analysis is a method for synthesising and analysing the research literature on a particular topic

The essence of good science is replicable and generalisable results.

Increasingly sophisticated

Continuously evolving

For more information about the meta-analysis training courses that we offer, please see http://education.ox.ac.uk/research/resgroup/self/training.php

28