implications for meta-analysis literature comparison of weights in meta-analysis under realistic...
TRANSCRIPT
Implications for Meta-analysis Literature
Comparison of Weights in Meta-analysis Under Realistic Conditions Michael T. Brannick Liu-Qin Yang Guy Cafri
University of South Florida
Abstract Study Design
Important Notes
Study Purpose
• The overall effect size in meta-analysis is a weighted mean. Does it matter what weights we use?
Study Background— Other Weighting Schemes
• Hedges & Vevea’s (1998) approach in r
• Shrunken Estimates in r (Empirical Bayes)
• Combined Estimates in r: REVC by H&S; by H&V
• Unit Weights in r: The baseline
• Hunter & Schmidt use N; Hedges converts to z and uses N-3
• Study Background— Realistic Simulation• This simulation was based on published meta-analyses, so that values of k, N, rho ( ), and REVC ( ) would be representative of I/O meta-analyses.
We compared several weighting procedures for random-effects meta-
analysis under realistic conditions. Weighting schemes included unit,
sample size, inverse variance in r and in z, empirical Bayes, and a
combination procedure. Unit weights worked surprisingly well, and the
Hunter and Schmidt (2004) procedures worked best overall.
Results
• Published M-As• AMJ, JAP and Personnel Psychology; 1979-2005
• Inclusion criterion: effect sizes (r) available or available after conversion
• 48 M-As and 1837 effect sizes
• Inter-rater reliability: 1.0 – Ns; .99 – effect sizes (r)
• Simulation conditions formed by characteristics of published meta-analyses• Average N (N_bar) and the skewness of N distribution (N_skew) for each M-A
• A median of 168.57 for the distribution of N_bar (sampling distribution)
• A median of 2.25 for the distribution of N_skew (sampling distribution)
• Four conditions along the medians (Figure 1)
• Sampling studies for the Monte Carlo• A published M-A was randomly chosen, its K and Ns were used for that
simulation. The parameters for the simulations were chosen from:
• Choice of parameters
• The distribution of | |: 10th, 50th , and 90th percentile = .10, .22, .44, respectively
• The distribution of : 10th, 50th, and 90th percentile = .0005, .0128, and .0328
• 3 ( ) by 3 ( ) of parameter conditions• Therefore, the parameters in the simulation represent published studies
• Data generation• A Monte Carlo program written in SAS IML
• Picked an M-A under one condition of Figure 1, then picked a parameter combination
• Sampled r from a normal distribution of that and
• Meta-analyzed those sampled r(s); repeated 5000 times
• Estimators• H&S (2004) in r, H&V (1998) in z, and the other 4 approaches as described earlier
• Data analysis• and were estimated with each of 6 approaches
• Root-mean-square-difference (RMSR) between the parameter and the estimate
•Skewness in the distribution of Ns was shown to have little effect, and so simulations were rerun with only the high/low levels of N considered
•Figures 2, 4, and 6 show the empirical sampling distributions of the population mean estimates
•Figure 3, 5, and 7 show the empirical sampling distributions of the REVC estimates
The design elements had their generally expected impacts on the estimates
•The empirical sampling distributions were generally more compact with big Ns
•The means got larger when the underlying parameters increased
•The variance of the distribution increases as increases
•Provided a database and quantitative summary of published M-As of interest
•Monte Carlo simulation based on representative study characteristics
•Weights only matter when k and N are small
Conclusions
•Unit weights had surprisingly good estimates, esp. when and are large
•H&V (1998) in z performed as expected— slight overestimates
•H&S (2004) in r worked best for estimating overall mean and REVC
Study Purposes and Study Background
•Random-effect models were applied in the current study
•Sampled actual numbers of studies (K) and sample sizes (N) from the published M-As
•Used population parameters representing published M-A data
22 )1(
)1(
i
ii r
Nw
1
)1( 22
t
M N
rV
2ˆ1
MM Vw
Mi
Miii wN
rwrNEB
LN_LSLN_HSHN_LSHN_HS
1000
800
600
400
200
0Stu
dy
sa
mp
le s
izes
fro
m e
ac
h m
eta
-an
aly
sis
2ˆ
44.,22.,10. 18.,11.,02.
2
HVz1
C2C1
EB2EB1
HVz2HVr2HVr1
SH2SH1
UW2UW1
.100
0.75
0.6
0.45
0.3
0.15
0.0
Est
imat
ed R
ho
Rho=.1
.030 .017 .023 .013.028.013.043.013.024.013.027.013 RMSR
Mean of Rho estimates.099 .105.102.101.105.100
HVz1C2
C1EB2
EB1HVz2HVr2
HVr1SH2
SH1UW2
UW1
0.75
0.6
0.45
0.3
0.15
0.0
Est
imat
ed R
ho
Rho=.22
.028.037.040.055.028.034.028.033.036.036 .037.028 RMSR
Mean of Rho estimates.217 .231.236.224.231.218.219 .224.230.223.224.219
HVz1
C2C1
EB2EB1
HVz2HVr2HVr1
SH2SH1
UW2UW1
0.75
0.6
0.45
0.3
0.15
0.0
Est
imat
ed R
ho
Mean of Rho estimates
RMSR
Rho=.44
.043 .049.174.051.049.053.040 .041.159.049.041.057
.435 .458.571.464.458.437.439 .446.553.462.446.439
HVz1 C2
C1EB2
EB1HVz2HVr2
HVr1SH2
SH1UW2
UW1
0.12
0.11
0.1
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.0
Est
imat
ed t
au-s
qu
are
Tau-square= .0005
RMSR.001.004.000.016.001.004.001.03.001.017 .003.006
.011 .001.002.000.003.001.002.001.008.001.001.004
RMSR
Mean of Tau-sqaure estimates
HVz1 C2
C1EB2
EB1HVz2HVr2
HVr1SH2
SH1UW2
UW1
0.12
0.11
0.1
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.0
Est
imat
ed ta
u-sq
uare
Tau-square= .013
Mean of Tau-sqaure estimates
.016 .008 .005.006.007.007.005 .007.013.009.025.007 RMSR
.021 .015 .011.010.014.013.011 .012.012.014.020.011
HVz1 C2C1
EB2EB1
HVz2HVr2HVr1
SH2SH1
UW2UW1
0.12
0.11
0.1
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0.0
Est
imat
ed ta
u-sq
uare
Tau-square= .033
Mean of Tau-sqaure estimates
RMSR.017 .014.018.057.032.013
.039
.011 .014.016.058.026.013
.033 .032.034.067.045.030 .031.032.066.042.029
Good estimator
Good Estimator
Distributions of sample sizes from published meta-analyses
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7