publication bias in meta-analysis selection method approaches
DESCRIPTION
PBTRANSCRIPT
Publication Bias in Meta-Analysis:Selection Method Approaches
Michaela Paul
Biostatistics UnitInstitute for Social and Preventive Medicine
University of Zurich
PhD seminar, 26 May 2009
Introduction Selection models using weight functions Copas selection model Example Summary
Outline
1 Introduction
2 Selection models using weight functions
3 Sensitivity approach of Copas
4 Example: The effects of environmental tobacco smoke
5 Summary
2 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Introduction
Effect size model
specifies what the distribution of the effect size would be if therewere no selection
Selection model
specifies the mechanism by which effect estimates are selected tobe observed
The selection model involves unknown parameters that govern theselection process:
estimate from observed effect size data (if possible)
assume specific values and carry out a sensitivity analysis
3 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Effect size model
Any model can be used, e.g. the random effects model
θi = θi + σiεi , θi ∼ N(θ, τ2), εi ∼ N(0, 1)
with parameters
θi estimated effect size
θ overall (true) treatment effect
σ2i within-study sampling variance
τ2 between study variance
Usually, the within-study sampling variance is assumed to beknown as σ2
i ≈ s2i , where si is the estimated standard error (SE).
4 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Selection models using weight functionsDenote
T ? random variable representing the effect estimateirrespective of selection
f (t|θ) density function of T ? (with true effect size θ)
w(t|ω) non-negative weight function depending onparameters ω
The weighted density of the observed effect size T is given by
g(t|θ, ω) =w(t|ω)f (t|θ)∫∞
−∞ w(t|ω)f (t|θ)dt
When w(t|ω) 6= const and a monotonic increasing function of t,the sampling distribution of T differs from that of T ?:
E[T ] = θ + bias︸︷︷︸>0
5 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Suggested weight functions
Idea
Decisions about the conclusiveness of research results are oftenbased on statistical significance.
⇒ The chance of a study being included in the meta-analysisdepends only on the p-value, i.e. on the ratio θi/si
Examples: (see Sutton et al.; 2000)
simple dichotomised weight function with p-value p = 0.05as cutpoint
parametrically decreasing weight function with cutpointp = 0.05
step function with (psychologically motivated) prespecifiedcutpoints (Hedges; 1992; Hedges and Vevea; 2005)
. . .
6 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Example: Hedges’ stepped weight functions
0.0
0.2
0.4
0.6
0.8
1.0
p−value
Pro
babi
lity
of o
bser
ving
effe
ct
0 0.1 0.25 0.35 0.5 0.65 0.75 0.9 1
weak 1−tailedweak 2−tailed
strong 1−tailedstrong 2−tailed
7 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Hedges’ stepped weight functions
If selection is a function of the one-tailed p-value pi = Φ(− θi
si
),
then
w(·) =
ω1 if 0 < pi < a1 ⇔ −siΦ−1(a1) < θi ≤ ∞
ωj if aj−1 < pi < aj ⇔ −siΦ−1(aj) < θi ≤ −siΦ
−1(aj−1)
ωk if ak−1 < pi < 1 ⇔ −∞ < θi ≤ −siΦ−1(ak−1)
where Φ is the standard normal cdf.
Set ω1 = 1: Weights must be relative rather than absolutebecause number of studies before selection occurs is unknown.
Parameter estimation via ML (Newton-Raphson).
Test for publication bias based on likelihood ratio testH0 : ωi = 1.
8 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
How well do weight functions work?
Number and location of cutpoints
locations not completely arbitraryat least one observed p-value must be within each interval
Performance of estimation procedure
reasonable starting values requirednumerical problems if number of studies is smallstandard errors of estimates can be very large
Alternative:Specify several weight functions and carry out a sensitivity analysis.
9 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Why go to the trouble of implementing somethingthat is complex and may not even work?
Simulation studies have shown (Hedges and Vevea; 2005):
Even when the selection model is poorly estimated, the associatedadjustment to the effect estimate can be quite accurate(provided the effect size model is well specified).
10 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Copas selection model
Idea (Copas and Shi; 2000a, 2001)
Keep as close as possible to usual ML random effects model,but add two parameters which describe study selection.
1 effect size model
θi = θi + σiεi , θi ∼ N(θ, τ2), εi ∼ N(0, 1)
2 selection model
Zi = γ0 +γ1
si︸ ︷︷ ︸ui
+δi , δi ∼ N(0, 1), corr(εi , δi ) = ρ
where residuals (εi , δi ) are assumed to be jointly normal andindependent across studies
θi is observed only when latent variable Zi > 0.
11 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Copas selection model
The observed treatment effects are modelled by the conditionaldistribution of θi in 1) given that Zi > 0.
E(θi |Zi > 0, si ) = θ + ρσiλ(ui ) where λ(·) = ϕ(·)Φ(·) is Mill’s ratio
Interpretation of parameters:
γ0, γ1(> 0) Inestimable parameters that control the marginalprobability that a study with within-study SE si ispublished.
ρ = 0 No publication bias: θi and Zi are independent.
ρ > 0 Selected studies will have Zi > 0⇒ δi , εi , θi are morelikely to be positive, overestimating the true mean θ.
12 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Estimation
The log-likelihood of Copas selection model is given by
l(θ, ρ, τ, γ0, γ1) =∑
i
log(P(θi |Zi > 0, si ))
=∑
i
log
(P(Zi > 0|θi , si ) P(θi )
P(Zi > 0|si )
)= . . .
where P(Zi > 0|θi , si ) = Φ
ui + ρσiθi−θσ2
i +τ2√1− ρ2σ2
i /(σ2i + τ2)
The nuisance parameters σ2
i = Var(θi |θi ) are replaced by their
sample estimates based on s2i = Var(θi |Zi > 0)
Asymptotic inference about θ for given γ0, γ1
13 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Goodness of fit test for residual publication bias
For a specific pair (γ0, γ1) for the selection model, we need tocheck that the resulting model gives a reasonable fit.
The expected values give a good fit to the data if theysatisfactorily predict any observed trend in the funnel plot.
This can be tested using an extended model for the treatmenteffect
θi = θi + σiεi + βsi
Likelihood ratio test H0 : β = 0.
14 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Sensitivity analysis
Marginal selection probability of a typical study with SE s:
Ps = P(Z > 0|s, γ0, γ1) = Φ(γ0 +γ1
s)
> # Convert selection probabilities into values (gamma0, gamma1)> tran <- function(p, se){+ q1 <- qnorm(p[, 1]); q2 <- qnorm(p[, 2])+ gamma1 <- (q1 - q2)/(1/min(se) - 1/max(se))+ gamma0 <- q1 - gamma1/min(se)+ return(cbind(gamma0, gamma1))+ }> (ps <- matrix(c(.99,.8,.6,.4,.2,.8,.5,.3,.1,.01), ncol=2))
[,1] [,2]
[1,] 0.99 0.80
[2,] 0.80 0.50
[3,] 0.60 0.30
[4,] 0.40 0.10
[5,] 0.20 0.01
15 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Sensitivity analysis
> hackshaw <- read.table("../data/data2_hackshaw.txt",header=T)> gamma01 <- tran(ps, se = hackshaw$se)
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0.0
0.2
0.4
0.6
0.8
1.0
se
sele
ctio
n pr
obab
ility
For any pair (γ0, γ1), θ can be estimated by ML.16 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Epidemiological evidence on lung cancer and passivesmoking
Meta-analysis reported by Hackshaw et al. (1997), reanalysed inCopas and Shi (2000b):
meta-analysis consists of 37 published epidemiological studies
each study provided an estimate of the odds ratio
Objective:Asses the epidemiological evidence for an increase in the risk oflung cancer resulting from exposure to environmental tobaccosmoke
Outcome measure:Relative risk of lung cancer among female lifelong non-smokers,according to whether her partner was a current smoker or a lifelongnon-smoker
17 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Funnel Plot
0.5 1.0 2.0 5.0
0.6
0.4
0.2
0.0
Odds Ratio
Sta
ndar
d er
ror
●●
●●●
●●● ●●●
●
●●●
●●●● ●● ●
●
●●
●●●
●●●
●
●●
●
●
●
18 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Analysis using copas
Copas selection model is implemented in the R package copas(Carpenter et al.; 2009).
> # loading library, requires library meta> library(copas)> # fit model> metaH <- metagen(lnOR,se,data=hackshaw, sm="OR")> copH <- copas(metaH,+ gamma0.range = NULL,+ gamma1.range = NULL,+ ngrid = 20, # grid for contourplot+ levels = NULL, # levels for contourplot+ slope = NULL # slope of ’orthogonal’ line+ )
19 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Explore sensitivity of conclusions for selection mechanismsof varying strength
> plot(copH, which = 2)
0.06 0.08
0.1 0.12
0.14
0.14
0.16
0.1
8
0.2
−0.45 0.04 0.53 1.02 1.51 2
00.
030.
060.
090.
120.
15
Values of gamma0
Val
ues
of g
amm
a1●
●●
●
●
●
Contour plot
20 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
How does treatment effect change with varying selection?
> plot(copH, which = 3)
1 0.9 0.8 0.7 0.6 0.5
00.
050.
10.
150.
20.
250.
3
Probability of publishing the trial with largest sd
log
OR
●
●
●
●
●
●
●
Treatment effect plot
21 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Which probabilities of publishing the trial with largest SEare most consistent with the model?
> plot(copH, which = 4)
●
●
●
●
●
●
●
1 0.9 0.8 0.7 0.6 0.5
00.
20.
40.
60.
81
Probability of publishing the trial with largest sd
P−
valu
e fo
r re
sidu
al s
elec
tion
bias
P−value for residual selection bias
22 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Results
> summary(copH)
Summary of Copas selection model analysis:
publprob OR 95%-CI pval.treat pval.rsb N.unpubl
1.00 1.2416 [1.1256; 1.3697] < 0.0001 0.037 0
0.93 1.2218 [1.1072; 1.3482] < 0.0001 0.0633 1
0.86 1.1973 [1.0863; 1.3197] 0.0003 0.1145 3
0.81 1.1737 [1.0678; 1.2902] 0.0009 0.167 5
0.69 1.1504 [1.0248; 1.2914] 0.0176 0.2452 11
0.59 1.1271 [0.9972; 1.2740] 0.0555 0.3583 17
0.48 1.1047 [0.9666; 1.2625] 0.1438 0.4975 27
Copas model (adj) 1.1973 [1.0863; 1.3197] 0.0003 0.1145 3
Random effects model 1.2378 [1.1294; 1.3566] < 0.0001
Legend:
publprob Probability of publishing the study with the largest SE
pval.treat Pvalue for hypothesis that treatment effect is equal in both groups
pval.rsb Pvalue for hypothesis that no further selection remains unexplained
N.unpubl Approx. number of studies the model suggests remain unpublished
23 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Conclusion
A modest degree of publication bias leads to a reduction inthe relative risk.
Although the number of unpublished studies is unlikely to belarge (see Copas and Shi; 2000b),the possibility of publication bias cannot be ruled out.
The published estimate of the increased risk needs to beinterpreted with some caution.
24 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
Summary
Simpler methods such as Trim and Fill can give misleadingresults if funnel plot asymmetry is due to other factors thanpublication bias.
Selection method approaches can account for such factorsand provide considerable insight into the problem ofpublication bias.
However, they are rarely used in practice partly due to thecomplexity of the methods and the rareness of user-friendlysoftware.
Carpenter et al. (2009) showed the practical utility of theCopas selection model and recommend using the modelroutinely for systematic reviews.
25 / 26
Introduction Selection models using weight functions Copas selection model Example Summary
References
Carpenter, J. R., Schwarzer, G., Rucker, G. and Kunstler, R. (2009). Empiricalevaluation showed that the Copas selection model provided a useful summary in80% of meta-analyses, J Clin Epidemiol 62(6): 624–631.
Copas, J. B. and Shi, J. Q. (2000a). Meta-analysis, funnel plots and sensitivityanalysis, Biostatistics 1(3): 247–262.
Copas, J. B. and Shi, J. Q. (2000b). Reanalysis of epidemiological evidence on lungcancer and passive smoking, BMJ 320(7232): 417–418.
Copas, J. B. and Shi, J. Q. (2001). A sensitivity analysis for publication bias insystematic reviews, Stat Meth Med Res 10(4): 251–265.
Hackshaw, A., Law, M. and Wald, N. (1997). The accumulated evidence on lungcancer and environmental tobacco smoke, BMJ 315(7114): 980–988.
Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis, Statist.Sci. 7(2): 246–255.
Hedges, L. V. and Vevea, J. L. (2005). Selection method approaches, in H. Rothstein,A. Sutton and M. Borenstein (eds), Publication Bias in Meta-Analysis: Prevention,Assessment Adjustments, Chichester, West Sussex: Wiley, pp. 145–174.
Sutton, A. J., Song, F., Gilbody, S. M. and Abrams, K. R. (2000). Modellingpublication bias in meta-analysis: a review, Stat Meth Med Res 9(5): 421–445.
26 / 26