workshop semiparametric frailty models coxph and coxme
TRANSCRIPT
WorkshopSemiparametric frailty models
coxph and coxme
The data set: Time to first insemination
• Database of regional Dairy Herd Improvement Association (DHIA)– Milk recording service– Artificial insemination– Select sample– Subset of 2567 cows from 49 dairy farms
Fixed covariates data setinsemfix.dat
Time-varying covariates data setinsemtvc.dat
Fitting semiparametric modelswith fixed covariates
setwd("c://docs//onderwijs//survival//Flames//")insemfix<-read.table("insemfix.dat",header=T)coxph(Surv(timeto,stat)~heifer+frailty(herd))
Call:coxph(formula = Surv(timeto, stat) ~ heifer + frailty(herd)) coef se(coef) se2 Chisq DF p heifer -0.24 0.0432 0.0430 30.9 1.0 2.7e-08frailty(herd) 205.3 40.9 0.0e+00
Iterations: 10 outer, 23 Newton-Raphson Variance of random effect= 0.123 I-likelihood = -16953
Degrees of freedom for terms= 1.0 40.9 Likelihood ratio test=281 on 41.9 df, p=0 n= 2579
HR=exp(-0.24)=0.79
Exercise
• Can we maximise the penalised partial likelihood at once?
• Depict the penalised partial likelihood as a function of theta
• The PPL is the second item of the named list loglik, e.g., for our example– insemfix$loglik[2]
Penalised partial log likelihood
0.06 0.08 0.10 0.12 0.14
-16
89
8-1
68
96
-16
89
4-1
68
92
-16
89
0
theta
pe
na
lise
d p
art
ial l
og
like
liho
od
detloglik<-function(x){coxph(Surv(timeto,stat)~heifer+frailty(herd,theta=x),data=insemfix)$loglik[2]}thetlist<-seq(0.05,0.15,0.005)logliklist<-sapply(thetlist,detloglik)plot(thetlist,logliklist,type='l',xlab='theta',ylab='penalised partial loglikelihood')
Exercise
• Do we obtain the maximum profile likelihood?
• Depict the profile likelihood as a function of theta– Theta can be fixed at x by frailty(herd,theta=x)– The marginal likelihood can be obtained as
insemfix$history[[1]]$c.loglik
Profile log likelihood
0.06 0.08 0.10 0.12 0.14
-16
95
4.5
-16
95
4.0
-16
95
3.5
-16
95
3.0
-16
95
2.5
-16
95
2.0
theta
pro
file
log
like
liho
od
detloglik<-function(x){coxph(Surv(timeto,stat)~heifer+frailty(herd,theta=x),data=insemfix)$history[[1]]$c.loglik}thetlist<-seq(0.05,0.15,0.005)logliklist<-sapply(thetlist,detloglik)plot(thetlist,logliklist,type='l',xlab='theta',ylab='profile loglikelihood')
Adjusting outer number of iterations
coxfit<-coxph(Surv(timeto,stat)~heifer+frailty(herd,eps=0.00000001),data=insemfix,outer.max=100)coxfitCall:coxph(formula = Surv(timeto, stat) ~ heifer + frailty(herd, eps = 1e-08), outer.max = 100)
coef se(coef) se2 Chisq DF p heifer -0.241 0.0431 0.043 31.1 1.0 2.4e-08frailty(herd, eps = 1e-08 194.5 38.5 0.0e+00
Iterations: 21 outer, 39 Newton-Raphson Variance of random effect= 0.0878 I-likelihood = -16952.1 Degrees of freedom for terms= 1.0 38.5 Likelihood ratio test=277 on 39.5 df, p=0 n= 2579
Time-varying covariates data
Contribution to the denominator:
tij=10 : 2.29tij=55 : 2.61
Fitting Cox models withtime-varying covariates (1)
#Read datasetwd("c://docs//onderwijs//survival//Flames//")insemtvc<-read.table("insemtvc.dat",header=T)coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001), outer.max=100,data=insemtvc)
Fitting Cox models withtime-varying covariates (2)
> coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001), outer.max=100,data=insemtvc)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + frailty(herd, eps = 1e-07), data = insemtvc, outer.max = 100)
coef se(coef) se2 Chisq DF p ureum -0.0525 0.0187 0.0186 7.85 1 0.0051frailty(herd, eps = 1e-07 1238.47 165 0.0000
Iterations: 35 outer, 82 Newton-Raphson Variance of random effect= 0.334 I-likelihood = -69868.9 Degrees of freedom for terms= 1 165 Likelihood ratio test=2066 on 166 df, p=0 n= 93608
Alternative semiparametric models
• Unadjusted and marginal semiparametric model
• Stratified semiparametric model
• Semiparametric frailty model
Unadjusted semiparametric modelureumtv.unadjust <-coxph(Surv(begin,end,stat)~ureum,data=insemtvc)summary(ureumtv.unadjust)Call:coxph(formula = Surv(begin, end, stat) ~ ureum, data = insemtvc)
n= 93608 coef exp(coef) se(coef) z pureum -0.0273 0.973 0.0162 -1.68 0.093
exp(coef) exp(-coef) lower .95 upper .95ureum 0.973 1.03 0.943 1.00
Rsquare= 0 (max possible= 0.779 )Likelihood ratio test= 2.83 on 1 df, p=0.0926Wald test = 2.82 on 1 df, p=0.0929Score (logrank) test = 2.82 on 1 df, p=0.0929
Marginal semiparametric modelureumtv.marg<-coxph(Surv(begin,end,stat)~ureum+cluster(herd),data=insemtvc)summary(ureumtv.marg)Call:coxph(formula=Surv(begin,end,stat)~ureum+cluster(herd),data=insemtvc) n= 93608 coef exp(coef) se(coef) robust se z pureum -0.0273 0.973 0.0162 0.0285 -0.957 0.34
exp(coef) exp(-coef) lower .95 upper .95ureum 0.973 1.03 0.92 1.03
Rsquare= 0 (max possible= 0.779 )Likelihood ratio test= 2.83 on 1 df, p=0.0926Wald test = 0.92 on 1 df, p=0.338Score (logrank) test = 2.82 on 1 df, p=0.0929, Robust = 0.91 p=0.339
Stratified semiparametric modelureumtv.strat<-coxph(Surv(begin,end,stat)~ureum+strata(herd),data=insemtvc)summary(ureumtv.strat)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + strata(herd), data = insemtvc)
n= 93608 coef exp(coef) se(coef) z pureum -0.0588 0.943 0.0198 -2.97 0.003
exp(coef) exp(-coef) lower .95 upper .95ureum 0.943 1.06 0.907 0.98
Rsquare= 0 (max possible= 0.444 )Likelihood ratio test= 8.86 on 1 df, p=0.00291Wald test = 8.83 on 1 df, p=0.00296Score (logrank) test = 8.83 on 1 df, p=0.00296
Semiparametric frailty modelureumtv.frail<-coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001),outer.max=100,data=insemtvc)summary(ureumtv.frail)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + frailty(herd, eps = 1e-07), data = insemtvc, outer.max = 100)
n= 93608 coef se(coef) se2 Chisq DF p exp(coef) lower.95 upper.95ureum -0.0525 0.0187 0.0186 7.85 1 0.0051 0.949 0.915 0.984frailty(herd, eps = 1e-07 1238.5 165 0.0000
Iterations: 35 outer, 60 Newton-Raphson Variance of random effect= 0.334 I-likelihood = -69868.9 Degrees of freedom for terms= 1 165 Rsquare= 0.022 (max possible= 0.779 )Likelihood ratio test= 2066 on 166 df, p=0Wald test = 7.85 on 166 df, p=1
Conclusions
• With adjustment, either by stratification or frailties, significant ureum effect
• Without adjustment, either with or without variance adjustment, no significant ureum effect
• Why?• Plot herd specific ureum effect as a
function of mean herd ureum concentration
Exercise
• Plot percentage events as a function of mean herd ureum concentration
• Using following scheme– Derive average ureum conc. for each cow
• avureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)
– Derive average ureum conc. for each herd– Derive percentage events– Plot
Mean ureum and %insemination
setwd("c://docs//onderwijs//survival//Flames//")insemtvc<-read.table("insemtvc.dat",header=T)herd<-insemtvc$herdnr;timeto<-(insemtvc$end*12/365.25)stat<-insemtvc$stat;heifer<-insemtvc$par2avureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)herd.cow<-tapply(insemtvc$herd,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)numinsem.herd<-tapply(insemtvc$stat,insemtvc$herd,sum)totanim.herd<-tapply(insemtvc$stat,insemtvc$herd,length)percinsem<- 100*numinsem.herd/totanim.herdplot(avureum.herd,percinsem,xlab="Ureum",ylab="Insemination (%)")
Exercise
• Plot frailty term as a function of mean herd ureum concentration
• Using following scheme– Save frailties from Cox frailty model– Derive average ureum conc. for each herd– Plot
Mean ureum and frailties
ureumtv.frail<-coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001),outer.max=100,data=insemtvc)avureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)herd.cow<-tapply(insemtvc$herd,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)stat.herd<-tapply(insemtvc$stat,list(insemtvc$herd),sum)results<-data.frame(frailty=ureumtv.frail$frail,ureum=avureum.herd,stat=stat.herd)results0<-results[results$stat==0,]plot(results$frailty,results$ureum,xlab="Frailty",ylab="Mean ureum concentration")points(results0$frailty,results0$ureum,pch=3)
Mean ureum and frailties plot
-4 -3 -2 -1 0 1
2.0
2.5
3.0
3.5
4.0
4.5
Frailty
Me
an
ure
um
co
nce
ntr
atio
n
Exercise
• Derive herd specific ureum effects– Fit the model for each herd separately
Herd specific treatment effectavureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)herd.cow<-tapply(insemtvc$herd,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)
fitmod<-function(herdnr){coxph(Surv(begin,end,stat)~ureum,subset=(insemtvc$herd==herdnr),data=insemtvc)$coef}
ureumeffect.herd<-sapply(herd.cow,fitmod)Error in fitter(X, Y, strats, offset, init, control, weights = weights, : Can't fit a Cox model with zero failures
Exercise
• Refit the frailty model and the marginal model leaving out the herds without inseminations
Results deleting farms without inseminations
insemtvcr<-insemtvc[insemtvc$herd!=1302100819 & insemtvc$herd!=1302310377 & insemtvc$herd!=3103311094 & insemtvc$herd!=3402102723 & insemtvc$herd!=3403801839 & insemtvc$herd!=4300110030,]ureumtvcr.unadjust<-coxph(Surv(begin,end,stat)~ureum,data=insemtvcr)summary(ureumtvcr.unadjust)Call:coxph(formula = Surv(begin, end, stat) ~ ureum, data = insemtvcr)
n= 90232 coef exp(coef) se(coef) z p lower .95 upper .95ureum -0.0433 0.958 0.0163 -2.65 0.008 0.927 0.989 Rsquare= 0 (max possible= 0.789 )Likelihood ratio test= 7.04 on 1 df, p=0.00795Wald test = 7.02 on 1 df, p=0.00804Score (logrank) test = 7.02 on 1 df, p=0.00805
Results deleting farms without inseminations - alternative
insemtvcr<-insemtvc[insemtvc$herd!=1302100819 & insemtvc$herd!=1302310377 & insemtvc$herd!=3103311094 & insemtvc$herd!=3402102723 & insemtvc$herd!=3403801839 & insemtvc$herd!=4300110030,]ureumtvcr.unadjust<-coxph(Surv(begin,end,stat)~ureum,data=insemtvcr)summary(ureumtvcr.unadjust)Call:coxph(formula = Surv(begin, end, stat) ~ ureum, data = insemtvcr)
n= 90232 coef exp(coef) se(coef) z p lower .95 upper .95ureum -0.0433 0.958 0.0163 -2.65 0.008 0.927 0.989 Rsquare= 0 (max possible= 0.789 )Likelihood ratio test= 7.04 on 1 df, p=0.00795Wald test = 7.02 on 1 df, p=0.00804Score (logrank) test = 7.02 on 1 df, p=0.00805
Deleting farms without inseminations
#alternativeherdstat<-data.frame(herd=names(stat.herd),numinsem=stat.herd)insemtvcr2<-merge(insemtvc,herdstat,by="herd")
Exercise
• Fit the frailty model assuming the lognormal distribution for the frailties
Fitting the shared normal frailty model
coxph(Surv(begin,end,stat)~ureum+frailty(herd,dist="gaussian",eps=0.00001),outer.max=100,data=insemtvc)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + frailty(herd, dist = "gaussian", eps = 1e-05), data = insemtvc, outer.max = 100)
coef se(coef) se2 Chisq DF p ureum -0.0513 0.0188 0.0186 7.48 1 0.0062frailty(herd, dist = "gau 1581.56 165 0.0000
Iterations: 6 outer, 21 Newton-Raphson Variance of random effect= 0.411 Degrees of freedom for terms= 1 165 Likelihood ratio test=2030 on 166 df, p=0 n= 93608
Lognormal frailties versus normal random effects
• Assume U has lognormal distribution, with U=exp(W) and W~N(m,s2)
• The gamma density with mean 1, variance q is given by
• For q = 0.1, 0.5, 1, 1.5, plot the gamma density function and lognormal density chosing values for m and s2 so that– Var(exp(W))= q and m = 0– Var(exp(W))= q and E(U)=1
2
2
2 2
logexp
2
1)(
i
i
iU
u
uuf
1
exp1
11
iiiU
uuuf
1exp2expVar
5.0expE22
2
U
U
Lognormal frailties versus normal random effects (2)
• With m equal to 0, we have
• With E(U)=1, we have
1expexpVar
5.0expE22
2
U
U
2
411log2
1log
5.02
2
1exp2expVar
5.0expE22
2
U
U
Plotting densitiestheta<-c(0.1,0.5,1,1.5)s1<-log((1+sqrt(1+4*theta))/2)s2<-log(theta+1)m2<- -0.5*s2u<-seq(0,3,0.01)for (i in 1:length(u)){dlnorm.theta0.1.EWeq0<-dlnorm(u,0,sqrt(s1[1]))dlnorm.theta0.5.EWeq0<-dlnorm(u,0,sqrt(s1[2]))dlnorm.theta1.EWeq0<-dlnorm(u,0,sqrt(s1[3]))dlnorm.theta1.5.EWeq0<-dlnorm(u,0,sqrt(s1[4]))dlnorm.theta0.1.EUeq1<-dlnorm(u,m2[1],sqrt(s2[1]))dlnorm.theta0.5.EUeq1<-dlnorm(u, m2[2],sqrt(s2[2]))dlnorm.theta1.EUeq1<-dlnorm(u, m2[3],sqrt(s2[3]))dlnorm.theta1.5.EUeq1<-dlnorm(u, m2[4],sqrt(s2[4]))dgam.theta0.1.EUeq1<-dgamma(u,1/theta[1], 1/theta[1])dgam.theta0.5.EUeq1<- dgamma(u,1/theta[2], 1/theta[2])dgam.theta1.EUeq1<- dgamma(u,1/theta[3], 1/theta[3])dgam.theta1.5.EUeq1<- dgamma(u,1/theta[4], 1/theta[4])}
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.0
0.5
1.0
1.5
2.0
u
Den
sity
theta=0.1
gammalnormEU=1lnormEW=0
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.0
0.5
1.0
1.5
2.0
u
Den
sity
theta=0.5
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.0
0.5
1.0
1.5
2.0
u
Den
sity
theta=1
0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.0
0.5
1.0
1.5
2.0
u
Den
sity
theta=1.5