Genetic risk prediction for CHD: will we ever get there or are we already there?

Themistocles (Tim) Assimes, MD PhD Assistant Professor of Medicine

Stanford University School of Medicine

WHI Investigators meeting May 7, 2015

Disclosure: CRA with Telomere Diagnostics Inc.

Adapted from Thanassoulis, G. and R.S. Vasan, Genetic cardiovascular risk prediction: will we get there? Circulation, 2010. 122(22): p. 2323-34.

• Initial GWAS: Complex diseases have more complex genetic architectures then expected • not the best situation for risk prediction

Most widely used and recognized test of discrimination – C statistic

• AKA Receiver-operating-curve (AUC) – Sensitivity vs 1-Specificity – Probability among a randomly selected case and

control, that the case will have a higher model-based predicted probability of an event

• 0.5 = chance 1.0 = perfect

• “standard” metric for binary outcomes • Limitations

– Big differences in risk = small differences in risk

Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29 –36.

AUC tough to budge when its already reasonably good

Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008;54:17-23.

Pitfall of relying only on AUC: some TRFs would not be included in current scores

+ indicates the addition of each variable separately to the model with age, SBP, smoking only

Adapted from Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928-35.

General concept of reclassification Who moves and to where with new model?

Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928-35.

But what if you moved a subject inappropriately? E.g. move case into a lower category of risk

New discrimination tests – NRI and cNRI introduced in 2008

• Consider absolute predicted risk of individuals (“reclassification”) – ≥ 2 category Net reclassification index (NRI) – clinical NRI (cNRI)

• The NRI for the intermediate category of risk only

• US Preventative Services Task Force endorsed concept of reclassification

Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72; discussion 207-12. Cook, N.R., Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine. Stat Med, 2008. 27(2): p. 191-5. Helfand M, Buckley DI, Freeman M, et al. Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force. Ann Intern Med 2009;151:496-507.

NRI for adding HDL to Framingham

Adapted from Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72; discussion 207-12.

AUC: 0.762 (without HDL), 0.774 (with HDL), ΔAUC p-value=0.092.

Bias in the cNRI

• Clinical NRI – widely used but found to be biased

• Overall NRI could be negative and the clinical NRI highly positive (including several GRS papers)

– Correction for Clinical NRI correction • Wiped out signal in 2 high profile CVD risk predictions

papers

Paynter, N.P. and N.R. Cook, A Bias-Corrected Net Reclassification Improvement for Clinical Subgroups. Med Decis Making, 2012

cNRI bias in CHD risk prediction

Ripatti S, Tikkanen E, Orho-Melander M, Havulinna AS, Silander K, Sharma A, et al. A multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses. Lancet. 2010 Oct 23;376(9750):1393-400. PubMed PMID: 20971364.

Questioning the utility of the NRI

• NRI is not a proportion – Only the NRIevents and NRInonevents – Combining as a simple sum or not appropriate and

potentially misleading • If one weights overall prevalence of events and non-events,

NRI can easily move from positive to negative territory • look at each separately and consider clinical consequences

• ≥3 category NRI doesn’t consider large jumps in risk any differently than small jumps

Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology. 2014 Jan;25(1):114-21. PubMed PMID: 24240655 Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med. 2014 Jan 21;160(2):122-31. PubMed PMID: 24592497

Large and Small Values for NRI>0 Are Undefined

Kerr, K.F., et al., Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology, 2014. 25(1): p. 114-21.

Questioning that statistical properties of NRI>0

• high false positive rate even with independent test set – Can Make Uninformative New Markers

Appear Predictive – Especially if models not well calibrated – But not the case for AUC or for likelihood ratio

testing

Kerr, K.F., et al., Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology, 2014. 25(1): p. 114-21. Pepe, M.S., H. Janes, and C.I. Li, Net risk reclassification p values: valid or misleading? J Natl Cancer Inst, 2014. 106(4): p. dju041

Concerns with testing the nulls for NRI

• H0 : NRI = 0, z-statistic has never been validated • t t

• For 2-category NRIevent or NRInon-event at a given risk

threshold – cannot reject H0: NRIevent = 0 and H0 : NRInon-event = 0

on the basis of Y being a risk factor. – Tests not yet established for these nulls

Kerr, K.F., et al., Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology, 2014. 25(1): p. 114-21. Pepe, M.S., H. Janes, and C.I. Li, Net risk reclassification p values: valid or misleading? J Natl Cancer Inst, 2014. 106(4): p. dju041

Back to the AUC??

• recent insights on testing whether a new model is better than the old one – EQUIVALENT NULL HYPOTHESES

• H0: risk (X,Y) = risk (X) • H0: AUC(X,Y) = AUC(X)

• Recommend standard regression statistics – No need to test null > 1 x – Superior power with Highly developed likelihood-

based tests – Avoid inconsistent results from inference that has not

been worked out as well for other methods

Pepe, M.S., et al., Testing for improvement in prediction model performance. Stat Med, 2013. 32(9): p. 1467-82.

Back to the AUC ??

• Testing the AUC – Much more work needed re: properties of tests – Delong or resampling based tests do not adjust for

variability in est. regression coefficients – VERY CONSERVATIVE (low power) - even after

bootstrap – Is this why AUC is insensitive to improvements in

prediction performance?

Pepe, M.S., et al., Testing for improvement in prediction model performance. Stat Med, 2013. 32(9): p. 1467-82.

Performance of GRS for CHD today

• Many examples of robust and relatively consistent association with GRS using ~45-50 GWAS SNPs Cohort GRS RR (95% CI) Comparison # events

ARIC 1.29 (1.2-1.4) Per SD GRS 620

Finnish Cohorts 1.27 (1.20–1.35) Per SD GRS 1093

6 Swedish Cohorts 1.54 (1.24-1.87) 4th quartile to 1st 781

Goldstein, B.A., et al., Simple, standardized incorporation of genetic risk into non-genetic risk prediction tools for complex traits: coronary heart disease as an example. Front Genet, 2014. 5: p. 254 Tikkanen, E., et al., Genetic risk prediction and a 2-stage risk screening strategy for coronary heart disease, in Arterioscler Thromb Vasc Biol. 2013. p. 2261-6. Ganna, A., et al., Multilocus Genetic Risk Scores for Coronary Heart Disease Prediction. Arterioscler Thromb Vasc Biol, 2013. 33(9): p. 2267-2272.

Then what?

• Need to estimate the extent of improvement – Big debate as to how to quantify improvement – One strong recommendation: net benefit (NB)

• Good news, if you reject – H0: risk (X,Y) = risk (X)

• You also reject – H0: NB(X,Y) (t) = NB(X) (t) where t is risk threshold – Test of equality of decision curves

Pepe, M.S., et al., Testing for improvement in prediction model performance. Stat Med, 2013. 32(9): p. 1467-82. Vickers, A.J. and E.B. Elkin, Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making, 2006. 26(6): p. 565-74.

Net benefit analysis of treating with rosuvastatin in the JUPITER trial

Dorresteijn, J.A., et al., Estimating treatment effects for individual patients based on the results of randomised clinical trials. BMJ, 2011. 343: p. d5888.

Comparison of HRs for the last CHD risk factor added to the model

Goldstein, Salfati, Yang, Assimes, under preparation

My optimistic viewpoint for genetic risk prediction in CHD

• ++ Markers – reproducible and stable • ++ Safe and effective interventions • We have already overcome initial analytic challenges

– GRS with robust association – comparable to other risk factors, will continue to improve – Net benefit likely present when it comes to statin Rx – clinical trial to test GRS ?

• Value to just having a better calibrated model to convey risk?

• the main impediment to implementation is cost • genotyping / sequencing • Technical – genetic data & lgorithms into EHRs