Risk Factors and Risk Models: Why We Need Them, and How

We Develop and Use ThemMarch 21, 2018

David M. Shahian, MDVice-President, Center for Quality and Safety, MGH

Professor of Surgery, Harvard Medical SchoolChair, STS Quality Measurement Task Force

October 24, 2019

No relevant disclosures

“…so many operations of such and such a nature, without reference to age, sex, or cause of the operation, followed by so many deaths, without reference to age, sex, or complications. Given these elements, divide the one by the other, and you get the mortality…

A statistical proceeding such as this can at best lead to loose approximations. It can convey but a very imperfect idea of the real state of the case. And one thing is quite certain, that it can lead to no practical result whatever, either as regards the true causes of the mortality, or how these might be mitigated.”

Nightingale 1863, cited by Spiegelhalter 1999

Nightingale on the need for risk adjustment

VA quality assurance legislationDecember 3,1985

The Chief Medical Director shall compare the mortality and morbidity rates … with the national mortality and morbidity standards …. and analyze any deviation between such rates and such standards in terms of (i) the characteristics of the respective patient populations; (ii) the level of risk for the procedure involved, based on (I) patient age; (II) the type and severity of the disease; (III) the effect of any complicating diseases; and (IV) the degree of difficulty of the procedure; and (iii) any other factor that the Chief Medical Director considers appropriate

“Apart from the proportion of a hospital’s

cases in 80 DRG’s, the predictive models

had no measure of case severity based on

diagnosis or procedure…Model bias also

favored lower-risk hospitals.”

Ann Thorac Surg 1988;45:348-349

“In March 12, 1986, the Department of Health and Human Services Health Care Financing Administration (HCFA) released a list of hospitals whose mortality rates for Medicare patients allegedly exceeded ”predicted mortality rates” for those hospitals, either overall or for nine specific diagnostic categories. …the implication was clearly made that these raw mortalityrates were equated with quality of care in the institutions in question…an incorrect perception of care in certain communities…. All of the risk factors that are predictive of operative mortality must be identified and subjected to appropriate statistical analysis before comparisons of mortality rates between institutions can be made”

Ann Thorac Surg 1989

“…prompted by the release by HCFA…of raw mortality data for Medicare patients undergoing coronary artery bypass grafting procedures without respect to any of the then-known risk factors …”

Risk models are a core elementof the STS quality program

Risk factor data fields

Risk model considerations

• How do we initially pick risk factors for the Database? Endpoint (e.g., mortality)for which we are adjusting should have at

least moderate incidence (e.g., >1%) Permissible risk factors for different uses Rare risk factors very difficult if not impossible to model Low risk factor missingness, < 5% (preferably < 1-2%) Bivariate associations interesting but not determinative Harrell’s 10:1 (endpoints:variables) rule helps avoid overfitting

• Model development and testing full versus parsimonious models forward or backward selection, stopping rules, penalties, clinically

“supervised” selection calibration, discrimination

• Model uses

Permissible risk factors

Profiling• Only include risk factors present before treatment begins• Do not include treatments or complications• SES currently being studied, may be applicable to some

outcomes such as readmission

Performance improvement (e.g., readmission prediction and mitigation) or shared decision making• Permissible and desirable to use all available data

Hg A1c MELD score 5 meter walk Aortic etiology Aortic stenosis gradient Mitral etiology

Some candidate variables with high missing rates

Logistic regression—logit and probability forms

log odds (logit) intercept coefficients

log[p/(1-p)] = b0 + b1x1 +….. + bkxkrisk factors

Probability form of the same equation1

intercept coefficients

p =

Risk models: why do we talk about risk in terms of coefficients, odds and odds ratios?

Odds: probability of an event occurring divided by probability of not occurring (p/1-p)

Odds of tails on a coin toss is 0.5 / 0.5 = 1

Odds of rolling a 2 with a single six-sided dice:1/6 ÷ 5/6 = 1 ÷ 5 = 0.2

Risk models: What are odds ratios?

Odds ratio (OR): odds of the event when the risk factor is present compared to when it is not present

Consider an endpoint (e.g., death) occurring 2% of the time when some specific risk factor (e.g., a recent heart attack) is absent• Odds = 2% ÷ 98% = 0.0204• If the odds ratio for that risk factor is 2, it means

that the odds of death doubles to 0.0408 when the risk factor is present

Derivation of odds ratios from logistic equations

log[p/(1-p)] = b0 + b1x1 +….. + bkxk

Conveniently, odds ratios are mathematically directly related to the coefficients (b) in the logistic model

Odds Ratio (OR) = exp (b) = eb

and e0 = 1

Severe chronic lung disease

Coefficient = 0.85513

Odds ratio = e0.85513 = 2.35

What information does the risk factor odds ratio provide?

• Whether it has a positive or negative effect on the outcome:OR > 1 (beta coefficient +): when risk factor is present, outcome more likelyOR < 1 (beta coefficient -): when risk factor is present, outcome less likely

• How strong is its independent effect in comparison with all the other predictors when they are all considered simultaneously

Testing risk models Calibration (good fit: observed versus expected)

Discrimination (classification—fatality or survivor)

Calibration assesses prediction accuracy—of the days we predict 40% probability of rain, it should rain 40% of the time

Discrimination is whether or not it will rain (0 or 1, measured over the range of threshold probabilities—i.e., “Classify a day as rainy if the probability is greater than x %”)

Can have high discrimination but poor calibration, and vice versa

Graphical assessment of calibration

ROC curve area, c-index

Grunkemeier et al, ATS 2001

Multiple uses

Identification patient features that impact outcomes Compare risk factor profiles across hospitals Assist in the selection of the best procedure for a

particular patient Shared decision making with patient Guide risk mitigation and improvement activities Target specific patients for special attention, such as

aggressive post-discharge monitoring Performance measurement

Risk-adjusted outcomes Risk adjustment essential for outcomes measures

• Fair and accurate performance assessment—accounts for higher risk patients, mitigates risk aversion

• Face validity and provider acceptance, irrespective of statistical considerations

From a purely statistical perspective, risk adjustment appropriate if:• Presenting patient characteristics associated with outcomes• Prevalence of patient characteristics varies among providers

Percentiles of Risk Factor Distribution

1st 5th 10th 25th 50th 75th 90th 95th 99th

Isolated CABG (2009)

Age ≥80 0.0% 2.7% 3.7% 5.6% 8.1% 10.9% 13.7% 15.4% 19.5%

Preop Dialysis 0.0% 0.0% 0.0% 1.2% 2.1% 3.7% 5.6% 7.1% 11.2%

Creatinine ≥ 2 0.0% 0.0% 0.5% 1.3% 2.1% 3.2% 4.4% 5.4% 7.7%

CPR/salvage 0.0% 0.0% 0.0% 0.0% 0.0% 0.7% 1.5% 2.2% 3.7%

Cardiogenic Shock 0.0% 0.0% 0.0% 0.7% 1.5% 2.9% 4.5% 5.7% 9.6%

PVD 2.7% 6.1% 7.9% 10.9% 14.4% 17.9% 22.3% 26.0% 32.6%

PCI ≤ 6 Hours 0.0% 0.0% 0.0% 0.0% 0.7% 1.5% 2.6% 3.3% 6.0%

MI ≤6 Hours 0.0% 0.0% 0.0% 0.4% 1.3% 2.4% 4.1% 5.6% 9.0%Reoperation 0.0% 0.0% 0.4% 1.7% 3.2% 4.8% 6.8% 8.3% 12.3%

Shahian et al, Ann Thorac Surg 2013;96:718-26

CABG risk factor prevalence—national STS data

Massachusetts CABG programsRisk factor prevalence FY 2014

• 44 Veterans Affairs Medical Centers• 87,078 major non-cardiac operations• October 1, 1991- December 31, 1993

• 93% of hospitals changed rank after risk adjustment (some up, some down)

50% by more than 525% by more than 10

Risk adjustment makes a difference

Appropriate uses of risk-adjusted results

YES--Comparing a hospital’s performance for its specific case mix with what would have been expected based on the national benchmark population

NO—Directly comparing two hospitals with each other

Why?• Patient risk versus case-mix• Indirect versus direct standardization

Epidemiology: Direct Standardization

Apply study hospital’s stratum-specific rates to reference population

Direct hospital to hospital comparisons often possible

Typically used in epidemiologic studies focusing on a limited number of strata (e.g., age, sex, etc.)

Profiling: Indirect Standardization

Often too many variables (strata) to use direct standardization

Some study hospital patient strata with zero observations

Apply reference population rates to the study hospital’s patients

Profiling: Indirect Standardization Provider’s results for their specific patients compared with

what would have been expected for the same patients based on the performance of all providers who contributed to the benchmark population

Mortality lower than expected (O/E < 1), greater than expected (O/E > 1), or as-expected

O/E ratio < 1 at a hospital treating mainly low risk patients not directly comparable with O/E ratio at a hospital treating mainly high risk patients

A desirable O/E ratio does not necessarily extrapolate to a different mix of patients (i.e., much higher risk)

Provider-level “expected” outcomes

Individual patient-level probabilities for specific outcomes are summed for all patients of a given provider

This yields the total expected number of those outcomes

Patient Probability of death1 0.0312 0.0233 0.0544 0.016

….. ……100 0.035

Total N =100 Sum of probabilities of death = 4.3

Provider-level observed outcomes and O/E ratios

Expected total deaths compared with the observed deaths to estimate observed to expected ratio (O/E) • >1 indicates worse than expected• < indicates better than expected• Always look at the confidence intervals, which

reflect our certainty about the point estimate!

Patient Probability of death Actual outcome

1 0.031 Alive2 0.023 Alive3 0.054 Dead4 0.016 Alive

….. …… ….100 0.035 Dead

Total N =100 Sum of probabilities of death = 4.3

Sum of observed deaths = 5

Provider-level observed outcomes and O/E ratios

= O/E ratio = 5/4.3 = 1.16Observed Expected

Risk-adjusted outcomes

The O/E ratio can be multiplied by the average population mortality to give the risk-adjusted mortality

If O/E 1.16 and population average mortality 2.1%, then risk-adjusted mortality rate = 2.44%

• Indirectly standardized, risk-adjusted outcomes basedonly on the patients a particular hospital treated

• Should not be used for direct hospital to hospital comparisons

Do risk models protect providerswho care for the sickest patients?

Most risk models over-predict risk among highest risk patients

>10% Predicted Risk ofmortality

Overall

Center with highest risk patients consistently had lowest O/E ratios

Risk adjustment

Essential for most outcomes measures Need to collect desired factors with low missing % Intended use determines permissible risk factors Models should be transparent and peer reviewed In most instances, direct hospital to hospital

comparisons not appropriate, even with risk-adjusted results

Watch for extreme differences in case mix

Thank You!