presented at the flinders centre for epidemiology and biostatistics (fceb)

19/04/2023 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

1

Statistics in MedicineResearch Article

Received 18 June 2012, Accepted 30 January 2013 Published online in Wiley Online Library(wileyonlinelibrary.com) DOI: 10.1002/sim.5779

Evaluating the performance of Australian and New Zealand intensivecare units in 2009 and 2010

J. Kasza, J. L. Moran, P. J. Solomon for the ANZICS Centrefor Outcome and Resource Evaluation (CORE) of the Australian

and New Zealand Intensive Care Society (ANZICS)

Presented at the Flinders Centre for Epidemiology and Biostatistics (FCEB)Journal Club – 9th May 2013

byDr Wayne Clapton

Public Health Physician, Lecturer Critical Care Medicine,Flinders University


2

SUMMARY-1

• Australian and New Zealand Intensive Care Society (ANZICS) Adult Patient Database (APD)

– Collects Individual Admissions Data - Aust & New Zealand Intensive Care Units (ICU’s)

– Used for monitoring and comparing performance of ICU’s

– Identifies unusual ICU’s via Standardised Mortality Ratios with 95% or 99% confidence intervals or Funnel Plots with 95% or 99% prediction limits.

• Theoretical and computational challenges arise from this approach.


3

SUMMARY-2

• Specific statistical issues often neglected in ICU comparison studies:

– Expected number of deaths must be estimated appropriately

– ICU casemix adjustment must be adequate

– Must account fully for statistical variation

– Must adjust appropriately for multiple comparisons


4

SUMMARY-3

• Authors’ approach:

– Fit a random coefficient hierarchical logistical model for the in-hospital death of each patient, with patients clustered within ICU’s.

– Anticipate that the majority of ICU’s will be estimated as performing “usually” after adjusting for important clinical covariates.

– Start with Ohlssen et al’s ideas and estimate an appropriate null model that ICU’s would be expected to follow (A Frequentist, rather than a Bayesian approach)


5

SUMMARY-4

• Results of Authors’ methodology:

– Rigorously account for the statistical issues

– Determine if any ICU’s on the ANZICS APD have comparatively unusual performance

– Investigate annual ICU performance

– Estimate changes in individual ICU performance between 2009 and 2010 by adjusting for regression-to-the-mean.


6

STANDARDISED MORTALITY RATIOS

• Used as performance indicators• Expected number of deaths for each ICU is estimated by

summing patient mortality probabilities adjusted for severity of the patient’s conditions.

• Despite some inclusion of other covariates, other clinically important covariates are not present.

• Expected number of deaths are assumed fixed when calculating SMR’s variances.

• Unusual ICU performers are detected when:– SMR confidence intervals do not contain one– SMR’s fall outside prediction limits of the Funnel Plots


7

HIERARCHICAL MODELS - 1

• Standard use for comparing healthcare providers; but not always used with patients clustered within ICU’s for comparing ICU performance.

• Two approaches for comparing providers:

– Fit a random effects distribution that encompasses all variation between providers and identify those providers that have extreme random effects with respect to the model.

– Fit a random effects distribution that the majority of providers are expected to follow and identify any providers who have outlying random effects with respect to this model, using hypothesis testing.


8

HIERARCHICAL MODELS - 2

• Most ICU’s will have usual performance; but not be identical.

• The interest is in identifying unusually poor or good performance.

• Estimate a random coefficient hierarchical logistic regression model for the in-hospital death for each patient.

• Clustering patients within ICU’s

• Random effects distribution that describes mortality experience in an ICU with usual performance.


9

MODEL - CASEMIX ADJUSTMENT

• Included in the model:

– Age, sex, patient severity score, patient diagnostic category, patient surgical status, patient ventilation status, source of admission to the ICU, geographical locality, ICU level, annual ICU volume, clinically meaningful interactions (see Table 2 in the supplementary information)

– Random intercept

– Random coefficient for patient severity score

• Patient severity score: Acute Physiology and Chronic Health Evaluation, third revision (APACHE III) – worst clinical values in the first 24 hours post-admission


10

MODEL - OTHER

• Log-SMR’s, rather than SMR’s, used as performance indicators to obtain confidence intervals with better coverage properties.

• Uncertainty in both observed and expected numbers of deaths are accounted for when calculating log-SMR variances.

• The multiplicity of hypothesis tests is accounted for by controlling the False Discovery Rate (FDR).

• Funnel plots are used to display results.

• Thresholds controlling the FDR are placed on the plots.

• Yearly performance plus change between 2009 and 2010.


11

ANZICS APD & CHOICE OF RISK-ADJUSTED MODEL

• ANZICS APD – Individual patients’ admissions data, large, voluntary, data quality checks, extracts to CORE

• Study dataset exclusions: – Unknown hospital mortality status and discharge date (4023

patients), – Patients with an ICU stay less than 4 hours (2110),– Patients less than 16 years old (3822)– 10170 separate hospital readmissions– ICU’s with fewer than 150 patients in 2009 or 2010

• Of 139 ICU’s in 2009, 24 excluded• Of 142 ICU’s in 2012, 27 excluded.

• Final dataset: 163,795 patients from 115 ICU’s


12


13

HIERARCHICAL LOGISTIC REGRESSION MODEL - 1

• Mortality experience is not independent year to year (Continuity of hospital-level and ICU level factors from year to year, many of which are outside the control of the ICU.)

• So cannot fit separate models for 2009 and 2010.

• Thus models incorporate data from both years, with patients clustered within ICU’s.

• 30 day-after-hospital-discharge mortality would be more appropriate; but no APD linkage with death registers to enable this.


14



15



16

HIERARCHICAL LOGISTIC REGRESSION MODEL - 4• Building an appropriate model for risk-adjustment:

– Patients split into an 80% training set and a 20% test set.– Model parameters estimated using xtmelogit (Stata 12), with

the Laplace approximation used in the model-building stage.– Model 1: All covariates and two-way interactions; but

excluded a random coefficient for APACHE III score.– Model 2: Included a random coefficient for APACHE III score.– Model 3: Removing interactions insignificant at the 5% level

(ICU source by each of APACHE III score and surgical status; and ventilation status by each of age, sex and surgical status.)


17



18


• Model 3 was chosen for risk adjustment.

• No interactions were insignificant at the 5% level.

• 110 fixed effects parameters and

• Three random effects parameters.


19

IDENTIFYING ICU’s WITH UNUSUAL PERFORMANCEUSUAL APPROACHES - 1

• Usual approaches (which have problems):

1. Estimate mortality probabilities using a scoring algorithm, e.g., APACHE III.• ANZICS Centre for Outcome and Resource Evaluation

(ANZICS CORE) uses this method with APACHE III-J

2. Fit a logistic regression model similar to Equation 1, where the log odds of in-hospital patient deaths are dependent upon a severity score.• Can extend to account for hierarchical nature of patient-level

data, plus include covariates and a random coefficient for severity score.


20

IDENTIFYING ICU’s WITH UNUSUAL PERFORMANCEUSUAL APPROACHES - 2

Problems with the usual methods:

• Method 1: – Fails to account for patient clustering within ICU’s.– Scoring systems may not be calibrated for the population under study.

• Method 2:– Adjustment for risk is inadequate, because many clinically important covariates often are not

included.

• Both:– The presence of unusual ICU’s is not accounted for in the estimation of a model describing the

mortality experience of ICU’s with usual performance.– A model estimated using data from both usually and unusually performing ICU’s is likely to have

over-inflated random effect variance estimates, which leads to overestimation of the usual levels of variation in SMR’s.

– Risk of misinterpretation of confidence intervals.– Expected number of deaths assumed to be fixed in SMR variance estimations for funnel plot

construction.– Multiple comparisons are not addressed.


21

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 - 1


22

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 – 2 STAGE 1


23



24



25



26



27

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 2


28



29



30



31



32



33

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 - STAGE 3


34

IDENTIFYING ICU’s WITH RECENT CHANGES IN PERFORMANCE


35



36



37



38

DISCUSSION - 1

• Key concepts and principled statistical methods• Estimating the null model for in-hospital patient

mortality and adjusting for important casemix factors Rigorously monitored 2009 & 2010 APD ICU’s.

• Formally compared 2009-2010 changes via regression-to-the-mean techniques.

• Reduced the risk of incorrectly identifying ICU’s as unusual.

• Reduced the risk of failing to identify ICU’s performing unusually, be the performance unusually good or unusually poor.


39

DISCUSSION - 2

• Suggested that random intercept captures unknown ICU-level explanatory variables (possibly related to performance).

• Therefore, recommend a null random-effects density.

• ICU’s detected by outlying log-SMR’s should be investigated further - as per CORE policy.

• Further years of data may be required to check performance outliers re whether this is a real effect.

• Results here differ from the ANZICS CORE Report – suggest more rigorous statistical methods by CORE.


40

DISCUSSION - 3

• Adding further interactions and/or covariates significantly adds to computer run times and may over-adjust for risk?

• 30-day mortality after hospital discharge would be better for survival analyses – need linkage.

• Comparisons for periods more than two years is of interest and would require extensions of the methods described here.

• Models more than two years incorporating autocorrelation and seasonal effects are of interest – develop at the risk- adjustment stage of the modelling process.


41

COVARIATES USED IN THE MODELS

presented at the flinders centre for epidemiology and biostatistics (fceb)

Documents