www.icnarc.org analysis and presentation of quality indicators dr david harrison senior...
TRANSCRIPT
![Page 1: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/1.jpg)
ww
w.icn
arc.o
rg
Analysis and presentation of quality indicators
Dr David HarrisonSenior Statistician, ICNARC
![Page 2: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/2.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Analysis and presentation of QIs⢠Principles of statistical process
control⢠Comparison among providers⢠Continuous monitoring over time
![Page 3: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/3.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Analysis and presentation of QIs⢠Principles of statistical process
controlâ Common cause variationâ Special cause variationâ Control limits
⢠Comparison among providers⢠Continuous monitoring over time
![Page 4: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/4.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg Principles of statistical process
control⢠Common cause variation
â Variation cannot be eliminatedâ Some variation is inherent to any
processâ This is termed âcommon cause
variationââ To reduce common cause variation
we need to change the process
![Page 5: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/5.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Five signaturesâŚ
![Page 6: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/6.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
They are not identicalâŚ
![Page 7: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/7.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
They are not identicalâŚ
âŚbut they are all my signature
![Page 8: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/8.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
We could rank themâŚ
âŚbut this doesnât make much sense!
1.
2.
3.
4.
5.
![Page 9: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/9.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg We could reject some as low
qualityâŚ
âŚbut they are still my signature!
![Page 10: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/10.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
This is common cause variation
![Page 11: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/11.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg Principles of statistical process
control⢠Special cause variation
â Some variation is the result of external factors acting on a process
â This is termed âspecial cause variationâ
â To reduce special cause variation we need to identify the source and eliminate it
![Page 12: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/12.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Now we have a sixth signatureâŚ
![Page 13: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/13.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Now we have a sixth signatureâŚ
âŚitâs a good try, but I think you can tell which one is the forgery!
![Page 14: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/14.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
This is special cause variation
![Page 15: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/15.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Control limits⢠Statistical process control is all
about making allowance for common cause variation to detect special cause variation
⢠To do this we place control limits around a process
⢠Control limits represent the acceptable range of common cause variation
![Page 16: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/16.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Control limits⢠Typically control limits of 2 and 3
SDs represent âalertâ and âalarmâ⢠If a system is in control:
â 95.4% of values within 2 SDsâ 99.7% of values within 3 SDs
![Page 17: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/17.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Analysis and presentation of QIs⢠Principles of statistical process
control⢠Comparison among providers
â League tablesâ Caterpillar plotsâ Funnel plotsâ Over-dispersion
⢠Continuous monitoring over time
![Page 18: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/18.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Comparison among providers⢠Iâll assume we have a binary event
(e.g. death) and an associated risk estimate (e.g. predicted risk of death)
⢠Most common QI is:observed events / expected
events⢠(for mortality this is the standardised
mortality ratio)⢠How should we compare this QI
among providers (e.g. critical care units)?
![Page 19: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/19.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
League tables⢠Journalists love them
â High impactâ Everyone wants to know who is first
and last
![Page 20: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/20.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Seven deadliest hospitals identified
in damning Dr Foster reportDaily Telegraph, 29 November 2009
Twelve NHS trusts slammedThe Sun, 29 November 2009
Patient safety at Scarborough
Hospital âsecond worst in countryâ
Scarborough Evening News, 29 November 2009
![Page 21: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/21.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
League tables⢠Journalists love them
â High impactâ Everyone wants to know who is first
and last
⢠Statisticians hate themâ Overemphasise unimportant
differencesâ Even if there is no true difference,
someone will be first and someone lastâ No account of role of chance
(common cause variation)
![Page 22: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/22.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Marshall & Spiegelhalter, BMJ 1998⢠League table of 52 IVF clinics
ranked on live birth rate⢠Monte Carlo simulation to put 95%
CI on ranks
![Page 23: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/23.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Marshall & Spiegelhalter, BMJ 1998
![Page 24: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/24.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Marshall & Spiegelhalter, BMJ 1998
⢠Kingâs College Hospital â sixth from bottom â is the only one that can reliably be placed in the bottom 25%
![Page 25: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/25.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Marshall & Spiegelhalter, BMJ 1998
⢠BMI Chiltern Hospital â seventh from bottom â may not even be in the bottom 50%
![Page 26: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/26.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Marshall & Spiegelhalter, BMJ 1998
⢠Five clinics can confidently be placed in the top quarter
*
****
![Page 27: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/27.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Marshall & Spiegelhalter, BMJ 1998
⢠Southmead General â ranked sixth from top â may not be in the top 50%
![Page 28: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/28.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Caterpillar plots (or forest plots)⢠Plot of QIs with CIs in rank order⢠Still a league table really⢠But at least acknowledges
variation by including CIs
![Page 29: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/29.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Caterpillar plot â IV clinics
![Page 30: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/30.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Caterpillar plot â ANZICS
⢠SMRs by APACHE III-J for 106 adult ICUs in Australia and New Zealand, 2004(Cook et al. Crit Care Resusc 2008)
![Page 31: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/31.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Funnel plots⢠Larger sample = greater precision⢠If you plot QI against sample size,
you expect to see a funnel shape⢠We can plot funnel shaped control
limits
![Page 32: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/32.jpg)
ww
w.icn
arc.o
rg
|
![Page 33: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/33.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Funnel plot â ANZICS
⢠SMRs by APACHE III-J for 106 adult ICUs in Australia and New Zealand, 2004(Cook et al. Crit Care Resusc 2008)
![Page 34: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/34.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Funnel plot â ANZICS
⢠Note: use of normal distribution can result in negative confidence intervals â better methods exist
![Page 35: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/35.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Funnel plot â ANZICS
⢠Note: as SMR is a ratio measure, we would advocate plotting on a log scale (i.e. SMR=2 and SMR=0.5 are equidistant from SMR=1)
![Page 36: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/36.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Funnel plot â SICSAG
⢠SMRs by APACHE II for 25 adult ICUs in Scotland, 2009(SICSAG Audit of critical care in Scotland 2010)
![Page 37: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/37.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Funnel plot â SICSAG
⢠Note: as the model is poorly calibrated, most units are âbetter than averageâ â the funnel has been centred on the average SMR not 1
![Page 38: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/38.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Over-dispersion⢠Variability more than expected by
chance⢠Suggests important factors that
vary among providers are not being taken into account
⢠Too many providers classified as âabnormalâ (i.e. outside the funnel)
![Page 39: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/39.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg Over-dispersion â hospital
readmissions
(Spiegelhalter. Qual Saf Health Care 2005)
![Page 40: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/40.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Over-dispersion â what to doâŚ?⢠Donât use the indicator?⢠Improve risk adjustment⢠Adjust for it
â Estimate âover-dispersion factorâ by âWinsorisationâ
⢠Use random effects modelsâ Assumes each provider has their
own true rate from a distribution
![Page 41: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/41.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Example â over-dispersion factor
0.25
0.50
1.00
2.00
Mor
talit
y ra
tio -
ICN
AR
C (
200
9) m
odel
0 500 1000 1500Number of admissions
⢠SMRs by ICNARC model for 171 adult ICUs in England, Wales & N Ireland, 2009
![Page 42: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/42.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Example â over-dispersion factors
0.25
0.50
1.00
2.00
Mor
talit
y ra
tio -
ICN
AR
C (
200
9) m
odel
0 500 1000 1500Number of admissions
Note: Overdispersion factor 1.4 based on 10% Winsorised
⢠Over-dispersion factor estimated at 1.4⢠Funnel widened
![Page 43: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/43.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Analysis and presentation of QIs⢠Principles of statistical process
control⢠Comparison among providers⢠Continuous monitoring over time
â RAP chartâ EWMAâ VLADâ R-SPRTâ CUSUM
![Page 44: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/44.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Continuous monitoring over time⢠Various approaches⢠In general, they consist ofâŚ
â an indicator that is updated for each consecutive patient
â control limits
![Page 45: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/45.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg Example for continuous
monitoring⢠Queen Kate Hospital⢠Fictitious critical care unit⢠Random sample of 2000 records
from the Case Mix Programme Database
⢠After 1000 records, outcomes changed so that an extra 6% of patients (selected at random) die
⢠Risk adjustment by the ICNARC (2009) model
![Page 46: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/46.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Queen Kate Hospital â SMRs
0.7
0.9
0.8
1.0
1.2
1.4
1.6
Mor
talit
y ra
tio (
95%
CI)
Consecutive blocks of 250 patients
![Page 47: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/47.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
RAP chart⢠Risk-adjusted p chart⢠Cohort divided into discrete blocks
(e.g. 100 patients)⢠Indicator is observed mortality⢠Control limits are predicted mortality
+/- 2 or 3 SDs⢠Pro
â Displays observed and expected mortality
⢠Conâ Still in blocks, not sensitive
![Page 48: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/48.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Queen Kate Hospital â RAP chart
10%
20%
30%
40%
Mor
talit
y
0 500 1000 1500 2000Number of admissions
Observed Predicted 2 SDs 3 SDs
![Page 49: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/49.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
EWMA⢠Exponentially weighted moving
average⢠Similar to RAP but uses all data up
to the current timepoint⢠Data weighted by a smoothing
factor so that most recent data are given most weight
![Page 50: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/50.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
EWMA⢠Pro
â Displays observed and expected mortality
â Estimates updated continuously not in arbitrary blocks
⢠Conâ Choice of smoothing factor is
important â too little smoothing and plot is unreadable, too much and plot is insensitive to changes
![Page 51: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/51.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Queen Kate Hospital â EWMA
20%
25%
30%
35%
40%
Mor
talit
y
0 500 1000 1500 2000Number of admissions
Observed Predicted +/- 2 SD 3 SD
![Page 52: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/52.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
VLAD⢠Variable life adjusted display⢠Cumulative observed minus
expected deaths⢠Pro
â Nice easy interpretation
⢠Conâ Control limits are complex to
calculate curved functions
![Page 53: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/53.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Queen Kate Hospital â VLAD
-20
0
20
40
60
Cum
ulat
ive
obse
rved
min
us e
xpec
ted
deat
hs
0 500 1000 1500 2000Number of admissions
![Page 54: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/54.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
R-SPRT⢠Resetting sequential probability
ratio test⢠Tests evidence for/against a
specific hypothesis (e.g. odds of death are double that predicted by the model)
⢠Plot of log likelihood ratio⢠If bottom line is reached (strong
evidence against hypothesis) then line resets to zero
![Page 55: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/55.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
R-SPRT⢠Pro
â Nice statistical propertiesâ Control limits are horizontal lines
⢠Conâ Choice of hypothesis to test is
arbitrary â should we test for an OR of 2, 1.5,âŚ?
![Page 56: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/56.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Queen Kate Hospital â R-SPRT
-10
-5
0
5
10
RS
PR
T fo
r a
doub
ling
of o
dds
0 500 1000 1500 2000Case number
0.01 0.001 0.0001alpha = beta =
![Page 57: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/57.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
CUSUM⢠âCumulative sumâ⢠Log likelihood ratio â same as R-
SPRT⢠âAbsorbing barrierâ at zero (i.e.
never goes below zero)
![Page 58: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/58.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
CUSUM⢠Pros/Cons as for the R-SPRT plusâŚâ˘ Pro
â Does not allow credit to build up (as in R-SPRT) so alerts earlier
â Negative CUSUM (e.g. OR=0.5) can be plotted on the same axes
⢠Conâ Cannot detect evidence against
hypothesis
![Page 59: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/59.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Queen Kate Hospital â CUSUM
0
5
10
15
Log
likel
ihoo
d ra
tio fo
r O
R=
2
0 500 1000 1500 2000Number of admissions
0.01 0.001 0.0001alpha = beta =
![Page 60: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/60.jpg)
Analysis and presentation of quality indicators | Dr David Harrison
ww
w.icn
arc.o
rg
Which method(s) to use�⢠Comparison among providers
â Funnel plot
⢠Continuous monitoring over timeâ EWMAâ or R-SPRTâ or CUSUMâ (VLAD can be used as a display in
conjunction with, e.g., CUSUM for monitoring)
![Page 61: Www.icnarc.org Analysis and presentation of quality indicators Dr David Harrison Senior Statistician, ICNARC](https://reader036.vdocuments.site/reader036/viewer/2022062517/56649ed95503460f94be6eb4/html5/thumbnails/61.jpg)
ww
w.icn
arc.o
rg
|