the use of funnel plots & multi- year cumulative data to track hospital performance herbert ma,...
TRANSCRIPT
The Use of Funnel Plots & Multi-Year Cumulative Data to
Track Hospital Performance
Herbert MA, Hamman BH, Roper KL, Ring WS, Edgerton JR, Texas Quality Initiative
The American Association of Thoracic SurgeonsApril 26, 2015
Seattle, Washington
Nothing to Disclose
The Texas Quality Initiative• 27 Hospitals in North Texas agreed to share
clinical and administrative data• All participated in the STS Database• 26,634 cardiac procedures from 1/2008 -12/2012• 13,379 isolated CABG were analyzed for observed
to expected (O/E) operative mortality• There was a need to graphically represent the
data– Simple– Easy to understand
The Funnel Plot
Methods
• A funnel plot is centered on a benchmark with 95% confidence intervals drawn on the graph.
• To assess operative mortality and allow for risk correction, the observed to expected (O/E) ratio is used.
• The case volume is plotted on the horizontal axis • O/E ratio on the vertical axis; • either annual data or multi-year data can be
shown.
The Funnel Plot• X Axis: Volume of Cases (CABG)• Y Axis: O/E Ratio for isolated CABG• An O/E of 1 is expected• 95% Confidence intervals surround “1”
The Funnel Plot
Worse than 1, but not statistically different
Better than 1, but not statistically different
Outlier for poor performance
Outlier for good performance
The Funnel Plot• At low volume it is very hard to become an outlier• In fact, at less than 200 cases, you cannot become an outlier for
good performance• The Problem:• Most hospitals analyze their data on an annual basis• Most Hospitals do less than 200 cases per year • They cannot reveal themselves as outlier due to wide confidence interval at low volume• Year after year they find that their results are “OK”
The Funnel Plot
PROBLEMNo Problem
How can we account for this problem?
• Plotting running totals moves the result to the right, where the funnel is narrower
• The results can begin to show statistically significant differences from “1”
• More important…..• Trends become visually apparent• Poorly performing hospitals can be identified– Even before the results reach statistical significance– Urgent interventions can be put in place
Plotting Running 5 Year Totals
Year
1
Year
1 + 2
+ 3
Year
1+ 2
Year
1 +
2 +
3 +
4Ye
ar 1
+ 2 +
3 + 4
+ 5
Plotting Running 5 Year Totals Year 6
Year
2
Year
2 + 3
+ 4
Year
2+ 3
Year
2 + 3
+ 4 +
5Ye
ar 2
+ 3 +
4 + 5
+ 6
How Does This Help
• Let’s see some examples from the TQI Data• These are real data from real hospitals• Some of the examples are from different time
intervals, because…• I Selected graphs to illustrate different scenarios• Colored dots represent annual data• Green line represents the running 5 year total
Even with excellent outcomes, a hospital with case volumes under 200 cannot become an outlier for good performance
However, Cumulative data will reveal excellence (in 1 more year)
Annual Data tightly clustered: O/E doesn’t change muchWith Cumulative Data …
The Curve is flat, but at higher volume becomes an outlier
Annual Data is all within the funnelHospital perceives “No Problem”
Cumulative Data Unmasks Outlier for Poor Performance
Annual Data is Inconclusive: 3 out of 5 years are within the funnelCumulative Data Slope of the Curve is Predictive of Poor Performance
At Year 3 Intervention is Needed, This is even before hospital becomes an outlier in year 5
Annual Data: 4 of 5 years O/E is > 1Cumulative Data Shows a downward slopeWe have no concerns about this hospital
Conclusions• The use of funnel plots allows easy comparison
of individual programs• Analyzing only annual data can lead to a false
sense of satisfaction • The plotting of a five year running total will
provide sufficient volume to reveal an accurate assessment
• the trend (slope) may give an indication of effectiveness of quality improvement programs in place.