quality indicators: past and present michael a noble md frcpc professor medical microbiology and...
TRANSCRIPT
Quality Indicators:Past and Present
Michael A Noble MD FRCPC
Professor
Medical Microbiology and Infection Control, Vancouver Coastal Health
and
Chair, Clinical Microbiology Proficiency Testing program,
Chair, Program Office for Laboratory Quality Management
Department of Pathology and Laboratory Medicine
University of British Columbia
Quality Indicators:Past and Present
• History• Quality Indicators and ISO• Characteristics of Indicators – strong and weak• Quality Indicator Inventories – USA and BC• Examples of Quality Indicators• Summary
A really good, inexpensive
reference book
Amazon.com$30.00
Two excellent (essential) references from CSA
A Short History of Metrics in Quality Management
Innovator Date Cycle
Walter A Shewart 1920’s Plan-Do-SEE
J Edwards Deming 1940’s Plan-Do-CHECK-Act
Bob Galvin 1980’s Define-MEASURE-Analyze-Improve-
Control
The Quality Cycle
Plan
DoAct
CHECK
Each step is essential to keep the quality cycle
cycling
Each step is essential to keep the quality cycle
cycling
Quality IndicatorsA workable definition
• Established measures used to determine how well an organization meets needs and operational and performance expectations.– Objective– Measurable– Repeatable
Metrics and ISO 9001:2000
Factual Approach to Decision Making5.4.1
• Top management should ensure that quality objectives, including those needed to meet requirements for product, are established at relevant functions and levels within the organization. The quality objectives shall be measurable and consistent with the quality policy.
Metrics and ISO 9001:2000 (2)8.4
The organization shall determine, and collect and analyze appropriate data to demonstrate suitability and effectiveness of the quality management system and evaluate where continual improvement of the effectiveness of the quality management system can be made. This shall include data generated as a result of monitoring and measurement and from other relevant sources.The analysis of data shall provide information relating to:– Customer satisfaction– Conformity to product requirements– Characteristics and trends of processes and products including
opportunities for preventive actions, and – suppliers
Metrics and ISO 15189:2003
• 4.12.4Laboratory management shall implement quality indicators for systematically monitoring and evaluating the laboratory’s contribution to patient care. When this program identifies opportunities for improvement, laboratory management shall address them regardless of where they occur. Laboratory management shall ensure that the medical laboratory participates in quality improvement activities that deal with relevant areas and outcomes of patient care.
So…
Quality Indicators are measured information that• Indicates the performance of a process.• determines quality of services.• highlights potential quality concerns, • identifies areas that need further study and
investigation, and • track changes over time.
Measuring PerformanceMark Graham Brown
• Fewer is better.
• Link measures to the factors needed for success.
• Measures should be based around customer and stakeholder needs.
• Measures should start at the top and flow down to all levels of employees.
• Measures should change as the environment and strategy changes
• Measures should have targets or goals established that are based on research rather than arbitrary values.
Keeping ScoreUsing the Right Metrics to Drive World Class
Performance1996
Many organizations spend thousands of hours collecting and interpreting data. However many of these hours are nothing more than wasted time because they analyze the wrong measurements, leading to inaccurate decision making.
– Mark Graham Brown.
As true today as it was then !
Indicators?You want Indicators?
We’ve got LOTS of Indicators!
AHRQ
IQLM
American Nurses Association
American Psychiatric Association
RAND
WHO
Leapfrog
National Quality Forum
JCAHO
ISQua
OECD
ASQ
Characteristics of Good Metrics
Timedshort and long term
Timedshort and long term
Engagingall levels
Engagingall levels
Balancedfull cycle
Balancedfull cycle
Actionableaction oriented
Actionableaction oriented
Interpretablespecific
Interpretablespecific
Achievablecontained
Achievablecontained
Measurableobjective
Measurableobjective
GoodMetricsGood
Metrics
Indicators of Good Indicators
Measurable Can you count it, time it, record it?
Achievable Can you actually capture it?
Interpretable When you’ve got it, what does it mean?
Actionable Can you do something about it?
Timed Does your set cover both the short and long term?
Engaging Does your set involve all laboratory personnel?
Balanced Does your set cover the full cycle of events?
Assessing Quality Indicators• Importance Potential for Improvement
• Scientific Acceptability Reliability and Validity
• Feasibility Implementation and cost
• Usefulness Comprehensive
Having Quality Quality Indicators
Total Testing Cycle forMedical Laboratories
Report Interpretation
Report Creation
Data Capture
Report Transport
AnalysisQuality Control
AnalyticPost
-Analy
ticP
re-A
naly
tic
Menu
Ordering Rules
Patient ID
Acceptance Criteria
Collection
Transport
Baldrige Award Criteria
• Balanced Metrics– Customer satisfaction– Employee satisfaction– Financial performance– Operational performance– Product and Service quality– Supplier performance– Safety and environment and public responsibility
Most organizations focus 80% of metrics on finance and operations.
IQLM Indicators• Diabetes monitoring (system) • Hyperlipidemia screening (system) • Test Order Accuracy and Appropriateness
• Patient Identification (pre-analytic)• Adequacy and Accuracy of Specimen Information (pre-analytic)• Blood Culture Contamination (pre-analytic / system)
• Accuracy of point-of-care testing (analytic) • Cervical cytology/biopsy correlation (analytic)
• Critical Values Reporting• Turnaround time (postanalytic) • Clinician satisfaction (system/postanalytic) • Clinician followup (system/postanalytic)
CMPT Metrics Scorecard• Balanced Metrics Percent
– Customer satisfaction 25– Employee satisfaction 5– Financial performance 10– Operational performance 20– Product and Service quality 30– Supplier performance 5– Safety /environment /
public responsibility 5
Characteristics of Weak Metrics
• Focus only on measures easy to count
• Focus only on measures easy to achieve.
• Metrics with arbitrary targets.
• Measures that don’t change with experience
Computer Nonsense Metrics
[urine culture] * [glucose] * [INR]
[NUPA hr] * [Telephone minutes] X100
Just because a computer can
calculate a value, doesn’t mean that
it should.
Computerese Quality Indicators
• Unit Producing Activity per Paid Hour• Unit Producing Activity per Worked Hour• Unit Producing Activity per Total FTE• Non-Unit Producing Activity per Paid Hour• Non-Unit Producing Activity per Worked Hour• Non-Unit Producing Activity per Total FTE
• Crude Turn-Around-Time
A Cautionary Note
• Measures that drive the wrong performance.
Measuring professionals is tough because intellectual work is difficult to measure objectively. Looking for factors that can be counted may not be what is really important. Meaningful outputs such as ideas, information, and problems avoided may be difficult but more relevant.
Mark Graham Brown
Caution about patient outcome indicators
Theoretically, outcomes best assess quality, but they are the most difficult to measure – too many variables and confusers
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data – Need long collection periods.
David HsiaMedicare Quality Improvement Bad Apples or Bad Systems? JAMA. 2003;289:354-356.
Are you an Indicator Glutton?
Monitoring more than 10-12 indicators is rarely successful
Mark Graham Brown 1996
Quality Inventory:US Medical Laboratories
2004
• In 2004 the Institute for Quality in Laboratory Medicine (IQLM) and the Clinical Laboratory Managers Association (CLMA) undertook an on-line quality inventory of laboratories with CLMA members.
• Approximately 400 laboratories responded.• The study was voluntary, self-reported, with a
validated questionnaire. • Information provided was not verified by a second
method
In British Columbia…
The Program Office for Laboratory Quality Management and the Provincial Laboratory
Coordinating Office have organized to perform a similar, but improved inventory
in 2005.
10 Most Common Procedures MonitoredBC Quality Inventory 2005
7678808284868890929496
Pre-AnalyticSystemAnalytic
Post Analytic Procedures MonitoredBC Quality Inventory 2005
626466687072747678
Satisfaction MonitoringBC Quality Inventory 2005
0102030405060
Other Achievable Indicators
• Blood culture volumes:Blood culture false negative results occur when bottles contain insufficient (<3 mL) or excessive (>15 mL) blood.
Insufficient or excessive blood collection is a collection non-conformity.
Under and Overfill Blood Cultures 2001-2004
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Percent Over
Percent Under
Underfill Blood Collections (As a percent of collections per site)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
RF2
RF1
ER
ICU
Phlebotomists
Baldrige Award Criteria
• Balanced Metrics– Customer satisfaction– Employee satisfaction– Financial performance– Operational performance– Product and Service quality– Supplier performance– Safety and environment and public responsibility
Most organizations focus 80% of metrics on finance and operations.
Eight Steps to DevelopingSuccessful Indicators
1. Objective
2. Methodology
3. Limits
4. Interpretation
5. Limitations
6. Presentation
7. Action plan
8. Exit plan
Developing IndicatorsObjective What are you trying to measure
Methodology 1. How to capture the data2. Who (or what) to capture the data3. How often to capture the data
Limits Acceptable, Concern, Unacceptable Critical
Presentation Graphic or Text
Interpretation What does it mean?Does it reflect on YOUR quality?
Limitations Unintended variables
Action Plan What will I do if it indicates acceptable performance?
What will I do if it does not?
Exit Plan When can I stop measuring?
PresentingQuality Indicator Information
0
10
20
30
40
50
60
70
80
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
High
Low
Average
The BIG SECRET for Quality Indicator Team
Engage the folks who do the work, because they know what they do!
Microbiology Indicators
Collected and Monitored by Vancouver General Hospital Division of Medical Microbiology and Infection Control
Many thanks to:Diane Roscoe Anita KwongMedical Microbiology team
Contmination Rate: Blood Culture Sets
0.0%
1.0%
2.0%
3.0%
4.0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time Period
Per
cent
2002-2003 2003-2004 2004-2005
Contmination Rate: Blood Culture Sets
0.0%
1.0%
2.0%
3.0%
4.0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Time Period
Perc
ent
2002-2003 2003-2004 2004-2005
Limits: Below 2%
Interpretation: Meeting accepted limits all the time
Limitations Definition may include some true infections and may miss others
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.
Exit plan: Reactivate with cause
Objective: to ensure that blood culture results reflect sepsis.
Methodology: Count single bottle positives of common skin flora/ total sets
Underfill Blood Collections (As a percent of collections per site)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
RF2
RF1
ER
ICU
Phlebotomists
Limits: Below 2% (?)
Interpretation: Wards with inexperienced collectors have problems
Limitations Some frail and elder people have very weak veins and may be impossible to collect
Presentation: Linear time graph
Action plan: Identify and educate blood collector group.Exit: Continue on selective basis
Objective: to ensure that blood culture are properly
filled.
Methodology: Count underfilled bottles / total bottles collected
Underfill Blood Collections (As a percent of collections per site)
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
RF2
RF1
ER
ICU
Phlebotomists
Certification Performance
Year Event Measures MAJOR
NC
Minor
NC
2002 Pre-Certification (EI) 100 1 2
2002 Certification (E) 100 0 2
2003 Pre-Certification (EI) 100 0 0
2003 Certification (E) 100 0 0
2004 Certification (E) 100 0 0
2005 Pre-certification (EI) 100 0 1
2005 Re-Certification (E) 100 0 0
Limits: No Major above 1Below 2%; No Minor above 3
Interpretation: Meeting accepted limits all the time
Limitations May indicate things are better than they are if inspector is not diligent
Presentation: Linear time table
Action plan: Maintain program, respond through OFI and Corrective Actions
Exit plan: Compile with each inspection
Objective: to monitor CMPT quality preparedness
Methodology: Monitoring External assessment values
Year Event Measures MAJOR
NC
Minor
NC
2002 Pre-Certification (EI) 100 1 2
2002 Certification (E) 100 0 2
2003 Pre-Certification (EI) 100 0 0
2003 Certification (E) 100 0 0
2004 Certification (E) 100 0 0
2005 Pre-certification (EI) 100 0 1
2005 Re-Certification (E) 100 0 0
Composite Indicators
• Reflecting a single subject with a number of sub-components
When the finished value is greater than just the sum of the parts
Creating Composite Quality Indicators
1
2
34
5
25
üIdentify individual componentsüWeight the componentsüDefine LimitsüMeasure and CombineüMonitor for trend
CMPT Client Satisfaction Composite Score
Factor Weighting
Survey Score +10
Open Comments –Positive +5
Open Comments – Negative -10
New Contracts +10
Contract Renewals +25
Contract Cancellations -100
Consults +5
Complaints -10
Lim
its
Su
rvey
Positiv
e O
pin
ion
s
Neg
ativ
e O
pin
ion
s
New
Con
tracts
Con
tract R
en
ew
als
Con
tract
Can
cella
tion
s
Con
su
lts
Com
pla
ints
VA
LU
E
90 30 5 1 1 0 1 0 105
80 30 16 0 0 0 0 1 76
Year
Su
rvey
Positiv
e O
pin
ion
s
Neg
ativ
e O
pin
ion
s
New
Con
tracts
Con
tract R
en
ew
als
Con
tract
Can
cella
tion
s
Con
su
lts
Com
pla
ints
2002-2003 90 24 6 0 0 0 5 2
2003-2004 85 22 10 4 0 0 5 0
2004-2005 85 22 6 6 0 0 3 0
2005-2006 85 20 2 2 1 0 4 1
CMPT Client Satisfaction Composite Score
96.5 96.5
103.5
100
70
80
90
100
110
2002-2003 2003-2004 2004-2005 2005-2006
CMPT Composite Satisfaction Score
• Objectives: To indicate customer satisfaction
• Methodology: Examination of 5 independent variables
• Presentation: Composite score
• Interpretation: Score associated with satisfaction
• Limits: 76-105 calculated weighted score
• Limitations: Arbitrary
• Action plan: Root Cause Analysis of deficiencies
• Exit: Annual for 5 years and evaluate
In Summary
• Quality Quality Indicators are a required component of a quality management system.
• Quality Quality Indicators can be characterized and distinguished from Weak and Terrible Quality Indicators.– Watch out for the weak ones– Avoid the terrible ones
• Quality Quality Indicators provide the information and opportunity essential for POSITIVE action.
Setting Relevant Ranges
• Set Objectively
• Validate by Study
• Clinical Relevancy
• Customer Expectation
• Matched Benchmarks
• Regulation 60 minutesRelevant or Easy?
Quality Indicators and Timing
Use an indicator only as long as
it providesyou with
useful information.
Don’t get tied to your indicators
Caution about patient outcome indicators
Theoretically, outcomes best assess quality, but they are the most difficult to measure – too many variables and confusers
• Age, underlying conditions, therapy, circumstance
– require high volumes of detailed data – Need long collection periods.
David Hsia
The BIG SECRET for Quality Indicator Team
Engage the folks who do the work, because they know what they do!
QualityCountsBalanced
Measurable Actionable
Interpretable
Engaging
Timed
Achievable