participant 2 participant 3 - aomevents.com nsm 2014/thomson.pdfthat rcpa qap and other external...
TRANSCRIPT
18/08/2011
1
0.860.82
0.73
0.62
0.28
0.03 0.01
0.17
0.08
0.49
0.8
0.68
0.07
0.14
0.020.01
0.05 0.090
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10
KPI
All analyte KPI Indicator analyte KPIBest
Worst
Participant 1Chemical Pathology QAP
Participant 2Chemical Pathology QAP
6
9 9
6
8
3
5
4
5
2 2
1
3
2
4 3
2 1 1 2 1 4 3 2 2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Concordant Minor Discordance Discordant
2008_1 2008_2 2008_3 2009_1 2009_2 2009_3 2010_1 2010_2 2010_3Average Median Score
Median Score 70% 92% 93% 68% 86% 55% 62% 62% 67% 73%
% Rank 3% 45% 50% 11% 57% 1% 1% 2% 3% 3%
Participant 3Anatomical Pathology QAP
18/08/2011
2
Participant 4Cytopathology QAP
2004 2005 2006 2007 2008 2009 2010
Major error 0 3 0 3 0 0 1
Unacceptable response 0 0 0 0 1 0 3
No slides 2444 2772 2821 3266 632
PM 1 2.50 2.78 4.08 2.27 2.37
PM 2b 1.6 1.4 2.8 1.1 0.4
PM 3a 25.0 60.0 70.0 69.2 100.0
PM 3b 33.3 46.2 36.4 22.2 100.0
PM 4 0.0 8.0 2.4 12.0 7.7
Participant 5Cytopathology QAP
2004 2005 2006 2007 2008 2009 2010
Major error 1 0 0 0 1 0 0
Unacceptable response 1 0 0 0 1 1 2
LBC major 0 0 0 0 0 1 1
LBC unacceptable response
0 0 1 1 0 0 1
No slides 54529 53891 50464 51385 50579 50909
PM 1 1.08 0.97 0.78 0.94 1.25 1.1
PM 2b 0.4 0.6 0.6 0.5 0.9 0.6
PM 3a 79.4 75.7 77.6 81.7 64.3 76.8
PM 3b 59.4 60.0 53.8 63.6 57.0 63.2
PM 4 1.0 1.7 3.2 0.5 2.3 1.1
Participant 6Haematology QAP - Haemoglobinopathy
Participant 7Haematology QAP - Haemoglobinopathy
18/08/2011
3
Participant 8Haematology QAP - Morphology
2001
• In 2001 the Commonwealth Department of Health and Ageing sought proposals to undertake an evaluation of Australian pathology laboratory accreditation arrangements.
• This was the first comprehensive evaluation since the introduction of accreditation in 1986. This research resulted in the 2002 Corrs Chambers Westgarth report ‘Evaluation of the Australian Pathology Laboratory Accreditation Arrangements’.
18/08/2011
4
Evaluation of the Australian Pathology Laboratory Accreditation Arrangements
for the
Commonwealth Department of Health and Aging
REPORT
17 July 2002
Chapter 5. External Quality Assurance Programs
• Recommendation 5.1That the DHA and the HIC seek the cooperation of the RCPA QAP to establish explicit external quality assurance performance criteria, initially in chemical pathology and gynaecological cytology, and a mechanism for the RCPA QAP to identify relatively poorly performing laboratories.
• Recommendation 5.2That RCPA QAP and other external quality assurance providers regularly submit to NATA reports identifying laboratories that are poorly performing according to these agreed performance criteria.
KPI Project• From this recommendation, in 2004/2005 the KPI project was
established by the RCPA and RCPA QAP.
• Key Performance Indicators were setup for Chemical Pathology and Cytopathology with the hope that it could be used as a mechanism to identify unacceptable laboratory performance.
• In 2006/2007 a collaboration was established between RCPA, RCPA QAP and NATA.
• Peer Review Committees were established by NATA to review a limited amount of KPI data.
KPI Project conclusions
• The KPIs and Peer Review Committee process can identify laboratories with poor EQA performance but this does not necessarily equate to poor ratings at on-site NATA assessment.
• The KPIs for Chemical Pathology have continued to be sent to laboratories every 6 months.
• A guide for laboratories on which results should be reviewed was established and distributed to laboratories.
• Cytopathology - KPIs are not a valid tool to use for identifying poorly performing laboratories and therefore it was recommended that they not be implemented in the existing format.
18/08/2011
5
Department of Health and Aging
0.13
0.55
0.13
0.01
0.09
0.01
0.24
0.75
0.03
0.18
0.96
0.080
0.050.01
0.53
0.67
0.020
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10
KPI
All analyte KPI Indicator analyte KPI
Chemical Pathology QAP
Worst
Best
EQA is one aspect of accreditation
Accreditation
EQA
18/08/2011
6
‘The role of External Quality Assurance in identifying laboratory performance’
• Review four years of Chemical Pathology KPI data retrospectively
• Develop and evaluate KPIs for :– Anatomical Pathology– Transfusion Medicine– Cytopathology
• Develop triggers of concern using the external quality assurance data
• Trial the protocols to determine whether early indicators of unacceptable performance can be developed around the regular EQA that laboratories perform as part of the accreditation process.
• Establish mechanisms to use EQA data to help monitor quality of test kits
Review process
• Established a methodology / scoring system
• Retrospective data analysis
• Set criteria– Test performance– Participation
• Reviewed criteria against individual reports
• Assessed fit to framework (workability)
1
12
8
10
54
8 8 87
19
11
14
9 8
1213 14
22
25
2223
1917
18 19
35
29
36
26 26
21
2322
27
46
0
5
10
15
20
25
30
35
40
45
50
Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10
Percentage returned Chemical Pathology KPI
Freq <50% returned Freq <60% returned Freq <70% returned Freq <80% returned
3
15
1
106 8 6 8 88
19
2
1813 12 12
1715
24
42
9
32
24 2534
28 28
39
64
24
50
3843
4743
54
137
148
119
128 125
159
172 170
162
0
20
40
60
80
100
120
140
160
180
200
Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10
Percentage late Chemical Pathology KPIFreq >50% late Freq >40% late Freq >30% late Freq >20% late Freq >10% late
18/08/2011
7
12
01 1 1 1 1
02
5 54
35
87 7
1311
15 15
7
21
17
2122
36
25
44
28
21
41
37
41
49
0
10
20
30
40
50
60
Dec‐06 Jun‐07 Dec‐07 Jun‐08 Dec‐08 Jun‐09 Dec‐09 Jun‐10 Dec‐10
Percentage amended Chemical Pathology KPIFreq >20% Freq >15% Freq >10% Freq >5% Anatomical Pathology
0.00
0.50
1.00
1.50
2.00
2.50
3.00
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
100% 94% 81% 72% 67% 50% 45% 30% 19% 9%
Average Discordan
t Sum
Average Med
ian Score
Overall Participant Rank
General 2007 – 2010 excluding nonparticipation (433 participants)Average score over all surveys Average sum over all surveys
Participation 2007 –2010
2007Survey 1
2007Survey 2
2007Survey 3
2007Survey 4
2007Survey 5
2008Survey 1
2008Survey 2
2008Survey 3
2009Survey 1
2009Survey 2
2009Survey 3
2010Survey 1
2010Survey 2
2010Survey 3
Completed surveys 291 302 293 301 282 298 295 265 301 303 286 276 279 270Survey Nonparticipation 21 12 18 13 32 23 26 47 24 27 44 33 47 56Case Nonparticipation 1 1 3 3 2 2 3 11 3 4 5 8 6 5
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
x1 x2 x3 x4 x5 x6 x7 x8 x9Case 32 3 3 0 0 0 0 0 0Survey 104 52 21 6 2 1 1 0 1
0
20
40
60
80
100
120
Nonparticipation ‐ number of times
Non-participation
18/08/2011
8
Gynaecological Cytopathology
2007 2008 2009 2010
x1 Major Error 11 7 7 9
x3 Unacceptableresponse in 12 months 3 1 0 1
x1 PerformanceMeasure outside national standard
11 9 11 ‐
x1 Non return of QAP results 0 0 1 0
x1 Non return of PerformanceMeasure 0 0 0 0
• Number of participants outside criteria
Draft Framework
4. Criteria: Participants falling outside the criteria set for acceptable performance in the next survey or no response has been received within four weeks from the second letter sent, will be referred to the RCPA QAP
Performance Review Committee.
Action: A copy of the follow‐up letter or letter, EQA results and participants
response is sent to NATA.
Action: The Chairperson or nominated Pathologist will contact the Medical
Director directly.
3. Criteria: Persisting unacceptable performance is referred to the Program Performance Review Committee.
Action: A copy of the letter, EQA results and participant response is
sent to NATA.
Action: A letter is sent to the participant and nominated
supervisor(s) from the Performance Review Chairperson.
2. Criteria: Identification of results falling outside the criteria set for acceptable performance.
Action: Initial letter sent to participant by the Program
Manager.
1. Enrol in QAP.
Participants Results, letter a
nd re
spon
se re
ferred
to
NAT
A
Participants Results m
onito
redfor o
ne year
Results re
view
ed and
actione
d by
participant w
ithin 4 weeks
5. Findings and discussions with the manufacturer, their sponsor or TGA referred to RCPA QAP Review Committee.
4. Manufactoring company or their sponsor informed of findings by the Performance Review Committee Chairperson
3. Results outside the criteria set for acceptable performance of an Invitro Medical Device reviewed by the Program Performance Review Committee (IVD representative invited)
2. Results outside the criteria set for acceptable performance of an Invitro Medical Device identified by the QAP Program
1. Enrolment in QAP
Referred
to TGA
Where to from here
• Participant consultation
• Letter to participants
• Letter from Programs re: criteria
• Pilot 2012
18/08/2011
9
Survey – have your say! Acknowledgments
• Department of Health and Aging - QUPP
• RCPA QAP Program Managers
• Penny Petinos