Quality Control and Patient RiskCurtis A. Parvin, Ph.D.Manager of Advanced Statistical ResearchQuality Systems Division

2 |

Laboratory QC and Patient Risk

• For the clinical laboratory patient risk is related to the reliability of the patient results they produce and report.

• Laboratory Quality Control principles have been connected to the concept of patient risk for decades.• Specify the quality required of a patient result.• Determine the magnitude of “critical” out-of-control

conditions with unacceptably high probability of producing patient results that fail to meet the required quality.

• Design QC procedures with adequate power to detect “critical” out-of-control conditions.

3 |

Risk Management

• A number of risk management guidelines have been published recently.

• For manufacturers• ISO 14971: Medical devices – Application of risk

management to medical devices• CLSI EP18: Risk management techniques to identify and

control laboratory error sources• For the laboratory

• ISO 22367: Medical laboratories – Reduction of error through risk management and continual improvement

• CLSI EP23: Laboratory quality control based on risk management

4 |

The Risk Management Process

• Risk Management provides a formal approach to• Identify potential failure modes in the lab

• Rank the identified failure modes in terms of their risk• Establish policies and procedures to prevent or reduce

(mitigate) the risks• Focus on the high ranked risks

• This is generally a qualitative process• Risks are computed as a combination of

• The probability of occurrence of harm• The severity of harm

5 |

Statistical QC and Risk Management

• The question that we will address today is:

How do statistical QC design principles that address questions such as

• How many QCs should I examine?• What QC rule(s) should I use?• How frequently should I schedule QC evaluations?

fit into the overall risk management process?

6 |

Risk Assessment (EP23 Figure)

Hazard Identification

• Create a process map• Identify potential failures in each process step• Determine mechanisms in place to prevent or detect a failure

Risk Estimation

• Assess the likelihood or probability of harm for each failure• Assess the severity of harm to a patient from each failure

Risk Estimation

Is the residual riskof harm clinically

acceptable

The Laboratory’s QCP

• Compile set of QC process into QCP• Review QCP for conformance to regulatory and accreditation requirements• Document and implement the set of control processes as the laboratory’s QCP

No

Yes

Risk Control

What control processes are needed to lowerrisk to an acceptable level

7 |

Risk Assessment (EP23 Figure)

Hazard Identification

• Create a process map• Identify potential failures in each process step• Determine mechanisms in place to prevent or detect a failure

Risk Estimation

Assess the likelihood or probability of harm for each failure• Assess the severity of harm to a patient from each failure

Risk Estimation

Is the residual riskof harm clinically

acceptable

The Laboratory’s QCP

• Compile set of QC process into QCP• Review QCP for conformance to regulatory and accreditation requirements• Document and implement the set of control processes as the laboratory’s QCP

No

Yes

Risk Control

What control processes are needed to lowerrisk to an acceptable level

8 |

Sequence of Events Creating Risk of Harm for a Patient

Probability of Harm

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reportedMisdiagnosis

Hazardousmedicalaction

Patientharmed

P1 P2 P3 P4 P5 P6

Hazardous Situation

• Estimates of the probability of harm are desirable• For each potential mode of failure

• Collectively for all possible failure modes

EP23-A, Figure 6ISO 14971, Figure H.1ISO/TS 22367, Figure B.1

9 |

Estimating Probability of Harm

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reportedMisdiagnosis

Hazardousmedicalaction

Patientharmed

P1 P2 P3 P4 P5 P6P1

P1: The probability that a testing process failure occurs

10 |

P1: Probability of a Failure Mode

• P1 = how likely is a given failure mode going to occur

or more commonly • P1 can be the mean time between failures (MTBF)

Information regarding MTBF may come from•Manufacturer•Historical failure data• Literature

• MTBF can be converted to mean number of patient results between failures (MPBF)• MTBF = 30 days• 150 patient results examined per day (on average)• MPBF = 30 * 150 = 4500 patient results

11 |

Estimating Probability of Harm

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reportedMisdiagnosis

Hazardousmedicalaction

Patientharmed

P1 P2 P3 P4 P5 P6MPBF P2MPBF

P2: The probability that a testing process failure generates incorrect patient results

12 |

P2: Probability of Producing an Incorrect (Unreliable) Result

• The number of unreliable patient results produced during the existence of an out-of-control condition depends on• The quality required for patient results• The type and magnitude of the out-of-control condition• The power to detect the out-of-control condition• When (how often) QC evaluations are performed

• An out-of-control condition may result in many unreliable patient results being produced

13 |

Unreliable Patient Results

• ISO 15189 Clause 5.6.1: Laboratory QC should assure that patient results meet the quality required for their intended use

• The quality of a patient result depends on the difference between the correct value and the value reported.

• If the error in a patient’s result exceeds the allowable total error (TEa)

• The result is considered unreliable (or incorrect).• It creates a hazardous situation for the patient.

14 |

Expected Number of Unreliable Patient Results Produced: E(NU)

• The number of unreliable patient results produced during an out-of-control condition (red asterisks) will depend on• The magnitude of the out-of-control condition• The power of the QC rule – Detection• The number of patient specimens between QC events

• E(Nu) = Expected number of unreliable patient results generated during an out-of-control condition

E(Nu)

15 |

Estimating Probability of Harm

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reportedMisdiagnosis

Hazardousmedicalaction

Patientharmed

MPBF P2 P3 P4 P5 P6E(Nu)E(Nu) P3

P3: The probability that incorrect results that are generated will be reported

16 |

P3: Probability of Reporting an Unreliable Patient Result

• The probability of reporting an unreliable result that leads to an incorrect action depends on• The number of unreliable results produced because of an

out-of-control condition• How and when results are reported

• The likelihood of identifying and correcting a reported unreliable result before an incorrect action is taken

17 |

E(Nu)

E(Nuf) E(Nuc)

• The patient results produced during an out-of-control condition can be divided into•Results prior to the last accepted QC event (Pre)•Results since the last accepted QC event (Post)• Bracketed QC: results aren’t reported• Immediate reporting: results should be repeated and updated in a

timely fashion

• E(Nuf) = Expected number of unreliable “final” results

• E(Nuc) = Expected number of unreliable “correctable” results

Expected Number of Unreliable Patient Results Reported

18 |

Estimating Probability of Harm

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reportedMisdiagnosis

Hazardousmedicalaction

Patientharmed

MPBF E(Nu) P3 P4 P5 P6E(Nuf)E(Nuf) P4 P5 P6

P4: The probability that an incorrect result leads to a misdiagnosis

P5: The probability that a misdiagnosis leads to a hazardous medical action

P6: The probability that a hazardous medical action leads to patient harm

P4*P5*P6: The probability than an incorrect result leads to patient harm

19 |

Probability of Patient Harm from an Unreliable Patient Result

• The probabilities (P4, P5, P6) are associated with activities outside the laboratory

• Their product equals the probability of patient harm given that an unreliable result has been reported

Ph|u = P4*P5*P6

• Estimates of the probability of harm given an incorrect result are based on;• Medical literature• Medical judgment• Consultation with clinical colleagues

20 |

Estimating Probability of Harm

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reportedMisdiagnosis

Hazardousmedicalaction

Patientharmed

MPBF E(Nu) E(Nuf) P4 P5 P6Ph|u

21 |

Estimating Probability of Harm: Risk Management + Statistical QC

Initiatingcause

Testingprocessfailure

Incorrectresult

generated

Incorrectresult

reported

Patientharmed

MPBF E(Nu) E(Nuf) Ph|u

Risk ManagementStatistical QC

22 |

Harmed Patients from Failure Mode Occurrence

MPBF

E(Nuf)

E(Nu)

E(Nuc)

** *

Ph|u*E(Nuf)

• Probability of patient harm is• The expected number of patients harmed due to incorrect

results from a failure mode: Ph|u* E(Nuf)

•Divided by the average number of patient results examined between failure mode occurrences: MPBF

Probability of Harm = Ph|u* E(Nuf) / MPBF

23 |

• The quality requirement for the analyte

• Allowable Total Error (TEa)

• Test method performance• Instrument imprecision and bias

• QC procedure• QC rule(s) used• Number of QC levels evaluated at each QC event• QC schedule (average number of patients examined for the

analyte between QC events)

What Info Do We Need to Predict E(Nu) and E(Nuf)?

• The quality requirement for the analyte

• Allowable Total Error (TEa)

• Test method performance• Instrument imprecision and bias

• QC procedure• QC rule(s) used• Number of QC levels evaluated at each QC event• QC schedule (average number of patients examined for the

analyte between QC events)

Needed to compute theprobability of anunreliable patient result

• The quality requirement for the analyte

• Allowable Total Error (TEa)

• Test method performance• Instrument imprecision and bias

• QC procedure• QC rule(s) used• Number of QC levels evaluated at each QC event• QC schedule (average number of patients examined for the

analyte between QC events)

Needed to compute theexpected # ofaffected patient results

24 |

Probability of an Unreliable Result Due to a Systematic Error Condition

25 |

Probability of an Unreliable Result Depends on TEa, Method CV and Bias

26 |

Expected # Affected Depends on QC Rule and QC Schedule

27 |

Expected # of Unreliable Patient Results

27 |

28 |

Ways to Reduce (or Manage) Patient Risk

Estimate Risk Reduce Risk

Rate of Occurrence of Failure Modes

Yes: manufacturer info, technical bulletins, product alerts

Some: lab environment/processes training

Very little: Medical devices

Number of Incorrect Results Reported

Yes: Statistical QC design & evaluation

A lot: Detection in analytical phase

A little: Detection in pre- and post-analytical phases

Probability of Harm from an Incorrect Result

Yes: Medical literature, medical judgment, consultation

Very little: Report formatting, education

29 |

Ways to Reduce (or Manage) Patient Risk

• Manage the risk of unreliable patient results in the worst case• Limit the maximum values for E(Nuf) and E(Nuc)

• Manage the overall expected risk of unreliable patient results• Limit the areas under the E(Nuf) and E(Nuc) curves

• Manage the “defect rate” for unreliable patient results• Requires an estimate of the rate at which out-of-control conditions occur

30 |

Summary

• The laboratory has two main mechanisms to estimate and reduce the risk of patient harm:

1. Identify as many potential failure modes as possible and seek ways to reduce occurrences of the identified failure modes

Risk management

2. Implement QC strategies that minimize the number of incorrect patient results that are reported when a failure mode does occur

Statistical QC

• The combination enables a laboratory to address the full spectrum of patient risk implications for their operations

31 |

References

• Parvin CA. Assessing the impact of the frequency of quality control testing on the quality of reported patient results. Clin Chem 2008;54:2049-54.

• Parvin CA, Yundt-Pacheco J, Williams M. Analytical assessment in the clinical laboratory: assessing analytical quality goals when the same analyte can be tested on multiple systems is explored. Adv Admin Lab 2011;20(1):28-31.

• Parvin CA, Yundt-Pacheco J, Williams M. The focus of laboratory quality control: why QC strategies should be designed around the patient, not the instrument. Adv Admin Lab 2011;20(3):48-9.

• Parvin CA, Yundt-Pacheco J, Williams M. Designing a quality control strategy: in the modern laboratory there are 3 questions that must be answered. Adv Admin Lab 2011;20(5):53-4.

• Parvin CA, Yundt-Pacheco J, Williams M. The frequency of quality control testing: QC testing by time or number of patient specimens and the implications for patient risk are explored. Adv Admin Lab 2011;20(7):66-9.

• Parvin CA, Yundt-Pacheco J, Williams M. Recovering from an out-of-control condition: the laboratory must assess the impact and have a corrective action strategy. Adv Admin Lab 2011;20(11):42-4.

• Parvin CA, Yundt-Pacheco J, Williams M. Sigma metrics, total error budgets, and quality control: Make sure your test system performance and quality control

procedures are aligned with your quality goals. Adv Admin Lab 2012;21(1):40-4.

32 |

33 |

34 |

THANK YOU!