Method Validation & Verification

(Introduction to Method Validation)

Wilai Chalermchan Senior Laboratory Advisor (Contractor)

DGHT/Global HIV/AIDS Program Thailand MOPH – U.S. CDC Collaboration (TUC)

1

Contents

• Introduction Standard requirements

• Validation/Verification Parameters, Experiments and Process

• Samples of method validation/verification plan and experiments

• Summary

2

Validation and Verification (MT Std)

Validation (การสอบทวน ) หมายถงกจกรรม หรอกระบวนการ เพอพสจนวา ขนตอนการทดสอบ กระบวนการทดสอบ ระบบการทดสอบ เครองมอ หรอวธการทดสอบทน ามาใช สามารถใชงานไดตามวตถประสงคทก าหนดไว

Verification (การทวนสอบ ) หมายถงการยนยนโดยการตรวจสอบและมหลกฐานแสดงวาเปนไปตามขอก าหนด

3

Validation and Verification (ISO 15189)

VERIFICATION: confirmation, through provision of objective evidence, that specified requirements have been fulfilled การจดหาหลกฐานทยนยนวาวธทดสอบมคณสมบตเปนไปตาม..... ….ความตองการเฉพาะ

VALIDATION: confirmation, through provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled การจดหาหลกฐานทยนยนวาวธทดสอบมคณสมบตเปนไปตาม...... ….ขอก าหนดตางๆในการน ามาใชงานหรอตามความมงหมายในการใชงานทก าหนด

4

Standard Requirements (MT Std.)

5.4.1.1 ....หากใชวธวเคราะหทก าหนดขนเอง ..ไมใชวธสากล หรอดดแปลง ตองมขอมลผานการสอบทวน (validation) เทยบเคยงกบวธทสากลยอมรบในขณะนน บนทกผลการสอบทวนและการทวนสอบ คมอวธปฏบตการวเคราะห • ขอมลการสอบทวน (validation) ไดจากผผลต หรอผพฒนาวธวเคราะห ขอมลตองแสดง ถงความเทยงตรง ถกตอง แมนย า สามารถระบความคลาดเคลอน ความจ าเพาะ ความไว ชวงวเคราะห และความสอดคลองกบอาการทางคลนกได

Standard Requirements (MT Std.) How….

• ขอมลการทวนสอบ (verification) ไดจากขอมลเชงประจกษเพอยนยนไดวาเหมาะสมกบการใชงานในงานบรการ

verification parameter…..? 5.4.1.2 ทบทวนวธวเคราะห อยางนอยปละ 1 ครง... บนทก 5.4.1.3 มการทบทวนคาอางอง อยางนอยปละ 1 ครง

5.4.2 คณลกษณะของแตละวธวเคราะหตองเหมาะสมสอดคลองตามวตถ ประสงคการใชงาน และตามความตองการของผใชบรการ

6

Method Evaluation

Validation

• Establishment of performance characteristic by defining and characterizing the magnitude of the analytical error present

• Performed by the developers of the method (e.g. manufacturers)

• Requires extensive studies to reveal and assess the error present

Verification • Confirmation of known

performance parameters published by the developers of the method.

• Performed by the laboratory to explore error in system

• Requires studies to confirm manufacturer's claims

7 Slide from Anna Murphy, SLAMTA program

Why method need to be validated/verified?

8

Propose of Method Validation/Verification

• Identification of Sources and Quantitation of Potential errors

• Determination if Method is Acceptable for Intended Use • Establish Proof that a Method Can be Used for Decision

Making • Satisfy Regulatory Requirements

9

Process for Establishing a Routine Test

Establishing a Laboratory Routine Test

Select, Evaluate Diagnostic Test

Select Method of Analysis

Validate/Verify Method Performance

Implement Method

Perform Test

Method Improvement

- Maintain Method

- Prevent Problem

Check with Statistical QC

Acquire Specimen

Report Results

A Routine Laboratory Testing Process

Define test SOP - Select appropriate QC -

Personnel training -

10

Why method need to be validated/verified?

11

Validation Parameters and Experiments

12

Common Validation Parameters for a Quantitative Test

• Precision • Accuracy • Linearity Analytical Measurement range Clinical Reportable Range

• Sensitivity • Limit of detection, Limit of quantification

• Specificity • Reference Interval

13

Performance Characteristic required by Thai FDA on HIV Test

• Analytical sensitivity • Diagnostic sensitivity • Diagnostic specificity • Reproducibility • Inhibition • Cross contamination • Whole system failure rate

Precision

14

Performance Characteristics must be relevant to the laboratory intend use. 5.4.2 คณลกษณะของแตละวธวเคราะหตองเหมาะสมสอดคลองตามวตถ ประสงคการใชงาน และตามความตองการของผ ใชบรการ

15

What should I do?

The Guideline

CLIA (Clinical Laboratory Improvement Amendment) Non-waived test

• Application: after April 2003 • Unmodified, FDA-approved test

1. Performance specifications comparable to the manufacturer • Accuracy • Precision • Reportable range

2. Verify manufacturer’s reference interval (normal range)

16

Performance Characteristic to consider Example:

Application

• diseases monitoring

• For diagnosis of infectious diseases

• Blood safety

• High workload

Performance Characteristic

• Reproducibility

• Specificity, False positive

• Sensitivity, LOD

• System failure rate

17

Performance Characteristic and Experiments Performance Characteristic

Experiments

Accuracy • Comparison methods to determine systemic error (SE) or bias

Precision • Repeatability

• Intermediate precision

• Reproducibility

• Replication experiment

Reportable range • Linearity experiment

Reference internal • Study sample from normal/healthy person

• Adopt from the publication or reference lab 18

Verification Process • Establish General Validation/verification procedure; • Define the application, purpose and scope of the method use; • Method selection; (from manufacturer claim) • Establish validation/verification plan/protocol;

• Define the validation/verification parameters and experiments; • State of the acceptability;

• Perform pre-validation experiments: • Adjust method parameters and/or acceptance criteria if

necessary: • Perform full validation experiments: • Record, report with statement of acceptance; • Method approval 19

Define Laboratory Application Requirements

Three types of test characteristic can be considered •Application Characteristic: Cost, type an vol of spec,

TAT, workload, space, portability, personnel •Methodology Characteristic: principle contribute to

the best performance, chemical reaction, rigor of analytical procedure

•Performance Characteristic: reportable range, precision, accuracy, detection limit 20

Sample of defined POC Performance Characteristic or specific requirements

Application

• WB from finger stick or with anti coagulant

• Ease of use for non lab-personnel

• RT° storage

• Minimal maintenance and downtime

• Cost less than..

• Size, weight, portable

Performance

• Specified reportable range

• Reproducibility less than 3%CV

Methodology

• Enzymatic reaction

• Built in calibration and traceable

• TAT of 10 mins or less

21

Considerations Prior to Method Validation/Verification

Consider Suitability of • Instrument

Status of Qualification and Calibration • Materials

Status of Reference Standards, reference material, Reagents lots

• Analyst Status of Training and Qualification Records

• Documentation approved protocol with pre-established acceptance criteria and worksheet 22

Samples of Method Verification Experiment

23

Sample 1: Method evaluation/verification at National level

AIM: Compare two new NAT methods for use in routine infectious screening……….

STUDY DESIGN AND METHODS:

The sensitivity, specificity, and robustness were determined by testing 486,676 seronegative blood donations within one year.

Samples from each day of collection were divided into two sets; the odd-numbered samples were tested individually on the TIGRIS and

the even-numbered samples were tested in pools of 6 on the cobas s 201.

The status of reactive samples was confirmed by… - duplicate testing of samples from the plasma bag to

calculate the test specificity. - tested on the alternate system and followed up.

Reference: Phikulsod et al. One-year experience of nucleic acid technology testing for human immunodeficiency virus Type 1, hepatitis C virus, and hepatitis B virus in Thai blood donations. TRANSFUSION, Volume 49, P 1126-1135 24

Evaluation Parameters

Analytes: HIV-1, HBV and HCV nucleic acid

Analytical sensitivity WHO International Standards for HCV (NIBSC 96/798), HBV (NIBSC 97/746), and HIV-1 (NIBSC 97/760). The standards were diluted to the 95% detection limit for each assay. • 3, 10, and 50 IU/mL for HBV, HCV, and HIV-1 for the Roche MPX test. • 3, 10 and 30 IU/mL for HBV, HCV, and HIV-1, for the Ultrio test IU/mL

for HBV, HCV, and HIV-1, respectively • testing individually 24 replicates

25

Cross contamination studied by including five samples containing high titers of HBV (>108 IU/mL), HCV (105 IU/mL, NIBSC 96/798), and HIV-1 (280,000 copies/mL, determined by the COBAS TaqMan HIV-1 test for use with the High Pure s ystem), which were randomly distributed in the batches of routine blood donor samples that were tested in 1 day.

Genotype detection (detectability) Genotypes of the three viruses, isolated in Thailand, were - 41 HIV-1 isolates, 12 subtype B and 29 subtype A/E (CRF01_AE); - 20 HBV isolates, 8 genotype B and 12 genotype C; and - 20 HCV isolates, 4 genotype 1a, 14 genotype 3a, and 2 genotype 3b, were tested.

26

Clinical specificity and sensitivity - The initial-reactive and repeat-reactive rates for each test were calculated. - Follow-up study of donors with reactive donations All donors with confirmed reactive donations were followed up until seroconversion or until a conclusion about the donor status could be reached. (Sensitivity) - 5000 donor samples, found nonreactive by each test, were retested using the alternate test. The testing was included during the routine testing (Specificity)

27

Whole system failure rate The overall failure rate for the two systems was monitored. Any failure that resulted in the inability to release donor results was regarded as a system failure. In addition, the human errors and the invalid controls (both run controls, internal controls, and in the case of the TIGRIS, the calibrators) were monitored.

28

Sample 2: Verification of new Chemistry Analyzer Validation Plan 1) Aim: The verification will be conducted on the X Chemistry Analyzer , serial

number …. for AST (Aspartate Amino Transferase) compare with in-service method Y during dd/mm/yy…. to dd/mm/yy ……

2) Description …….background, method, scope of use and specific requirements

3, Parameters, experiment methods and result record a. Precision: Repeatability, Intermediate precision

- Description, detail procedure, material and acceptance criteria…….

b. Accuracy/correlation ………. c. Linearity: ………. d. Sensitivity (LOD and LOQ): e. Specificity/interferance: ………. f. Reference Ranges: ……….

4. Study Conclusion and Method Approval: ………. 5. Reference: ……….

29

Sample: Experimental worksheet

30

Analyst

MethodManufacturerAnalyteUnits of MeasureSample name/Description

QC Multi 2; ; Lot# QC223; exp 30-Aug 20XXMaterial 1Material 2Material 3

Precision ExperimentShort Term (within-run)/Repeatability

Miss USA3-Nov-15Date

XYZ Cheistry AnalyzerABCASTU/L

QC Multi 1; Lot# QC123; exp 30-Aug 20XX

Run # Material 1 Material 2 Material 3 Material 1 Material 2 Material 31 35 120

2 34 119 Mean 35.0 U/L 120.0 U/L3 35 120 SD 0.8 U/L 1.5 U/L4 35 121 %CV 2.20% 1.25%5 35 120

6 36 121 Manufacturer's Claim7 35 120 Mean 36.6 U/L 128 U/L8 34 118 SD 0.3 U/L 1.0 U/L9 36 120 %CV 0.80% 0.40%

10 35 122

11 35 119 25% of CLIA Allowable error is 5%12 35 120 Short term precision is acceptable13 35 121

14 35 117

15 34 119

16 35 118

17 35 120

18 37 122

19 36 123

20 34 120

Experiment Results

(acceptable/unacceptable)

Preliminary Estimate of Precision, Short Term

Laboratory

What’s next?

31

On-going Verification

• Test performed by multiple systems required comparison of test results (Comparability study)

• Monitoring of test precision through QC • Monitoring of accuracy PT/ EQA results

No PT program- at least semi annual re-validate/verified

• Kit lot verification

32

Re-validation

When

• Instrument repaired, relocate

• Method performance have been significant

changed

• Scope of application has been change

33

What are the evidence that an assessor looking for?

Minimum • General method validation/verification procedure • Validation/Verification plan/protocol • Records • On-going verification measure(s) and records

34

Summary Method verification • Procedure, plan and records; • Meet specific intend use and extensive; • before conducting the study, qualify instrument and

ensure personnel competency; • observe errors that may have in our laboratory

system; • First step in establish quality control procedure • on-going verification, re-validation

35

References

36

Questions & Comments

37