test method validation (tmv) - cbinet.com1).pdf1 test method validation (tmv) ... how to conduct tmv...

Test Method Validation (TMV) | Mar 2016 | 1

Mai Nguyen-Misra and George Omae

Medtronic

Test Method Validation (TMV)

How to Conduct TMV for Medical Device Industry


Presenter Bio’s

Mai Misra, Ph.D. Sr. Principal Quality Engineer, Medtronic Ph.D., Polymer Science; M.S., Physics (Solid State); M.S., Chemical Engineering; B.S., Applied Mathematics; B.S., Chemical Engineering Dr. Mai Misra, a chemical engineer by training, has diverse education & work experience. Mai received her PhD from University of Akron in Polymers Science in 1995. She also has a BS degree in Chemical Engineering and Applied Mathematics as well as a MS in Chemical Engineering and a MS in Solids State Physics. She has worked for over 20 years in process and product development for various applications serving diverse industries: US EPA (Reformulated Gasoline and Alternative Fuels), H.B. Fuller (Packaging, Window, Footwear etc), Boston Scientific (Cardiovascular Drug-Eluting Stents), and Medtronic (CRDM Pacing Leads). Mai is a co-inventor of 4 patents and has authored many peer-reviewed papers. She has been in the Medical Device industry for 9 years. Mai has been in the current role as a SME on TMV for the validation group (part of Neuromodulation Ops Quality) since 2014.

George Omae, M.S. ASQ CSSB, CQE Sr Engineering Manager, Medtronic B.S. Mechanical Engineering, Nairobi University M.S. Manufacturing Systems Engineering, Univ. of St. Thomas M.S. Applied Statistics, Rochester Institute of Technology MBA, Carlson School of Management, University of Minnesota George Omae, a mechanical engineer by training, has over 19 years experience in the Medical device industry and has worked in roles of increasing responsibility in Quality, Process Development and R&D. He currently manages a test development group. He has had significant experience developing processes and products for 510k and PMA while working at Guidant, Boston Scientific, American Medical Systems and Medtronic. As a Six Sigma Black Belt, he has trained and mentored engineers and other technical professionals in the application of quality engineering tools. He is also an experienced trainer for Test Method Validation, Process Validation and Sampling Plans. George is also an adjunct instructor at the St Cloud State University graduate school (Medical Technology Quality (M.S.))


Course Objective & Approach

Objectives: • Provide an overview of what is required to plan and execute

TMV activities

• Provide guidance on how to document TMV records to be in compliance with prescribed validation requirements

• Share best practices

Approach: • Presentation,

• Case Study,

• Audience Interaction


Course Flow

Section 1: TMV Overview

Presenter: Mai Nguyen-Misra

Section 2

Case study (Variables TMV)

1. TMV Planning

2. Executing & analyzing

3. Documenting TMV

4. Maintaining the validated state

Presenter: George Omae

Lessons Learned Mai Nguyen-Misra

1 2

3


Section 1

Test Method Validation (TMV) Overview


Benefits: • Assures consistence & compliance in the output • Ensures requirements are accurately tested throughout the

Product Lifecycle • Builds basis for evidence of requirements lifecycle traceability

TMV Purpose and Benefits

Purpose:

To demonstrate that a test or inspection method is suitable for its intended purpose and is capable of producing valid results.


Definitions

• Test Method: A definitive procedure that produces a test result. It includes any sample preparation, measurements, observations and calculations.

• Test Method Validation (TMV): The process by which a test method has been demonstrated with objective evidence to be suitable for its intended purpose and capable of producing valid results.


TMV Overview (inputs & output) – Risk-based


TMV Types

Analytical

QMSWI16303

Pharma

PHAP1191

Fluke A Fluke B


Group Discussion – Sharing best practices

• What types of TMV does your organization commonly work with?

• Does your organization create separate work instruction for each type of TMV?


Factors that influence TM Variability

Workpiece (Part)

Measurement System

Variability/Error

Cause Effect

Standard/Reference Instrument

(Gage)

Person (Appraiser)

Environment

Traceability Design

Maintenance

Skill

Procedure

Ergonomics (e.g. lighting)

Vibration

Processing variability

Cleanliness Stability

Influencing factors to consider for control

Stability


TMV Process Flow

The governing procedure defines the process for demonstrating that a test or inspection method is suitable for its intended purpose and is capable of producing valid results.


Process Activities: Lifecycle Approach

4 TM Phases:

I – Good science/engineering

II – Preliminary Test Method

III – Characterized Test Method

IV – Validated or Verified Test Method

• TMs at Phase I,II,III require full documentation of characterization elements

• Phase IV TMs require validation to either Level 1 or Level 2 based on the Risk Identification

No requirement to go through these sequentially or document evidence of prior phase completion


TM Minimum Requirements


Process Inputs (pre-requisites) of TMV

Big 6!

Requirements

Equip qualified

Software validated

Tooling/Fixtures released

Harm/Risk classification

Documented test method

Training documented

Pre-requisites Tool/Source Comment

1

2

4

5

6

Product/print specE.g. Lead body tensile ≥ 7 lbf,

DC resistance ≤ 107Ω, Length: 45cm ± 2cm

E.g. IQ NRP1086-35723 (protocol), NRP1086-36375 (report),

OQ NRP1086-35724 (protocol), NRP1086-36376 (report)

Critical/Major/High risk need TMV,

Provides validation level & sample sizes

Example of content in QMS16001

Train to TM, store in Saba/other system

Per QMS1848E.g. MEDN-3285 (IQ)

E.g. PCPWI8362

E.g. QMS1799, DFMEA/PFMEA/PCRW

QMS16001MEDN-0002

E.g. MEDN-31261 MEDN-0093

Inspection report

3


TMV Level (Risk-based)

Determine TMV level based on risk:

Level 1 TMV – Moderate RL Gathering & documenting evidence to demonstrate TM capability Documentation : Report only Level 2 TMV – High RL Level 1 + additional study data to demonstrate TM capability Documentation: Protocol and Report


Maintaining a Validated State

Active TMs only

• Control Plans are required for Active Test Methods


Controls and Monitoring

• Document: • Type of controls, including frequency & criteria

• Required general controls (e.g., training, equipment calibration & maintenance)

• TM-specific controls where applicable (e.g., periodic re-training, quality control sample checks)

• Type & Location of evidence of implementation

• Document controls and actions to be taken if controls fail in TMV report

• Conduct Periodic Reviews


Maintaining a validated state (cont’d)

To Manage /Control Changes:

• Assess impact of changes – Consider impact of cumulative effects of all changes since validation

• Re-validate or provide justification

• Document decision and/or results

Some examples of changes

• TM changes: TM parameters, Equipment/Tooling, SW

• TM Intended Use Requirement Changes: Specifications, Risk level


Group Discussion – Sharing best practices

• What is your process for Maintaining Validated

State?


Section 2

Variables Test Method Validation


Module 2: TMV - Variables Key Elements Covered:

• Evidence required to demonstrate completion of pre-requisites

• How to design the validation study (sample size, sample part selection, appraiser selection, validation study steps)

• Case Study: Manual DC Resistance Test Method

Section Agenda


Variables TMV Overview

Full verification (100% inspection) – L2b Process control – L2c Acceptance sampling – L2a

Population

Inference

Sample


Module 2: TMV Performance Characteristic Requirements

Key differences


Variables Test Method Case Study: Manual DC Resistance


Case Study - Introduction

Twist-Lock DC Resistance Test Method

A test engineer is getting ready to conduct 100% incoming inspection and design verification testing. One of the requirements (specification) is to test the manual DC resistance of an improved cable used for initial evaluation for patients that have overactive bladder. The resistance per conductor must be between 0 - 10Ω. The risk level is high for this requirement.

Measure this resistance Verify Evaluation System


Case Study - Introduction

TMV Questions:

1. What is the minimum validation level required?

2. What is the %Resolution?

3. What is the %Tolerance?

4. What is required to maintain the validated state of the TM?

Measure this resistance Verify Evaluation System


Case Study - Select Validation Level

Start

Phase IV (100% Inspection) High Risk

Level 2 TMV (Variables nondestructive)


Case Study - Conduct TMV

Start


Case Study - Plan the study

Key planning items Requirement

Pre-requisites: Ensure all are released and controlled

Number of appraisers: 2

Number of parts: 10

Number of trials x3 per appraiser

Part selection: Select from typical/nominal processing conditions or surrogates with rationale

Label & randomize: Label & create randomization scheme using Minitab

Measurement location: Identify location of measurement

Sample size

Other considerations



MXXXX-XXX Report Template used

Question: What validation level is selected for 100% inspection?



Brief Description of the TM and Requirements



Sample size & justification

Equip info


Case Study - Execute protocol

Identify: Validation execution roles

Document: Step-by-step procedure for executing validation/verification. Include role performing each step.

Plan Study Verify Pre-requisites

Execute Protocol

Analyze & Determine

Acceptability

Document results


Case Study - Execute protocol

Randomize Data

Collection

Blind Data Collection


Case Study - Analyze

Resolution

Show your calculations


Execute Protocol

Analyze & Determine

Acceptability

Document results


Case Study - Analyze

% Tolerance

%Tolerance = 4.27


Do:

• State how the data was analyzed

• Present results with enough detail to show how acceptance criteria were met

• State where evidence is located (attaching in Appendices preferred)

Case Study - Document


Execute Protocol

Analyze & Determine

Acceptability

Document results



Cite references used for the validation

Must have been

mentioned in report

List supporting data attached to report

Must be mentioned in

the report


Case Study - Maintain the Validated State

Workpiece (Part)

Measurement System

Variability/Error

Cause Effect

Standard/Reference Instrument

(Gage)

Person (Appraiser)

Environment

Traceability Design

Maintenance

Skill

Procedure

Ergonomics (e.g. lighting)

Vibration

Elastic deformation

Cleanliness Stability

Influencing factors to consider for control

Calibration & Preventative maintenance schedule

e.g. Annual recertification



Use SWIPE model to help document

controls

Describe general controls

Describe TM specific controls

State where change impact assessment is captured when

changes occur


Raw Data Attachment

For Legibility: Use the Snipping Tool or similar to capture and attach evidence in your TMV record


Case Study - Document (raw data)

Equipment used identified

Legible raw data

Clear sample identification

Randomization demonstrated


Lessons Learned


Lessons Learned

Key Challenges:

• Inconsistent record quality

Solutions:

• Provide templates to ensure consistency and completeness

• Training and proficiency assessment


TMV Templates

• Ensure requirements were met

• Provide consistent format to “tell the story”

Help Ensure TMV Records Are:

• Compliant – ensure Author, Approver or someone else can present/defend the work

• Clear/Concise - make it easy for readers to review evidence and understand your thinking; include rationale; legible

• Complete – verify all requirements addressed; attach (training, raw data) or reference (pre-requisite reports) evidence, include data analysis records

• Correct - templates can not replace the need for Authors and Approvers to carefully read and check the work or to think critically


TMV Documentation Tips - Protocols

General Considerations » Good planning and a clear and complete protocol is key to successful validation

» Use tables, flow charts, graphs, photos to create clear & concise records

Samples » Represent intended use (e.g., at nominal, bracket specification limit(s))

» Explain creation of “surrogate” samples and why they are representative of

actual samples

Avoid Ambiguity » Include equations with units

» Explain how data will be collected in compliance with GDP; consider creating

data collection forms


TMV Documentation Tips – Execution

Attributable • Individual: Signature, name/Employee ID

• Test Method: TM Title/Number • Equipment: Equip ID

Legibility • Clear, readable, concise

• Changes must not hide/obscure original record

Contemporaneous • Recorded & signed on date performed

Original • Original record or verified copy.

Accurate • Recorded as observed. No reprocessing. Meaning

maintained through life of data

During Execution:

• Follow the protocol

• If deviation occurs: STOP, document deviation, investigate, document root cause & action plan, obtain approval, then PROCEED

• When collect data:


Group Discussion – Sharing Best Practices Raw Data Integrity …

• Electronic Data and Audit Trails – What do you do?


Key Points for Conducting Successful TMV

Say what you do Written procedures

Do what you say

Follow procedures

Record what you did

Keep records

Check the results

Did ‘it’ pass or fail

Act on the difference

Procedures for deviations/departures


Wrap-up and Final Questions


Group Discussion – Sharing Best Practices Closing Thoughts …

• What are the key challenges (in validation areas) has your organization encountered?

• … and what have you put in place to effectively deal with those challenges?

• TMV Training: How do you ensure your TMV authors/approvers are properly trained on the TMV process?

• Finally - Any other best practices to share?


TMV TOOLS AND RESOURCES


References

• Measurement System Analysis (1995), 4th Ed, Automotive Industry Group, Southfield, MN

o ANOVA Method – for Level 2b & 2c (variables)

o Range Method – for Level 2a (variables)

• ASTM E2782-11e1 – Standard Guide for Measurement System Analysis (MSA)

test method validation (tmv) - cbinet.com1).pdf1 test method validation (tmv) ... how to conduct tmv...

Documents