1

Shadya George, MLS(ASCP)CM

GMP Quality Manager Cell Therapy Manufacturing

2

Why are We Here?

*

3

Purpose & Use of Quality Indicators

Critical Areas to Monitor

Indicator Types & Desired Characteristics

Design of Balanced Monitoring Program

Metrology (Measurement) Selection

SPC Analysis – Statistical Process Control

Ground Up Challenges & Strategies

4

Provide Evidence-Based Quality Assurance, Risk Prediction, & Corrective Action

5

Predict Acceptability of Clinical Outcomes

Treatment/Collection Decisions

Predict Risk of Adverse Reactions

Development of Process Understanding

Process Improvement/Optimization (DOE)

Alerts for Needed Corrective Action

Regulatory Compliance

6

Key Process Areas for Optimization

• Potency

• Purity

• Identity

• Efficacy

• Cell Dose Enumeration

Technical Manipulations: Cell Loss/Recovery

Clinical Outcomes: Engraftment

Regulatory Based Priorities:

• Historic Weak Compliance Areas

Critical Areas of Risk: FMEA

7

Team Brainstorming Process Failure Risk Study:

List Possible Process Failures Assign Risk Score

Severity

Likely/Frequency of Occurrence

Detectable

8

Balance a Mix of Types of Indicators based on

your Process and System Risks

9

10

# Name Categor

y Level Type Dimension

Department Responsibility

Definition Formula Units Data

Location Collection Frequency

Report Type

16 DMSO

Exposure Time Internal Process

Activity Process Safety Manufacturing Time from addition of DMSO

freeze media to cells until step advancement for freeze

protocol

Time of freeze initiation - Time of DMSO addition min

Product Batch

Record

Quarterly Run Chart

17 Cell Viability Post Thaw

Internal Process

Activity Process Effective Manufacturing % of living cells as determined

by validated & controlled release testing assay

(# viable cells/total cells)100%

Average product type viability, SD, CV% Annually

% Product Batch

Record

Quarterly

Run Chart

Bar Graph with Annual Averages, SD, CV%

14

Adverse Product Infusion

Reactions

Internal Process

Activity Outco

me Safety Clinical

Graded seriousness of reactions from 0 for no

reactions to 5 for complications resulting in patient death.

Charted for each infusion on graph quarterly, calculate

average, standard deviation and %CV annually

Average: Sum of all infusion reaction grades/total # infusions performed within calendar year SD: %CV: (SD/Average)100%

Grade/Infusion

Annual Grade Average

Annual Grade

SD

Annual Grade %CV

Product Infusion

Sheet Quarterly

Run Chart,

Scatter Plot linear Correlation against unit characteristics

Annually

Patient Mortality

Internal Process

System Outco

me Safety Clinical

Tracking % patient

deaths for any cause

within 1 year of treatment

(# Patient Deaths/Total

Patients Treated)100%

% Clinical

Outcome Report

Annual Bar

Graph

Manufacturing Cost

Finance Activit

y Struct

ure Efficient Manufacturing

Average Cost to manufacture 1 unit, to include supplies, labor,

release testing

Sum of (manufacturing supplies used, testing services, testing

supplies, office supplies, proportion of equipment & facility certification costs, &

labor)/#Units manufactured. Performed for each individual

product type.

$/unit Finance

Department

Annual Run

Chart

11

Can We Monitor Too Much?

*What type of processes do you

perform?

Autologous HPC

Allogeneic HPC

Clinical Trial Cell Manufacturing

All of the Above

12

*How many indicators do you track

on average for each manufacturing

processes?

1-4

5-10

11-20

>20

13

*How many indicators total are

tracked and reported quarterly

within your facility?

1-9

10-24

25-50

>50

14

15

Benefit vs Time & Cost Considerations

Risk Loss Focus on Critical Areas - CLUTTER

Frequency Decisions? How Often?

Should Our Indicators of Focus Change?

Effective Quality Improvement

Requires Prioritizing!!

*How frequently do you revise your

monitoring system (adding/omitting

indicators)? Quarterly

Annually

Rarely (> 5 years)

Never done it 16

17

Feasible Valid Understandable Specific Sensitive Reliable: Repeatability, Robustness, Reproducibility Mainz, J. (2003). Defining and Classifying Clinical Indicators for Quality Improvement. International Journal for Quality in Health Care, 15(6), 523-530.

18

How can my indicator be

valid if I am not sure my

measurement is?

19

20

21

Balanced Quality Cost Budget

Intended Use Risk Assessment

COST

CAPABILITY FMEA

22

How Much Error can I Tolerate? EXAMINE WHAT YOU TOLERATE…

vs

23

Total Error Tolerated?

Accuracy—true/traceable to a standard? Do I always need this?

Bias/Systematic Error (SE)-why do different machines give

me different results?

Linearity – Define Limit Reportable range of results?

Specificity-measuring the quality I intended despite potential

interferences?

24

RANDOM VARIATION Error Tolerated?

Precision: Random Error (RE) Repeatability/Robustness –

Sensitivity-Noise to Signal Ratio See through the static? Resolve small differences in quality despite assay

imprecision?

25

Compare Tolerance of Error with

Measurement Capabilities

26

Is my process in a state of statistical control? - Does it matter?

Quality Control Charts

27

Control Charts – Levey Jennings

Defining Limits?? Mean, SD, %CV

Trends & Patterns

Shifts/Drifts of Quality

Reporting & Visualization Data

Shewart, Demming, Westgaurd (lab), etc….

28

Common Cause vs Special Cause Variation

DEMMING SAYS: DON’T TAMPER

29

Measurement Methodology of Clinical Trial Cells Data is limited –experience and publication minimal

Limited understanding of the process, what variables are really

affecting my quality?

How do you define quality in this new product? Who defines it?

Lacking manufacturer controls, manufacturer validated test kits,

established standards, clinical reagents, proficiency surveys, etc.

30

Clinical Trial Cell Challenges… Is cell count/viability/dose always the most meaningful

indicator to predict therapeutic outcomes?

Do we always know what the true target cell marker is?

New indicators may be needed in order to measure the

most meaningful quality attributes

New Measurement Systems may need developed

Need Controls that meaningfully monitor variation in

assays-(i.e. frozen cell bank controls)

31

Standardization Needs: Community peer

comparison initiatives?

Collaboration with Manufacturers &

Standardization Organizations for Support

32

Develop Balanced Facility Quality Dashboard

Review Plan Optimize for Improvement Focus

Use SPC Understand Measurement Capabilities

Balance Investment in Capabilities according to Risk- Determined Tolerances

Seek Inter-laboratory Comparison Initiatives

Petition Manufacturer Development Support

33

Questions

34

Thank You!

shadya.george@louisville.edu

Invitation to continue the conversations….

35

Post Thaw Viability Scenario….

Facility began post thaw 7-AAD viability by Flow Cytometry for HPC, Apheresis products after a limited validation of 10 testing events (due to cost and low volume), all performed by one tech.

Upon implementation, noticed methods averaged 80% Viability post thaw as seen in validation, but rare outliers 50% post thaw viability with no manufacturing cause discovered during investigation. Engraftment outcomes were as expected.

Has this ever happened to you???

36

Indicator Purpose:

Monitor stability of freeze/thaw manufacturing

process to detect clinically significant differences

in quality and functionality of cells.

WARNING FLAG for problems,

Evidence of acceptability/improvement during

process changes

37

Having assay standardized to a gold

standard is not as important ….

The question is not….

Is the Viability of cells

as they are IV infused into the patient

really 50% or 80%?

38

BUT is my assay capable to indicate

when the clinical quality of my cells

increases or decreases…

or is there too much “NOISE to hear”?

39

To Be Meaningful Assay Must….

Exhibit a Large Signal

(sensitive to real changes in quality)

Have Small “Noise”

(comparably small amount of random error uncertainty and imprecision in the result)

Desired Tolerance: +/- 10% Viability

Discovered

Assay NOT ROBUST—

Minor technique difference in tech preparation

vast differences in results

Poor Repeatability due to cell clumping

No meaningful QC available

Investigation annual process data: CV 40%

40

IMPROVEMENT ACTION:

NOISE --Random Error

pipette measurement capability &

standardized rate diluent delivery electronic

positive displacement technology

Optimized “Thaw Diluent”: clumping

Coordinated 1 hour analysis time limit

Increased detail SOP

Performed training

41

Next years processing data:

CV% Process decrease from 40 to 17%

Repeats gave near identical results

between operators.

Indicator now had value for intended use.

42

43

*Do we need to transfuse

PLTs or NOT?

Cancer center following engraftment of patients post

autologous HPC, Apheresis transplant. Patient CBCs

tested on small office Horiba Micros60 analyzer, but

sometimes sent to hospital analyzed on LH750.

Transplant nurses noticed significant discrepancies

occasionally (i.e. > 30 PLTS difference) between

machines.

QC was acceptable both instruments.

Manufacturer says they are both functioning within

spec??? 44

45

Discrepant Results Monitoring patient

engraftment PLTs Transfusion Decisions?

Tolerance Limits

CLIA allows a tolerance limit of 25%

total error for PLT counts on hematology analyzers for

acceptable PT testing criteria as printed in the Federal Register Feb. 28, 1992; 57(40):

7002-186. Theses ranges are commonly used to define Analytical Quality Requirements.

46

Tolerance Limits

However, for critical medical decisions,

biological cut-offs for 13% Total Allowable Error

have been defined in the literature for PLTs

(or +/- 2.5 at a true PLT value of 20). See Desirable Specification for Total Error, Imprecision, and Bias, derived from intra- and

inter-individual biologic variation and Current databases on biologic variation: pros, cons and

progress. Scand J Clin Lab Invest 1999; 59: 491-500. Ricos C, Alvarez V, et al.

Also,Clinical Decision Levels for laboratory Tests, Second Edition (Oradell NJ; Meical

Economics Books, 1987; Statland BE.)

47

TRUE CAPABILITY ANALYZER HIDDEN:

Both Manufacturers State: Linearity down to 0.0 PLTs

PT testing in US and standard manufacturer QC does

NOT monitor precision/accuracy at critical PLT decision

points: 0-20 PLTs!

Most standard linearity kits provide limited testing at this

level and CLIA acceptability range is wide.

48

UK NEQAS (United Kingdom national external Quality Assessment Scheme for Haematology)

General Haematology Study presented by Barbara De la Salle , 2009.

Thrombocytopenic Platelet Counting

Low Platelet Counting in EQA- the UK Experience

Discrepancies observed in automated Counting EQA: What do they mean?

49

Using 13% TE cutoff…After Service!!!

50

51

# 0 1 2 3 4 5 7 8 9 10 112 9.3 20 23.1 37.6 95.7 134.1 191.2 227.8 622.8 1471 3730

3 10 18.5 24.3 37.2 95.5 139 190.1 232.9 605.7 1447 3731

4 9.7 19.4 23 38.4 95.1 134 201.9 229.2 615.6 1483 3506

5 9.3 19.7 22.9 38.8 97.3 136.5 193.2 237 610 1489 3726

Mean 9.58 19.40 23.33 38.00 95.90 135.90 194.10 231.73 613.53 1472.50 3673.25SD 0.34 0.648 0.66 0.73 0.97 2.37 5.36 4.12 7.39 18.57 111.52

CV% 3.55% 3.34% 2.81% 1.92% 1.01% 1.74% 2.76% 1.78% 1.21% 1.26% 3.04%

Expected

Results 9.705 19.41 23.1725 38.82 97.05 135.87 194.1 231.725 625.6575 1506.2125 3475.875

Range 7.2 - 12.2 16.91 - 21.91 20.67 - 25.67 33.82 - 43.82

Absolute Bias -0.13 -0.01 0.15 -0.82 -1.15 0.03 0.00 0.00 -12.13 -33.71 197.38% Bias -1.34% -0.05% 0.66% -2.11% -1.18% 0.02% 0.00% 0.00% -1.94% -2.24% 5.68%

Absolute

Accuracy

Limits 2.50 2.50 2.50 5.00 13.00 18.21 26.01 31.05 83.84 201.83 465.77

% Accuracy

Limits 25.76% 12.88% 10.79% 10.00% 13.40% 13.40% 13.40% 13.40% 13.40% 13.40% 13.40%

SD Limit @ 0

Bias 0.83 0.83 0.83 1.67 4.33 6.07 8.67 10.35 27.95 67.28 155.26

Acceptable? Passed Passed Passed Passed Passed Passed

Reference

Point:

Precision

Passed

Reference

Point:

Precision

Passed

Passed Passed Passed

Testing Performed by: Judy Charlton, MT, Hematology Specialist Date: 7/21/2011

Calculations & Graphical Analysis Performed by: Shadya George, MLS (ASCP)CM

Date: 7/25/2011

PLT Statistics Study: LH 750 AK06092

Total Error as %

-30%

-20%

-10%

0%

10%

20%

30%

0 500 1000 1500 2000 2500 3000 3500

Expected Result

% E

rro

r

% Bias

Upper Limit

Low er Limit

52

PLTs Ultra Low: Total Error as %

-40%

-30%

-20%

-10%

0%

10%

20%

30%

40%

0 10 20 30 40 50 60 70 80 90 100

Expected Result

% E

rro

r

% Bias

Upper Limit

Lower Limit

53

54

• Ceased monitoring engraftment on Horiba Micros60

• Routine purchase of ultra-low PLT linearity kit for LH750

• Within 2 years increased instrument capability,

Replaced Hospital LH750 with DXH800

Replaced office Horiba Micros60 with Sysmex XS-1000i

Will the real HCT please

stand up! 55

*Transplant Center has been using the LH750 to

analyze HCT of cell therapy HPC, Apheresis

product for use in optimization of Apheresis

collection. The hospital replaced LH750 with the

advanced DXH800. Shortly after, Cancer Center

installed a new Sysmex XS-1000i to for better

precision of very low PLT count monitoring. The

Processing facility began a validation study for

comparison, but having problems with analyzer

agreement on HCT. Which one is “right”? 56

57

Beckman Type Technology: Calculated HCT

HCT = MCV x RBCs MCHC=HGB x 100

10 HCT

Technology Review:

58

Beckman Type Technology: Calculated HCT

Technology Disadvantages:

WBCs are counted in same chamber as RBCs.

High WBC /low RBC common in cell therapy

products interfere with all RBC indices

HCT is significantly falsely elevated in WBC

counts > 100 x103 cell/uL

59

Beckman Type Technology: Calculated HCT Technology Disadvantages: Example Typical Interferences

60

CALCULATED CORRECTION POSSIBLE LH750 & Similar Technologies:

Assuming 100% WBC interferences, acceptable correction can be

obtained mathematically. Improvement in use of corrected HCT

for optimization of collection, with adjustment in ranges.

• Use patient peripheral MCV, • Subtract WBC from RBC count • Manually Recalculate HCT

61

Beckman Type Technology: Calculated HCT

DXH800 technology:

Auto-Correction of Interferences

Advances in algorithms developed for

abnormal peripheral bloods; less than

satisfactory for cell therapy products.

62

DXH800 technology: Disadvantages: Manual calculated corrections impossible

due to differing degrees of failed “auto-correction”

technology. Significant unpredictable error, erratic results,

limited usefulness for collection optimization

63

Manual Spun Hematocrit

Significant Interferences:

WBCs appeared trapped in the RBC fraction,

poor separation –

--falsely elevating the apparent HCT value.

Additional limitations: poor resolution, difficulty

reading due to large buffy coat, and trapped

plasma

64

Sysmex-Direct Measured HCT

Based on the cumulative pulse heights of all

the RBCs counted as proportional to cell

volume. No calculation interferences

65

HCT Counts: Technology Comparison

LH750 #1 LH750 #2DXH800 #1

Manual Mode

DXH800 #1

Cassette

(2mL)

DX800#2

Manual

Mode

DX800#2

Cassette

(2mL)

Patient

Peripheral

Concurrent

MCV

LH750

Average

DXH800

Average

6 Ralph Tallent #2 11/9/2011 2.21 2.18 1.78 1.78 1.78 1.78 89.2 2.19 1.78

5 Ralph Tallent #3 11/9/2011 2.46 2.24 1.87 1.87 1.87 1.87 89.2 2.35 1.87

1 Donna Moore #2 11/7/2011 3.01 3.09 2.92 2.73 94.3 3.05 2.83

13 Wanda Miller #2 10/12/2011 3.35 3.10 2.99 106.8 3.35 3.04

2 Donna Moore #3 11/7/2011 3.48 3.33 2.92 2.83 94.3 3.41 2.88

14 Wanda Miller #3 10/12/2011 3.48 2.78 2.88 106.8 3.48 2.83

8 William Wells #3 11/14/2011 3.88 1.92 1.92 91.3 3.88 1.92

15 Wanda Miller #2 10/13/2011 4.27 3.80 3.80 105.5 4.27 3.80

3 Ralph Tallent #2 11/8/2011 4.52 4.24 3.89 3.80 4.15 88.4 4.52 4.02

17 Wanda Miller #2 10/14/2011 4.52 5.48 5.48 105.4 4.52 5.48

4 Ralph Tallent #3 11/8/2011 4.75 4.15 4.42 4.51 4.51 88.4 4.75 4.40

16 Wanda Miller #3 10/13/2011 5.25 5.24 4.43 4.64 105.5 5.25 4.54

7 William Wells #2 11/14/2011 5.50 2.47 2.56 91.3 5.50 2.51

18 Wanda Miller #3 10/14/2011 8.43 8.01 8.01 105.4 8.43 8.01

Average 4.22 3.22 3.27 3.92 3.04 3.62 4.21 3.57

Difference

r

LH750 vs DXH800

Calculated Corrected HCT : LH750 corrected using corrected RBC (Reported RBC - WBC) and patient concurrent MCV. DXH800 corrected

using patient concurrent MCV only (Instrument provides RBC auto correction in presence of high WBCs)

0.65

0.8517

DatePatient

Note: Wells sample contains Giant Plts flag which could interfere with the RBC count. DXH800 may be better at correcting

for this error, hence the two points that do not agree well. Without the well's points superior agreement.

Conclusion: DXH800 runs just slightly lower corrected HCT on average, clinically insignificant. Corrections equivalent.

#

66

HCT counts: Technology Comparison

HCT Comparison

y = 0.9172x - 0.2971

R2 = 0.7254

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

0.00 2.00 4.00 6.00 8.00 10.00

LH750

DX

H800

67

WHICH ANALYZER IS REPORTING

THE “RIGHT” HCT?

Purpose: optimization of Apheresis collection to

limit granulocyte contamination and increase

efficiency collection.

68

Again…

Having assay standardized to a gold

standard is not as important ….

The question is not

Is the HCT really 3, 2, 9 or 12%?

69

BUT is my assay capable or SENSITIVE enough

for my intended use?

Increase and decrease proportionately to

the actual product HCT?

Specific admidst fluctuating interferences?

Can it be used to direct in apheresis

settings to improve collection?

Outcome:

*LH750: Nurses report acceptable indicator use in setting

collection parameters using corrected values. However,

machine is gone.

*DXH800: Nurses report frustration and lack of

consistency in applying corrections based on product HCT.

Corrected and uncorrected HCT reported as not helpful.

*Recommended switching to HCT analysis using the

Sysmex XS-1000i, and re-setting new acceptable

expected ranges using statistical analysis.

70

Stability Program

You prepare a comprehensive Stability Program. Your

program includes the cryopreservation of 6 vials to

be tested at: 30, 60, 90 days and 1, 3 years. One of

the products included in program was collected on

9/22/2015. You start acquiring data and have the

following results:

The first 4 specimens yielded the following results:

71

Pre

Freeze TNCx1010 (Vol x WBC) =

7.10

7AAD (%

Alive)=

99.49

CFUx104/

kg=_____

Not

Done

CD34 %

=1.03

CD34x106/kg=

10.23

Post

Thaw WBC

(x106/ml)

TNCx1010

(Vol x WBC)

Total 7AAD

(% Alive)

CFU

(Growth) CD34%

CD34

Only

%Viabilit

y

Viable CD34 Dose

(x106/kg) (%

Recovery)

Tech Initials

30 days 276.0 7.73 52.48 Yes

No 1.54 68 7.81(43%) LC/MM`

60 days 290.0 8.12 49.55 Yes

No 2.74 92.7 13.78(76%) LC/RR

90 days 282.0 7.90 50.62 Yes

No 3.07 96.5 15.34(84%) LC/LCR

1 year 276.0 7.73 56.70 Yes

No 2.82 81.9 15.45(85) JM/YY

72

1. Is the difference in results due to different staff, different technique?

2. Should I check the gating technique used for all samples?

3. Should I trust these results? What this is really telling me?

1. Is the difference in results due to different staff, different

technique?

*We decided to assign a team that does most of the post-thaw together. We

stated to get better results, more “credible”

2. Should I check the gating technique used for all samples?

*When we went back to check gating, we noticed we were all over in our

technique. With time, more cells (grans) will die and gating became more

challenging. We decided to “standardize” gating by using similar viability

gating for all vials. Now we checked results from previous thawing and

compare gating.

73

74

ORIGINAL GATING RE- GATED

75

ORIGINAL GATING RE- GATED

Pre

Freeze TNCx1010 (Vol x WBC) =

7.10

7AAD (%

Alive)=

99.49

CFUx104/

kg=_____

Not

Done

CD34 %

=1.03

CD34x106/kg=

10.23

Post

Thaw WBC

(x106/ml)

TNCx1010

(Vol x WBC)

Total 7AAD

(% Alive)

CFU

(Growth) CD34%

CD34

Only

%Viabili

ty

Viable CD34

Dose (x106/kg)

(% Recovery)

Tech Initials

30 days 296.0 6.22 90.1 Yes

No 0.93 93.4 5.74(103%)

60 days 300.0 6.30 88.8 Yes

No 1.01 96.1 6.23(117%)

90 days 288.5 6.06 90.6 Yes

No 0.92 96.4 5.56(104.7)

76

With standardization we started to see “better” results. One thing we noticed:

1. We continue to recover >100% of cells. Why?

2. Have you experience a change in methodology, ex. new cell counter with different technology,

and results for WBC from post-thaw are very different from counts made with previous instrument:

a. Did you stop the stability testing for the rest of the vials?