ispe quality metrics initiative - regulations.gov

ISPE Quality Metrics InitiativeJune 2015

A Report from the Pilot Project – Wave 1

© 2015 International Society for Pharmaceutical Engineering

All rights reserved

ISPE 600 N. Westshore Blvd., Suite 900

Tampa, FL 33609 USA

Tel: 813-960-2105 – Fax: 813-264-2816 – [email protected]

www.ISPE.org

mailto:ask%40ispe.org?subject=

http://www.ISPE.org

Table of Contents

1 Executive Summary 41.1 Main Findings: Summary 51.2 Next Steps for ISPE’s Quality Metrics Initiative 8

2 Background 93 Pilot Design 13

3.1 Project Governance Model 133.2 Choice of Metrics for the Wave 1 Pilot 153.3 AchievingaStandardizedDefinitionsSet 173.4 Additional Surveys Included in the Wave 1 Pilot 193.5 EstimatesofDataCollectionandSubmissionEffort 203.6 Data Collection Period 21

4 Operational Processes for the Wave 1 Pilot 234.1 McKinsey Operational Process 234.2 Experiences from Participating Companies 24

5 Findings from the ISPE Quality Metrics Initiative Wave 1 Pilot 255.1 Sample Size 255.2 MetricandSurveyDataAnalysisandDiscussion 275.3 Wave 1 Pilot Quality Metric Data Analysis 285.4 Wave 1 Pilot Quality Culture Survey Data Analysis 315.5 DataCollectionandSubmissionEffortData 355.6 Process Capability Survey 425.7 EstablishingStatisticallySignificantRelationships 445.8 StatisticallySignificantRelationshipsinWave1PilotData 465.9 RelationshipsatLowerLevelsofSignificance 535.10 ComparisonsWhereMetricsAreNotDifferentiatedorAreInconclusive 565.11 DiscussionofRelationships 575.12 Complaints Analysis 605.13 Analysis of Product-Based Metrics 615.14 McKinseyAnalyticalEffortandObservations 63

6 Output and Lessons Learned from ISPE Quality Metric Wave 1 Pilot 646.1 Success Factors 646.2 Definitions 656.3 Metrics Collection by Site and by Product 656.4 IndustryEffort 666.5 McKinseyAnalyticalEffort 66

7 Proposals 677.1 RationaleforMetricsProposedasaStartingSetforWave2Pilot 67

8 Conclusions 699 References 70Appendix 1: DefinitionsofQuantitativeMetricsUsedinPilot 72Appendix 2: SurveyQuestions 77Appendix 3: ExamplesforSiteandProductDataCollectionTemplates 79Appendix 4: Case Study Company A 80Appendix 5: DetailedAnalysisofDataandRelationshipsforEachIndividualMetric 84

Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 4

1 Executive Summary

ISPE commenced its Quality Metrics Initiative in June 2013 after the US Food and Drug Administration (FDA) announced its Quality Metrics Program in a February 2013 FederalRegisternotice[15]

To assist in the evaluation of product manufacturing quality, FDA is exploring the broader use of manufacturing quality metrics.

Throughanextensiveseriesofengagementswithindustryandotherkeystakeholdersoverthepasttwoyears,FDAhasfurtherindicatedthat“anobjectivesetofqualitymetrics”wouldbereportabletosupporttheirrisk-basedinspectionprogram,asgiveninsections704–706oftheUSFDASafetyandInnovationAct(FDASIA)[9]. The FDA Quality Metrics Program is also intended to move both industry and the agency towardthedesiredstate[18] for pharmaceutical manufacturing. The FDA Quality Metrics Program, including the set of metrics selected, is expected to be published for public comment in 2015.

InawhitepaperdeliveredtoFDAinDecember2013[12], ISPE recommended thatapilotprogramshouldbeconductedwithinindustrytofurtherunderstandtheimplementationopportunities,challengesandbenefitsavailablefromsuchaqualitymetricsprogram.ISPE,incooperationwithMcKinseyandCompany,undertookthisproject.TheresultwastheISPEQualityMetricsPilotProject—Wave1.DesignedanddevelopedbytheISPEQualityMetricsCoreTeam,theprojectdrewontheknowledgeand experience of cross-functional industry representatives, ex-regulators and academicians,withfurtherinsightgainedindetaileddiscussionswithavarietyof industry associations at many industry meetings.

The ISPE Wave 1 Pilot ran from June through November 2014 and included:

f Data collected at 44 sites from 18 participating companies f Datawascollectedretrospectivelyfor12monthsandprospectivelyfor3months

at each site. f AWave1setofquantitativequalitymetrics f Nearlyallmetricscollectedwerereportedatsitelevel;threewerecollectedatproductlevelwithineachsite.


1

1.1 Main Findings: Summary

The Wave 1 Pilot met its overall objectives. A summary of the insights gained include:

f It is feasible to collect and submit a standardized set of metrics. f Themajorityofcompaniesthatparticipatedreportedthefollowingbenefits:

– Gainingadeeperunderstandingofthestandardizedmetricsdefinitionsand design

– Establishing a centralized submissions process trial – Developingaccesstoabenchmarkingreportthatallowedthemtoexamine

their progress against aggregated data from their peers

f Centralcollectionandsubmissionofmetricswillcreateaburdenforindustry,primarilybecausestandardizedmetricswillinevitablydifferfromcurrentcompany metrics.

f Manycompanieswillperformmetricscollectioninadditiontotheirestablished programs.

f Understanding organizational context is crucial to interpreting results. f The Wave 1 Pilot also provided some key insights in relation to the prevailing qualityculturewithinanorganizationthatmeritfurtherexploration.

ThesuccessoftheWave1Pilotcanbetracedtothefollowingfactors:

f Usingastandardizedsetofmetricswithclearandspecificdefinitionsprovidedfor each of the metrics measured.

f Excellentcollaborationbetweenallstakeholders. f Frequentanddirectinteractionforguidanceandqueryresolutionbetween

the McKinsey support team and participating sites. f Leveraging McKinsey’s experience and capability in metrics program delivery. f Leadership from the ISPE Quality Metrics team and the sponsorship from

the leaders at the participating sites. f Ongoingdialogandtrust-buildingwithFDAthroughoutthepilotperiod.


1

TheWave1Pilotalsoidentifiedseveralchallengesthatarepresentinrollingouta centrally reported standardized metrics program. These include:

f Theindustryanditssectorshavenottraditionallysharedacommondefinitions.Consequently,definitionsofeachmetricmustbespecific,clearlyunderstoodand meaningful across the range of organizations under consideration to ensure theprogram’ssuccess.Thiswillrequiredetailedup-frontdesignandongoingoperational support.

f Thelevelofeffortrequiredfordatacollectionandsubmissiononbehalfoftheindustry, and data analysis and support on behalf of the agency cannot be underestimated.RefertoSection 5 and Section 5.14 of the report for estimates oftheeffortinvolved.However,theseestimatesmaybeconsideredconservativebecause they do not include several factors, such as: – A“goodenough”situationwasappliedtodatasubmissionintheWave1Pilot.SubmissiontoFDAwouldrequiremorethoroughandcompletedatacollection,additionalmanagementreviewanddataverification.

– Pilotparticipantshadtheflexibilitytoprovidetheirmostpragmaticdataset(e.g.,allproductsatthesiteoronlythosefortheUSmarket);thiswouldnotbe the case in a formal submission process.

– Pilotparticipantstypicallyhadmaturesystemsandcapabilitiesandwerefromdevelopedcountries;thiswouldnotbethecaseforallsitesundera centralized reporting initiative.

f Understandingthevariationinrangesforinterpretationofthedatawillrequirelonger timeframes to assess than those examined in the Wave 1 Pilot.


1

Details of the statistical analysis, main outcomes and recommendations arising from the Wave 1 Pilot can be found in Section 5 and Section 6 of this report. A summary ofthesefindingsisasfollows:

f Evenwith44sitesreportinginthisinitialphaseoftheISPEQualityMetricsInitiative,thesamplesizesallowedstatisticalanalysiswithsomelimitations;these are outlined in more detail in Section 5.3 of this report.

f TheanalysisoftheWave1Pilotdataidentifiedanumberofstatisticallysignificant(lessthan5%likelihoodofacoincidence)relationshipsbetweenthemetricscollectedandoverallqualityoutcomesatthesites.

f Theseinitialfindingsofstatisticallysignificantrelationshipsdonotimplycausation,thereforethisreportdoesnotattempttodrawconclusionsfromthisphaseofthedataanalysis.Instead,thisreportisintendedtosharethefindings,identifyrecommendationsandputforwardproposalsforthenextphaseofthestudy.

f To meet one of its objectives, this pilot targeted a set of metrics collected at both the site and the product level to understand the current capacity to collect metrics.Product-levelmetricsrequireanunderstandingofthechallengesofaggregating metrics across the supply chain for multisite products. In addition, thedefinitionof“product”asrelatedtoan“applicationnumber”presentedissues for over-the-counter (OTC) products. This element of the pilot has led to some key learnings and recommendations for both industry and FDA, and these are included in the report.

f Quality metrics reporting alone should not be the basis for action (either positiveornegative)withoutunderstandingthecontextofthedataandthe originating company.

f Choosinganappropriatemetricsetwillhelpidentifycontinualimprovement opportunities.

f TheknowledgegainedfromtheWave1PilothasbeenleveragedtodeveloparevisedsetofstartingmetricsthatISPEnowproposesforfurtheranalysisina Wave 2 pilot program. Details of this proposal can be found in Section7 of this report.

f LearningsfromtheWave1PilothavealsobeensharedwithFDAforconsiderationinthedesignofagency’sfinalsetofobjectivemetrics.


1

1.2 Next Steps for ISPE’s Quality Metrics Initiative

BasedonthefindingsfromtheWave1Pilot,ISPEnowrecommendsthefollowingset of starting metrics:

1. Lot acceptance rate (normalized by lots dispositioned), collected at site level2. Lot acceptance rate (normalized by lots dispositioned), collected at product level

withinasite3. Critical complaints (normalized by packs released), collected at product level

byeachproductapplication,notbrokendownbysite4. Critical complaints (normalized by packs released), collected at site level,

undifferentiatedbyproduct5. Deviations rate at site level

FollowingthepresentationoftheWave1PilotresultsattheISPEQualityMetricsSummitinBaltimoreon21–22April2015,itwasbroadlyagreedthatthereisacontinuingappetitewithinindustryforadditionallearningwithrespecttoqualityperformance measures.

ISPE has therefore initiated planning for a Wave 2 Pilot, to commence in the second halfof2015.Thissecondphasewilltestthestartingsetofmetricsonanextendedsampleandtimeperiodtoincreasetherangeanddurationoftheknowledgebaseand enable more in-depth statistical data analysis to examine correlations and dependences.TheWave2Pilotwillalsoexploretheinclusionofotherpotentialmetricsofinterestandfurtherstudyoftheassessmentofqualitycultureatparticipating companies.

Itishopedthatcontinuingthisworkwillenablethepharmaceuticalindustrytoundertakethe“qualityrevolution”[16] proposed by Dr. Janet Woodcock at the ISPE Quality Metrics Summit to truly enhance the future state of pharmaceutical manufacturing.

Sinceregratitudeisextendedtotheparticipatingcompaniesandtheirstaffforthe excellent input, support and enthusiasm given throughout this Wave 1 Pilot.


2 Background

This section describes the:

f ISPE Quality Metrics Initiative background f Quality Metrics Pilot Program, a major component of this project f Wave 1 Pilot background, rationale, and key milestone dates

TheFDA’svisionfortwenty-firstcenturymanufacturinginthepharmaceuticalindustryisoftenquotedasthe“desiredstate”andis:

A maximally efficient, agile, flexible pharmaceutical manufacturing sector that reliably produces high quality drugs without extensive regulatory oversight.

There have been many guidelines issued by FDA [1][2][3][4] and the International ConferenceonHarmonisationofTechnicalRequirementsforRegistrationofPharmaceuticalsforHumanUse—Q8[5], Q9 [6], Q10 [7] and Q11 [8]. These provide a regulatoryframeworkthatallowindustryandregulatorstomovetowardthisdesiredstate.Despitetheintroductionofthenewguidance,recentacknowledgementsconfirmthatmoreworkisrequiredbyboththeindustryandtheregulatorycommunitytoattainthe desired state.

TohelpFDAandindustryworktowardthetwingoalsofensuringproductqualityina global supply chain and reducing drug shortages, FDASIA [9] (July 2012) gave the agencynewauthoritytoenhancethesafetyofthedrugsupplychainandcreatedlegislativemandatesaffectingcurrentgoodmanufacturingpractices(CGMPs).FDASIAalsorequiredFDAtoimplementarisk-basedinspectionprogramofpharmaceuticalmanufacturingsitesratherthanthecurrenttwo-yearinspectioncycleineffect.

Ofrelevancetoarisk-basedinspectionprogramaresections704,705and706ofFDASIArelatingtoadvancedprovisionofinformation(e.g.,qualitymetrics).

f Section704“…enablesFDApersonneltosearchthedatabasebyanyfieldofinformationsubmittedinaregistration…”

f Section705requires“risk-basedschedulefordrugs”andlists“riskfactors”as:(A) The compliance history of the establishment.(B) The record, history, and nature of recalls linked to the establishment.(C) The inherent risk of the drug manufactured, prepared, propagated,

compounded, or processed at the establishment.(D) Theinspectionfrequencyandhistoryoftheestablishment,includingwhether

theestablishmenthasbeeninspectedpursuanttosection704withinthelast 4 years.

(E) Whether the establishment has been inspected by a foreign government or an agency of a foreign government recognized under section 809.

(F) Any other criteria deemed necessary and appropriate by the Secretary for purposes of allocating inspection resources.

Section706requires“…recordsorotherinformation…beprovided...inadvanceorinlieuofaninspection...”These“recordsorotherinformation”areinterpretedasprovisionofqualitymetricsdataaspartofotherinformationwhichcouldpotentiallyberequested.


2

InresponsetotheFDAinitiativeonqualitymetrics,ISPEestablishedaProductQualityLifecycle Implementation®(PQLI)–sponsoredQualityMetricsprojectwithateamthat consisted of representatives from a variety of pharmaceutical companies.

Guiding principles established at the commencement of this project stipulated that anyproposedmetricswouldbe:

f Clearlydefinedtoallowconsistentreportingacrosssites f Objective and meaningful f Easy to capture f Easy to report f Normalizedbyfactorssuchasprocessdifferencesandtechnicalcomplexity f Abletodriveacceptable,notunwantedbehaviors

Thisteamstartedworkatawell-attendedsessionoftheISPE–FDACGMPconferenceinBaltimoreon12June2013,atwhichthebothFDAandindustrywererepresented.TheinitialobjectiveforISPE’sprojectteamwastoanalyzeandusetheoutputfromthediscussionatthismeetingasinputtoawhitepapertobeissuedfordiscussionwiththeFDA.AsummaryofmainmilestonesfortheQualityMetricsPilotProgramisshowninFigure1.

Figure 1: Major Milestones for the ISPE Quality Metrics Project

Quality Metrics Progress

Jan2013 Apr AprJul Oct Jul Oct

Jan2014 Apr Jul

Jan2015

Initialindustryresponses

FDA FR noticerequestingmetrics

Brookingsstakeholdermeeting

FDA metrics of “potential interest”

Kick offISPE QMteam

Pilotkickoff

LauchWave 2?

Pilotdata lock

ISPE QMSummit

Industry whitepapers


2

TheISPEwhitepaper,publishedinDecember2013,proposedalistofmetricsacceptable to industry that could be reportable to FDA to support a risk-based inspectionprogram.Thewhitepaper’smainrecommendationswereto:

f Conductapilottofleshoutstandarddefinitionsandapproach. f Initiatewithsitemetricscollection,withthepotentialtomovetoproduct

metrics later.

Based on these recommendations, ISPE announced its intention to conduct a Quality Metrics Pilot Program on 12 March 2014.

During this period, FDA expressed a desire for industry input on the development of FDA’squalitymetricsprogram.FDAthenparticipatedindiscussionswithindustryattheMeasuringPharmaceuticalQualitythroughManufacturingMetricsandRisked-Based Assessment meeting held 1–2 May 2014 and hosted by the Engelberg Center forHealthCareReformattheBrookingsInstitution.[10]Anextstepidentifiedatthismeetingwas:

That the pilot quality metrics programs currently under development by the International Society for Pharmaceutical Engineering … may yield important lessons

for FDA as it moves forward with its own program.

In preparation, ISPE explored the options of conducting the Quality Metrics Pilot Programincooperationwithasuitableindependentpartnerthatcouldprovideoperationalexpertiseandassureparticipantconfidentialityduringthepilot.ISPEsubsequentlyagreedtopartnerwithMcKinseyandCompany,duetotheirexperienceofconductingindustry-benchmarkingprograms,specificallythePharmaOperationsBenchmarking of Solids (POBOS) series of programs, [11]whichhavebeenoperatingsince2004.Itwasrecognizedthatbenchmarkingprogramsrequiresignificantexpertiseto succeed, such as:

f Developmentoftemplatestoallowforeaseofinputofdata. f Structureddatasubmissionwithdetailedguidanceonhowtoreportthedata. f Ability to comment on data points to enable interpretation. f Experienceddedicatedsupportforquestionsandclarificationsduringand

throughout the data collection. f Built-indatavaliditychecksandjointreviewtoensuredataconsistency

and accuracy. f Development and operation of supporting IT systems. f Relevanthigh-levelstatisticalexpertisetoassistindatainterpretation. f Autonomyandconfidentialityindatacollection,reviewandanalysis.

ISPEannouncedthelaunchoftheQualityMetricsPilotPrograminpartnershipwithMcKinseyandCompanyattheISPE–FDACGMPconferenceon2June2014.Itwasintendedthatthepilotprojectwouldhavephases:Wave1andWave2.Thisreportsummarizes the Wave 1 Pilot and proposes recommendations for programs and metrics for consideration in a future Wave 2 Pilot.


2

Wave1Pilotwasintendedtodemonstratethefeasibilityandvalueofstandardqualitymetrics.Someimportantprimaryobjectiveswereto:

f Testtheharmonizationofdefinitionsforasetofindustrymetricsthatrepresentboth leading and lagging indicators.

f Testthefeasibilityofcentralizeddatacollectionacrosscompaniesatdifferentmaturitylevelswithintheirowninternalmetricsprograms.

f Exploreindustrypracticesintheareasofqualitycultureandprocesscapability. f Inform continued industry input to FDA.

Industryparticipantswereintendedtogainthebenefitsof:

f InfluencingtheoutputfromtheISPEQualityMetricsPilotProgramintermsofchoiceofmetrics,definitionsandeaseofdatacollectionbasedonactualexperience.

f Receivingablindedcomparisonorbenchmarktotheparticipatingsiteindustryaverageandtosimilartechnologyplatformpeers(providedsufficientsamplesize is achieved).

f Havinganopportunitytodeveloporenhanceinternalproceduresformetriccollectionalongwithasetofmetricdefinitions.

f Gaining insight into the implications of external metric reporting.

Duringtheperiodfromthewhitepaper’sissuetoinitiationoftheWave1Pilot,considerableattentionwasgiventochoiceofmetricstobeincluded.ConsiderationwasgiventodiscussionfromtheBrookingsmeeting,aswellasinputfromFDAandfromotherindustryassociations.ThefinalWave1Pilotmetricschosenwereselectedtomeasureobjectivequalityperformanceofasite.TheyincludeallthemetricsidentifiedbyFDAattheBrookingsmeeting,twotechnology-specificmetricsandtwosurveys,oneonqualitycultureandoneonuseofprocesscapability.MorediscussiononchoiceofmetricsandtheirassociateddefinitionsisgiveninSection 3.


3 Pilot Design

3.1 Project Governance Model

To manage the Wave 1 Pilot project, ISPE and McKinsey established a project governancemodel,whichisshowndiagrammaticallyinFigure2.

Figure 2: ISPE and McKinsey Project Governance Model

Data flow

Communicationand guidance

Communication

FDA Quality MetricsProgram Leader

Quality Metrics Pilot Program Core TeamLeader

McKinsey POBOS Expert

McKinsey partner

Representatives from - Originator Companies, small and large molecules - Generic companies - OTC company

Academic

ISPE staff and advisor

McKinsey Operational TeamData input and analysis specialists

Participant CompanyCompany leadSite leadsSenior leaders

ISPE Quality Metrics Subteams DefinitionsInfluencing/CommunicationQuality Culture

ISPE and McKinsey Sponsor TeamISPE President and CEOMcKinsey PartnerISPE International Leadership Forum of senior leaders representing - Originator Companies, small and large molecules - Generic companies - OTCISPE staff and advisor


3

Key features of this project governance model are:

f Data from individual companies are seen only by McKinsey personnel f ISPE project team has access only to aggregated data across all companies or tosubsetsofcompanieswherenumbersaresufficienttomaintainanonymity

f The ISPE Quality Metrics Core Team, representing a broad spectrum of the pharmaceuticalbusiness,meetregularly—usuallyweekly.

f The ISPE Sponsor Team consists of ISPE’s president and CEO, senior leaders of pharmaceutical companies in ISPE’s International Leadership Forum and a McKinsey partner.

f The ISPE Core Team seeks communication, guidance and decisions from the ISPE Sponsor Team at about approximately monthly intervals.

f SubgroupsoftheISPECoreTeamheldregularteleconferenceswithparticipantcompany leads and site leads to: – Brief them on progress – Provideanoverviewofthedataanalysisfortheirreview – Seek their input

f SubgroupsoftheISPECoreTeamheldinformalmeetingswiththeFDAQualityMetrics Program leader to: – Seek input to choice of metrics and overall design of the Wave 1 Pilot – Provide update on progress of the Wave 1 Pilot – Provide early readouts of summary Wave 1 Pilot results – Bepresenttosharefindings

Inaddition,theCoreTeamtaskedsubteamswithchartersanddeliverablestoprogressparticularelementsoftheprojectindependently.Subteamswereestablishedfor:

f Definitions f Communications f Influencingandindustryengagement f Quality culture f Process capability

Theimportanceofdefiningmetricscarefullywasidentifiedearlyintheproject.TheDefinitionsSubteamdevelopedtheworkdescribedinSection 3.3.Itwasespecially key to:

f Developingresponsestofrequentlyaskedquestions(FAQs) f ProducingthedefinitionsgiveninAppendix 1 f Leading the process to develop the surveys given in Appendix 2.

TheInfluencingandIndustryEngagementSubteam’srolewastoencouragecompaniestoenrollfortheWave1Pilotandtoarrangeteleconferenceswithpilotleadindividualsand senior leaders in participant companies. Additionally, this subteam took the lead in preparing material for presentation at the FDA meetings.


3

TheQualityCultureSubteamexplorednewideasandpotentialleadingqualitymetrics.GiventhefindingsfromtheCoreTeamandthelevelofpublicinterestexpressedatmanymeetings,thissubteamfocusedonsharingcurrentqualityculturebestpractices.TheteaminitiallyplannedtoexplorewhetheraquantitativeQualityCultureIndexcouldbeestablished.Subsequentwork,however,includingdiscussionandengagementacrossindustryandacademia,hassuggestedthatqualitycultureevaluationrequiresaholisticapproach,andcentralizedreportingofastandardizedassessment is not desirable. The Quality Culture Subteam is developing a cultural excellenceframeworkentitledThe Six Dimensions of Quality Culture;futurepublicationsare also planned.

TheProcessCapabilitySubteam,workingunderthewiderPQLIumbrella,wasestablished based on a recommendation from the Quality Metrics Core Team. Its objectivesaretoproduceaseriesofarticlesand/orwhitepapers,aswellascasestudies, a potential baseline guide and industry sessions at ISPE meetings to examine the use of process capability measurements by the pharmaceutical industry globally. Thisteamalsocontributedtotheprocesscapabilitysurveyquestionsconductedas part of the Pilot Wave 1.

3.2 Choice of Metrics for the Wave 1 Pilot

This section outlines the list of metrics used in the ISPE Quality Metrics Pilot Program, Wave 1andtheirassociateddefinitions.

The choice of metrics for the Wave 1 Pilot evolved over time by taking account of manyinfluencesandinput.OutputfromtheBrookingsmeetingwasconsideredbytheISPEprojectteam,aswellasFDA’srequestforproduct-basedmetrics.Therewasalsoastrongdesirebymostpartiestostarttounderstandtheimpactofqualityculture.Additionallytwotechnology-specificmetricsforsterileproductmanufacturewereincluded.AsummaryofthelistofmetricsusedintheISPEQualityMetricsPilotProgram, Wave 1andtheiroriginsisshowninFigure3.


3

Figure 3: Summary of Final Metrics Collected During ISPE Industry Wave 1 Pilot

Quantitative metrics Technology specific metrics

Additional survey-based metrics

Media fill (for sterile aseptic sites) failuresEnvironmental monitoring (for sterile aseptic sites)

Process capabilityQuality culture

Consensus Industry Metrics

Metric proposed in Brookings meeting

Product and site-based metric

Lot acceptance rateComplaints rate (total and critical)Confirmed OOS rateUS recall events (total and by class)Stability Failure rateInvalidated (unconfirmed) OOS rateRight first time (Rework/Reprocessing) rateAPQR reviews completed on timeRecurring deviations rateCAPA effectiveness rate

2 more quantitative metrics calculated from data collected for Wave 1:• Deviations rate• Incoming material OOS

3.2.1 A Note on Product-Based Metrics

Thefollowingmetricswerecollectedonbothsiteandproductbasestohelpgainanunderstandingofthedifferencesbetweentheseapproaches:

f Lot acceptance rate f Total and critical complaints rate f Confirmedout-of-specification(OOS)rate

Product-basedmetricsintheWave1Pilotwerecollectedonrelevantunitoperationsperformedataspecificsite.Forthispilot,metricswerenotaggregatedacrossmultiplesitestothefinalpackagedandlabeleddosageformlevelortoanewdrugapplication(NDA) level.

Whenreviewingproduct-levelmetricsreporting,it’simportanttoensurethatdefinitionsandexpectationsareclearlydefinedandunderstood.USNDAs,forexample,caninclude multiple dosage form strengths, and each strength may be assembled into aseriesofpacks.ThismeansthatoneNDAmayincludemany“products”–ifa“product”isdefinedasonedosageformstrengthassembledintoonepack.


3

3.3 Achieving a Standardized Definitions Set

Experience from the project team, feedback at ISPE public meetings and preparation ofthewhitepaperallidentifiedtheimportanceofdefiningmetricsthat:

f Are clear to the project team. f Are understood by participants. f Match those currently used by companies as closely as possible. f Measurequalityperformanceaccurately. f Reducetheopportunityfor“gaming”. f Minimizeunintendedconsequences.

TheDefinitionsSubteamestablishedathoroughandrobustapproachtoderivethedefinitionsusedintheWave1Pilot,usinganiterativeprocesstoreachconsensus.This process is depicted in Figure 4.

Figure 4: Process to Derive Definitions for Wave 1 Pilot

ISPE Dec 2013 white paper

Brookings meeting

POBOS quality definitions

ISPE Quality Metrics Definitions subteam

Metrics definitions examples from individual companies

ISPE Quality Metrics Core team

Templates with definitions of each

data point

Inputs from multiple broadindustry sources

Development of precise definitions for each data point

For review Feedback


3

TheDefinitionsSubteamreceivedinputsfrommultiplesources.Outputswerecollatedintoasetofagreed-upondefinitionsintheExceldata-collectiontemplatesproducedby the McKinsey team, then given to participating companies for completion.

Oneexampleofthiscomplexseriesofinteractionscanbeseeninthedefinitionof“lotdispositioned.”Thisisacriticalterm,whichrequiredconsensus,asitisthe denominator for several metrics collected in the pilot. A representation of this consensus building process is given in Figure 5.

Figure 5: An Example of Challenges Defining “Lot Dispositioned”

"Total number of lots for commercial use produced

and/or packaged on site that went through final disposition

during the period, i.e. were released for shipping or

rejected (for destruction). Rejections should be counted as final disposition regardless at what production stage the rejection occurred. Release is only final release for shipping. Excludes lots that have been

sent for rework or put on hold/quarantined in this

period and hence are not finally dispositioned. Excludes

lots that are not produced or packaged on site, but

released for CMOs."

Final definition of "lots dispositioned"If several formulation lots are aggregated in one “combo” packaging lot, and it is rejected at the packaging stage, does it count as 1 or several rejections?We count every batch rejected as a rejection, since it counts rejections at any stage of the production process but in the denominator as the final lots dispositioned.

We do release work for other sites, but do not produce those batches. do these count?Excludes lots released for other entities, not produced on site.

Do we count intermediate batches?No, only batches shipped unless we ship intermediates.

If a formulation is split into several packaging lots, does each lot count as a new "attempt" or does it remain counted as 1 attempted lot?Yes it counts as multiple lots, Splitting or aggregating of lots happens – it only matters what is finally dispositioned not earlier changes.

Do we count products that were produced in previous years and we now release?Yes, counts on holds or rework in the period when finally dispositioned.

?Even with standardized definition, there will be remaining unclarities (e.g., what do we do with continuous manufacturing?)

Inadditiontodevelopinganddesigningdefinitions,theDefinitionsSubteamalsoassessedanyquestionsraisedbyparticipantsduringthepilotkickofforintheearlyphasesofcompletingthedata-collectiontemplates.ThesubteamissuedaweeklylistofclarificationsintheformatofaFrequentlyAskedQuestionsdocument(FAQs)to all participants.


3

WhileanalyzingdataintheWave1Pilot,itbecameapparentthattwoadditionalmetrics – deviations rate and incoming material OOS rate – could also be calculated automatically,sincethedatarequiredforthesemetricswasalreadybeingcollected.This brought the number of metrics analyzed to 14.

DefinitionsforallquantitativemetricsusedintheWave1PilotaregiveninAppendix 1 of this report.

3.4 Additional Surveys Included in the Wave 1 Pilot

TwoqualitativesurveyswerealsoconductedaspartoftheWave1Pilot.Theseexploredtheprevailingqualitycultureatthesiteandexaminedtheuseofprocesscapability monitoring and trending on the site.

3.4.1 Quality Culture Survey Development

UsingMcKinsey’spreviousexperienceinconductingqualityculture–typesurveys,theDefinitionsandtheQualityCultureSubteamsreviewedanexistingPOBOSQuality Culture Shop Floor Survey for inclusion in the Wave 1 Pilot. Using this survey, thesubteamsdevelopeda15-questionassessmenttoolthatmeasuredfiveculturalelements: Leadership, Governance, Integrity, Mindset and Capabilities.

Althoughcompletingthisqualityculturesurveycouldbeasubstantialamountofworkforparticipants,itwasconsideredanecessarytooltotestthehypothesisofhowqualityculturemayimpactthequalityperformanceoutcomesatagivensite.

3.4.2 Process Capability Survey Development

PreviousworkundertakenbytheISPEQualityMetricsCoreTeamandadditionalreviewwithindustrycolleaguesindicatedtwothings:

f Applicationofprocesscapabilitymeasureswasnotwidespreadinindustry f Sitesthatdidmeasureprocesscapabilityusedawidevarietyofapproaches

Itwasdecided,therefore,todevelopasurveytoassessthetoolsandprocessesused to monitor process capability by the participating sites.

Both the Quality Culture Survey and the Process Capability Survey are included in Appendix 2 of this report.


3

3.5 Estimates of Data Collection and Submission Effort

Companieswereaskedtoestimatetheperson-hoursofeffortthattheyusedtosetupandcarryouttheirpartoftheWave1Pilot.Eachsitecompletedatemplatewithestimatesofthetimeandeffortspentcollectingeachindividualmetric:

f Time (hours) spent to collect individual metric data at both site and product levels, for both the retrospective and the prospective periods.

f Degreeofdifficultyonascaleof1to4(easiesttomostdifficult)forcollectingeach metric at both site and at product levels.

f Siteratingsofwhetherthedatawasavailableintherequestedformorrequiredrecalculation/aggregation,orwascollectedfromfragmentedsources.

Companiesdidnotindicatehowmucheffortwasrequiredtocompletesurveyquestions.TheQualityCultureSurvey,estimatedtotakeapproximatelyfiveminutesperrespondent,wascompletedbymorethan10,000staffinWave1.Europeansurveysrequiredapprovalbyunionrepresentatives;thiswasalsonotincludedintheestimationofeffort.

Inaddition,Wave1Pilotdatacollectionandsubmissionsiteswereallowedtoprovide“goodenough”data.ThismeansthattheymaynothaveconductedallthecheckingandapprovalstepsthatwouldotherwiseberequiredforformalsubmissiontoFDA.

McKinseyestimatedtheoperationaleffortrequiredtosetupandsupporttheWave1Pilot, including:

f Preparing submission templates f Establishing the database f Definingthecollectionprocessfordatasubmission f Supporting companies in submitting their data f Analyzing the data

TheseestimatesdidnotincludethetimespentbyMcKinseypersonnelworkingaspartofISPE’sprojectteam,contributingtothepilotdesignanddefinitionsdevelopment,producing the report for ISPE and individual companies’ benchmarking reports.


3

3.6 Data Collection Period

Part of the announcement at the launch of the Wave 1 Pilot included details of the data collection period:

Companies to provide data [emphasis added] for approximately one year (historic) and 3-months “real-time,” but individual flexibility possible to accommodate data availability.

AprimarygoaloftheWave1Pilotwastohavefindingsaccruedbytheendof2014sothattheycouldbeavailableeitherbeforeFDAissueditsFederalRegisterqualitymetricnotificationorforconsiderationduringthepubliccommentperiod.Giventhistighttimeline,andtoensuremeaningfuldatawascollectedinshortorder,itwasdecidedtocollectdatausingtwodataperiods–retrospectivelyfor12monthsforcertainmetricswherecompanydataalreadyexisted,andprospectivelyfor3monthsfor metrics that may not have been previously collected or measured at a site. Templatesweredesignedtocollectdata.AnexampleofsitedatafrequencyandcollectionperiodisgiveninTable1,showinganonsterilefinisheddosageformasan example.

Table 1: An Example of Frequency and Period of Collecting Retrospective and Current (Prospective) Data in the Pilot

Data Points/MetricRetrospective (Nominal)

Current (Nominal)

Baseline Data

Production volume in units

Monthly for 12 months Monthly for 3 months

Packs released

Lots dispositioned

Lotstested—total

Lotstested—stabilityonly

Site head count

Annual 3 MonthsSitequalityheadcount

Number of products


3

Data Points/MetricRetrospective (Nominal)

Current (Nominal)

Site Data

Rejectedlots

Monthly for 12 months Monthly for 3 months

Reworked/Reprocessedlots

ConfirmedOOS—total

ConfirmedOOS—stabilityfailuresonly

UnconfirmedOOS

Total recall events

RecallEvents—ClassIandII

Rejectedlots

Total recalled lots

Total complaints

Critical complaints

ProductssubjecttoAPQRAnnual No data collected

APQRontime

NumberofCAPAswitheffectivenesschecks

3 monthly in 4 periods, April 2013 to March 2014 One 3 month period

NumberofeffectiveCAPAs

Number of deviations

Number of recurring deviations

Product Data

Total complaints for the product for the reporting year

Annual for individual products

Current period, typically 3 months for individual products

Total critical complaints for the product for the reporting year

Total packs released for the product for the reporting year

Total lots dispositioned for reporting year

Total lots tested for reporting year

Rejectedlots

ConfirmedOOS


4 Operational Processes for the Wave 1 Pilot

This section discusses some of the key operational aspects of the Wave 1 Pilot for both the McKinsey support team and the participant companies.

4.1 McKinsey Operational Process

TheMcKinseysupportprocessesforperformingtheworkassociatedwiththeWave1PilotwerebasedontheirexperiencegainedfromtheirPOBOSbenchmarking programs.

Anoverviewoftheprocessisasfollows:

f Preparing templates for data submission: – Differenttemplateswererequiredfordrugproducts,sterileandnonsterile

drug products, and for labs. – Templatesforeachmetricincludeddetaileddefinitions,fieldsforeachdatapointtimeperiod–monthly,quarterlyorannual–andacommentaryfield.

– Templateswerevalidatedusingbuilt-inchecksfordataconsistency(e.g.,totalOOSbyproduct=totalOOSforsite)andlockedbeforetheyweresentto sites.

f Setting up databases for input and analysis. f Definingthedata-collectionprocess. f Translatingthequalityculturesurveyintotheappropriatelanguageforeach

participating site. f Answeringexploratoryquestionsfrominterestedcompanies,suchas:

– Howmuchtimeandeffortandresourceswillweneedforthepilot? – Howmuchdoesitcost? – Whatisinvolved?

f Enrolling companies into the Wave 1 Pilot by arranging: – Confidentialityagreements – Purchasing orders – Explainingdatasubmissionrequirements

f Answeringquestionsduringthedata-collectionphaseandupdatingthe FAQs document.

f Reviewingandclarifyingdata(i.e.,potentialoutliers). f Processing the Quality Culture and Process Capability Surveys. f Analyzingdata,runningcorrelations,profilingmetrics. f ReportingresultsoftheWave1Pilotdataanalysis.


4

Usingagreed-upondefinitionsandsurveyquestions,projecttimelinesanddata-collectionfrequencies,McKinseypreparedthetemplatesinExcel.Anexampleisprovided in Appendix 3.

For the Wave 1 Pilot, companies completed the Excel spreadsheets manually and sent them to the McKinsey support team. An automated data-entry process may be considered for the future.

AdatalockwasappliedattheendofNovember2014,andallparticipatingcompanies complied.

4.2 Experiences from Participating Companies

Companies participating in the Wave 1 Pilot provided feedback throughout the data collectionandanalysisphases;PilotLeadsmeetingsheldbytheIndustryEngagementSubteam also provided feedback.

ThemajorityofcompaniesthatparticipatedintheWave1Pilotreportedbenefitsarisingfrom their involvement. These included:

f The opportunity to trial a centralized submissions process gave them a deeper understandingoftheimpactofstandardizedmetricsdefinitionsanddesign.

f Participation enhanced the maturity of their internal metrics programs. f Eachparticipatingsitereceivedaconfidentialbenchmarkingreportthatoutlinedtheirperformancewithrespecttotheirpeergroup(s).

Withrespecttoreporting“goodenough”data,someparticipantsnotedthatsomedatatheycollectedwerederivedfromnon-GMPsystems(e.g.,productportfolios).Discussions have indicated the need for a validated/cGMP-based metrics collection, storageandreportingsystemthatcouldbereviewedbyinspectionteamstoconfirmthe veracity of any metrics reported to FDA. Concerns raised about the potential burden associatedwiththiswillrequirefurtherconsideration.

A detailed example of some of the key aspects of one company’s experience of participating in the Wave 1 Pilot is provided in a case study in Appendix 4. A summary of the case study’s main points are:

f CompanyAhasalargeproductrangeandverycomplexsupplychains,whichmakeassigningproduct-levelmetricsextremelydifficultandtime-consuming.

f Changing their current IT systems to a standardized set of metrics that could produceproduct-leveldatawouldrequiresignificantinvestment.

f DatareportedintotheWave1Pilotwere“goodenough”toexaminethedata-collectionand-submissionsystems’,mechanics,buttheywerenotsubjectedtothereviewandcheckingthatwouldberequiredforofficialsubmissiontoFDA.


5 Findings from the ISPE Quality Metrics Initiative Wave 1 Pilot

ThissectiondiscussesthemainfindingsfromtheISPEQualityMetricsWave1Pilotand includes considerations of the sample size, metric and survey data analysis, collectionandsubmissioneffortdataandthekeyrelationshipsobserved.

5.1 Sample Size

The Wave 1 Pilot collected data from 18 participating companies at 44 individual sites. Distribution of the sites by technology, type of product/business, region and company size is given in Figure 6 and Table 2.

Figure 6: Sample Distribution of Participating SitesDiverseSample:18ParticipatingCompanieswith44Sites/Technologies

468 8

18

18

Other API Bio DS Sterile Solids

567

26

CMO, labs

Cons. health

Gx Rx

73

16

LA EMEA NA Asia

By technology By type of product

By region By company size

39

5

Large Small

Note:Ifasitehasmorethanonetechnologywecountthenumberofseparatetemplatestheywillfill,usuallyonepertechnology


5

Table 2: Figure 6 Abbreviations

Technology

Bio DS Biopharmaceutical or biological drug substance site

API Small-molecule drug substance (active pharmaceutical ingredient)

Type of product

Rx Originator company

Gx Generic company

Cons. health Consumer health or OTC

CMO Contract manufacturing organization

Labs Contract research and testing laboratories

Region

NA North America

EMEA Europe, Middle East and Africa

LA Latin America

Company size

Small < $1 billion in revenues

Large > $1 billion in revenues

The Quality Culture Survey sample size comprised 10,300 respondents from 37participatingsites.Thisdiffersfromthetotalpilotsamplesizeof44sitesbecausesomesiteshadtwodifferentproducttechnologieslocatedonthesamephysicalsite.ThesesitescompletedtwoWave1Pilotmetricstemplatesastwoseparatesites,yetsubmittedtheirqualityculturesurveyassessmentsasonesite.

TheWave1Pilotsamplewasconsideredsuitablydiverse,bothbyregionandtechnology, to facilitate a representative analysis. Other aspects of the sample, however,hadsimilaritiesthatshouldbeacknowledged:

f Most participant companies originated from developed countries, and therefore didnothaveanysignificantlanguageorinterpretationissueswiththestandardizeddefinitionsorthepilottemplatesubmissioninstructions.

f Allenrolledsitesmaybeconsideredingoodstandingwithrespecttoquality;theydidnothaveanyqualityorcompliance(e.g.,consent)issues.

f Themajorityofcompanieswereclassifiedaslarge. f All companies and their sites consented to enroll in the Wave 1 Pilot and, therefore,hadanopenandpositivedispositiontotheuseofqualitymetricstomonitoranddriveenhancedqualityperformance.


5

5.2 Metric and Survey Data Analysis and Discussion

Toassistwithdataanalysisandpresentation,metricscollectedinthepilotweregroupedintodifferentcategoriesshowninTable3.Theseinclude:

f External Quality Outcomes:Anoutcomethatmayaffectthepatientdirectly(e.g., the patient makes the complaint) or indirectly (e.g., a recall leads to product unavailability).

f Internal Quality Outcomes: An outcome observed by a company that could affectbusinessoutput(e.g.,arejectedproduct),productdoesnotleavethecompanycontrol,however,andisnotavailabletothepatient.

f Supplier Quality: ConfirmedOOSofanincomingrawmaterialisameasureofsupplierquality.

f Laboratory Quality: AnunconfirmedOOSisameasureoflaboratoryquality,whetherornotthecauseisidentified.

f Site Maturity: Metrics that could be considered measures of site maturity, suchasannualproductqualityreviews(APQRs)completedontime,highvaluesofcorrectiveandpreventiveaction(CAPA)effectivenessrateandlowvaluesofrecurring deviations rate.

Table 3: Categories of Metrics

Categories Metrics

External Quality Outcomes (market)

Totalrecallevents—US

RecalleventsClassIandII—US

Recalledlots—US

Total complaints rate

Critical complaints rate

Internal Quality Outcomes

Lot acceptance rate

ConfirmedOOSrate—release

ConfirmedOOSrate—stability

Deviations rate

Rightfirsttime(rework/reprocessing)rate

Mediafillssuccessful

Environmental monitoring (EM) action limit investigations rate

EM action limit rejects rate

Supplier Quality ConfirmedOOSrate—incomingmaterials

Laboratory Quality

Site Maturity

UnconfirmedOOSrate

APQRcompletedontime

CAPAseffectiverate

Recurringdeviationsrate


5

5.3 Wave 1 Pilot Quality Metric Data Analysis

Individualcompanydatawascomparedwiththetotalsampletodevelopabenchmarkingreportforeachcompany.Thesedatawereconfidentialtothecompany and McKinsey. ISPE did not have access to individual company data.

Total industry-level data for all metrics collected, relationship determinations and comments are provided in Appendix 5 of this report.

Totalindustry-leveldataforeachmetricweretestedandevaluatedasappropriate,using either scatter plots (correlation of a leading indicator vs. an outcome) or quartilesanalysis(rangeofoutcomevaluesagainstleadingindicatorvaluessplitintoquartiles)orprofiling(formetricswithdiscretevalues).

ManyfigurespresentedinAppendix 5 of this report also contain a summary ofthestatisticaltoolsappliedandincludeexplanation,wherenecessary.

Asperstandardstatisticalpractices,incompletedataandextremeoutlierswereexcludedfromanalyses.Theresultingsamplesizesallowedstatisticalanalysiswithsomelimitations,asfollows:

f Toallowforsufficientsamplesize,mostanalysesweredoneforfinisheddosagesites overall, not by technology.

f Productdatawascollectedonannualbases,anddidnotallowtime-laganalysistoseehowproductmetricscorrelateovertime.

f Anyrelationshipsidentifiedbetweenmetricswerestatisticallysignificant(lessthan5%likelihoodofacoincidence),however: – Thestrengthoftherelationshipsvaryandmayberelativelylow (e.g.,somemaycorrelatewithR2of30%or40%),sincethesemetricsareinfluencedbymultiplefactors.

– Correlation doesn’t imply causation. Understanding the underlying factors anddirectionofarelationshipwillrequirefurtherwork,ideallyonalargerdata set from a larger sample size.

From the total industry database, median ranges of individual metrics split bytechnologyaregiveninFigure7andFigure8below.


5

Figure 7: Metric Ranges by Technology: Solid Dosage and Sterile

External (market) quality outcomes

Internalquality outcomes

Supplier quality

Lab quality

Site maturity

� Recall events – US� Recall events class I and II – US� Recalled lots – US� Complaints rate (per million packs)� Critical complaints rate (per million packs)

� APQR on time (%)� CAPAs effective (%)� Recurring deviations (%)

� Lot acceptance rate (% released out of all finally dispositioned lots)� Confirmed OOS – release (per 000’ lots release-tested)� Confirmed OOS – stability (per 000’ stability lots tested)� Deviations rate (per 000’ lots dispositioned)� Rework rate (% lots dispositioned)� Media fills successful (%)� EM action limit investigations rate (%)� EM action limit rejects rate (%)

� Confirmed OOS – incoming materials (per 000’ RM/PM lots tested)

� Unconfirmed OOS (per 000’ lots tested)

Solids Steriles

0.00.00.0

0.00.00.0

0.00.00.0

60.2

16 240.4

1.00.51.5261.2 0

0.00.00.0550.1

0.00.00.0761.7

99.7% 99.4% 98.8% 99.6% 98.8% 96.1%0.8 1.5 2.9

0.0 2.3 6.20.5 1.0 8.3

0.0 2.5 6.3257 472 74494 134 230

0.0% 0.3% 0.7%0% 0.1% 1.0%

0.9 2.7 5.40.3 1.1 4.9

0.6 0.3 4.11.6 3.2 4.3

90.9%100% 100% 100% 100%96.2%

96.7% 94.9% 92.6%98.5% 97.4% 33.4%5.0% 9.5% 23.6%5.2% 12.9% 27.5%

100% 100% 100%0.6% 2.12% 8.8%

0 0 0.02%

TQ BQMedian TQ BQMedian


5

Figure 8: Metric Ranges by Technology: API, Bio and OtherMedian

0.00.00.0

0.00.00.0

N/A N/AN/AN/A

0.00.00.01110.9

100% 95.3% 98.1%

2.9 25.1 6.10.9 15.4 0.0464 9678 154

1.6% 0.2% 0.6%

3.6 2.2 3.7

4.2 1.7 1.4

100% 87.5% 100%95.8% 33.2% 64.9%7.5% 13.6% 20.7%

API Other FPBio


Internalquality outcomes

Supplier quality

Lab quality

Site maturity

� Recall events – US� Recall events class I and II – US� Recalled lots – US� Complaints rate (per million packs)� Critical complaints rate (per million packs)

� APQR on time (%)� CAPAs effective (%)� Recurring deviations (%)

� Lot acceptance rate (% released out of all finally dispositioned lots)� Confirmed OOS – release (per 000’ lots release-tested)� Confirmed OOS – stability (per 000’ stability lots tested)� Deviations rate (per 000’ lots dispositioned)� Rework rate (% lots dispositioned)� Media fills successful (%)� EM action limit investigations rate (%)� EM action limit rejects rate (%)

� Confirmed OOS – incoming materials (per 000’ RM/PM lots tested)

� Unconfirmed OOS (per 000’ lots tested)

AlthoughafewsitesexperiencedarecallduringtheWave1Pilotperiod,themedianforrecallsinbothFigure7andFigure8waszero.(Forfurtherdetailssee“RecallEvents” in Appendix 5.)

In addition to the data collected, participating sites provided very useful feedback on thedefinitionsusedinthepilot.Anexampleofthisincludestheextensivefeedbackreceivedonthedefinitionfor“recurringdeviationsrate,”showninFigure9.


5

Figure 9: Feedback on Definition of Recurring Deviations Rate

Pilot definition

Numberofdeviations(outofthetotalreportedinline49)forwhichduringthe12monthperiodprecedingeachdeviation,atleastoneotherdeviationhasoccurredwiththesamerootcausewithinthesameprocessand/orworkarea.

Feedback/ alternative definitions

f Periodconsideredfortherecurrencemaybebasedonthetypeofissueorworkarea(e.g.dependsontheoccurrenceofthespecificprocesswithdeviation),orlefttothequalitypersonneljudgment;

f Somesitesuse6monthsor2yearsaslookbackperiod; f Deviationmaybeconsideredrecurringifreoccurredanywhereintheplant,notjustinthesameworkarea f AtleastonesiteconsidersrecurringonlydeviationsthathavehadaCAPA(asrecurrenceindicatesineffectiveCAPAimplementation(otherdeviationswithsamerootcauseareconsidered“repeat”))

f Manysitesfeelthatfinalassessmentdependshowdeepyougointotherootcause–fromamoregeneral“operatorerror”toaveryspecificerrordescription(hastobethesameproduct,natureofincident,rootcausecategory)–whichwouldinfluencehowrecurrenceisidentified;

Recurringdeviationsratewasnottheonlymetricthatproducedlotsofquestionsregardingdefinitionandhowthedatashouldbecalculatedandsubmitted.Otherdefinitionsinneedofrefinementandfurtheralignmentarecriticalcomplaints,reworkrateandCAPAeffectivenessrate.

5.4 Wave 1 Pilot Quality Culture Survey Data Analysis

Forthequalityculturesurvey,eachofthe15questionscouldbeansweredusingoneoffiveresponseoptions:

f Strongly agree f Agree f Disagree f Strongly disagree f Ican’tanswerthisquestion


5

Tofacilitatedataanalysisandrelationshipmappingwiththequalitymetricsdataset,scoringmechanismwasestablishedbasedonthe“topboxes”approach.Foreachquestion,theproportionof“Stronglyagree”and“Agree”answerswascalculated.Topboxesanalysisassignsa1(or100%)ifallrespondentsreply“Stronglyagree”or“Agree,”and0(0%)ifallrespondentsreply“Disagree”or“Stronglydisagree.”Theseproportionalvalueswerethenplottedonaradardiagramforeachquestion,asshowninFigure10.

Figure 10: Quantitative Quality Culture Scores Plotted on a Radar DiagramTotalof10,300respondentsfrom37sites

0.6

0.7

0.8

0.9

1

Patientfocus

Training

Problem solving

Recognition

Metrics

Knowledge

Continual improvement

Coaching Dialogue

Gemba

Awareness

Responsibility

Openess

Ethics

Motivation

Score1

Capabilities

Governance

Leadership

Mindset

Integrity

1 Totalscorecalculatedas“topboxes”(shareof“agree”and“stronglyagree”responses)ratio.100%=allrespondentsagreeorstronglyagree,0%=nobodyagrees or strongly agrees.

Theresultsshowedthatatthesitelevel,questionspertainingtotheculturalelementsofCapabilitiesandIntegrityreceivedthehighestratings,whilethoseassociatedwithGovernanceandLeadershipscoredrelativelyweaker.Thetop-boxesapproachfacilitatedaquantitativeanalysisofthequalityculturefindingsandprovidedsomeinterestinginsights,butitisbroadlyagreedthatfurtherworkisrequired,potentiallyinaWave2Pilot,tounderstandthesefindingsbetter.


5

Therangeofvaluesassignedtotheresponsesforeachqualityculturequestionaregiven in Figure 11. Orange shading represents the range of site responses that fall belowtheindustryaverage,whilelightorangeshadingindicatestherangeofsiteresponses exceeding the industry average.

Figure 11: Range of Values for Quality Culture Responses

0.05 0.15 1.00 0.50 0 0.80 0.90 0.45 0.25 0.35 0.55 0.65 0.10 0.20 0.30 0.40 0.60 0.70 0.75 0.85 0.95

Maximum Minimum AverageSurvey Question Avg. And ranges

Min

dset

s In

tegr

ity

Lead

ersh

ip

Gov

erna

nce

Cap

abili

ties

Training: The training I have received clearly helps me to ensure quality in the end product

Patient focus: I know which parameters of our products are particularly important for patients

Problem solving: All line workers are regularly involved in problem solving, troubleshooting and investigations Recognition: We recognize and celebrate both individual and group achievements in quality Metrics: Up-to-date quality metrics (e.g. defects, rejects, complaints) are posted and easily visible near each production line Knowledge: Each line worker can explain what line quality information is tracked and why Continual improvement: We are regularly tracking variations in process parameters and using them to improve the processes Coaching: Supervisors provide regular and sufficient support and coaching to line workers to help them improve quality Dialogue: We have daily quality metrics reviews and quality issues discussions on the shop floor Gemba: Management is on the floor several times a day both for planned meetings and also to observe and contribute to the daily activities Awareness: Every line worker is aware of the biggest quality issues on their line and what is being done about them Responsibility: All employees see quality and compliance as their personal responsibility Openness: I am not afraid to bring quality issues to the management’s attention Ethics: People I work with do not exploit to their advantage inconsistencies or “grey areas” in procedures

Motivation: All employees care about doing a good job and go the extra mile to ensure quality

Responsestoquestionsonethics,knowledge(e.g.,metricstracking),andGemba(Japanesefor“atthesite,”itrefersobservationsofaprocessinaction,ormanagementpresenceontheshopfloor)hadthehighestvariationinresponses.

Toexplorewhethertherearedifferencesinresponsestoquestionswithineachoftheculturaldimensionexamined,thevaluesforeachdimensionwereplottedagainstvaluesoftotalqualitycultureresponses.TheseplotsaregiveninFigure12.


5

Figure 12: Plot of Total Quality Culture Values for Each Quality Culture Dimension

00.20.40.60.81.01.2

0.4 0.6 0.8 1.0 1.2

QTotal

QLeadership

00.20.40.60.81.01.2

0.6 0.8 1.0 1.2

QTotal

QMindset

00.20.40.60.81.01.2

0.6 0.8 1.0 1.2

QTotal

QIntegrity

00.20.40.60.81.01.2

0.7 0.8 0.9 1.0 1.1 1.2

QTotal

QCapabilities

00.20.40.60.81.01.2

0.4 0.6 0.8 1.0 1.2

QTotal

QGovernance

Leadership

Capabilities Integrity

Governance Mindset

R2=0.9974

R2=0.8935

R2=0.8192

R2=0.9283

R2=0.8781

1 Totalscorecalculatedas“topboxes”(shareof“agree”and“stronglyagree”responses)ratio

Thisanalysisshowsthattheculturaldimensionsincludedinthequalityculturesurveyarehighlyconsistentbetweeneachother—i.e.,thelineslopesaresimilarandeachdimensionishighlycorrelatedwiththeoverallvalue.

Thisanalysisconfirmedthatattemptstodeterminerelationshipsbetweenqualitycultureandotherqualitymetricsvaluescanusetotalqualityculturevalues.


5

5.5 Data Collection and Submission Effort Data

Estimatesofeffortexpendedbycompaniesasanindustrytotalwereprovidedandthe results of this analysis are provided in Appendix 5 for each individual metrics.

Theaveragetimeoneachsitetocollecttheannualdatawas88hours.Effortestimates(hours)splitbetweenthevarioustypesofbusinessarealsogiveninFigure13below.

Figure 13: Average Time to Collect Annual Quality Metric DataTotaltimespentoncollectingdata(12months),[Hours]

54

156 162

80

14

88

434

499

434422

55

120

14 138 84

Average (minimum and maximum as range)

Drug substance

Gx

Total¹

OTC

Rx

Labs

28

1 Excluding DS and Labs

TheresultsshowthatOTCandgenericsitestookthreetimeslongerthanoriginatorcompaniestocollectsimilardata.Participantsseemedtoindicatethisdifferencecould be due to issues such as increased number of products and (potentially) the increased complexity of supply chain for OTC and generic sites. Site volumes (packs or lots dispositioned) and site complexity (number of products) did not appear to influencethecollectionandsubmissioneffortrequired.

Drugsubstanceandlaboratorysitesrequiredsignificantlylesstimethanfinishedproduct sites because less data is collected. Even though the Wave 1 Pilot had limitedvisibilityinsuchsites,theirworkloadwascalculatedat45hourscollectionandreportingpersite,theaverageworkloadforbothlabanddrugsubstancesites.


5

Approximately12,000sitesgloballyhaveFederalEstablishmentIdentifier(FEI)numbers;[17] close to 6,000 are registered as Finished Dosage and API sites, and the remaining 6,000 sites include medical gases, medical feed, labs, Center for Biologics Evaluation andResearch(CBER)establishments,andothers.Wave1PilotdataestimatedtheaverageworkloadfortheAPIandFinishedDosagesitesat90hours,whiletheaverageworkloadfortheothersitesisestimatedat45hours.Usingatypicallaborcost(includingoverhead)of$75,000peryear,collectingthisamountofdatawouldcosttheindustryapproximately an additional $35 million annually.

Thisestimateisconsideredconservative,however,becauseitdoesnotincludeseveral factors, such as:

f Wave1Pilotsiteswereallowedtoprovide“goodenough”data.SubmissiontoFDAcouldrequiremorethoroughandcompletedatacollection,additionalreviewanddataverificationsteps,potentiallyatdifferentlevelsanddisciplinesandwouldhavetobeaccompaniedbyconsideredcomments.

f Timeforinternaldiscussions,managementreviewandabove-siteguidancewasnotincluded.

f Efforttodevelopandvalidatenew/modifiedITsystemswasnotincluded.(ThiswasnotrequiredfortheWave1Pilot.)

f Participantshadflexibilitytoprovidemostpragmaticdataset (e.g., for all products at site or only those for the US market).

f Datawereprovidedwithineachsite,notthroughfullproductsupplychain. f Participants had mature systems and capabilities. f Majorityofsiteswerefromdevelopedcountries.

Theadditionalcosttoproduceofficialsubmissionscouldbringtheannualcostof such a program to $100+ million.

The time spent to collect the 3 months of data for the three metrics collected at both productandsitebases(lotacceptancerate,confirmedOOSandcomplaints)aregivenin Figure 14.


5

Figure 14: Time Spent To Collect Product Level Data and Site Level Data for 3 MonthsTime spent on collecting site1andproductleveldata,[Hours,(15months)]

19.4

76.0

9.6

OTC Rx

Drug substance

Metrics collected at both site and product level:

Lot acceptance rate Confirmed OOS Complaints, ppm

17.5

87.0

5.0

Site level data1 – Hours

Product level data – Hours

1.9 -11 4.6

xx Difference between site and product data time

Note:Sitesthatdidn’tsubmitfullproductdatawereexcludedaseffortlikelytobeunderstated.Gxsamplewastoosmalltoreportresults1 Onlyformetricswithproductlevelgranularity

NodataisincludedintheanalysisshowninFigure14forGenericcompaniesduetoinsufficientsamplesize.Alsositesthatdidnotsubmitfullproductdatawereexcludedsincetheywerelikelytounderestimatetheeffortrequired.

ThefindingsfromtheanalysisshowninFigure14are:

f Rxsiteswereabletocollectthesemetricsatproductlevelaseasilyasatsitelevel(therewasanapproximately10%differenceintimeforthethreemetrics)due to several factors.

f SomeOTCsitesfoundproduct-leveldataforthesemetricsmoredifficulttocollectthansitedata(15%moretimeforthethreemetrics).


5

PotentialreasonswhyRxsiteswereabletocollectproductleveldataaseasilyas site level data include:

f Thesethreemetricswerechosenintentionallytobecollectedeasilybyproductlevel. f Some sites already had systems set up to collect product-level data. f FortheWave1Pilot,productdatawascollectedwithineachindividualsite

rather than across a full supply chain. f Mostsitesselectedforthepilothadgoodsystemsandproficientpersonnel;

some even had systems set up to allocate data by product and then aggregate it at site level.

f Definitionsallowedaccurateallocationbyproduct(e.g.,usinglotsdispositionedrather than lots attempted).

OTCsitesspentmoreeffortcollectingsitelevelmetricsfromFigure13andFigure14,andevenmoreeffortcollectingproductlevelmetricsfromFigure14,potentiallydueto:

f Separatingcomplaintsdowntoindividualformulationsratherthanataproductfamily(e.g.,severalflavorsorcolors)level.

f Countingpackorunitdata(vs.cases),anddisaggregatingdatafromAPR’sto match the time frame of the data collection period.

Using the full 15 months of data, the time to collect individual metrics is listed inFigure15,withcolumnsgivenforRx,GxandOTCcompanies.


5

Figure 15: Effort per Metric for Rx, Gx and OTC Companies15 months of data

Rx Gx OTC

Time [hours]

Thisanalysisindicatesthatthemosttime-consumingmetricstocollectwere;OOSforfinalproductandcomplaints,bothtotalandcritical.Theleasttime-consumingmetricstocollectwererecallsandAPQRsontime.

Inadditiontocapturingthenumberofhoursrequiredtocollecteachmetric,Wave1Pilotcompanieswerealsoaskedtoreportthedegreeofdifficultyofcollectingaparticularmetricusingascaleof1to4,(1beingeasiestand4mostdifficult).ThisanalysisisshowninFigure16,asestimatedbytheparticipatingcompaniesthemselves.


5

Figure 16: Degree of Difficulty of Collecting Each Metric

Rx Gx OTC

Difficulty[1(easy)-4(difficult)scale

1-Veryeasy,allthedatawehavein an electronic system/paper based documents in the form requiredhere

2 - Fairly easy, most of the data wehadready,somewehadtorecalculate/aggregate basing ontheprovideddefinitionsandourrawnumbers

3-Difficult-wehadtorecalcu-late/aggregate most of the data, sometimes from very fragmented sources

4-Verydifficult,wedidnothavethe data at all for retrospective period,weonlystartedcollec-ting it for current period

Average

3 quartile 1 quartile

Thisanalysisindicatesthatthemostdifficultmetricstocollectweretherecurringdeviationsrateandthetwosterilespecificmetrics.TheleastdifficulttocollectwereonceagainUSrecallsandAPQRsontime.

Figure17showsafurtheranalysisrelatingtoeaseofdataaccessandburdentocollect,plottingtheaveragedifficultyratingagainstthemediantimefordatacollectionforRxcompanies.Thisgroupwaschosenastheypresentedthelargestsamplesizein the Wave 1 Pilot.


5

Figure 17: Average Time for Collection of a Metric and Median Time for CollectionsFinisheddosageRxsample,15monthsofdata

1

2

3

0 5 10 15 20

Lot acceptance

Investigations related to action limit excursions

Confirmed OOS - product

CAPA effectiveness

APQRs on time

Media fill successful

Median time for data collection [hours]

Recalled lots (US)

Confirmed OOS - stability Rework rate

Average difficulty rating

Recurring deviations


Confirmed OOS - RM

Recall events - class I and II (US)

Unconfirmed OOS rate Lot rejects related

to action limit excursions


Deviations rate Recall events (US)

TheresultsfromFigure17showthatthemosttime-consumingmetricsarenotnecessarilyratedasthemostdifficult.

Theredcircleshighlightthemostdifficulttocollectmetrics:

f Investigations and lots rejected related to environmental monitoring for sterile products.

f Recurringdeviations:Themajorityofsitesreportedthattheydonotcurrentlyhave processes in place to collect this metric accurately. Where it is collected thereisbroadvariationinthedefinitionof“recurrence.”

AsshowninFigure15,OOSforproduct,andcomplaints,totalandcriticalwerethe most time-consuming metrics to collect.


5

5.6 Process Capability Survey

The results of the Process Capability Survey (see Appendix 2) are given in Figures 18, 19and20.Responsestothefirsttwohigh-levelquestionsinaregiveninFigure18.

Figure 18: Response to Process Capability Questions

No 5

Yes 95

Do you measure that the process remains in a state of control during manufacturing?

To what % of products is it applied? 1

26

431

4

30% - 40%

10% - 20%

70% - 80%

90% - 100%

0% - 10%

1 Whenonlysomeproductswerechosen,choicewasbasedonriskapproachtocustomerandimportanceforbusiness

FromFigure18weobservethat95%ofsitesapplyongoingmonitoringduringproductionprocessestoanaverage74%oftheirproducts.Whereprocessmonitoringwasappliedtoaselectedrangeofproducts,arisk-basedapproachwastypicallyusedtodeterminewhichproductsrequiredmonitoring.

Moredetailedquestionsaskedwhichprocesscapabilitystatisticaltoolwasmostusedandtowhatattributes[e.g.,criticalqualityattributes(CQAs)]orparameters[e.g.,criticalprocessparameters(CPPs)orin-processcontrols(IPCs)].ResponsestothesequestionsaregiveninFigure19andFigure20.


5

Figure 19: Process Capability Survey FindingsPercentage of sites using each type of metric1

12%

46%

5%19%19%21%

Other2

21%

Trending

42%

Box plots

0%

Tolerance interval

19%

PpK

14%

CpK

16%Applied to CQA

Applied to CPP

CQA

IPC

CPP

28%

77%

7%23%

35%39%

1 Outofsitesmonitoringcapabilityinanyway2 Othermentionedmetricswereparetocharts,monitoringviaexcursionstrending,I-chartsregression,3sigma,andRun/controlcharts

Theresultsindicatethattrendingismostthewidelyusedtool.Processcapabilityindex(CpK),processperformanceindex(PpK)andtoleranceintervalsareusedlessoften—by39%and22%ofsitesrespectively.While,asseeninFigure20,91%ofsitesmeasuretheircurrentstateofcontrolthroughCQAs,whileonly56%onIPCsand61%onCPPs.

Figure 20: Process Capability Tools in UsePercentage of sites monitoring each parameter type

CQA IPC CPP

91%

56% 61%

As anticipated the capability approach is variable across companies and in terms of use and applicability. The tool employed e.g. Ppk, Cpk, control charts etc. is contextual and there is no one tool that can be applied to all situations. These tools should be used internally for troubleshooting and identifying continual improvement opportunities rather than for monitoring compliance.


5

5.7 Establishing Statistically Significant Relationships

RelationshipsbetweenthecollectedmetricsweretestedandassessedusingthegroupingofquantitativemetricsshowninTable3(externalqualityoutcomes,internalqualityoutcomesandsitecultureandmaturity).ConnectionsbetweeneachofthestandardizedqualitymetricsandQualityCultureSurveyvaluesaredepictedbytheorange lines in Figure 21.

Figure 21: Relationship Testing

Total complaints

Critical complaints US recalls


Deviations recurrence Culture APQR on

time CAPA

effective

Site culture and maturity (leading indicators)

Internal quality outcomes

Relationship testing – annual and quarterly data, overall and by technology

Lot acceptance

Deviations rate

Action limits excursions

(sterile)

Confirmed OOS

Invalidated OOS

Stability failures Rework

Asbefore,incompletedataandextremeoutlierswereexcluded.Theresultingsamplesizesallowedstatisticalanalysiswithsomelimitations:

f Toallowsufficientsamplesizemostanalyseswereperformedforfinisheddosage sites overall, not by technology

f Productdatawerecollectedonannualbasis,notallowingtimelaganalysistoseehowproductmetricscorrelateovertime

Identifiedrelationshipsbetweenmetricsweredeemedstatisticallysignificantwhentherewasalessthan5%likelihoodofacoincidence.Thestrengthoftherelationshipsmayvary,however,andinsomecasesarerelativelylow(e.g.,somemaycorrelatewithR2of0.30(30%)or0.40(40%).ThesizeoftheWave1Pilotdatasetisacknowledgedinthesefindingsanditisalsonotedthatthesemetricsunderexaminationareinfluencedby multiple factors not currently included in this analysis.


5

R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationship)to1(perfectlinearrelationship).Pearsoncoefficient(R)measurestheextenttowhichtwovariablesmoveinthesamedirection.Itvariesfrom 0 (random relationship) to 1 (perfect linear relationship).

TherelationshipsdeterminedfollowingthisanalysisaresummarizedinFigure22.

Figure 22: Significant Relationships

Total complaints

Critical complaints US recalls


Deviations recurrence

Culture

APQR on time

CAPA effective

Site culture

Lot acceptance

Deviations rate

Action limits

excursions (sterile)

Confirmed OOS (release)

Invalidated OOS

Stability failures Rework

Internal quality outcomes

Site maturity (leading indicators)

Demonstrated relationship within the Wave 1 sample Statistically significant Significance >0.05

Theorangelinesshowrelationshipsthatarestatisticallysignificant(<0.05%),whilethegreylineshowsrelationshipsthathaveasignificancelevelexceeding0.05%butarestillconsideredworthexaminingfurthergiventhelevelofvariabilityandrelativelylowsamplesize.

Multivariatecorrelationanalysiswasnotperformedsincethesamplesizewasinsufficientforthis.

A statistically significant relationship does not imply causation. Causation can only be proposed after studying underlying factors, which requires further work, as does understanding the degree of correlation and direction of a relationship.


5

5.8 Statistically Significant Relationships in Wave 1 Pilot Data

Statisticallysignificantrelationshipswerefoundbetweenthefollowingmetrics,orqualityculturevalues,andthosemetricsasshownintheappropriateanalysisfigure.

f Critical complaints and deviations rate (Figure 23). f US recalls and deviations recurrence (Figure 24). f Lotacceptancerateandrework(Figure25). f Lotacceptancerateandqualityculturevalues(Figure26). f Lotacceptancerateanddeviationsrecurrence(Figure27). f Actionlimitexcursions(sterileproducts)andqualityculturevalues(Figure28). f APQRontimeandCAPAeffectivenessrate(Figure29).

Figure 23: Critical Complaints and Deviations RateCritical complaints in selected intervals of deviations rateSampleof21plants,finisheddosage,averageannualvalues

Median level of critical complaints for sites with higher and lower deviations rate ppm

8.13

1.340.880.34

<0.16 <0.12 <0.39 >0.39 Deviations rate Per dispositioned lot

Sample size 5 6 6 5

Significance: 0.045

Significancelevelisaresultofindependentsamplesmedian(chi-square)test.Valuebelow0.05meansthatthemediansofVariablearesignificantlydifferentbetweencategories.


5

Criticalcomplaintsrateincreaseswithincreaseofdeviationsrate.Valuesfordeviationsratearequartileboundaries.Thecriticalcomplaintsratevaluesarestatisticallydifferent(0.045%)usingthechi-squaredtest.

Figure 24: US Recalls and Deviations RecurrenceRecurringdeviationsforplantswithandwithoutrecallsSample of 35 plants, all technologies, average annual values

Median level of recurring deviations % of closed deviations

38

13

US no recalls US recalls Recalls

Sample size 31 4

Significance: 0.001

+192%

Significancelevelisaresultofindependentsamplesmedian(chi-squared)test.Valuebelow0.05meansthatthemediansofvariablearesignificantlydifferentbetweencategories.

Thereisarelationship(significance0.001%)betweenUSrecallsandrecurringdeviations.Because only four of the 35 sites in this analysis had a recall, causal relationship betweentherecallandrecurringdeviationsratehasnotbeenestablished.


5

Figure 25: Lot Acceptance Rate and ReworkReworkrateinselectedintervalsoflotacceptanceSample of 38 plants, all technologies, average annual values

Median level of rework rate % lots dispositioned

0.62

0

>99 <99 Lot acceptance rate

% lots dispositioned

Sample size 19 19

Significance: 0.001

Significancelevelisaresultofindependentsamplesmedian(chi-square)test.Valuebelow0.05meansthatthemediansofvariablearesignificantlydifferentbetweencategories.Significancerepresentsnon-lineardependency.

Higherreworkrateisassociatedwithhigherlotacceptancerate(fewerrejects)atasignificancelevelof0.001%.Samplesizesaresplitequallybetweentwolevelsofreworkrate.Higherreworkratesarealsoassociatedwithfewerrejects(higherlotacceptancerate),thereforereworkratemayhaveadditionalvalueasbalancingmetric,butthesitepracticesdrivingthisrelationshipstillrequirebetterunderstanding.


5

Figure 26: Lot Acceptance Rate and Quality Culture ValuesSample of 34 plants, all technologies, average annual values

93

94

95

96

97

98

99

100

0.6 0.8 1.0

Lot acceptance rate % of lots dispositioned

Quality culture overall Top 2 boxes

R2 = 0.29 R = 0.54

P-value = 0.001

R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationshipbetweenXandY)to1(perfectlinearrelationship).Pearsoncoefficient(R)isameasuretowhatextenttwovariablesmoveinthesamedirection.Itvariesfrom0(randomrelationship)to1(perfectlinear relationship) or -1 (perfect negative linear relationship). P-value is probability that correlation is zero (in this case this means there is no linear correlation betweenXandYvariables),valuebelow0.05indicatessignificantresults.

Strongerqualityculturevalues(total)areassociatedwithhigherlotacceptancerate.Thesignificancelevelof0.001%isstrong,however,thelevelofcorrelationisweak(R2=0.29)sincelotacceptancerateisalsoinfluencedbyotherfactorsbesidesqualityculture.


5

Figure 27: Lot Acceptance Rate and Deviations RecurrenceSample of 25 plants, all technologies, average annual values

97

98

99

100

0 5 10 15 20 25 30 35

Recurring deviations % of closed deviations

R2 = 0.22 R = 0.47

P-value = 0.017

Lot acceptance rate % of lots dispositioned


Ahigherdeviationsrecurrencerateiscorrelatedtolowerlotacceptancerateatasignificancelevelof0.017%,withhigherqualityculturevaluesalsoassociatedwithlowerrecurringdeviationsrate.Thelevelofcorrelation(R2=0.22),however,isweak,sincelotacceptancerateisinfluencedbyfactorsotherthandeviationsrecurrence.


5

Figure 28: Action Limit Excursions (Sterile Products) and Quality Culture ValuesSample of 6 plants, steriles, average annual values

0

0.05

0.10

0.15

0.20

0.84 0.88 0.92 0.96

Action limit excursions Per sterile lot dispositioned

Quality culture capabilities Top 2 boxes

R2 = 0.69 R = 83

P-value = 0.04


Strongerqualityculturevalues(capabilitydimension)alsolinktofeweractionlimitexcursionsforsterileproductmanufactureatasignificancelevelof0.04%.


5

Figure 29: APQR on Time and CAPA Effectiveness RateSample of 24 plants, all technologies, average annual values

60

80

100

40 60 80 100

CAPAs effective % of evaluated CAPAs

APQR on time % all APQRs

R2 = 0.44 R = 0.66

p-value < 0.001


CAPAeffectivenessrateisstronglyrelated(significance<0.001%)toAPQRsontime.This relationship may possibly be driven by similar culture and capabilities. This has notbeendemonstrated,however.

Inconclusion,thefollowingmetricsandsurveyvalueshavestatisticallysignificantrelationshipseitherdirectlytothemainexternalqualityoutcome(USrecalls)orviaaninternalqualityoutcomeasshowninFigure22:

f Critical complaints f Lot acceptance rate f Deviations rate f Recurringdeviationsrate f Quality culture values

Reworkrateisstronglylinkedtolotacceptanceratebutisnotincludedintheabovelist.


5

5.9 Relationships at Lower Levels of Significance

Inadditiontotheanalysisoutlinedabove,relationshipsataweakersignificancelevel(morethan5%chancethattherelationshipiscoincidental)werealsoseenbetween:

f Critical complaints and lot acceptance rate (Figure 30) f Critical complaints and US recalls (Figure 31) f Deviationsrecurrencerateandqualityculturevalues(Figure32) f CAPAeffectivenessrateandqualityculturevalues(Figure33)

Figure 30: Critical Complaints and Lot Acceptance RateCritical complaints in selected intervals of lot acceptanceSampleof22plants,finisheddosage,averageannualvalues

Median level of critical complaints for sites with higher and lower lot acceptance rate

>99.64 <99.64

0.54 0.42

<99.24

1.20

<98.50

2.78

Lot acceptance rate %

ppm

Sample size 6 5 5 6

Lot acceptance rate may have a relationship to critical complaints. Values for lot acceptancerateinFigure30arequartileboundaries.


5

Figure 31: Critical Complaints and US RecallsComplaintsforplantswithandwithoutrecallsFinished dosage, average annual values

Critical complaints ppm

0.88

0.54

recalls no recalls

+63%

Sample size 5

26

35

no recalls recalls

-26%

Total complaints ppm

25 5 24 Sample size

Figure31showsthatcriticalcomplaintstrendisconsistentwithnumberofUSrecallsfromtheleft-handgraph,whiletotalcomplaintsgoesintheoppositedirection.Apossiblereasonforthisdifferenceisthatcriticalcomplaints,althoughdifficulttodefineprecisely,couldbemuchlessvariablethantotalcomplaints,whichincludesmany categories, such as subjective defects.


5

Figure 32: Deviations Recurrence Rate and Quality Culture ValuesRecurringdeviationsinselectedintervalsofQualitycultureoverallSample of 9 plants, solids, average annual values

Median level of recurring deviations rate % closed deviations

8.3

20.0

>0.8 <0.8 Quality culture overall Top 2 boxes

Sample size 4 5

Figure32indicatesthatthereissomeevidencethatqualityculturevaluesarerelatedtodeviationsrecurrencerate.Whenthesampleissplitintotwoparts,thosesiteshavingaqualityculturevaluesgreaterthan0.8arefoundtoalsohavealowerrecurringdeviations rate (8.3 versus 20.0)


5

Figure 33: CAPA Effectiveness Rate and Quality Culture ValuesCAPAeffectivenessinselectedintervalsofqualitycultureSampleof74averagequarterlyvalues,alltechnologies

Median level of CAPA effectiveness for sites with higher and lower culture score

ppm

0.89

>0.8

0.95

<0.8

Sample size 19 18

Significance: 0.06

Quality culture overall Top 2 boxes

+7%

UsingthesamequalityculturevalueasgiveninFigure32of0.8,theremayalsobeadifference(7%observed)inCAPAeffectivenessrate.Thesignificancelevelforthisrelationshipiscloseto,butabove,0.05%(0.06%)andthisisshowninFigure33.

Note:Thequalityculturevalueof0.8alsosplitsthesampleintotwoalmostequalparts.

5.10 Comparisons Where Metrics Are Not Differentiated or Are Inconclusive

Somemetricswerenotdifferentiatedorhadnoconclusiveresultsandtheseincluded:

f APQRontime,whichisalsonothighlydifferentiating,reportedas100%bythemajority of sites.

f Forstabilityfailures,confirmedOOS(productreleaseandincomingmaterials)andUnconfirmedOOS,thepilotsampledidnotyieldclearconclusions.

f Thetechnology-specificmetricswerenotsufficientlytestedduetosmallersamplesize,andsomeofthem(mediafills,environmentalrejects)werenotdifferentiatedbetweensites.


5

5.11 Discussion of Relationships

Combinedfindingsonmetrics,qualityculturesurveyvaluesintermsofrelationships,timespenttocollectmetrics,difficultyofcollectionanddifficultyofdefinitionaresummarizedbelowinTable4.

Table 4: Overview of Findings from PilotSite level data

Value Time Difficulty1Definitions discussion2

US recalls Relationship Low 1

Total complaints rate Inconclusive yet High 2

Critical complaints rate Relationship High 2 Yes

Lot acceptance rate Relationship Moderate 2

Reworkrate Inconclusive yet Moderate 2 Yes

ConfirmedOOSrate Inconclusive yet High 2

Stability failure rate Inconclusive yet Moderate 2

Deviations rate Relationship Moderate 2

Invalidated OOS rate Inconclusive yet Moderate 2

APQRontime Inconclusive yet Low 1

Recurringdeviationsrate Relationship Moderate 3 Yes

CAPAeffectivenessrate Inconclusive yet Moderate 2 Yes

Successfulmediafills Notdifferentiating Low 2

Action limits excursions Inconclusive yet Moderate 3

Environmental rejects Notdifferentiating Moderate 3

Culture Relationship

1 1=veryeasyto4=verydifficult;-siteratingsrelatedtowhetherthedataareavailableintherequestedform,orrequiresrecalculation/aggregation,orcollectionfromfragmentedsources;

2 Metricswheredefinitionsvarysignificantlyacrosscompanies,andfeedbackwasprovidedthroughpilot


5

FurtherdiscussionofeachcolumninTable4isgivenbelow.

5.11.1 Column 1: Value

AsillustratedinFigure22thefollowingmetrics/valueshavestatisticallysignificantrelationships (as discussed in Section 5.8)directlyorindirectlywithanexternalqualityoutcome(USrecallsorcriticalcomplaints):

f US recalls (outcome) f Critical complaints (outcome) f Lot acceptance rate f Deviations rate f Recurringdeviationsrate f Quality culture

Incolumn1,theabovemetrics/valuesareshownwithgreyshading.

Critical complaints is related to US recalls and may be a relevant measure of external quality,however,therewasmuchfeedbackfromsiteleadersfromparticipantcompaniesthatthedefinitionofcriticalcomplaintsledtodifficultyininterpretationofvaluestosubmitandasnotedearliertherewasaonlyaverysmallnumberofrecallsfromthepilot study.

Lotacceptancerate,deviationsrate,deviationsrecurrencerateandqualityculturevaluesallhaverelationships(somestatisticallysignificant)toqualityoutcomes(critical complaints and US recalls).

As stated at the beginning of this section, it cannot be stressed too strongly that the statistically significant relationships observed in the Wave 1 Pilot sample do not indicate cause.Furthertestingandstudiesarerequiredtobetterunderstandtheserelationships.

SomemetricshadinconclusivecomparisonswithothermetricsandfurthertestingisrequiredtoindicatethepresenceorabsenceofrelationshipsasshowninTable4.

Somemetriccomparisonsfortechnology-specificmetrics(successfulmediafillsandenvironmentalrejects)werenotdifferentiated;thesearehighlightedwithorangeshading in column 1.


5

5.11.2 Column 2: Time

Timespentcollectingeachindividualmetricistabulatedincolumn2.Metricswithlowestamountoftimeareshadedgreyandthosewiththehighesttimeareshadedorange.Somepossiblereasonsfordifferencesare:

f Complaintsareusuallyreceivedcentrally,andeffortisrequiredtounderstandthe complaint and distribute to a particular site in a supply chain, and then for that site to categorize the complaint and submit a value. Also some complaints mayneedtobeforwardedtoanumberofdifferentsites(e.g.,toapackagingsite if the complaint is related to the packaging operations).

f AconfirmedOOSmetricmaybehardtocollectonasitebasissinceitisusuallycollectedonaproductbasisandeffortisrequiredtoaggregatetoasitelevel.

f APQRsontimemetricsareprobablyeitheralreadycollectedoreasytocollect.

f USrecallsarealownumberandofsuchhighimportancethatthemetriciseasyto collect.

f Forsterileproductmanufacturingsites,thesuccessfulmediafillsmetricisarelativelylownumberandeasytocollect.

EstimatesoftimetocollectQualityCultureSurveydatawerenotcollectedintheoveralleffortestimates,sinceeventheapproximately5minutesrequiredforeachindividualresponsewouldstillpresentalargeeffortgiventheoverallnumber(10,300)of survey respondents.

5.11.3 Column 3: Difficulty

Theserankingsaretheparticipatingsites’estimatesofthedifficultyofcollectingametricasshowninFigure16.

Mostdifficultwastherecurringdeviationsrate,sincethismetricdoesnotappeartobecollectedroutinelybymostsites.AsshowninFigure17,thereisnotnecessarilyacorrelationbetweendegreeofdifficultyofcollectingametricandmedian time for collection.

5.11.4 Column 4: Definitions

Inthelastcolumn,metricsaregivenwheretheMcKinseysupportteamspentmosttimeexplainingametricdefinitionandwhere,fromfeedback,definitionsvariedmostbetweencompaniesandbetweenacompanydefinitionandthedefinitionusedinthepilot.Themetricsmostdifficulttodefinewere:

f Critical complaints rate f Reworkrate f Recurringdeviationsrate f CAPAeffectivenessrate


5

5.12 Complaints Analysis

Thedatabasewasexaminedtoevaluaterelationshipsbetweencriticalcomplaints,lotacceptancerateandtypeofproductandtechnology.Datawereplottedinascatter plot as given in Figure 34.

Figure 34: Scatter Plot of Critical Complaints, Lot Acceptance Rate, Technology and Type of ProductProductleveldata,retrospectiveperiod(12m.),finisheddosageplants

0

200

400

600

800

1,000

100 99 98 97 96 95 94 93 92 91 90

Critical complaints rate1 ppm

Lot acceptance rate 2

% lots dispositioned

SOURCE: ISPE Quality Metrics pilot, January 2015

Size of bubble indicate volume of product in packs

High rejects but few critical complaints. This “conservative” segment represents: 16% of overall volume 41% of Gx volume 56% of steriles volume

High critical complaints, high rejects. This “issues” segment represents: 36% of overall volume 53% of OTC volume 63% of liquids and creams volume

High critical complaints, low rejects. This “misses” segment represents: 15% of overall volume (with roughly proportionate share within each technology or type of product)

Few critical complaints, low rejects. This “healthy” segment represents: 33% of overall volume 48% of Rx volume 63% of solids volume

1 Productswithover1000criticalcomplaintspermilionpackshavebeenomittedonthechart2 Productswithlessthan90%lotacceptancehavebeenomittedonthechart

The red lines represent the 80th percentile boundaries for lot acceptance rate (99.24,lowertotheleft)andcriticalcomplaints(0.12,lowerbelowline).

Thisshowsthatproductsvaryintheirrejectsandcomplaintsprofiledependingonboththetechnologyandthetypeofproduct.Itisthereforehardtodrawanyspecificconclusions at this time from this analysis and available data set.


5

5.13 Analysis of Product-Based Metrics

Product-basedmetricswereanalyzedforthecomplaintsandlotacceptanceratemetrics.Theanalysisindicatedthat“tail”products(90%ofproductsrepresenting30%ofvolume)representadifferentlevelofcriticalcomplaintsandrejectsdependingontechnologyasshowninFigure35andFigure36.

Figure 35: Critical Complaints and Volumes for Product-Based DataProduct level data1

7058

3042

Critical complaints

Volume (packs)

72

28

28

72

Volume (packs)

Critical complaints

70

47

30

53

100%

Volume (packs)

Critical complaints

Solids Steriles Other

High-volume products XX # products in group

100

7 25

157 458

63

1 Excludesinactiveproducts(withnopacksreleasedinreportingperiod)

Themultipletailproducts,representing30%ofvolume,resultin42%ofcriticalcomplaintsforsolids,70%ofcriticalcomplaintsforsterileproductsand53%of critical complaints for other technologies.


5

Figure 36: Rejects (from Lot Acceptance Rate) and Volumes for Product-Based DataProduct level data

High-volume products xx # products in group

Solids API and biologics OtherSteriles

30

70

34

66

30

70

75

25

29

71

69

31

30

70

37

63

Volume(lots disp.)

Lotsrejected

Volume(lots disp.)

Lotsrejected

Volume(lots disp.)

Lotsrejected

Volume(lots disp.)

Lotsrejected

558

102

138

92

18

98

14

52

100%

Forrejectsobservedaspartoflotacceptancerate,30%ofthevolumegave34%rejectsforsolids,70–75%ofrejectsforbothsterileproductanddrugsubstancesitesand37%rejectsforothertechnologies.


5

5.14 McKinsey Analytical Effort and Observations

AnalysisbyMcKinseyregardingtheamountofeffortthattheirsupportpersonnelhadperformedwasestimatedas:

f Eachsiterequiredonaverage22hoursofdedicatedsupport—explainingthedatarequirements,answeringquestions,reviewingandverifyingthesubmitteddatathroughdiscussionswiththesite.

f The more structured and supported the data submission process is, the more accurate the received data.

f Checkingsubmitteddataforcoherence/accuracywasrequiredevenafterseveral submission cycles.

f Preparation of submission templates, databases and process, and analyzing thegathereddatarequiredapproximately400additionalhours

TheabovemayindicatethelevelofsupportthatFDAwillberequestedtoprovide,at least for early rounds of data collection.

DuringthecourseofthePilotthereweremanyinteractionsbetweenparticipatingcompaniesandMcKinseypersonnelandthefollowingsuccessfactorswereobserved:

f Good definitions: To make data submission as easy as possible and minimize theneedforfollow-updiscussion,metricsmustbedefinedtoaveryhighstandard.Industry-consensusdefinitionsalmostalwaysdifferfromthosebeingusedbycompanies;henceitisnecessarytoallowcompaniestoadjust.

f Strong support:Experienced,dedicatedsupportwasrequiredthroughoutthecollectionperiodtoanswerquestionsquicklyandmakenecessaryclarifications.

f Engaged and committed participants: Given that participating in the pilot wasvoluntary,itwasclearthatparticipantcompanystaffwerecommittedandknowledgeable;mostcompaniesalsohadmaturesystems.UsingMcKinseypersonnel and teleconferences of participant company site leaders permitted muchsharingandlearningbetweencompanies.

f Built-in checks: Submitteddatarequiredcarefulcheckingtoensureconsistency and accuracy.

f Feedback from participating companies:McKinseycollectednonconfidentialandcross-companycommentsaboutpilotdesignandoperation.DefinitiondifferencesbetweenISPEandparticipatingcompaniesfor“recurringdeviationsrate” generated considerable feedback, for example.

Thistypeoffeedbackreinforcestheneedtohaveprecise,agreeddefinitions.Thereisthepossibilitythatevenwithprecisedefinitions,differencesinproductandprocessflow,couldleadtoundesirablevariationincompanies’submissions.


6 Output and Lessons Learned from ISPE Quality Metric Wave 1 Pilot

ThissectiondiscussessuccessfactorsfortheWave1Pilotandanoverviewoffindingssuchasasummaryofeffortrequiredandcomparisonofsite-andproduct-based data.

TheISPEQualityMetricsProjectTeamhaveproducedthefollowinglessonslearnedandoutputsfromtheWave1PilotbasedonfindingsanddatageneratedduringthePilot,commentsmadebyparticipantsaswellascommentsbyMcKinseypersonnelcomparingtheirexperienceswiththisISPEPilotwithotherbenchmarkingexercises.

6.1 Success Factors

Thefollowingarelistedassuccessfactorsforthepilot:

f Metrics lists prealigned during industry meetings. f Precisedefinitions,developedwithinputfrommultiplecompanies. f Engagedandcommittedparticipantswithmaturesystems. f Acceptabletosubmit“goodenough”data. f Strongcollaborationacrosscompaniesandbetweenexpertstosupportaprocessthatallowslearningandcontinuouslyimprovingdataaccuracy.

f Strong McKinsey Support, providing: – Structureddatasubmissionwithdetailedguidanceonhowtoreportthedata. – Ability to comment on data points to enable interpretation. – Experienced,dedicatedsupportforquestionsandclarificationsduringandthroughoutthedatacollectioneffort.

– Built-inchecksandjointreviewbetweenMcKinseyandtheparticipatingcompany to ensure consistency and accuracy of data.

– Transparencythroughouttheprocess,engagingparticipantsinfrequentdebriefs and discussions.

Thesefactorssupportstartingwithaqualitymetricsprogramcontainingarelativelysmallnumberofwell-definedmetricswithalearningperiodtorefinescope,definitionsandprocess.


6

6.2 Definitions

Definitionsareextremelyimportant:

f Definitionsmustbeexact:Denominatorsinparticulararehighlysensitivetoissuesaroundlotaggregationandfinaldisposition.

f Evencommontermslike“lot,”“deviations,”“complaints”and“reviews”mustbespecifiedingreatdetailtominimizemultipleinterpretations.

f Evenwithdetaileddefinitions,supportandansweringquestionsthroughoutthe process is necessary to ensure more accurate data submission.

f Standardizeddefinitionswilldifferfromcurrentcompanydefinitions,thusrequiringadditionalwork.

f Productandprocessdifferenceswillgeneratedifferencesandvariationsin metric ranges.

f Commentary on data points is essential to interpretation and analysis. f Somevariationmustbeexpectedduetodifferencesinproduct,processflows

and product/process complexity.

Thepilotshowedthatstandardizingmetricsdefinitionsacrosscompaniesisfeasible.

6.3 Metrics Collection by Site and by Product

Attemptsweremadetogenerateproduct-leveldata,however,therearechallenges:

f Thepilotcollectedmetricsonlywithinagivensite,notacrossmultiplesitesor across a full supply chain.

f Acompany’sabilitytocollectdatabysiteand/orbyproductdiffersbymetricandhowtheirsystemsaresetup.

f Most companies collect data at site level and may have to manually disaggregate data in order to report at a product level.

f Some metrics data collected at sites cannot be easily allocated to a product (e.g., deviation rate).

f Some companies report some metrics (e.g., OOS) at a product level and have to aggregate data to report at a site level.

f SomedatacollectedatproductlevelmaybederivedfromAPRs. f Dependingonthesizeandcomplexityofthesite,APRpreparationmaybespreadbyproductondifferentcycletimesduringtheyear.

f Complaints data are generally gathered centrally (corporate level) and distributed to sites for investigation.

Insummary,fewcompaniesaggregatemetricsacrossthesupplychaintobeableto report at product/application level.


6

6.4 Industry Effort

Industryeffortissummarizedas:

f On average the participating sites in the Wave 1 Pilot spent ~90 hours in pure datacollectiontime.For12,000siteswithFEInumber(6,000API/FinishedDosageand6,000Labs/DrugSubstance/‘other’)attypicalqualitylaborcost,collectingthisamountofdatawouldcost the industry an additional ~$35 million annually. – This conservative estimate does not include several factors that could

bring the cost of such a program to $100+ million, such as: f The 90 hours observed might be underestimated for sites that did not report

all products at that site. f Workloadestimatesfor“goodenough”datasubmissionvs.anofficialsubmission. f Timeforinternaldiscussions,managementreviewandabove-siteguidance

not included. f Theneedfornew/modifiedITsystemswasnotincludedinWave1. f Participantshadflexibilitytoprovidethemostpragmaticdataset

(e.g., for all products at site or only those for the US market). f Datawasprovidedwithineachsiteandnot through the full product supply chain. f Most participants had mature systems and capabilities regarding

performance measurement. f Majorityofsiteswerefromdevelopedcountries.

Theprogramscopeanddesigncaninfluencetheseindustrycostssignificantly.

6.5 McKinsey Analytical Effort

TherewassubstantialeffortfromMcKinsey:

f Eachsiterequiredonaverage22hoursofdedicatedsupport—explainingthedatarequirements,answeringquestions,reviewingandverifyingthesubmitteddatathroughdiscussionswiththesite.

f The more structured and supported the data submission process is, the more accurate the received data.

f Checkingincomingdataforcoherence/accuracywasrequiredevenafterseveraldata submission cycles.

f Preparation of submission templates, databases and process, and the analysis ofthegathereddatarequiredapproximately400additionalanalyticalhours.

As already mentioned in Section 5.14, the above points may indicate the level of support thatFDAwillberequestedtoprovide,atleastforearlyroundsofdatacollection.


7 Proposals

BasedonfindingsfromtheWave1Pilot,intermsofeaseandeffortofcollectionandsubmissionofmetrics,itissuggestedthatfuturequalitymetricsprogramsstartwitharelativelysmallnumberofstandardizedmetrics.

FromtherelationshipfindingsdiscussedinSection 5a“startingset”offivemetricsareproposedforinclusioninfutureindustryqualitymetricsprograms:

1. Lot acceptance rate (normalized by lots dispositioned) at site level.2. Lotacceptancerate(normalizedbylotsdispositioned)atproductlevelwithin

a site.3. Critical complaints (normalized by number of packs released) at product level

byapplication,notbrokendownbysite.4. Criticalcomplaints(normalizedbypacksreleased)atsitelevel,undifferentiated

by product.5. Deviations rate at site level.

RationaleforincludingthesemetricsinaWave2Pilotaregivenbelow.

7.1 Rationale for Metrics Proposed as a Starting Set for Wave 2 Pilot

f InmostcasesthemetricwasalreadycapturedintheWave1Pilot,andcontinuedmonitoring is desired over a longer timeframe and broader set of companies, technologies, regions.

f Themetricselecteddemonstratedastatisticallysignificantrelationshiptooneofthefollowing: – Deviations recurrence – Quality culture values – Critical complaints – Lot acceptance rate

f It has been demonstrated to be relatively easy to collect and submit. f Itwasdeemedanimportantmetricfordeterminingsitequalityperformance. f Itwillassistthecompanytoidentifycontinualimprovementopportunities. f While the critical complaints metric (normalized by number of packs released atproductlevelbyapplication,notbrokendownbysite)wasnot included in the Wave 1 Pilot, it is thought to have merit and should be explored by product application.


7

ThestartingsetofmetricsarenowbeingconsideredforfurtheranalysisinanISPEQualityMetricsWave2Pilotthatwillincludeanextendedtimeperiodofprospectivedata collection and increased sample size of participating companies. The momentum andknowledgegainedfromtheWave1PilotwillbeleveragedtofacilitateanearlytransitiontoWave2.AttheQualityMetricsSummitinBaltimore,April2015,notificationwasgivenforbothnewandcontinuedparticipationinaWave2PilotwithatargetenrollmentcommencementdateofJune2015.FurtherdesignoftheWave2Pilotwillthengetunderway.

Inadditiontothestartingsetoffivemetricslistedabove,othermetricsofinterestunder consideration for a Wave 2 Pilot include:

f Deviations recurrence rate: Howtoachieveaconsistentandacceptedindustrydefinitionandpractice.

f Unconfirmed OOS: Test on a bigger sample its relation to culture, particularly useful for laboratories, and assess against outcomes.

f Quality culture: Howbesttoassesstheinfluenceofqualitycultureonqualityperformance outcomes at industry scale.

Briefingsessionswithexistingparticipatingcompaniesareunderwaytodiscusstheimplications for ongoing participation. Many other companies in attendance at the QualityMetricsSummitwhoexpressedawillingnesstogetinvolvedinfuturepilotstudieswillbecontactedintheneartermregardingpotentialenrollment.

TheWave2Pilotwillgiveparticipatingcompaniesadeeperunderstandingofmetricsdefinitions,theopportunitytocontributetothefinalpilotstudydesign,andthechanceto experience the challenges of a centralized metrics submissions process. In addition, participantswillhaveaccesstoanindustrybenchmarkingreportthatwillallowthemtoexaminetheirprogresswithrespecttotheirpeers.Finally,participationalsoprovidesopportunities to enhance the maturity of their internal metrics programs.

Those interested in receiving further details should contact the ISPE Quality Metrics Initiative at [email protected].

mailto:PQLI%40ispe.org?subject=


8 Conclusions

TheobjectivesoftheWave1Pilotwereachievedandconsequentlythepilotisconsidered a success.

Evidenceofseveralstatisticallysignificantandinterestingrelationshipswasfoundbetweencompanyqualityperformance,prevailingqualitycultureandkeypatient-relatedqualityoutcomes.TheinsightsandknowledgegainedfromthisWave1Pilotconfirmthebenefitsofutilizingqualitymetricstomonitor,createtransparencyanddrive enhancement in pharmaceutical manufacturing.

Industryengagementthroughoutthisprocesshasconfirmedthatmanysitesandcompanies, including those participating in the Wave 1 Pilot, are already using metrics to drive and monitor internal continual improvement programs. This pilot wasanattempttodevelop,collectandanalyzeastandardizedsetofmetricsandhas indicated that this is achievable. While there are inherent challenges and costs associatedwiththisexercise,thepilotwasabletoidentifyseveralkeybenefitsalso.The pilot therefore accomplished its primary objectives and has set the stage for additionalworkinWave2.

TheserecommendationswerediscussedwithindustryrepresentativesandFDAattheISPEQualityMetricsSummitandbroadsupportwasgiventotheoutcomesandfutureplans.Asignificantopportunityfortrustbuildingbetweentheindustryand the agency has also been realized though this Wave 1 Pilot.

CommunicationswillcontinuewithallpartiesastheISPEQualityMetricsInitiativefurther develops its plans for the Wave 2 Pilot.

Ten years ago Dr. Janet Woodcock outlined the ultimate goal of the desired state [18] for pharmaceutical manufacturing. This Quality Metrics Initiative Pilot has provided earlysupportofthisjourneytowardpharmaceuticalexcellence.


9 References

1. USDepartmentofHealthandHumanServices:FoodandDrugAdministration,etal.“GuidanceforIndustry:QualitySystemsApproachtoPharmaceuticalCGMPRegulations.”September2006. http://www.fda.gov/downloads/Drugs/.../Guidances/UCM070337.pdf

2. .“PharmaceuticalGMPsforthe21stCentury—ARisk-BasedApproach.”FinalReport.September2004. http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/Manufacturing/ QuestionsandAnswersonCurrentGoodManufacturingPracticescGMPforDrugs/UCM176374.pdf

3. .“GuidanceforIndustry:PAT—AFrameworkforInnovativePharmaceuticalDevelopment, Manufacturing, and Quality Assurance.” September 2004. http://www.fda.gov/downloads/Drugs/Guidances/ucm070305.pdf

4. .“GuidanceforIndustry:ProcessValidation:GeneralPrinciplesandPractices.”Revision1.January2011. http://www.fda.gov/downloads/Drugs/Guidances/UCM070336.pdf

5. InternationalConferenceonHarmonisationofTechnicalRequirementsforRegistrationofPharmaceuticalsforHumanUse.ICHHarmonizedTripartiteGuideline.“PharmaceuticalDevelopment:Q8(R2).”Step4version,August2009. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q8_R1/Step4/Q8_R2_Guideline.pdf

6. .QualityRiskManagement:Q9.”Step4version,9November2005. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q9/Step4/Q9_Guideline.pdf

7. .“PharmaceuticalQualitySystem:Q10.”Step4version,4June2008. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q10/Step4/Q10_Guideline.pdf

8. .“DevelopmentandManufactureofDrugSubstances(ChemicalEntitiesand Biotechnological/Biological Entities): Q11.” Step 4 version, 1 May 2012. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q11/Q11_Step_4.pdf

9. GovernmentPrintingOffice.FoodandDrugAdministrationSafetyandInnovation Act (FDASIA). Publication L. 112–144. July 9, 2012. http://www.gpo.gov/fdsys/pkg/PLAW-112publ144/pdf/PLAW-112publ144.pdf

10. BrookingsInstitution.“MeasuringPharmaceuticalQualitythroughManufacturingMetricsandRisked-BasedAssessment,”ExpertWorkshop,1–2May2014 http://www.brookings.edu/events/2014/05/01-measuring-pharmaceutical-quality

11. McKinsey&Company.“PharmaOperationsBenchmarkingofSolids(POBOS).” http://solutions.mckinsey.com/pobos/

12. ISPE.“ISPEProposalsforFDAQualityMetricsProgram—Whitepaper.”20 December 2013. www.ISPE.org

http://www.fda.gov/downloads/Drugs/.../Guidances/UCM070337.pdf

http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/Manufacturing/QuestionsandAnswersonCurrentGoodManufacturingPracticescGMPforDrugs/UCM176374.pdf



http://www.fda.gov/downloads/Drugs/Guidances/ucm070305.pdf

http://www.fda.gov/downloads/Drugs/Guidances/UCM070336.pdf

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q8_R1/Step4/Q8_R2_Guideline.pdf

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q8_R1/Step4/Q8_R2_Guideline.pdf

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q9/Step4/Q9_Guideline.pdf




http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q11/Q11_Step_4.pdf

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q11/Q11_Step_4.pdf

http://www.gpo.gov/fdsys/pkg/PLAW-112publ144/pdf/PLAW-112publ144.pdf

http://www.brookings.edu/events/2014/05/01-measuring-pharmaceutical-quality%20%20

http://solutions.mckinsey.com/pobos/

http://www.ISPE.org


9

13. USDepartmentofHealthandHumanServices:FoodandDrugAdministration.“GuidanceforIndustry:Self-IdentificationofGenericDrugFacilities,Sites,and Organizations.” August 2012 http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm316721.htm

14. ISPE.“ISPEDrugShortagesPreventionPlan:AHolisticViewfromRootCauseto Prevention.” October 2014. www.ISPE.org/DrugShortagesPreventionPlan

15. USDepartmentofHealthandHumanServices:FoodandDrugAdministration.“FoodandDrugAdministrationDrugShortagesTaskForceandStrategicPlan;RequestforComments.”Federal Register, Docket No. FDA-2013-N-0124. 12 February 2013. http://www.gpo.gov/fdsys/pkg/FR-2013-02-12/html/2013-03198.htm

16. Woodcock,Janet.“TheQualityRevolution:TheFutureofPharmaceuticalManufacturingandItsRegulation.“PresentedattheISPEQualityMetricsSummit, Baltimore, MD, 22 April 2015.

17. http://rx-360.org/Portals/1/Guidance/FDAAnnualInspectionReportonEstablishments/2015AnnualReportonInspofEstablishments inFY2014Final.pdf

18. Woodcock,J.“TheConceptofPharmaceuticalQuality.”American Pharmaceutical Review47(6):1–3,2004.

http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm316721.htm

http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm316721.htm

http://www.ISPE.org/DrugShortagesPreventionPlan

http://www.gpo.gov/fdsys/pkg/FR-2013-02-12/html/2013-03198.htm

http://rx-360.org/Portals/1/Guidance/FDA%20Annual%20Inspection%20Report%20on%20Establishments/2015%20Annual%20Report%20on%20Insp%20of%20Establishments%20in%20FY%202014%20Final.pdf




Appendix 1Appendix 1Definitions of Quantitative Metrics Used in Pilot

Lot acceptance rate

Metric Definition

Lot acceptance rate = Total lots released for shipping out ofthetotalfinallydispositionedlotsfor commercial use in the period

f Total lots dispositioned = total number of lots for commercial use produced and/orpackagedonsitethatwentthroughfinaldispositionduringtheperiod,i.e.werereleasedforshippingorrejected(fordestruction).Rejectionsshouldbecountedasfinaldispositionregardlessatwhatproductionstagetherejectionoccurred.Releaseisonlyfinalreleaseforshipping.Excludeslotsthathavebeensentforreworkorputonhold/quarantinedinthisperiodandhencearenotfinallydispositioned.Excludeslotsthatarenotproducedor packaged on site, but just released for CMOs.

f Totallotsrejected=totalfulllotswererejectedforqualityreasons.Rejectedmeansintendedfordestructionorexperimentaluse,notforreworkorcommercialuse.Rejectionsshouldbecountedregardlessatwhatproductionstage the rejection occurred.

f Total lots released (accepted) = total lots dispositioned less total lots rejected.

Complaints Rate (Total and Critical)

Metric Definition

Total complaints rate = Total complaints received in the reporting period, related to the qualityofproductsmanufacturedinthe site, normalized by the number of packs released

f Packsreleased=Totalnumberofpacks(finalproductformthatleavestheplant, one level less than tertiary packs, most usually it is secondary packaging unit e.g. pack of blisters or bottle in carton pack) released in the period.

f Total complaints = All complaints received in the reporting period, related tothequalityofproductsmanufacturedinthesite,regardlesswhethersubsequentlyconfirmedornot.Allcomplaintsreceivedbythesiteshouldbecounted,evenifacomplaintaffectsmorethan1site,orifeventuallytherootcause analysis attributes the issue to another site. Complaints related to lack ofeffectshouldbecountedaswell.

Critical complaints rate = All critical complaints, normalized by the number of packs released

f Criticalcomplaints=Post-distributionproductqualitycomplaintswhichmayindicateapotentialfailuretomeetproductspecifications,mayimpactproduct safety and could lead to regulatory actions, up to and including product recalls. Critical complaints include those that potentially could lead toFDAnotification(e.g.,FieldAlertReports,BiologicalProductDeviationReports).Critical(orexpedited)complaintsareidentifieduponintake,whethersubsequentlyconfirmedornot,basedonthedescriptionprovidedby the complainant, and include, but may not be limited to: – i. Information concerning any incident that causes the drug product

or its labelling to be mistaken for, or applied to, another article. – ii.Informationconcerninganybacteriologicalcontamination,oranysignificant

chemical, physical, or other change or deterioration in the distributed drug product, or any failure of one or more distributed batches of the drug producttomeetthespecificationestablishedforitintheapplication.


Appendix 1

OOS Rate, Stability Failure, Invalidated (Unconfirmed) OOS Rate

Metric Definition

ConfirmedOOSrate= TotalconfirmedOOS(testresultsthatfalloutsidethespecificationsor acceptance criteria), out of all lots dispositioned by the lab during the period

f Total lots tested/dispositioned by the lab = total number of lots used for commercial production that are tested and dispositioned out of the lab in the period, i.e., have a QC pass or fail decision on them. Includes: – Lots for release testing (counted as 1 lot, even if sampled separately for

chemical and microbiological testing, or for in-process analytical testing inlaboronshopfloor).

– Lots of incoming materials for analytical testing (count 1 per each analyticallytestedrawmaterialand/orpackagingmateriallot).Includeswaterusedasrawmaterial.

– Lots for stability testing in that period (counted as 1 per each timepoint and condition sampled per the approved stability protocol).

– Does not include environmental monitoring samples. f ConfirmedOOS=alltestresultsthatfalloutsidethespecificationsoracceptancecriteriaestablishedindrugapplications,drugmasterfiles(DMFs),officialcompendia,formularyorappliedbythemanufacturerwhenthereisnotan‘official’monograph.

Stability failure rate = TotalconfirmedOOSrelatedto stability testing

f SubsetoftheConfirmedOOSrate–basedonstabilitylotstestedandconfirmedOOSrelatedtostabilityonly.

Invalidated(unconfirmed)OOSrate=TotalunconfirmedOOS,outofalllots tested during the period

f UnconfirmedOOS=allOOSminusconfirmedOOS(seethedefinitionofconfirmedOOS).

US recall events (Total and by Class)

Metric Definition

Recallrate f Recallevents=allUSmarketrecallevents. f By class = all US market recall events, class I and II. f Recalledlots=Includelotsrecalledeithervoluntarilyorbyregulatoryorder(recallimpliesphysicalremovalofproductfromfield,notjustafieldactionor correction). Include US market recalls only.


Appendix 1

Right First Time (Rework/Reprocessing)

Metric Definition

RFT(rework/reprocessingrate)=Total lots released that have not beenreworkedorreprocessedoutofthetotalfinallyreleasedlotsforcommercial use in the period

f Total lots released (accepted) = total lots dispositioned less total lots rejected (seethedefinitionofLotacceptancerate).

f Totallotsreworkedorreprocessed=alllotsthathavegonethroughrework(using alternative process) or reprocessing (using again the original process) beforethatfinaldispositioninordertomeetrequirementsforrelease.Onlycountreworkorreprocessingnecessitatedbyqualityissues(forexamplecontractmanufacturingsitesshouldexcludereworkduetocustomerorderchanges).Ifalotwassentforreworkandreceivedanewlotnumber,itshouldstillbecountedasundergonereworkwhenfinallydispositioned.

APQRs on Time

Metric Definition

APQRcompletedontime=Number of Annual Product Quality Reviewsintheperiodthatwerecompleted by the original due date, normalized by all products subject toAPQR

f ProductssubjecttoAPQR=TotalnumberofproductssubjecttoAnnualProductQualityReviews-annualevaluationsofthequalitystandardsofeach drug product to verify the consistency of the process and to highlight any trends in order to determine the need for changes in drug product specificationsormanufacturingorcontrolprocedures(asrequiredbyCFRSec.211.180,Generalrequirements,section(e)andICHQ7,GMPsforAPIs,section 2.5 or EU Guidelines for Good Manufacturing Practice for Medicinal ProductsforHumanandVeterinaryUse,Chapter1,PharmaceuticalQualitySystem, section 1.10). Does not include the data packages that a site preparestoitscustomerswhenactingasaCMO.

f NumberofAnnualProductQualityReviewsontime=completedbytheoriginal due date.


Appendix 1

Recurring Deviations Rate

Metric Definition

Recurringdeviationsrate= Number of deviations that have re-occurred during the preceding 12 month period out of all closed deviations

f Number of deviations = Any major or minor unplanned occurrence, problem, or undesirable incident or event representing a departure from approved processes or procedures, also includes OOS in manufacturing or laboratory or both. Please count only deviations that have been closed/resolved in the period.Deviationsfromoneperiod,forwhichtheinvestigationwasclosedin the next period, should be counted in the latter period.

f Recurringdeviations=Numberofdeviationsforwhichduringthe12monthperiod preceding each deviation, at least one other deviation has occurred withthesamerootcausewithinthesameprocessand/orworkarea.Ifredundant/duplicativeprocessesorequipmentexist,pleaseconsiderdeviationeventscommontothegrouping/workcenterasrecurring(stillwithinthe12 month timeframe). For example, if a deviation for missing desiccant occurstwice,ontwoseparatepackaginglineswithcomparableequipment/systems, it should be counted as recurring (i.e. as 2 same deviations, rather than1differentforeachline).

CAPA Effectiveness Rate

Metric Definition

CAPAeffectivenessrate= NumberofCAPAseffectiveoutofallCAPAswitheffectivenesscheckin the reporting period

f CAPAswitheffectivenesscheck=NumberofCAPAsevaluatedforeffectivenessinthereportingperiod.AllCAPAsshouldbecounted,includingthose related to inspection or audit observations.

f CAPAseffective=thoseevaluatedCAPAswherethequalityissuesubjectoftheCAPAwasresolved,and/orhasnotreoccurred,andtherehavebeenno unintended outcomes from the CAPA implementation.

Media Fill Failures (sterile/aseptic only)

Metric Definition

Mediafillrate= Numberofmediafillsdispositionedassuccessfuloutofallmediafillsto support commercial products dispositioned during the period

f Mediafills=Totalnumberofmediafills(regardlessofnumberofrunsineach)tosupportcommercialproductsthatweredispositioned(assuccessfulorfailed)duringtheperiod.Ifthemediafillwasdispositionedasfailureandarerunwasneeded,thatrepeatiscountedasaseparatemediafill.Includesallmediafills-bothforinitialandperiodicqualifications.

f Successfulmediafills=Allmediafillsthatwerenotdispositionedasfailures.


Appendix 1

Environmental Monitoring (sterile/aseptic only)

Metric Definition

Environmental monitoring

f Lotswithactionlimitexcursions, normalized by all sterile dispositioned lots

f Lots rejected due to environmental monitoring reasons, normalized by all sterile dispositioned lots

f Steriledispositionedlotsduringtheperiod(seedefinitionforLotacceptance rate).

f Lotswithlimitexcursions=Allsteriledispositionedlotsduringtheperiodthat had associated investigations related to exceeding environmental monitoring action limits. If a lot had more than 1 such investigation please countonly1perlot.Ifaninvestigationhasaffectedmultiplelots,pleasecount each lot separately. Action limit is an established microbial or airborne particlelevelthat,whenexceeded,shouldtriggerappropriateinvestigationand corrective action based on the investigation.

f Rejectedlotsduetoenvironmentalmonitoringreasons=Allsteriledispositionedlotsduringtheperiodthatwererejectedforexceedingenvironmentalmonitoringactionlimits.Rejectedmeansintendedfordestructionorexperimentaluse,notforreworkorcommercialuse.Rejectionsshouldbecountedregardlessatwhatproductionstagetherejection occurred.

Deviations Rate

Metric Definition

Deviations rate = Number of deviations closed during theperiodoutofthetotalfinallydispositioned lots for commercial use in the period

f Number of deviations = Any major or minor unplanned occurrence, problem, or undesirable incident or event representing a departure from approved processes or procedures, also includes OOS in manufacturing or laboratory or both. Count only deviations that have been closed/ resolved in the period. Deviationsfromoneperiod,forwhichtheinvestigationwasclosedinthenext period, should be counted in the latter period.

f Deviations rate = Number of deviations closed during the period out of the totalfinallydispositionedlotsforcommercialuseintheperiod.

f Total lots dispositioned = total number of lots for commercial use produced and/orpackagedonsitethatwentthroughfinaldispositionduringtheperiod,i.e.werereleasedforshippingorrejected(fordestruction).Rejectionsshouldbecountedasfinaldispositionregardlessatwhatproductionstagetherejectionoccurred.Releaseisonlyfinalreleaseforshipping.Excludeslotsthathavebeensentforreworkorputonhold/quarantinedinthisperiodandhencearenotfinallydispositioned.Excludeslotsthatarenotproducedor packaged on site, but just released for CMOs.

Report from ISPE Quality Metrics Pilot Program / June 1 – April 2015 77

Appendix 2 Survey Questions

Quality Culture Questions

Strongly agree Agree Disagree

Strongly disagree

I can’t answer this question

Cap

abili

ties

Patient focus:Iknowwhichparametersofourproducts are particularly important for patients

Training: The training I have received clearly helps metoensurequalityintheendproduct

Problem Solving:Alllineworkersareregularlyinvolvedin problem solving, troubleshooting and investigations

Gov

erna

nce

Recognition: We recognize and celebrate both individualandgroupachievementsinquality

Metrics:Up-to-datequalitymetrics(e.g.defects,rejects, complaints) are posted and easily visible near each production line

Knowledge:Eachlineworkercanexplainwhatlinequalityinformationistrackedandwhy

Continual Improvement: We are regularly tracking variations in process parameters and using them to improve the processes

Lead

ersh

ip

Coaching:Supervisorsprovideregularandsufficientsupportandcoachingtolineworkerstohelpthemimprovequality

Dialogue:Wehavedailyqualitymetricsreviewsandqualityissuesdiscussionsontheshopfloor

Gemba:Managementisonthefloorseveraltimesa day both for planned meetings and also to observe and contribute to the daily activities

Min

dset

s

Awareness:Everylineworkerisawareofthebiggestqualityissuesontheirlineandwhatisbeingdoneabout them

Responsibility:Allemployeesseequalityandcompliance as their personal responsibility

Inte

grity

Openness:Iamnotafraidtobringqualityissuesto the management’s attention

Ethics:PeopleIworkwithdonotexploittotheiradvantage inconsistencies or ‘grey areas’ in procedures

Motivation: All employees care about doing a good jobandgotheextramiletoensurequality

Report from ISPE Quality Metrics Pilot Program / June 1 – April 2015 78

Appendix 2

Process Capability Questions f Doyoumeasurethattheprocessremainsinastateofcontrol(thevalidatedstate)duringcommercialmanufacturing?(yes/no) f Forwhat%ofproductsaretheyapplied(basedonyourtotalnumberofproductsasreportedindatabysite)-excluding

packaging operations f Ifnotappliedon100%ofproducts,howdoyouchoose/segregate/prioritizeonwhichproductstoapplythesemetrics?(openquestion)

Please indicate which metric or metrics do you use for ongoing monitoring and to what parameters do you apply them CpK Ppk

Tolerance interval Box Plots

Trending of CQAs

Other (specify which in the comments field)

AppliedtoCQA(criticalqualityattributes tested in the lab)

Applied to IPC (in-process control) checks

Applied to CPP (critical process parameters)


Appendix 3 Examples for Site and Product Data Collection Templates


Appendix 4Case Study Company A

Introduction

Company A, a participant in the ISPE Quality Metrics Wave 1 Pilot, has a large portfolio of over-the-counter (OTC) products manufactured and distributed globally:

f 8,000+rawmateriallotstestedperyear. f 20,000+ intermediate bulk, packed, and labeled products tested annually. f 750+millionpacksproducedandreleased.

Given the extent of site-level activity, one of Company A’s motivations for participating inaqualitymetricsprogramisitspotentialtoreduceinspectionfrequencies.Unlikeotherpilotparticipants,however,CompanyAwillnotbenefitfromexpeditedapprovalsor improved efficiency in post-approval changes due to the nature of its OTC- and monograph-based business.

CompanyAalsounderstandsthatcollectingqualitymetricsmayhelpreducedrugshortages. Indeed, the ISPE Drug Shortages Prevention Plan [14]states“well-definedmetricstailoredtoproactivelyidentifythepotentialriskofashortage,willhelpmitigateloomingshortages.”Theplanacknowledgesthataprescriptivelyselectedsetofmetricsmaynotalwaysidentifydrugshortages,however.Forthisreason,itrecommends that each metrics program should be tailored to the supply chain situation it addresses.

Finally, because developing a healthy corporate culture is a particular focus for CompanyA,itknowsthatawell-designedsetofperformancemetricshelpsidentifyimprovement opportunities.

Pilot Experiences

Company A uses a detailed set of operational performance metrics. The definitions ofsomeofthesemetrics,however,differfromthoseusedintheWave1Pilot.ThesedifferencesrequiredCompanyAtoperformadditionalworksoitcouldreportitsdatain the Wave 1 Pilot standardized format.

MetricsreportedbyCompanyA(andallWave1Pilotparticipants)were:

f Data mined retrospectively from the previous 12-month period. f Measured prospectively for the current 3-month period. f Evaluatedatanoperational/siteleveland(whererequired)aggregated

by product/formula. f ReportedtoMcKinseyona“goodenough”basis;datawasnotsubjectedtotherigorousreviewandcheckingrequiredforofficialsubmissiontoFDA.


Appendix 4

Understanding Supply Chain Complexity

CompanyAoperationsinvolvesignificantsupplychaincomplexity,asshownin Figure A1.

Figure A1: Supply Chain Complexity

To patient/customer

To patient/customer

Filling in plant Pack to brown case

Manipulated on another line Manipulated

for local customization

Ship tocustomerWHSE

CMO

Filling in plant Pack to brown case

Manipulated on another line Manipulated

for local customization

Ship tocustomerWHSE

CMO

United States

CentralAmerica

WHSE:WarehouseCustomer: Pharmacy, grocery chain or consumer store

SupplychainsfortwodifferentformulasareshowninFigureA1:

f A red formula (red star) is a bulk drug product made in a manufacturing facility in Country 1 (United States).

f Ayellowformula(yellowstar)isabulkdrugproductmadeinbothCountry1(United States) and Country 2 (Central America).


Appendix 4

Asthefigureshows,packaging,labelling,secondarypackagingandcustomizingoperations add significant complexity:

f Theyellowformulacanbetransportedinbulktoacontractmanufacturingorganization (CMO) for primary packing, secondary (outer) packaging, and in some cases customization.

f TheyellowformulacouldalsobesenttooneofCompanyA’sexistingplantsineithercountrytobefilled,packaged(primary),dispositioned,andsenttothe customer.

f The primary package could also be moved to another line for manipulation or customization.

f Addingsecondarypackagingintoanoutercartonand/orbrowncasecouldalso include being shipped to a contract manufacturer to be customized.

f Customization can be very complex so that for example, a 24 pack case could consistofmanyproduct/packvariants–24differentformulas,2packsof12 formulas, 6 packs of 4 formulas, or 8 packs of 3 formulas

f A product could be shipped from a location in Country 2 to a location in Country1tobemoveddownthecustomersupplychain.

f Eachprimarypackagehasalotcodeandexpirydatetoallowfulltraceability.

ThiscomplexityrelatesdirectlytohowdatawasreportedbyCompanyAintheWave 1 Pilot: Because the back end of the supply chain includes variation-based choices of the final disposition point (i.e. data at the consumer pack level depends onwherethefinaldispositionpointinthesupplychainisallocated),plant-leveland/oraggregated formula-level metrics that are normalized on a per-pack basis can be affected. Understanding this is critical to ensuring that metrics are captured and reported accurately.


Appendix 4

Supply Chain Complexity in Quality Metric Reporting: Challenges

Given the potential range of primary packs that a secondary (dispositioned) pack might contain for Company A, allocating a specific complaint to a particular product and/or supply chain is particularly difficult.

Assigningconsumercomplaintswoulddependonwherethefinaldispositionwasdesignatedandwheretheconsumerpurchasedthe24-,12-,orsix-packcase.

For these and other reasons, complaint rate results in the Wave 1 Pilot are normalized for a product based on the dispositioned lot.

Other Product-Based Complexity Challenges

In addition to supply chain complexity, Company A also has product line complexity. Whileitisrelativelyeasytogetinformationandqualitymetricdataonproductfamiliesprepared using a base formula differentiated by flavor, for example, this becomes much morecomplicatedwhentherearemultipleformulaswithinonefamily.

CompanyA’sannualproductqualityreviews(APQRs)arebracketedbyformulafamiliesthathaveacommonbase.Onebaseformulawithaparticularflavorcanbepacked into many secondary-pack variations. This makes assigning product-based metrics extremely difficult.

Disaggregatingqualitymetricdata(complaintsrate,forexample)toproductsattheuniqueformulalevelortoaparticularmanufacturingsiteinasupplychainisextremelychallenging;existinginformationtechnology(IT)systemsatCompanyAdonotcurrently facilitate this.

Conclusion

Company A has a large product range and very complex supply chains. This makes assigning metric data to a product level extremely difficult and time-consuming. Changing IT systems to a standardized set of metrics that could produce product- leveldatawouldrequiresignificantinvestment.


Appendix 5 Detailed Analysis of Data and Relationships for Each Individual MetricLot acceptance rate

Lot acceptance rate defined as difference between 100% and ratio of rejects vs. all dispositioned lots

Effort difficulty

Preferred frequency of reporting

Comments

1.7 2.0 2.1 1.7

2.0 2.3 2.0 1.7

Lots dispositioned

7

13 14

6

43

9

4

95 4

2

Lots dispositioned

Lots rejected

Overall

Difficulty for other components Allocation between components

MONTHLY

Data pulling was not really hard but time consuming (manual pull) (few comments)

We would like to have an option to nclude partial batch rejections

Time consumed [h]2

Lot acceptance rate correlates with: Culture Deviations recurrence Critical complaints Rework

Rejects difficulty1

Drug subst. OTC Gx Rx

1 Onascalefrom1-easiestto4-mostdifficult 2 Retrospective+current

Lot acceptance rate%releasedoutofallfinallydispositionedlotsLot acceptance rate

98.9

87.9

97.0

100.0 99.7

Bio

API

Steriles

Solids

Other 1

Lab

1 OtherincludesCreams,LiquidsandOther


Appendix 5

Confirmed OOS – product

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


Confirmed OOS – product defined as (total confirmed OOS – confirmed stability OOS – cofirmed RM OOS)/(total lots tested – stability lots tested – RM lots tested)

1.92.5

1.9 1.720

23

915

Confirmed OOS – product correlates with: Total complaints (lag) Critical complaints (lag)

Collecting separately release OOS and lots tested would simplify and speed up the process (2 components instead of 6 for KPI) (McK.)

Numbers for lots tested get huge for multi-product plant or when lots of water is tested (few comments)

Would like more clarity which lots to include in lots tested (e.g. placebo, micro samples)

For OOS could be specifed‚ count when investigation closed’ as only then it can be‚ confirmed’

Total confirmed OOS difficulty1

2.42.12.42.11.8

2.52.3

2.5

2.32.0 1.81.81.72.21.72.32.02.12.02.0

RM lo

ts

test

ed

Stab

ility

lots

test

ed

Tota

l lot

s te

sted

Con

firm

ed

RM O

OS

Con

firm

ed

stab

ility

OO

S

4.61.8

7.0

1.41.2

4.0 1.53.84.64.6

7.5

1.40.9

2.3

1.21.0 2.81.4

4.6

1.21.14.2

RM lo

ts

test

ed

1.5

Tota

l lot

s te

sted

Tota

l lot

s te

sted

Con

firm

ed

RM O

OS

Con

firm

ed

stab

ility

OO

S

Tota

l co

nfirm

ed

OO

S

2.4


1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current

Confirmed RM OOS – productNo unit provided

Confirmed OOS – product 150.6

0.1 0

0.4

0 0

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Confirmed OOS – stability

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


Confirmed OOS – stability defined as ratio of confirmed OOS for stability to stability lots tested

1.8 2.0 1.7 2.0

2.1 2.3 1.8 2.0

Lots dispositioned

4

8

43

2.11.4 1.9

6.0

2.02.2 1.41.1

Lots tested stability

Confirmed OOS stability

Data for stability stored separately/pulled manually (few comments)

At least for sites with only few products it makes little sense to collect monthly (GMP requires 1 batch/year/ product)

Confirmed OOS – stability correlates with: -

Confirmed OOS stability difficulty1



Confirmed OOS – stabilityPer 000’ stability lots tested

0 0

7.1

250.0

2.3

25.3

Bio

API

Steriles

Solids

Other

Lab



Appendix 5

Deviations rate

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


Deviations rate defined as ratio of deviations to lots dispositioned

1.62.3 1.9 1.5

2.0 2.3 2.0 1.7

Lots dispositioned

6

1310

6

52

94

10

14

1

Lots dispositioned

# of deviations

Deviations rate correlates with: Critical complaints

# of deviations difficulty1 Deviation def. is very broad and includes almost everything, should be narrowed, also adding OOS here confuses the picture Need more clarity if e.g. RM or buffer deviations are to be counted Issues with data pulling (e.g. separate systems for OOS and deviations and need for aggregation) (few comments)



Deviations ratePer 000’ lots dispositioned

13

15,667

256 128

683

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Rework rate

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


Rework rate defined as ratio of reworked and reprocessed lots vs. (lots dispositioned minus lots rejected)

1.9 1.52.0 1.7

2.01.7 2.32.0 2.02.1 1.71.7

Lots rejected Lots dispositioned

7

18 19 17

421

945

116

1

953

Rework rate correlates with: Lot acceptance rate

Lots reworked difficulty1We do not permit any rework at all (few comments)

Need more clarity what rework or reprocess mean


Lots rejected

Lots reworked

Lots dispositioned


Rework rate%lotsdispostioned

2.43

1.20

0.48

00

25.41

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Confirmed OOS – raw materials

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


1

Confirmed OOS – raw materials defined as ratio of confirmed OOS for raw materials to raw materials lots tested

2.1 2.3 2.2 2.0

2.4 2.51.8 2.3

Lots tested - RM

4

85

4

2.90.9 1.9

6.0

2.72.3 2.51.0

Confirmed OOS - RM

Lots tested - RM

Confirmed OOS – raw materials correlates with: -

Confirmed OOS RM difficulty2 sites did not supply it Need more clarity what to include (e.g. which EM, why water is in and other EM not neccessarily) (few comments)

Split between technologies and/or pharma/cosmetics businesses difficult (few comments)



Confirmed OOS – raw materialsPer000’RM/PMlotstested

40.20

0

2.24

0.40

5.22

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Unconfirmed OOS

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


1

Unconfirmed OOS defined as ratio of unconfirmed OOS to total lots tested

2.1 2.31.6 1.9

2.4 2.51.7 2.1

Lots tested

10

37

7 12

72

1720

349

2

Lots tested Unconfirmed OOS

Unconfirmed OOS correlates with: -

Unconfirmed OOS difficulty

Data is automatically available together with confirmed OOS (few comments)

There was some manual aggregation needed due to scope (few comments)



Unconfirmed OOSPer 000’ lots tested

3.0

869.3

8.8

0 0.8

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Recall events US

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


1

Recall events US

1.2 1.5 1.3 1.30.5

1.30.7 0.7

Recall events US correlates with: Deviations rate Critical complaints

This is non-applicable for some drug substance sites

Time was short as we had no recalls, not sure if this reflects situation with recalls

We should not limit ourselves to US market


Recall events US# of events

1

000

3

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Recall events class I and II US

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


1

Recall events US class I and II US

Recall events US class I and II US correlates with: Not tested

1.2 1.3 1.3 1.30.3

1.40.7 0.5





Recall events class I and II US# of events

3

0000

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Recalled lots US

Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


1

Recalled lots US

Recalled lots US correlates with: Not tested

1.2 1.3 1.4 1.40.3

1.40.9 0.5





Recalled lots US# of lots

4

0000

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5


Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2

OTC Gx Rx

Total complaints rate defined as ratio of total complaints to total packs released

1.8 2.0 2.0

1.82.5 2.1

Packs released

1220

34

7513

7

29

5

Packs released

Complaints

Total complaints rate correlates with: Action limit excursions OOS product (lag)

Total complaints difficulty1 1 plant did not manage to collect packs data Had trouble to report packs due to different system set-up Difficult to split between technologies Time consuming due to many products (few comments)

Some complaints cannot be allocated to product at all Need more clarity – e.g. whether include market only or also internal complaints,if bulk complaint should be reported bybulk of packaging site, how to compare complaints on bulk with packs (few comments)

Little value added in that, confirmed complaints would be better



Total complaints ratePer million packs

26

86

14

1,078

1

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5


Effort difficulty


Comments

Overall

MONTHLY

Time consumed [h]2

OTC Gx Rx

1.82.3 2.1

1.82.5 2.1

Packs released

1020

231 1 plant did not manage to collect

packs data Had trouble to report packs due to different system set-up Difficult to split between technologies (few comments)Time consuming due to many products (few comments) Some complaints cannot be allocated to product at all


Critical complaints rate defined as ratio of total complaints to total packs released

73

137

18

5

Packs released

Critical complaints

Critical complaints rate correlates with: Deviations rate Lot acceptance rate OOS product (with lag)

Critical complaints difficulty

If we have a complaint on bulk how do we report corresponding packs? Had to manually assess each complaint for criticality Little value added in that, confirmed complaints would be better (few comments)


Critical complaints ratePer million packs

2.0

0.7

0.10

161.8

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

APQR on time

Effort difficulty


Comments Overall

ANNUALLY

Time consumed [h]2


1

APQR on time defined as ratio of APQR on time to all APQR closed in reporting period

1.42.0

1.1 1.3

1.42.0

1.3 1.4

Products subject to APQR

1.0

3.6

2.01.1

0.40.6

1.91.71.01.0 0.50.5

Products subject to APQR

APQR on time

We group our products for APQR (few comments)

Period may need to be adjusted for this point, products evaluated are different than ones produced in the period

APQR on time correlates with: CAPAs effective

APQR on time difficulty



APQR on timePercent

0

100.0

42.9

100.0 100.0

90.9

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

CAPA effectiveness

Effort difficulty


Comments Overall

QUARTERLY

Time consumed [h]2


1

CAPA effectiveness defined as ratio of CAPAs evaluated as effective to all CAPAs evaluated for effectiveness in the period

2.12.7 3.0

1.8

1.8 2.03.0

2.1

CAPAs with effectiveness check

5

20

2 4

23 4

16

11 22

CAPAs with effectiveness check

CAPAs effective

27% of plants did not collect enough data for this metric Manual extraction of data from e.g text files (few comments) We evaluate group of CAPAs as a whole and not single CAPAs Need clarity how to include partially effective CAPA Recurrence of the "quality issue" is secondary. CAPA target a specific root cause and to be effective should address the given root cause which subsequently prevents recurrence of the "quality issue"

CAPA effectiveness correlates with: APQR on time

Effective CAPA difficulty



CAPA effectivenessPercent

Bio

API

Steriles

Solids

Other 1

Lab 0

86.5

35.3

93.6 97.7

100.0



Appendix 5


Effort difficulty


Comments Overall

MONTHLY

Time consumed [h]2


1

Recurring deviations defined as ratio of recurring deviations (with 1 year look-back period) to all deviations closed

2.63.8

2.7 2.3

1.62.3 1.9 1.5


49

16

5

12 45 3

13

23



Recurring deviations correlates with: Lot acceptance Recalls Culture

Recurring deviations difficulty 15% of sites did not report it at all Deviation def. is very broad and includes almost everything, should be narrowed, also adding OOS here confuses the picture Need more clarity if e.g. RM or buffer deviations are to be counted Issues with data pulling (e.g. separate systems for OOS and deviations and need for aggregation, manual checking) (few comments) recurring Deviation should be limited to the same deviation (same specific departure from approved process or instructions) and independent of "root cause". The given definition of "deviation" is so broad many things that meet that definition likely do not require root cause analysis



Recurring deviationsPercent

13.5

3.90

62.3

25.4

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Media fills successful

Effort difficulty


Overall

ANNUALLY

Time consumed [h]2

Rx

1

Media fills successful defined as media fills successful to all media fills

1.8

1.8

Total media fills

1.1

0.40.7

Media fills successful Total media fills

Media fills successful correlates with:

Media fills successful difficulty



Media fills successfulPercent

100.0100.0

Bio

API

Steriles

Solids

Other 1

Lab



Appendix 5

Lots rejected due to action limit excursions

Effort difficulty


Overall

QUARTERLY

Time consumed [h]2

Rx

1

Lots rejected due to action limit excursions defined as ratio of lots rejected due to action limit excursions to total sterile lots dispositioned

2.5

1.9

Total sterile lots

dispositioned

3.5

1.81.8

Total sterile lots

dispositioned

Sterile lots rejected for limit exc.

Lots rejected due to action limit excursions correlates with: -

Sterile lots rejected for action limit - difficulty



Lots rejected due to action limit excursionsPercent

0.1

0

0.5

0

Bio

API

Steriles

Other 1

Lab



Appendix 5

Investigations related to action limit excursions

Effort difficulty


Overall

MONTHLY

Time consumed [h]2

Rx

1

2.8

1.9

Total sterile lots

dispositioned

3.6

1.81.8

Total sterile lots

dispositioned


Investigations related to action limit excursions defined as ratio of lots with action limit excursions to total sterile lots dispositioned

Lots with action limit excursions

Investigations related to action limit excursions correlates with: Total complaints Culture (capabilities)

Investigations difficulty


Investigations related to action limit excursionsPercent

11.7

00.8

2.8

15.4

Bio

API

Steriles

Other 1

Lab


ispe quality metrics initiative - regulations.gov

Documents