Click to edit Master title style

11/26/2015 1

Measuring Performance, Driving Excellence

ISPE Nordic, Stockholm WTC November 26th 2015

Dr. Nuala Calnan, Research Fellow, Dublin Institute of Technology

Driving Patient Focused Excellence

Patient Focused Excellence

© Nuala Calnan, PRST – All rights reserved

Understanding the DNA Building Blocks • The best potential to sustainably impact the outcomes that

matter to the patient lie in the integration of three criticalareas of influence for the pharmaceutical industryoperational excellence, knowledge excellence andcultural excellence.

• Although there are many programmes and initiatives ofvarying levels of maturity across these areas of influencewithin the industry, the majority are not well integrated.

• The opportunities for benefit to flow to the patient (and thebusiness) are therefore not fully realised.

© Nuala Calnan, PRST – All rights reserved

The Excellence Framework combines the cultural excellence of a learning organisation with excellence in knowledge creation and utilisation, in order to deliver operational excellence based on a relentless restlessness for improvement

The Excellence Framework : Knowing, Doing, Being

The desired “Culture of Excellence” is one based on a reflective Learning Culture where people become

proactive problem solvers and perpetual learners as members of Learning Teams

4 © Nuala Calnan, PRST – All rights reserved

11/26/2015 6

FDA Guidance Issued - What Did We See?• Key items…product v’s site• 10 data collection points• 4 metrics

• Covered Establishments / Reporting Est.

• Optional metrics• Data collection mechanism….

Source: IPQ July / August 2015

Click to edit Master title style

11/26/2015 9

ISPE Quality Metrics Initiative The Journey Thus Far“Start Small, Learn and Evolve”

11/26/2015 10

ISPE Wave 1 Pilot – What Did We Learn ?• Feasible to collect standardized set of metrics• Few companies aggregate metrics across supply chain to be able

to report at product, differentiated by site• Level of Burden not to be underestimated• Understanding organizational context is crucial

Start with a targeted set of “relatively well established” metrics

The Wave 1 Pilot also provided some key insights in relation to the prevailing quality culture within an organization that merit further exploration.

Key benefits reported from Pilot Participants:• Gaining a deeper understanding of the standardized metrics definitions and design• Establishing a centralized submissions process trial• Developing access to a benchmarking report that allowed them to examine progress

against peers

44 Sites, 18 Companies, 14 Metrics

ISPE Wave 2 Pilot – What are our Objectives?

Evolve data set across segments, geographies and time to further the learnings from Wave 1 and evaluate trending patterns

• Test the proposed FDA metrics • help develop robust definitions • understand data collection challenges

• Evaluate logistics and effort of gathering data at a product application level

• Continue to develop measures, tools and dialogue related to Quality Culture and Process Capability to facilitate ongoing industry self-development and assessment

• Enable continued objective and data driven dialogue with FDA and other Health Authorities

ISPE Wave 2 Pilot Metrics

Culture indicatorsQuantitative metrics

• CAPAs with preventive actions

• Planned maintenance

• Employee turnover

• Human error deviations

• Deviations with no assigned root cause

------

• CAPAs requiring retraining4

• APR approvals4

• Total complaints rate 1,4

• Critical complaints rate 1

• Recall events

• Lot acceptance rate 1,4

• Invalidated OOS rate 1,4

• Right first time rate

• Deviations rate 1

• Recurring deviations rate

▪ In addition to site data collection, participating companies will compile logistics and effort of gathering product data at application level (on 2-5 products) for complaints rates, lot acceptance rate and invalidated OOS

1 To be tested in multiple variants (e.g. lots attempted vs. dispositioned, packs released vs. lots released, lots tested vs. total tests performed)2 Only for sites which have not participated in wave 13 Will include additional questions to the ones tested in wave 14 Aligned with the Federal Register Notice guidance. Additionally OOS rate and Lots pending disposition for 30+ days can also be calculated from the

data5 questions to assess how standardized metrics can assist prediction of drug shortages

Survey-based metrics

• Quality culture survey2

• Process capability3

• Drug Shortages5

Wave 2 Pilot enrollment Data collection and submission

ISPE Quality Metrics Wave 2 – What’s Next ?

Individual reports

Nov 30

Industry analysis

Data lock for Wave 2 pilot analyses

Oct 30 Data lock for analyses of selected date for the FRN response

Launch Quality Metrics wave 2 Pilot

Feb-Mar 2016Jun 2015

We are here

Quality Metrics Vision – A Journey

• Reporting accessible to multiple agencies, based on firms granting view access

• Goal: facilitate regulatory convergence on requests for quality metrics

• Potential Benefit: Achieving transparency of quality performance across industry. Turning quality into a competitive advantage.

Additional measures to be discussed during inspections

Starter Set of Metrics

• Reported to FDA

• Benefit: Risk-based inspection frequency

• Potential Benefit: Simplification of regulatory process for sites in good standing

Industry led “work space”to further develop metric program

Click to edit Master title style

11/26/2015 15

Cultural ExcellencePromoting Behavior Based Quality

An Update From the ISPE Quality Culture Team

Thought Leadership…“There is the virtuous cycle of execute, monitor, measure, andimprove…that is the quality system part of the modern concept ofquality.You can do all these things and have wonderful systems andmeasurements, have wonderful procedures… but if you do not havea quality culture in which people feel free to speak up there isproblem.

• The people doing the work they have to own it.• They have to feel their relationship to the patient and that what they are

doing benefits the patient.• They have to feel safe to speak up if something is going wrong.• They have to feel empowered. That is really important.

Dr. Janet Woodcock, CDER Director

ISPE Quality Metrics SummitBaltimore, April 2015

Source: IPQ Monthly Update April/May 2015, Vol. 6, No. 3. www.ipqpubs.com

Thoughts from this side of the pond…

Source: https://mhrainspectorate.blog.gov.uk/author/davidchurchward/

Impact of organisational culture: is your company behaving well?The impact of organisational culture and senior management behaviour on data governance must not be underestimated. Indicators with relevance to data governance provide a measure of the workforce’s understanding and reporting behaviour, combined with the management’s receptiveness to ‘bad news’. Is error or system failure reported as an opportunity for improvement, or is there a mind-set around ‘not wanting to cause trouble’? To remove the incentive to manipulate, re-create or amend data, the managerial response to ‘bad news’ must be fair and consistent, and not based on a fear of consequences.

‘Led from the top; empowered from below’Organisational culture is not just addressed by senior management putting the right words in a mission statement. I have seen that communicating expectations clearly to staff at all levels in the company, and then living by these principles, is the key to success. Leadership, engagement and empowerment of staff at all levels in the organisation can then combine to identify and deliver systematic data integrity improvements where good practice becomes automatic.

“We are what we repeatedly do. Excellence, then, is not an

act but a

habit”.

Current ISPE Position on Quality Culture

• The Quality Culture sub-team comprises of a cross section ofIndustry large and mid-sized pharma, incorporating bothinnovator and generics organisations, key industry suppliersand academia.

• The team has met bi-weekly since July 2014 with an initialfocus on identifying and sharing the variety of practicescurrently used.

• A repository of approaches, practices and tools were capturedfor detailed review

Acknowledging the key influential role of QualityCulture with respect to the capture and reporting ofquality metrics the ISPE Quality Metrics Core Teamformed a Quality Culture Sub-Team in July 2014

Update from ISPE Quality Culture Team

No single tool or practice provides a quantitative or qualitative ‘silver bullet’

as a means to establish the current health of the Quality Culture within an

organization.

• Six “dimensions” teams were formed July 2015, developing aprogram of tools, training and templates for companies to useto inform their cultural excellence programs. These tools willbe published through ISPE during 2016 and 2017

• Six “dimensions” teams were formed July 2015, developing aprogram of tools, training and templates for companies to useto inform their cultural excellence programs. These tools willbe published through ISPE during 2016 and 2017

Who we are…

Six Dimensions of Cultural Excellence

• The ISPE Quality Culture Sub -Team proposes that the promotion of a ‘relentless restlessness’ towards improvement in Quality Culture necessitates a 360° approach to Cultural Excellence

• A holistic Quality Culture Framework has been developed, entitled the Six Dimensions of Cultural Excellence

What has the team been up to ?

Quality Culture In the News…

What is Keeping Us Busy Now?

Dimension 1 Leadership & Vision• Identify best-practices and leader-led behaviors most

directly linked to positive culture transformation. Tools under development include:

! “Shaping Cultural Excellence” Interview – Interview questions for Key Site Leaders on Cultural Transformation Programs in Quality, Safety or Operational Excellence Design: Program Name:___________________________________________________________ Purpose:_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Target Audience:_________________________________________________________________________________________________________________________________________________________________________________ _______ Where implemented and when:___________________________________________________________ Intended Results:__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Execution: Duration:_________________________________________________________ What was the messaging strategy: ________________________________________________________ ________________________________________________________________________________________________________________________________________ Were different approaches necessary or used for different organizational group or levels? ________________________________________________________________________________________________________________________________

! “Shaping Cultural Excellence” Interview – Interview questions for Key Site Leaders on Cultural Transformation Programs in Quality, Safety or Operational Excellence Design: Program Name:___________________________________________________________ Purpose:_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Target Audience:_________________________________________________________________________________________________________________________________________________________________________________ _______ Where implemented and when:___________________________________________________________ Intended Results:__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Execution: Duration:_________________________________________________________ What was the messaging strategy: ________________________________________________________ ________________________________________________________________________________________________________________________________________ Were different approaches necessary or used for different organizational group or levels? ________________________________________________________________________________________________________________________________

• Leader interviews• Intended to capture leader shared stories –

successes and learnings in their own words.• Looking for commonality in actions (leader-

led behaviors) or strategy (best practices).

• Leader Risk Assessment• Assesses risks associated with turnover of

key leaders and maturity / experience of current leaders.

Dimension 2 Attitudes and MindsetsThis group is focusing in on the relationship between the prevailing employee attitudes and mindsets and the actualbehaviors practiced in the day-to-day execution of tasks.

Tools under development include means of assessing the success of Engagement Strategies, Peer Involvement and Employee Ownership

The work of this group examines the effectiveness of the Leadership and Vision activities•Examining how much the employees “own it”•Establishing how each employee “feels” about their relationship to the patient •Understanding the level of “empowerment”

Maria Meier, Senior Advisor/Director, Senate Democratic Diversity Initiative

Provide connection for every employee about how their efforts matter to the patient and the

company quality mission.

Provide connection for every employee about how their efforts matter to the patient and the

company quality mission.

Dimension 3 Gemba Walks: Manage for daily improvement

• Gemba walks offer a key opportunity for coaching and mentoring, knowledge sharing and early issue intervention.

• Practicing Gemba helsp to understand the influences which inform the decision-making on the shop floor which directly impacts quality and the continuous improvement mindset

• The objective of this team is to define critical success factors and best practices for effective GEMBA walks as well as training, tools and templates that support the effective implementation of Gemba walks

Talking the Talk!Gemba is principally about the Dialogue

Opportunity for management engagement, learning, and recognition

Talking the Talk!Gemba is principally about the Dialogue

Opportunity for management engagement, learning, and recognition

Dimension 4 Leading Quality Indicators (LQI) & Triggers

27

“Indicators and triggers represent the information and actions taken at specific thresholds that guide incident recognition, response, and recovery”

Source: Crisis Standards of Care: A Toolkit for Indicators and Triggers, Dan Hanfling, John L. Hick, and Clare Stroud, Institute Of Medicine Of The National Academies

The Leading Quality Indicator Model:Changing Behaviours, Measuring ResultsA set of quantitative measures of the behavioural changes which enable the delivery of enhanced quality regulated products called the Leading Quality Indicators (LQI) Index.

These measures target behavioural change across the key operational areas of Patient, People, Places, Prevention and Performance

Reinforcement….not Enforcement Measuring what Matters

Monitoring for Risk Triggers

Reinforcement….not Enforcement Measuring what Matters

Monitoring for Risk Triggers

Designing Measures for IMPACT

Monitoring the Risks

Dimension 5 Oversight & Review• Focus on the necessary management monitoring system

(including reviews by Leaders of the right combination of key data and observations) needed to aid Leaders in forming a holistic assessment of an organization’s Quality Culture at an Enterprise level.

• This team will develop relevant and practical management monitoring system tools (e.g. best practices, templates, governance).

Providing the Link!Enabling the link between management oversight and risk management/risk prioritization through the use of these

tools.

Visibility, Ownership… Action.

Providing the Link!Enabling the link between management oversight and risk management/risk prioritization through the use of these

tools.

Visibility, Ownership… Action.

Example Oversight & Review Tool

• Horizontal view indicative of site performance• Vertical view indicative of cultural component performance from a global/enterprise

perspective.

Co

mp

on

en

t 1

Co

mp

on

en

t 2

Co

mp

on

en

t 3

Co

mp

on

en

t 1

Co

mp

on

en

t 2

Co

mp

on

en

t 3

Co

mp

on

en

t 1

Co

mp

on

en

t 2

Co

mp

on

en

t 3

Co

mp

on

en

t 1

Co

mp

on

en

t 2

Co

mp

on

en

t 3

Co

mp

on

en

t 1

Co

mp

on

en

t 2

Co

mp

on

en

t 3

Site A R R R R Y R R Y R R G Y R Y GSite B Y Y R G G G G G R Y R G Y Y GSite C G G G Y G G G G Y G G G G G GSite D Y Y R Y G Y G G R Y Y Y Y Y GSite E G G R G G G G G G Y G G G G GSite F R R R Y G Y Y Y G Y Y R Y Y GSite G R R R Y G Y Y Y Y Y R R R R YSite H G G G G G G G G G G G G G G GSite I G G Y G G G G G G Y G G G G GSite J Y Y R Y G Y Y Y Y Y Y Y Y Y GSite K G G R Y G G G G G Y G G G G YSite L G G R Y G Y Y G G Y G G Y Y GSite M R R R Y G R R R R Y R R R R G

Leadership & Vision Mindsets & Attitudes Assessment & GEMBA LQIs & Triggers Structural Enablers

Highlights best practices and areas of focus.

Dimension 6 Cultural EnablersFocus on Cultural Enablement:

1. Organizational Enablers:• Learning Organization Development- Enabling • Recognizing Change and Aligning Behaviors• Recognition System - Recognizing the right behaviors

2. Continuous Cultural Improvement§ Proactive Problem Solving§ Getting to True Root Cause

Enable a Learning Organization!

One that continuously improves by aligning key performance indicators for each system with behavioral indicators (throughout organizational

levels) enabling cultural change, recognizing and rewarding appropriate behaviors

Enable a Learning Organization!

One that continuously improves by aligning key performance indicators for each system with behavioral indicators (throughout organizational

levels) enabling cultural change, recognizing and rewarding appropriate behaviors

Click to edit Master title style

11/26/2015 32

Sincere gratitude to all the volunteers on the Quality Culture dimension teams

Interested in joining the effort?

Contact: nuala.calnan@dit.ie

“Spreading constructive beliefs and behavior from the few to the many”

Sutton & Rao, Scaling Up Excellence (2014)