Merz RBQM Journey

Dr. Nico Wegener, Senior Clinical Project Manager, Merz Pharmaceuticals

Dr. Johann Pröve, Chief Scientific Officer, Cyntegrity

Merz RBQM Journey (Abstract)

Many healthcare companies and CROs struggle with the best approach to the implementation of RBQM. This is due to the fact that there is only 'little history' available, i.e. not a lot to piggy-back on what other companies had developed and shared already. Cyntegrity built a whole course on RBQM using the 'belt' approach while Merz Pharmaceuticals was the first company to jump onto the bandwagon and used it for a full-fletched roll out into their R&D organization.The training had been split into four parts (white, green black and executive belt) with a different focus for the different staff levels. Merz staff took the white belt course to get comfortable with the basic idea of RBQM, while the green belt course had been tailored to a specific Merz study in order to link an actual study with the RBQM process. Eventually, the black belt course had been taken by staff in charge of RBQM management in the future, mainly covering the change management aspects of RBQM implementation. Finally, Senior Management had to be familiarized with the concepts and the return on investment of an RBQM implementation, covered by the 'executive belt' course. This session will address the lessons learned when developing such a course and the experience made by the participants. We will also discuss a real study as an example for RBQM and how Merz shall implement RBQM in future studies.

Slide 2

Merz RBQM Journey

1. Introduction (slide 4-5)

2. Introduction to the topic / challenge (slide 6-7)

3. Development of the training courses (slide 8-9)

4. Delivery of the white belt training (slide 10-12)

5. Challenge of the ‘Chinese Study’, brief intro to study (slide 13-17)

6. Review of protocol and development of risks / KRIs /metrics (slide 18-19)

7. Results of the analysis and development of the green belt course (slide 20-26)

8. Black belt and executive belt course development rationale (slide 27-30)

9. Conclusion (slide 31)

Slide 3

Disclaimer: The views and opinions expressed in the following slides and in the podcast are those of the individual presenters and should not be attributed to Cyntegrity or Merz Pharmaceuticals.

Introduction

Dr. Nico WegenerSenior Clinical Project Manager, Merz Pharmaceuticals, Frankfurt, Germany

❖ Since 2009 with Merz❖ Experience in international Phase III drug and medical device clinical trials❖ Currently, coordinating various clinical trials in China❖ In my opinion, running clinical trials especially in China

is full of surprises and therefore implementation of RBQM is crucial❖ The Sponsor-team needs to get behind the RBQM-idea for successful

implementation in the trial

Slide 4

Introduction

Johann Pröve, PhDChief Scientific Officer, Cyntegrity, Frankfurt, Germany

• 1980 – 2016 Clinical Data Management, Bayer Healthcare• CSO at Cyntegrity since 2016• From paper CRF to NCR copies, from Remote Data Entry via EDC to EMR /EHR • Currently, supporting customers getting from protocol to RBQM • Checking clinical trial data using edit checks is a great step, however, one easily

misses the patterns, issues in the audit trail, and flaws in the important data • Many companies do not know how to get started and make the first steps in the

RBQM world

Slide 5

Introduction to the topic

1. Many healthcare companies and CROs struggle with the best approach to the implementation of RBQM.• Start small and grow the implementation over time• Implement using a big-bang approach• Have others start and wait and see

2. Only 'little history' available, i.e. not a lot to piggy-back on what other companies had developed and shared already.• Many companies started using Excel spreadsheets• Some used the TransCelerate proposed approach• Some used the more advanced MCC RAMMT tool• Some try to use tools requiring a lot of programming• Hardly any company implemented a tool that covers all aspects of RBQM

3. The day-to-day activities related to RBQM are the challenge• What to do first, what second and whom to involve• Start with a completed study, with a new one, with an ongoing one

Slide 6

Introduction to the topic

4. It is not about RBM (SDV reduction) but rather RBQM (plan, act, do, analyze) for many areas• RBM would only focus on the monitoring aspects• RBQM involves many other functions and their risk management plans >>> IQRMP• Address the critical areas and do not worry too much about the less critical areas (modify protocol )

5. The process starts at project level, covers many functions and processes• What are the risk areas for the project and how would they boil down to the study level• Systems used (EDC, CTMS, PV, CDMS, IXRS, ePRO, etc)• Monitoring, DM, Stats, Safety, Study Management, Programming

6. Many data sources and many target destinations of the data• How do the data travel through the systems, from a CRO to the Sponsor, from Central Lab to CDMS

7. Merz approached Cyntegrity to develop a course8. Cyntegrity built a whole course set on RBQM using the 'belt' approach

Slide 7

Development of the training courses

1. Training required for the implementation of RBQM2. Basic training for all affected parties, preferably on-line, self paced, and available for new-comers3. To convince organization, develop training on a real, completed study including

• Protocol review and risk identification• Potential mitigation actions• Definition of key risk indicators• Analyses of the available data• Conclusions and recommendations for a new study

4. Develop a training for in-house RBQM experts (Black belt course)5. Deliver a training for senior management (Executive belt course) to ensure broad acceptance of the

approach across all affected functions in the organization

6. Why was Merz in need for such RBQM training?

Slide 8

Development of the training courses

Why was Merz in need of such a training course?

❖ Regulatory requirement (DIN EN ISO 14155, 2011, FDA Guidance for Industry, 2013, EMA Reflection paper, 2013,ICH GCP E6 (R2) Addendum, 2016)❖ e.g. ISO 14155 offers the options of remote monitoring and reduced SDV

❖ Control and improvement of Subject Safety and Data Quality❖ Best and better use of Sponsor resources

❖ All team members / different functions need to understand the basic idea of RBQM for efficient implementation and the need for regulatory compliance

Slide 9

Delivery of the white belt training

1. Online, self-paced, using the guidance documents ICH E6 R2 as the basis, for almost everybody involved in drug development• Explaining the paragraphs of the guideline

2. Theoretical background• What are KRIs and metrics• What are thresholds and where do they originate from• What can be changed during a study and what has to be kept stable

3. Practical, real life examples as video clips• Using the guideline and connecting the aspects with examples from studies• Explaining how the implementation of RBQM could have prevented certain risks to jeopardize an

entire study• Highlighting how predictions could have avoided crunches towards the end of a study• Addressing the question whether artificial intelligence could eliminate human intervention and

thinking about the results4. Quiz, multiple choice, at the end

Slide 10

Delivery of the white belt training

Slide 11

Merz´ perception of the white belt training

❖ Colleagues entered the white belt training course with very different experiencein risk monitoring and mitigations

❖ The white belt course gave the chance to bring everyone to the same level❖ Take-home messages from that course were discussed in all study teams:

❖ How can we implement this? Can we implement this in running trials?❖ ...but I also want to see this data-point absolutely...!❖ ...but if I don´t review 100%, how do I know the data is correct?❖ Does this realy save ressource on my end or for the company?❖ Are our studies always high risk?❖ ...

Delivery of the white belt training

Slide 12

Challenges of a study conducted in China,in preparation for the green belt trainingChina is a very challenging study environment with ever changing regulatory requirements garnished with some cultural differences

Hierachical thinking

Power and Responsibilityof individuals in key positions

Formalistic requirements

Language and Translation

❖ Medical History❖ HGRAC❖ ...

皮下 vs. 真皮下„Guanxi“ 关系

Slide 13

A medical device study to talk about...

❖ Aesthetic study with approved injectable filler❖ Split face and active comparator design❖ > 150 subjects enrolled at 4 sites in mainland China❖ 12 months study duration / follow-up ❖ Live rating on aesthetic scale by qualified evaluator❖ Optional touch-up treatment

Slide 14

Challenges of a study conducted in China,in preparation for the green belt training

A medical device study to talk about...

❖ Aesthetic study with approved injectable filler❖ Split face and active comparator design❖ > 150 subjects enrolled at 4 sites in mainland China❖ 12 months study duration / follow-up ❖ Live rating on aesthetic scale by qualified evaluator❖ Optional touch-up treatment

...and what did we get?

❖ Many lessons learned ❖ Inconclusive outcome❖ The chance to make it better

Understanding of:❖ Site-effects❖ Number of protocol deviations❖ Injection-procedures❖ Data-point quality❖ Use of scales

Slide 15

Challenges of a study conducted in China,in preparation for the green belt training

Challenges of the Chinese Study, in preparation for the green belt trainingCould RBQM implementation have identified a trend early in the study?

First, obvious questions were asked: o How to generate robust and consistent data in a challenging study environment?o What exactly led to inconclusive results in this study?

❖ Various sources of information were available, e.g.:

Monitoring & Audit Reports Final data set Merz „China“-experience „Outsiders“ unbiased view

How to improve study design and processes?

Slide 16

Challenges of the Chinese Study, in preparation for the green belt trainingCould RBQM implementation have identified a trend early in the study?

First, obvious questions were asked: o How to generate robust and consistent data in a challenging study environment?o What exactly led to inconclusive results in this study?

❖ Various sources of information were available, e.g.:

Second, the idea was born to include Cyntegrity and their experience on RBQM implementation into retrospective analysis of this study!

Monitoring & Audit Reports Final data set Merz „China“-experience „Outsiders“ unbiased view

How to improve study design and processes?

Slide 17

Review of the protocol, development ofrisks and KRIs / metrics1. In preparation for the green belt course, Cyntegrity only got to know about an ‘inconclusive’ outcome of

the study2. Cyntegrity to figure out the root cause of the outcome3. Cyntegrity reviewed the protocol and got access to the data

• Protocol review done many times and usually revealed >70% of the risks• Randomization• Stratification• In- and Exclusion criteria• Safety management / SAE reporting / Under-reporting of AEs / SAEs / missed SAEs• Timelines (enrolment, data entry, response to queries)

4. Identified the risks, defined the appropriate KRIs and thresholds, and linked them with the risks

Slide 18

Review of the protocol, development ofrisks and KRIs / metrics5. Analyzed the study retrospectively

• Enrolment• Identified risks from the protocol• Centralized Statistical Monitoring (distributions, outliers, patterns of data, Benford’s law etc)

6. Listed and prioritized critical areas in the analyses7. Agreed upon the risks and the analysis aspects to be presented to Merz

Slide 19

Results of the analysis and developmentof the green belt courseCyntegrity identified various reasons why study results were inconclusive,for example:

❖ Unexpected injection volume ratio between verum and comparator❖ High variability of injected volumes (intra/inter site)❖ Diversity in touch-up treatments❖ Amount of protocol deviations related to specific reasons❖ Dominance of enrollment by one site❖ Footprints of data duplication and data invention

Reason Check of Hypothesis Corrective Action Preventive Action(s)

Injection Volume Significant impact on data if outliers are

removed

Further check volume-data on

site-level

▪ Define volume-range in the protocol▪ Train on expected injection volume▪ Close monitoring of injected volumes

from the beginning

Slide 20

Results of the analysis and developmentof the green belt courseCyntegrity identified various reasons why study results were inconclusive,for example: High variability of injected volumes on site-level

Vo

lum

e p

er S

ub

ject

Month 1 Month 2 Month 3 Month 4

Slide 21

Results of the analysis and developmentof the green belt courseCyntegrity identified various reasons why study results were inconclusive,for example: High variability of injected volumes on site-level

Question 1:When would early monitoring of injection volume have revealeda potential issue?

Question 2:How can volume differences at each of the sites be detected?

Question 3:How closely should this parameter be monitored?

Vo

lum

e p

er S

ub

ject

Month 1 Month 2 Month 3 Month 4

Slide 22

Results of the analysis and developmentof the green belt courseCyntegrity identified various reasons why study results were inconclusive,for example: Footprints of data duplication and data invention

❖ Routinly, data quality and integrity is checked by Merz for all endpoints

xxxx

yyyy

xxxx

xxxx

❖ Unlikely that subjects have the same vital signs...

❖ ...but study specifics need to be understood

Slide 23

Results of the analysis and development of the green belt courseCyntegrity identified various reasons why study results were inconclusive,for example: Footprints of data duplication and data invention

❖ Routinly, data quality and integrity is checked by Merz for all endpoints

xxxx

xxxx

❖ ...but study specifics need to be understood

Question 4:How to discover such data-issuesand what do they tell us?

Question 5:What should be the consequence if such duplicates are discovered?

Slide 24

xxxx

yyyy

❖ Unlikely that subjects have the same vital signs...

Merz ApproachCyntegrity Approach

Results of the analysis and development of the green belt course

Injection Volume

Cyntegrity identified various reasons why study results were inconclusive

❖ Findings were discussed in detail during green belt course with Merz❖ High overlap of Cyntegrity’s and Merz’ findings on what contributed to the issues in the study

Touch-up ratio

Injection Volume ratio

Enrollment dominance

others

Data duplicates

Visualization of inter- and intra site-comparison

The „effect“ of an unknown product

SAE reporting rate

othersothers

Slide 25

Speed ofrecruitment

Results of the analysis and development of the green belt course1. Identification of well-working KRIs > several becoming a gold-standard since applicable across the board

• Enrolment• Under-reporting / Over-reporting of AEs and SAEs• Site differences• Data entry timeliness• Response to queries

2. Green belt course development with access to / analyses of existing data proved to be a good approach for a wider training• Sponsor company may already have an idea of the issues in a study• Some analysis results may come as a surprise• In particular the analysis of ‘by when could we have known’ (predictive analysis) creates an ‘ups’

responseThe outcome usually is: Now I understand the benefits of using RBQM

Slide 26

Black belt course, process changes

1. New processes require change management, mainly covered by black belt course2. Changes in processes and long-trained behaviors (SDV, checking everything, losing focus of the

important aspects)3. Change in incentives for monitors (bonus for few queries from data management)4. Change wrt addressing potential issues prior to the start of a study

• Joint review of a new protocol• Step-by-step walk through the different process steps• Review of the existing SOPs and potential identification of new ones / update of existing ones• Agreement on KRIs and QTLs• Brain-storming on mitigation actions

5. Train specific situations and how to handle them• Outlier interpretation• Full understanding of the patient population / therapeutic area• Differences between / amongst labs used• Many drop-outs / lost to follow up / informed consent withdrawn cases

Slide 27

Black belt course, process changes

6. Who analyzes the risks in future ?• Central Monitor? Clinical Leader? Both?• So who is a Central Monitor?

• CRA?• DM?• Clinical Analyst?• Medical Reviewer?• Risk Manager?• Does the CRA have no role in managing risk any more?

Slide 28

Black belt course, change management

Follow up activities within Merz teams

❖ Results of the retrospective Cyntegrity analysis were discussed within the Merz team(s):

❖ Discussion on necessity and feasibility of RBQM could be re-ignited❖ RBQM-approach will cost resources at the beginning...but save them at the end❖ RBQM-approach increases the chance to generate conclusive data❖ Development and implementation of Key Risk Indicators (KRIs) and Key

Performance indicators (KPIs) as a set agenda item in study teams

Having (visualized) data of a real study facilitated the individual acceptance and understanding process!

Reminder: It is good if Sponsor team understands and embraces RBQM-principles... but don't forget about the clinical CRO...

Executive Belt Course

1. Rationale for it, i.e. getting backing by senior management and justifying• Investment• Short decrease in performance• Final review of Merz report• Outsourcing guidance for RBQM capabilities• Calculating ROI RBQM investment• Success factors RBQM implementation• Quality by Design under the RBQM horizon• Managing organizational change

2. Familiarization with the long term benefits3. Return on investment calculation (Module provided by Cyntegrity)

Slide 30

Conclusion

❖ Training is important, should be generic on one hand but also very specific on the other hand

❖ RBQM understanding is important not only on sponsor-level

❖ Quality by design: Development of corrective actions for a new study protocol

❖ Practical examples helped Merz to fully grasp rationale for RBQM implementation

❖ Change management is an important component of training

❖ Acceptance of RBQM-demand on resource while having an overall benefit needs time

Slide 31

THANK YOU FOR YOUR

ATTENTION

Slide 32