exl clinical quality oversight forum
DESCRIPTION
Managing Risk and Ensuring Effective Oversight in Outsourced Clinical TrialsTRANSCRIPT
Managing Risk and Ensuring Effective
Oversight in Outsourced Clinical Trials
Clinical Quality Oversight Forum
2
Panel Members
Mitch Katz Executive Director Medical Research Operations, PURDUE PHARMA LP
David Marks Executive Vice President, Quality Management, RESEARCH PHARMACEUTICAL SERVICES, INC. (RPS)
David Nickerson Senior Director, Clinical Quality Management, PFIZER
Clinical Quality Oversight Forum
3
Today’s Topics
• Background: Challenges in quality management
• Part One: Proactive quality management and effective oversight
• Part Two: Risk sharing • Part Three: Risk assessment and risk management
Clinical Quality Oversight Forum
Background and Overview
5
Mission: To accelerate the development of best practice approaches and industry standards for the proactive quality
management of outsourced trials Lead Sponsors and CROs in optimizing their approaches to proactive quality management with an emphasis on bringing them into greater alignment.
Avoca Quality Consortium
6
Avoca Quality Consortium Pharma/Biotech Industry Participation to Date
**Roche is also Consortium Members
7
Avoca Quality Consortium CRO Industry Participation
8
Are there quality issues with outsourced
clinical trials?
Challenges in Quality Management
9
Challenges in Quality Management Disconnect between sponsors and CROs regarding perceptions of
quality delivered by CROs
Sponsor: Overall, how satisfied are you with the quality delivered by your
Clinical Service Providers (including but not limited to CROs)?
2%
45%
30%
20%
3% 17%
70%
10% 3%
CRO: Overall, how satisfied are you with the quality that your company
delivers for its sponsors?
N=245* N=200*
5%
13%
8%
8%
15%
15%
40%
10%
3%
11%
27%
95%
100%
67%
77%
83%
64%
58%
20%
60%
71%
57%
66%
60%
20%
15%
15%
20%
20%
20%
14%
23%
16%
10%
8%
6%
7%
20%
10%
14%
17%
7%
2%
0% 20% 40% 60% 80% 100%
A
B
C
D
E
F
G
H
I
J
K
-All-
Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied
Somewhat dissatisfied Very dissatisfied
*2013 Avoca Quality Consortium Assessment
10
Overview: Challenges in Quality Management
“Reactive” vs “proactive” approaches to quality management: a look at whether quality issues exist
Areas with relatively high satisfaction:
● Compliance with SOPs and other written procedures
● Data quality and integrity
● Audit plans and execution
Areas with relatively high dissatisfaction:
● Oversight of third party vendors
● Governance of quality
● Communications surrounding quality
● Availability of quality personnel for projects
● Efficiency/timeliness in achieving clean data
● Adherence to monitoring plan
3.3 – 3.5 (on scale of 1 to 5)
2.9 – 3.2 (on scale of 1 to 5)
11
Quality by Task: Comparison of Sponsor and CRO Results (2012 Avoca Quality Consortium Assessment)
Sponsors: On average, how satisfied are you with
your clinical service providers’…
CROs: On average, how would you rate your
company's…
Compliance with SOPs and other written procedures 3.5 3.7
Data quality and integrity 3.4 3.8
Audit plans and execution 3.3 3.6
Adequacy of Monitoring Plan 3.2 3.5
Document control 3.2 3.3
Adherence to Monitoring Plan 3.2 3.4
Efficiency/timeliness in achieving clean data 3.2 3.5
Monitoring of protocol compliance 3.1 3.7
Availability of quality personnel for my projects 3.1 3.2
Management of protocol compliance 3.1 3.6
Staff training 3.0 3.3
Site training 3.0 3.4
Governance of quality (e.g. accountability, management system, leadership support) 2.9 3.4
Oversight of third party vendors (e.g. labs, IVRS vendors, etc.) 2.9 3.3
Communications surrounding quality 2.9 3.4
12
Research Finding:
Strong correlation between taking
proactive approaches to quality management
and satisfaction with the quality of deliverables.
Sponsor Proactivity Drives Satisfaction
12
0
1
2
3
4
5
0 1 2 3 4 5
Sa*sfac*on
with
the "quality"
delivered
by clinical se
rvice providers
The (sponsor) teams with which I work iden*fy all processes/deliverables for which quality expecta*ons of clinical service
providers need to be set
0
1
2
3
4
5
0 1 2 3 4 5
Sati
sfac
tion
wit
h th
e "q
ualit
y"
deliv
ered
by
clin
ical
ser
vice
pr
ovid
ers
The (sponsor) teams with which I work effectively communicate their expectations regarding quality to their
clinical service providers
13
Clarity of expectations: scores range between 2.4 on the low end to 4.8 on the high end (based on a scale of 1 to 5)
Avoca Relationship Assessments
Clarity of Roles and Expectations vs. Sponsor’s Overall Satisfaction: Across Strategic Partnerships*
*p<.05
Statistically significant correlation across strategic
partnerships:
Clarity of expectations, roles, and responsibilities, and
overall satisfaction with the work performed.
14
Data suggests room for improvement in proactive approaches
Many sponsors companies have reported difficulty in operationalizing approaches for
identifying and communicating expectations. The (sponsor) teams with which I work effectively communicate their
expectations regarding quality to their clinical service providers.
15
Operationalizing Proactive Communication
16
Operationalizing Proactive Communication
17
Operationalizing Proactive Communication
Construction Industry ü Has operationalized
proactive, systematic and integrated communication strategies
Although buildings are more
complex and sophisticated, building failure is exceedingly rare:
(.00002 percent/year)
“The biggest cause of serious failure in this business is a failure of communication.”
Finbarr O’Sullivan, Project Executive, Moriarty & Associates, Project Executive, Russia Wharf office building on Boston Waterfront
18
Leuchten’s Art Studio/Garage
Subset of problems to date: ü Foundation footings
poured based on old set of drawings
ü Scheduling snafu’s between builder and roofer
ü Failed inspection due to issues with electrical and plumbing
Root causes of issues: Problems
with proactive communication,
collaboration and clarity of expectations
19
Themes to explore today
• Proactive communication • Collaboration • Clarity of expectations
...and impact on quality of outsourced clinical trials
Clinical Quality Oversight Forum
Part One:
Proactive quality management and effective oversight
21
• Research Topics – Micromanagement – Documentation around roles and responsibilities – Overseeing work at a strategic level rather than tactical – Definition of roles to minimize duplication of effort – Efficiency in use of resources – Use of technology to promote efficient oversight practices
Proactive Quality Management and Effective Oversight `
22
What do you think?
Polling question for audience: Sponsors: My company is efficient in the use of resources applied to outsourced projects. CROs: Sponsors that I work with are efficient in the use of resources applied to outsourced projects. A. Strongly agree B. Agree C. Neither agree nor disagree D. Disagree E. Strongly disagree
23
21%
13%
5%
6%
6%
2%
30%
42%
34%
34%
28%
18%
34%
21%
34%
34%
29%
31%
11%
20%
18%
18%
32%
34%
4%
5%
8%
8%
6%
15%
The operational teams involved in overseeing our CROs consistently micromanage to ensure that the quality of
deliverables meets expectations.
My company clearly documents CRO oversight practices, roles, and responsibilities.
I personally have received high-quality deliverables from CROs, even when overseeing their work at a "strategic"
rather than detailed level.
My company clearly defines the roles of internal and CRO staff so as to minimize duplication of effort.
My company is efficient in the use of resources applied to the oversight of outsourced trials.
The technology systems used by my company and its CRO partners promote efficient oversight practices.
Sponsor Perceptions of Oversight Practices Mean N
3.5 219
3.4 219
3.1 203
3.1 218
2.9 218
2.6 215
Perceptions of Oversight Practices Consortium Sponsor Findings
5 Strongly Agree
4 3 Neither Agree nor Disagree
2 1 Strongly Disagree
24
8%
5%
3%
4%
3%
4%
47%
34%
30%
14%
23%
22%
32%
34%
33%
44%
35%
29%
11%
23%
25%
32%
31%
29%
3%
5%
10%
5%
8%
16%
The sponsors with which I work consistently micromanage to ensure that the quality of deliverables meets
expectations.
The sponsors with which I work clearly document CRO oversight practices, roles, and responsibilities.
The technology systems used by my company and its sponsor partners promote efficient oversight practices.
The sponsors with which I work are efficient in the use of resources applied to the oversight of outsourced trials.
The sponsors with which I work clearly define the roles of internal and CRO staff so as to minimize duplication of
effort. There is a direct positive relationship between the
intensity of sponsor oversight and the quality of CRO deliverables.
CRO Perceptions of Sponsor Oversight Practices Mean N
3.5 157
3.1 153
2.9 159
2.8 152
2.8 149
2.7 158
Perceptions of Oversight Practices CRO Findings
5 Strongly Agree
4 3 Neither Agree nor Disagree
2 1 Strongly Disagree
25
Statements Evaluated by Sponsors Sponsor Average
CRO Average Statements Evaluated by CROs
The operational teams involved in overseeing our CROs consistently micromanage to ensure that the quality of
deliverables meets expectations. 3.5 3.5
The sponsors with which I work consistently micromanage to ensure that the quality of deliverables meets expectations.
My company clearly documents CRO oversight practices, roles, and responsibilities. 3.4 3.1 The sponsors with which I work clearly document CRO
oversight practices, roles, and responsibilities.
My company clearly defines the roles of internal and CRO staff so as to minimize duplication of effort. 3.1 2.8 The sponsors with which I work clearly define the roles of
internal and CRO staff to minimize duplication of effort.
I personally have received high-quality deliverables from CROs, even when overseeing their work at a "strategic"
rather than detailed level. 3.1 2.7
There is a direct positive relationship between the intensity of sponsor oversight and the quality of CRO deliverables.
My company is efficient in the use of resources applied to the oversight of outsourced trials. 2.9 2.8 The sponsors with which I work are efficient in the use of
resources applied to the oversight of outsourced trials.
The technology systems used by my company and its CRO partners promote efficient oversight practices. 2.6 2.9 The technology systems used by my company and its
sponsor partners promote efficient oversight practices.
N≥203 N≥149
Perceptions of Oversight Practices 5 = Strongly Agree, 1 = Strongly Disagree
Perceptions of Oversight Practices Sponsor and CRO Findings
26
Best Practices in Quality Management
Operationalizing approaches to
setting and communicating
expectations drives consistency and
increases quality
27
Operationalizing Proactive Approaches to Quality Management and Oversight
Governance / Organiza*onal Construct
Technical / Project
Oversight
Processes
Communica*on
Oversight Leadership
Requirements
Metrics / Analy*cs / Technology
Roles / Responsibili*es
Proac*ve Risk Management / Con*ngencies
Effective Oversight
Define
Plan
Lead
Decide
Measure
Deliver
Proactive Quality
Management
28
Verbatim Comments
“[Sponsors don’t give us] ownership of decisions that we should have. This delays our deliverables which Sponsors will hold us accountable for in the end.” [CRO]
“CRO's don't like to problem solve and offer suggestions. They'd rather just do as they are told.” [Sponsor]
“One Sponsor that I work with is new to fully outsource trials, so at the beginning of our working relationship, they micromanaged us. This has slowly relaxed as we showed that we were consistently meeting project deliverables. “ [CRO]
Panel Discussion
Part Two:
Risk-sharing
31
Risk-sharing between Sponsors and Providers
Types of Risk-sharing
● Provider bonuses for achieving milestones and targets
● Provider penalties for missing milestones and targets
● Guarantees of continued work/revenue stream in exchange for provider commitments
● Provider stake in outcome of study
(e.g., company stock)
32
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
Provider penalties for failure to achieve milestones/targets
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange for provider commitments
None of the above
Strategic partnerships/alliances (N=205) Preferred provider relationships (N=194) Transactional relationships (N=217)
Risk-Sharing Models Used by Type of Relationship
Risk-Sharing Model Usage Sponsor Findings
Risk-sharing models are used more commonly in strategic partnerships or preferred provider agreements than in transactional relationships.
33
What is your experience?
Polling question for audience: With which risk-sharing method have you had the most positive experience? A. Provider bonuses for achieving milestones and targets
B. Provider penalties for missing milestones and targets
C. Guarantees of continued work/revenue stream in exchange for provider commitments
D. Provider stake in outcome of study (e.g., company stock)
34
41%
30%
25%
50%
62%
21%
48%
61%
52%
47%
34%
55%
10%
9%
23%
3%
4%
24%
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange for provider commitments
Provider penalties for failure to achieve milestones/targets
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange for provider commitments
Provider penalties for failure to achieve milestones/targets
Experience/Satisfaction with Risk-Sharing Models Used
Primarily Positive
A mix of Positive and Negative
Primarily Negative
Risk-Sharing Satisfaction Sponsor and CRO Findings
N
87
94
104
92
74
95
Spon
sors
CR
Os Consor*um sponsors report more posi*ve experiences with the use of provider bonuses
compared to provider penal*es.
35
41%
30%
25%
50%
62%
21%
48%
61%
52%
47%
34%
55%
10%
9%
23%
3%
4%
24%
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange for provider commitments
Provider penalties for failure to achieve milestones/targets
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange for provider commitments
Provider penalties for failure to achieve milestones/targets
Experience/Satisfaction with Risk-Sharing Models Used
Primarily Positive
A mix of Positive and Negative
Primarily Negative
Risk-Sharing Satisfaction Sponsor and CRO Findings
N
87
94
104
92
74
95
Spon
sors
CR
Os
Similar results are reported by sponsors and CROs in regard to use of provider penalties for failure to achieve milestones/targets.
Panel Discussion
Part Three:
Risk Assessment and Risk Management
38
37%
18%
20%
13%
12%
Usage Frequency of Systematic Risk Assessment Processes
Systematic Risk Assessment Frequency Sponsor and CRO Findings
51-75% of trials
>75% of trials
25-50% of trials
1-24% of trials
Never
52%
21%
13%
13% 1%
Sponsors CROs
N=174 N=135
39
Use of Systematic Risk Assessment
“Please briefly describe the conditions under which a systematic risk assessment process is used.” – Verbatim Responses
● “Not consistent”
● “Don’t know”
● “Often done ‘in silos’”
● Only…
╸ During RFP process
╸ At governance level
╸ Ad hoc
╸ At program level, not study level
╸ For audit strategy
╸ Upon sponsor request
╸ For key clients
╸ When resource limitations are a big concern
● Done, but not necessarily “systematic and rigorous”
╸ Too high level and “standard”
╸ copy and paste from previous studies
╸ no time to think about study-specific nuances
40
Risk Assessment and Risk Management
“We do not consistently approach the work from a risk assessment perspective and we continually put ourselves in the position of being surprised by sponsor questions, demands or comments.” [CRO]
41
Insights from Avoca Data
5 = Very Satisfied, 1 = Very Dissatisfied CONSORTIUM GROUP AVERAGES
Performance Aspect In-house Teams
CRO Partners
CRO Self Assessment
Proactive identification of potential risks 3.5 2.8 3.5
Proactive risk analysis and evaluation 3.3 2.7 3.4
Compilation of risk-related trial information during a trial (observations, trends, etc.) 3.2 2.6 3.2
Frequency of review of risk-related trial information 3.3 2.7 3.3
Rigor of review of risk-related trial information 3.1 2.5 3.1
Communications regarding risk-related trial information 3.3 2.8 3.2
Appropriateness of measures suggested or taken in reaction to risk-related information 3.4 3.0 3.3
Overall performance on risk assessment and management related activities 3.2 2.7 3.3
Low to middling scores suggest
knowledge gaps in key areas related to
risk identification, analysis and management
Performance Satisfaction on Select Risk Assessment Variables
42
Systematic Risk Assessment Results Sponsor and CRO Findings
More efficient use of resources for your company and/or your partner?
Increased quality?
Q: In general, have your risk assessment and management approaches resulted in…
N
204
159
203
159
9%
16%
20%
28%
39%
40%
36%
43%
17%
16%
8%
6%
35%
28%
36%
23%
Sponsors
CROs
Sponsors
CROs
Yes Sometimes No Don’t Know / Too Soon to Tell
43
Systematic Risk Assessment Results Sponsor and CRO Findings
More efficient use of resources for your company and/or your partner?
Increased quality?
Q: In general, have your risk assessment and management approaches resulted in…
N
204
159
203
159
9%
16%
20%
28%
39%
40%
36%
43%
17%
16%
8%
6%
35%
28%
36%
23%
Sponsors
CROs
Sponsors
CROs
Yes Sometimes No Don’t Know / Too Soon to Tell
44
What is your experience?
Polling question for audience: How would you rate your understanding of best practices in risk assessment and management in clinical trials? A. Very strong understanding
B. Good understanding
C. Fair understanding
D. Poor understanding
45
Understanding of Best Practices in Risk Assessment/Management
8%
39% 42%
11% Very strong understanding
Good understanding
Fair understanding
Poor understanding
How would you rate your understanding of best practices in risk assessment and management in clinical trials?
9%
49%
35%
7% Very strong understanding
Good understanding
Fair understanding
Poor understanding
N=207* N=159*
Sponsor: CRO:
*2013 Avoca Quality Consortium Assessment
46
“Team members inability to understand the big picture of what their contribution means to the broader study and what the broader study means to the overall strategy for the product is a big contributor to failed risk management.” [Sponsor]
“We don't proactively manage risk effectively. Instead we micromanage our CROs in the name of sponsor oversight, which is not a good use of anyone's time or resources.” [Sponsor]
Verbatim Comments
Panel Discussion
Thank you!
Patricia Leuchten [email protected]
Mitch Katz
David Marks [email protected]
Dave Nickerson