Fulfill Requirements and Ensure the Reliability of Clinical Trial Results Using Risk-Based Monitoring Methods

Nathaniel Katz, MD

Chief Scientific Officer

WCG Analgesic Solutions

Why did my trial fail?

ME

Your

Trial

CONSEQUENCES OF FAILED TRIALS

• Need to repeat studies

• Massive waste of resources

• Premature death of programs

• Exposure of human subjects to risk

without gaining scientific knowledge

What is reliability?

JCGM, 2012

Reliability How close repeated

measurements are to each other

Accuracy How close a measurement is to

the true value

Are clinical trials reliable?

Zhang W et al, OA Cartilage, 2010; Vinik AI et al, Pain, 2007

CV=70%

Why are clinical trials unreliable?

• Poor reliability of subjective measures

• Daily life events perturb patients’ symptom intensity

• Placebo response

• Medication adherence

• Rescue medication

• Baseline score inflation

• Too many sites

• Inter-site variability

• Geographic variability

Neutral expectation High expectation

Katz, 2005; Harris, 2005; Wise, 2009; Dworkin, 2012, 2014; Treister, 2019

What is the goal?

Performance

issues that

impact accuracy

or reliability of

study results

Anomalous data

Non-anomalous

data

No impact

No impact

Propagated BP

measurements

in oncology trial

Medication

non-

adherence

Just monitoring safety, protocol violations, and

operational variables

DOES NOT ADDRESS RELIABILITY OF STUDY

RESULTS

Reliability of study results mentioned 11 times in ICH E6(R2)

The Risk-Based Quality Management Cycle: Supporting Reliability

2. Define a surveillance method and control limits

3. Establish satisfactory

initial performance

4. Root cause

analysis of deviations

5. Intervention and followup

6. Follow signal to

resolution

1. Identify variables that

impact primary endpoint

FDA, 2013; ICH E6(R2), 2016

Each element can be

evidence-based

Pick variables (KRIs) that predict the primary endpoint

Example:

• Variability in pain intensity was thought to be worth

monitoring because extremes of variability might lead

to unreliability of study results

• We performed a retrospective analysis to see if this

was true

• Patients with extremely high or extremely low variability

of pain intensity indeed failed to demonstrate efficacy

of drug vs. placebo

• Thus “variability of pain intensity” is a validated key risk

indicator for reliability of study results

• This process has been repeated to create a library of

“validated” key risk indicators for reliability of study

results

Setting alert thresholds (or control limits) that predict study failure

1. Sentinel events

• Single events that have such a high likelihood of predicting failure on the

primary endpoint that each event merits further assessment

2. Fixed validated thresholds

• Cutoffs on key risk indicators that have been demonstrated to predict

failure of the primary endpoint

• Generally accepted fixed performance thresholds (e.g. 80% medication

adherence)

3. Statistical process control (SPC) limits

• Limits imposed by control charts chosen to monitor for statistically

meaningful transgression of a key variable

• Same variable changes over time

• Differences between patient/site/region vs. others

Validated Fixed Thresholds: Daily Pain Variability

TRAINED UNTRAINED

Patients whose daily pain variability is higher than 1.2,

or lower than 0.5, do not separate drug from placebo

When you don’t have validated fixed thresholds, use SPC

Shewhart WA, 1939

UCL

LCL

Case 1: Fixed thresholdE-Diary Compliance <85% per week

Case 2: Statistical thresholdRescue Medication Consumption

Case 3: Statistical ThresholdStaircase-evoked pain: key efficacy endpoint

0

10

20

30

40

50

60

70

80

14011 14012 14013 14014 14015

Per

cen

t

Sites

StEPP Results by Site

Better No Change Worse

0

10

20

30

40

50

60

70

14013 14013after training

StEPP Results Before and After Retraining

Better No Change Worse

Conclusions

• RBQM methods can be used to support reliability of study

results, which is a regulatory requirement

• Each step of the RBQM cycle must be based on factors

validated to impact the primary endpoint of the trial

• Success is maximized by combining statistical,

psychometric, and clinical insights

Thank You

nkatz@wcgclinical.com