illustrating uncertainty in extrapolating evidence for cost-effectiveness modelling

The CLAHRC Yorkshire

& Humber

Illustrating uncertainty in

extrapolating evidence for cost-

effectiveness modelling

Laura Bojke


& Humber

Extrapolation project team

• Stephen Palmer, Andrea Manca, Ronan

Mahon, Miqdad Asaira (University of York)

• Alan Brennan (PI), John Stevens, Nick

Latimer, Paul Tappenden, Suzy Paisley,

Kate Ren (University of Sheffield)

• Keith Abrams (University of Leicester)

• Chris Jackson (University of Cambridge)


& Humber

• The need for extrapolation

• Extrapolation methods – Not extrapolation from surrogate outcomes

• Uncertainty in extrapolation

• Approaches to dealing with uncertainty in

extrapolation

– Examples

• Areas for further research

Structure of presentation


& Humber

“estimating beyond the original observation range”

• An appropriate time horizon for evaluation

– All (incremental) positive/negative effects observed

– Patients lifetime when there are mortality effects

• Evidence base falls short of this - censoring

– High costs of research

– Loss to follow up

– Early market entry

• Modelling to extrapolate short term outcomes

– Involves assumptions and may be data sparse

Impetus for extrapolation


& Humber

Evidence gap resulting from time

horizon mismatch

Just extrapolation? Temporal Uncertainty in Cost-effectiveness Decision Models. Mahon

R, Manca A, Bojke L, Palmer, S Jackson C, et al.


& Humber

When does the mismatch

matter? • Not just the time difference between observed

and unobserved

• Relates to data maturity

– Number of people that have experienced the event of

interest

• Earlier outcomes may be of more value

– Proportion of costs & QALYs attributable to observed

period

– Discounting


& Humber

Mature data

Mean overall survival gain with aflibercept plus FOLFIRI vs placebo plus FOLFIRI in

patients with previously treated metastatic colorectal cancer. F Joulain, I Proskorovsky,

C Allegra, J Tabernero, M Hoyle, S U Iqbal and E Van Cutsem.


& Humber

Immature data


& Humber

Extrapolating TTE parameters

• Assumptions about how trends will continue

– Progression free or overall survival in cancer trials

• Estimation involves: (1) modelling the observed

data (Kaplan-Meier); (2) modelling the

unobserved data period.

– Statistical models (often parametric) are fitted to

observed data

• Choice of model informed by goodness of fit parameters

– Fitted model is used to extrapolate the un-observed

period to determine the TTE

– Assume hazard trends will continue


& Humber

Choice of distribution

• Choice of parametric distribution to fit:

– Exponential, Weibull, log-normal etc.

• Some advocate use of exponential as the default

– Proportional hazards, constant hazards

• Different assumptions/ survival estimates

• Data limitations

– IPD or aggregate (constant hazard)


& Humber

Extrapolating non-TTE parameters

• Repeated measurement of individuals over time

– Genuinely discrete or genuinely continuous

• Similar principles to TTE

– Set of observations on one subject tends to be inter-

correlated

• Driven by data availability

– Aggregate = discretised (Markov models)

– IPD = patient experienced models, continuous

outcomes – regression, e.g. risk equations


& Humber

Extrapolating resources

use/costs & utilities

• Typically models employ a simplistic approach to

extrapolation of costs and utilities

– Homogeneous w.r.t both time and patients'

characteristics

– Follow the dynamics of associated TTE parameters

• Resource use/costs and utilities may be more

nuanced

– Adaptation to health state

– Uncertainty may not resolve over time/with further

evidence


& Humber

Uncertainty in extrapolation

• Others have termed this ‘temporal’ uncertainty

• Extent of uncertainty is difficult to determine • Uncertainty cannot be resolved (now), because we

cannot observe the future

• Efforts should be made to characterise any

uncertainty − Accurate estimates of long-term cost-effectiveness

− Better adoption decisions

− Assess the need for further research

− Input into the design of further research

− Delay decisions?


& Humber

Does uncertainty in extrapolation

matter?

Just extrapolation? Temporal Uncertainty in Cost-effectiveness Decision

Models. Mahon R, Manca A, Bojke L, Palmer, S Jackson C, et al.


& Humber

“primary source of extrapolation uncertainty

in decision model results is the choice of

survival projection function rather than

sampling variation”

Bagust & Beale, 2014 in response to Latimer,

2013


& Humber

How should uncertainty be

assessed?

• NICE recommends that any extrapolation should

be assessed by ‘‘both clinical and biological

plausibility of the inferred outcome as well as its

coherence with external data sources’’

– Does not suggest specific methods to do this

• Latimer, 2013 recommends an assessment of

plausibility

– Concerns may be context specific


& Humber

Model selection

• Assessment of goodness-of-fit for observed

– Log-cumulative hazard, residuals plots, AIC/BIC etc

– Suggested the focus should be on fit for stable portion

of data

• Lots of things that prevent good model fit

– More flexible model forms, generalised gamma

• A model that fits the sample data well may not

provide good long-term predictions

• Clinical plausibility

• Causally known associations


& Humber

Just scenarios?

• Where there is no evidence to form an opinion

about the plausibility of assumptions/do not want

to rely on this

– Deterministic sensitivity analysis

– Illustrate how conclusions may change if the evidence

were to change and possibly inform the need for

further validation

– Not easy for decision-makers to interpret

• May implicitly average different scenarios


& Humber

Model averaging

• Explore uncertainty in the true underlying

distribution for survival extrapolation

– Bayesian view of model averaging

– Likelihood used to measure fit of a model to observed

data (plausibility)

– Unlikely scenarios: down-weighted/excluded

– Recommended that models should ‘stake out the

corners in the model space’

• Implications for extrapolation uncertainty

– Model-averaged inferences will generally have

greater uncertainty


& Humber

Discrepancy parameters • Where there are many sources of uncertainty in a

model

– Strong, et al method to identify the most important

sources of structural uncertainty

– Discrepancy parameters are added to intermediate

outputs of the model, rather than model inputs.

• e.g. the years of life spent in a health state could be replaced

by LY + delta_LY, where the discrepancy parameter delta_LY

is represented as a probability distribution based on experts

priors

– PSA to quantify which of the discrepancy parameters

are associated with the greatest variations in net benefit


& Humber

Stability of survival extrapolation

• Negrin, et al, 2016 explore uncertainty about the

stability of extrapolated parameters over time

– Explore past behaviour of distributions

• Built intermediate data sets (4, 5, 6 and 7 year)

– Numerous survival models fitted to 8 year data for 2

hip prosthesis types separately (Charley, Spectron)

• Two additional distributions to reflect stability over past

behaviour (optimistic and sceptical)

• BMA(1) using BIC (1/6 weight to 6 distributions)

• BMA(2) questions the value of BIC to select or average

across models (1/3 weight assigned to optimistic, sceptical,

others)


& Humber

Negrin, Nam, Briggs. Bayesian solutions for handling uncertainty in

survival extrapolation. Medical Decision Making. 2016 forthcoming


& Humber

Using external sources of data

• Sources: administrative data, disease registries,

cohort studies

– Longitudinal data over an appropriate time horizon

with sufficient follow up points

– Cross sectional data with sufficient coverage of

patients

• Assumed relationship between internal and

external data (baseline and treatments)

• Combine using Bayesian estimation


& Humber

Elicitation

• Use formally elicited priors

• Represents current level of knowledge regarding

the uncertainty of interest

– Expressed quantitatively

• Uncertainty about how this can be implemented:

– Alternative survivor function?

– Weights for survivor functions, e.g. TIDI

– Synthesis with internal data?

– Could elicit at multiple time points – how?


& Humber

Elicitation (2)

• Complexity of elicitation task

– Who are the relevant experts

– How can we weight experts according to accuracy

– Complex parameters difficult to elicit

– Time requirements

– Synthesis

• Eliciting uncertainty about priors

– Needed to fully characterise uncertainty and inform

long term follow up


& Humber

Case study: Extrapolation project

• Cost-effectiveness of biologics for PsA

– Probabilistic cohort model

– Time horizon = 40 years (3-month cycles)

– Initial response using PsARC @ 3-months

– Associated HAQ gain

– Assumptions that biologics halt progression

– Constant rate of progression on biologics (mean = 0)

• Different ways in which this data can be utilised in the model


& Humber

1) Initial HAQ gain

due to treatment

2)Long term HAQ

trajectory while on

treatment

3)Rebound of HAQ

once withdrawn

from treatment

4)Long term HAQ

trajectory post

withdrawal from

treatment

0

0.5

1

1.5

2

2.5

3

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

HA

Q S

core

Time (Years)

2. HAQ on treatment

1. Initial

HAQ

Gain

Natural History

3. HAQ

Rebound

4. HAQ Post-

Withdrawal

Treatment Arm

Issues regarding HAQ

trajectory


& Humber

Impact of uncertainties in PsA

model

Cumulative NMB over time base case 3-month HAQ change of 0.

Within

trial period

Breakeven

point

etanercept


& Humber

Exploring long term HAQ trajectory

for treatment responders

• Key driver for cost-effectiveness.

• Assumed mean of 0 + expert elicitation exercise

(SD)

• But don’t know:

– Constant HAQ gradient?

– Independent HAQ gradient at each 3-month model

cycle?

• Limited by lack of external data.


& Humber

Impact of different HAQ trajectories


& Humber

Constant HAQ gradient over

time • Randomly draw HAQ change from a normal

distribution (mean = 0 and SD = 0.022)

• Apply this as the constant 3-month HAQ score

assuming perfect serial correlation.

• Process is repeated for each iteration of the

model


& Humber

Cumulative NMB over time – constant HAQ gradient over time.

Breakeven

point

etanercept

Within

trial period


& Humber

Independent HAQ gradient at

each cycle • Previously perfectly serially correlated HAQ

trajectories

• Independently sampled HAQ scores for each

cycle

• Repeatedly randomly draw HAQ changes from a

normal distribution with mean 0 and standard

deviation 0.022


& Humber

Cumulative NMB over time - independent HAQ gradient at each 3-month model cycle.

Breakeven

point

etanercept


& Humber

Comparing the two scenarios

Constant HAQ

gradient over

time

Independent

HAQ gradient

at each cycle

Palliative Care Probability

cost-effective

(£20,000 per

QALY)

0.30 0.38

Etanercept 0.31 0.39

Infliximab 0.04 0.00

Adalimumab 0.18 0.17

Golimumab 0.18 0.07

EVPI (population) £768 mil £378 mil


& Humber

Conclusions • Models may be well characterised over the

observed period, but there may be considerable

uncertainty regarding behaviour over time.

• None of the methods described can fully

describe the extent of extrapolation uncertainty.

– Statistical models for extrapolation cannot be fully

validated.

• Focus on which influence conclusions: adoption

decision, decision uncertainty and VOI.

– Degree of effort should be proportional to the

influence of uncertainty.


& Humber

Areas for further research • Ability to validate for ‘crucial’ uncertainties:

– Internal/external validity

– Use of external evidence/experts priors

• How to generate experts priors for extrapolation

– Which parameters to elicit

• Link to OIR/delayed decisions

– Break even point important

• Discounting out the problem


& Humber

(some) References • Methods of Extrapolating RCT Evidence for Use in Economic Evaluation

Models, MRC report.

• Extrapolating Survival from Randomized Trials Using External Data: A

Review of Methods. Jackson C, et al. Medical Decision Making, 2016.

• Survival Analysis and Extrapolation: Modeling of Time-to-Event Clinical Trial

Data for Economic Evaluation: An Alternative Approach. Medical Decision

Making. Bagust A & Beale S. 2014.

• Davies C, Briggs A, Lorgelly P, et al. The “hazards” of extrapolating survival

curves. Medical Decision Making. 2013; 33 (3).

• Negrin, Nam, Briggs. Bayesian solutions for handling uncertainty in survival

extrapolation. Medical Decision Making. 2016 forthcoming.

• NICE DSU Technical support document 14: Survival analysis for economic

evaluations alongside clinical trials: Extrapolation with patient-level data.

Latimer N. 2011.

illustrating uncertainty in extrapolating evidence for cost-effectiveness modelling

Health & Medicine