New Research in Economic Modeling

and Simulation

New Research in Economic Modeling

and Simulation

Greg Samsa PhDGreg Samsa PhD

Organization of the talkOrganization of the talk

3 questionsHow does your work contribute to

economic and comparative effectiveness modeling?

What’s new in economic modeling and simulation?

What’s changing in how these models are applied?

3 questionsHow does your work contribute to

economic and comparative effectiveness modeling?

What’s new in economic modeling and simulation?

What’s changing in how these models are applied?

Question 1Question 1

How does your work contribute to economic and comparative effectiveness modeling?

Short answer: You provide the inputs

How does your work contribute to economic and comparative effectiveness modeling?

Short answer: You provide the inputs

Question 2Question 2

What’s new in economic modeling and simulation?

Short answer: Very complex models are now computationally feasible – the debate is shifting from what is possible to what is desired

What’s new in economic modeling and simulation?

Short answer: Very complex models are now computationally feasible – the debate is shifting from what is possible to what is desired

Question 3Question 3

What’s changing in how these models are applied?

Short answer: Decision makers are starting to take these models more seriously, and to embed them within more general strategies for learning

What’s changing in how these models are applied?

Short answer: Decision makers are starting to take these models more seriously, and to embed them within more general strategies for learning

Background Background

The previous speakers have provided definitions and examples of economic models

I’ll discuss “complex decision and cost-effectiveness models” – sufficiently complex to require simulation to implement

The previous speakers have provided definitions and examples of economic models

I’ll discuss “complex decision and cost-effectiveness models” – sufficiently complex to require simulation to implement

Observation 1Observation 1

A CEA model is a peculiar thing It is a counting machine intended to clarify

trade-offs among things that users value (e.g., survival, quality of life, costs)

Inputs are obtained from different sources – “the model” isn’t something that can be directly observed

The usual principles of validation don’t apply– I’ll discuss what makes a good model later

A CEA model is a peculiar thing It is a counting machine intended to clarify

trade-offs among things that users value (e.g., survival, quality of life, costs)

Inputs are obtained from different sources – “the model” isn’t something that can be directly observed

The usual principles of validation don’t apply– I’ll discuss what makes a good model later

Observation 2Observation 2

A dirty little secret:A CEA model is no stronger than its

weakest linkAdvocates focus on a model’s strengths,

but the weaknesses are also of importance

A dirty little secret:A CEA model is no stronger than its

weakest linkAdvocates focus on a model’s strengths,

but the weaknesses are also of importance

A personal confessionA personal confession

“The Duke Stroke Policy Model combines data from the best sources – natural history from Framingham, costs from national claims, utilities from a large survey developed specifically for this purpose…”

“The Duke Stroke Policy Model combines data from the best sources – natural history from Framingham, costs from national claims, utilities from a large survey developed specifically for this purpose…”

All true, but…All true, but…

Its conclusions depend on how survival, quality of life and costs vary by disability levelThese parameters were derived by

extrapolating from small studies of inconsistent quality

The user is essentially relying on (a) the face validity of the parameter estimates; and (b) sensitivity analyses

Its conclusions depend on how survival, quality of life and costs vary by disability levelThese parameters were derived by

extrapolating from small studies of inconsistent quality

The user is essentially relying on (a) the face validity of the parameter estimates; and (b) sensitivity analyses

N=?N=?

Natural history n=5,000Costs n=500,000Utilities n=1,500

Natural history, costs, utilities by disease state – n is variable (e.g., only 100 hemorrhagic strokes)

Impact of disability level n=20, mostly interview rather than observation

Natural history n=5,000Costs n=500,000Utilities n=1,500

Natural history, costs, utilities by disease state – n is variable (e.g., only 100 hemorrhagic strokes)

Impact of disability level n=20, mostly interview rather than observation

Question 1Question 1

How does your work contribute to economic and cost-effectiveness models?

How does your work contribute to economic and cost-effectiveness models?

Data sources for model inputsData sources for model inputs

Traditional efficacy trialsEffectiveness trialsRegistriesAdministrative dataSurveysObservational studiesLiterature reviews

Traditional efficacy trialsEffectiveness trialsRegistriesAdministrative dataSurveysObservational studiesLiterature reviews

Study planningStudy planning

Design studies to improve estimates of parameters that are: Important (e.g., using sensitivity analysis)Currently estimated with bias or

imprecision

Design studies to improve estimates of parameters that are: Important (e.g., using sensitivity analysis)Currently estimated with bias or

imprecision

ImplicationImplication

CEA models can not only organize thinking, decision making and communication about a topic, but can also be used to help set an agenda for research

CEA models can not only organize thinking, decision making and communication about a topic, but can also be used to help set an agenda for research

Question 2Question 2

What’s new in economic modeling and simulation?

What’s new in economic modeling and simulation?

Bayesian approachBayesian approach

A general approach to CEA modeling All parameter estimates are based on prior

distributions Ideally, correlations among parameters are

considered Ideally, these distributions reflect the impact

of covariatesThe output – posterior distributions – reflects

the impact of uncertainty

A general approach to CEA modeling All parameter estimates are based on prior

distributions Ideally, correlations among parameters are

considered Ideally, these distributions reflect the impact

of covariatesThe output – posterior distributions – reflects

the impact of uncertainty

Example outputExample output

“Uncertainty in all the model parameters was addressed using (a) prior distributions; and (b) resampling – in >95% of replications of the simulation the ICER was <$20,000/QALY…”

“Uncertainty in all the model parameters was addressed using (a) prior distributions; and (b) resampling – in >95% of replications of the simulation the ICER was <$20,000/QALY…”

AdvantagesAdvantages

This is a general, intellectually coherent way of modeling

Now computationally feasibleEuropeans and analysts like itA more sophisticated treatment of

uncertainty than 1- and multi-way sensitivity analysis

This is a general, intellectually coherent way of modeling

Now computationally feasibleEuropeans and analysts like itA more sophisticated treatment of

uncertainty than 1- and multi-way sensitivity analysis

DisadvantagesDisadvantages

Possible loss of transparencyParameter estimates might not be

possible / practical to obtainEasy for the model to take on a life

of its own

Possible loss of transparencyParameter estimates might not be

possible / practical to obtainEasy for the model to take on a life

of its own

What makes a good model?What makes a good model?

Model structure focuses on core of the issue

As simple as possible, but not too simple

Model is transparent Model inputs can be collected at the

required level of precision / quality

Model structure focuses on core of the issue

As simple as possible, but not too simple

Model is transparent Model inputs can be collected at the

required level of precision / quality

OpinionOpinion

Are more structurally, technically and computationally complex models such as Bayesian CEA models “good”?My opinion: sometimes

Are more structurally, technically and computationally complex models such as Bayesian CEA models “good”?My opinion: sometimes

Question 3Question 3

What’s changing in how CEA models are applied?

What’s changing in how CEA models are applied?

Back in the dayBack in the day

“Your health care organization should place our acute stroke drug on the formulary because its ICER indicates that it is good value for the money…”

“Your health care organization should place our acute stroke drug on the formulary because its ICER indicates that it is good value for the money…”

ProblemsProblems

The decision maker doesn’t have the same societal perspective as the analyst

The analysis ignores silos Even with discounting, lifetime impact is less

important to the decision maker than short term impacts

Unless accompanied by a back-of-the envelope calculation, the result isn’t transparent

The decision maker doesn’t have the same societal perspective as the analyst

The analysis ignores silos Even with discounting, lifetime impact is less

important to the decision maker than short term impacts

Unless accompanied by a back-of-the envelope calculation, the result isn’t transparent

ExampleExample

A back of the envelope model Suppose that an acute stroke drug keeps 2

people per 100 out of nursing homes. If they survive 3 years at $50,000 per year, the excess cost is $300,000 per 100 patients, or $3,000 per patient. So long as it costs less than $3,000, an acute stroke treatment that is even marginally effective is likely to be cost-effective as well.

A back of the envelope model Suppose that an acute stroke drug keeps 2

people per 100 out of nursing homes. If they survive 3 years at $50,000 per year, the excess cost is $300,000 per 100 patients, or $3,000 per patient. So long as it costs less than $3,000, an acute stroke treatment that is even marginally effective is likely to be cost-effective as well.

Current trendsCurrent trends

With calculations becoming less burdensome, it is easier to produced customized models • (e.g., including only costs of interest to the

decision maker)

Model results are embedded within more realistic frameworks such as comparative effectiveness

With calculations becoming less burdensome, it is easier to produced customized models • (e.g., including only costs of interest to the

decision maker)

Model results are embedded within more realistic frameworks such as comparative effectiveness

Ideal frameworkIdeal framework

Transparent Includes as many of the elements

of interest to the decision maker as possible

CEA model is descriptive, not prescriptive

Transparent Includes as many of the elements

of interest to the decision maker as possible

CEA model is descriptive, not prescriptive

ExampleExample

As an example of a formal decision making process intended to satisfy these criteria, I’ll describe how the oncology clinics at Duke systematically learnCEA models are one (albeit not the only)

tool that we use

As an example of a formal decision making process intended to satisfy these criteria, I’ll describe how the oncology clinics at Duke systematically learnCEA models are one (albeit not the only)

tool that we use

Oncology modeling at DukeOncology modeling at Duke

Rapid learning cancer clinicsCombine sound data collection

with an explicit mechanism for learning

Rapid learning cancer clinicsCombine sound data collection

with an explicit mechanism for learning

DataData

A data warehouse is used to generate multiple views, typically derived from linked files (e.g., cancer type, treatments, outcomes)

The lynchpin is a data set of patient-reported outcomes (derived from the PCM)

A data warehouse is used to generate multiple views, typically derived from linked files (e.g., cancer type, treatments, outcomes)

The lynchpin is a data set of patient-reported outcomes (derived from the PCM)

Data qualityData quality

The PCM contains 70+ items on a 0-10 scale (e.g., level of nausea during last 7 days)

Filled out in waiting room using e-tablets – migrating to web

Results are reported to clinicians in real time – for example, highlighting issues to discuss during the visit

The PCM contains 70+ items on a 0-10 scale (e.g., level of nausea during last 7 days)

Filled out in waiting room using e-tablets – migrating to web

Results are reported to clinicians in real time – for example, highlighting issues to discuss during the visit

IncentivesIncentives

Patients: confident that their concerns won’t be overlooked

Physicians: saves time in performing a review of symptomsPrinciple: (Sufficiently) valid data are

produced by design, not by accident

Patients: confident that their concerns won’t be overlooked

Physicians: saves time in performing a review of symptomsPrinciple: (Sufficiently) valid data are

produced by design, not by accident

Formal learning structureFormal learning structure

Learning from the databases occurs within a formal PDCA cycleRelevant stakeholders are representedThe stakeholders determine the level of

accuracy / precision required to make decisions

The stakeholders determine study design (e.g., interventional, observational)

Learning from the databases occurs within a formal PDCA cycleRelevant stakeholders are representedThe stakeholders determine the level of

accuracy / precision required to make decisions

The stakeholders determine study design (e.g., interventional, observational)

Types of designsTypes of designs

Observational designs with undirected machine learning

Observational designs with pre-specified hypotheses

Pre-post designs with interventions

Randomized trials

Observational designs with undirected machine learning

Observational designs with pre-specified hypotheses

Pre-post designs with interventions

Randomized trials

ExampleExample

Is it “worth it” to refer patients with high levels of psychological distress to specialized counseling?

Is it “worth it” to refer patients with high levels of psychological distress to specialized counseling?

InputsInputs

Observational data – natural history of outcomes by level of distress

CEA model to estimate what level of improvement would justify use of specialized counseling resources

Literature review on expected impact of counseling

Pre-post design to assess impact of counseling in our setting

Observational data – natural history of outcomes by level of distress

CEA model to estimate what level of improvement would justify use of specialized counseling resources

Literature review on expected impact of counseling

Pre-post design to assess impact of counseling in our setting

Criteria for learningCriteria for learning

A practice is worth changing if the alternative is cost-effective

We use CEA models that are simple to moderately complex

A practice is worth changing if the alternative is cost-effective

We use CEA models that are simple to moderately complex

CommentComment

Our goal is to systematically and explicitly embed learning into our usual procedures

Our goal is to systematically and explicitly embed learning into our usual procedures

Final thoughtsFinal thoughts

Health economics has always been quantitative – now, it is becoming more explicitly “statistical” as well

A statistically-inspired literature on CEA is rapidly developing – a distinguishing characteristic is the ability to accommodate increasingly complex models through advances in computation

Health economics has always been quantitative – now, it is becoming more explicitly “statistical” as well

A statistically-inspired literature on CEA is rapidly developing – a distinguishing characteristic is the ability to accommodate increasingly complex models through advances in computation

Final thoughtsFinal thoughts

The danger in this literature is that, if its perspective is entirely statistical, it can become divorced from reality

A particular area of promise lies in integrating CEA modeling with modern systematic approaches to learning

The danger in this literature is that, if its perspective is entirely statistical, it can become divorced from reality

A particular area of promise lies in integrating CEA modeling with modern systematic approaches to learning