Markov Models, part II

Marcelo Coca Perraillon

University of ColoradoAnschutz Medical Campus

Cost-Effectiveness AnalysisHSMP 6609

2016

1 / 37

Outline

More examples

Markov model extensions

1 Incorporating time dependency2 Relaxing the Markov assumption (memoryless property)3 Patient-level simulation (microsimulations)4 Static and dynamic models

Summary

2 / 37

Big picture

In the last two classes we incorporated uncertainty intocost-effectiveness using decision analysis and Markov models

Decision trees without a Markov model within are less common

Markov models provide more flexibility; they incorporate uncertaintyand also model disease progression

Good setting for calculating life expectancy and costs

Today we will see more examples and talk about extensions toMarkov models

3 / 37

Recap

We needed very few elements to build a Markov model

1 Health states2 Cycles3 Transition probabilities4 Rewards (costs and benefits)

Even though there are few elements, Markov models can be fairlycomplex (in a good way)

4 / 37

Example 1: description

Cost-effectiveness of options for the diagnosis of high blood pressurein primary care, Lovibond et al (2011), Lancet; 378:1219-30

Compare three diagnosis strategies. Further blood pressuremeasurement 1) in clinic, 2) at home, 3) ambulatory monitor

Hypothetical primary care population aged 40 or older with ascreening blood-pressure greater than 140/90 and risk-factorprevalence of the general population (UK)

Cycle: 3 months; time-horizon: 60 years

Rewards: costs and QALYs

Stratified by age

Data: meta-analyses, risk of events using Framingham risk equations

5 / 37

Example 1: Model

6 / 37

Example 1: Model

Note that all paths lead to death

Different types of states: hypertension, diagnosed, events

Possible to go back to suspected hypertension in some cases

Certain events last only one cycle (MI, angina, stroke, TIA) (tunnelstates; more on this shortly)

Even though patients stay in that state for one cycle, this cycle isexpensive and lowers QALY (and it changes the probability of death)

One Markov model by intervention; used Excel, stratified by sex andage group

Transition probabilities were not fixed

7 / 37

Example 1: Conclusions

Ambulatory monitoring most cost effective

After an initial raised reading, it reduces the misdiagnosis and thussaves costs

Additional costs of monitoring are compensated by cost savings fromtargeted treatment

Recommended monitoring before starting antihypertensive drugs

8 / 37

Example 2: description

Objective: to estimate the cost effectiveness of herpes zoster(shingles) vaccine versus no vaccination

Herpes zoster: a reactivation of the chickenpox virus in the body(causes a painful rash). No cure so prevention is key

Cohort entered the model healthy at 50 y/o

Rewards: costs and QALYs; time horizon: 70 years (lifelong); cycle: 1year

9 / 37

Example 2: Model

10 / 37

Example 2: Model

11 / 37

Example 2: features

Model expressed as decision tree with Markov models

The decision node shows the options: vaccine or no vaccine

Sex is a chance node; similar to stratification in the hypertensionexample but in that example they did not show a tree

Their model is not very clear. It is easier to understand Markovmodels using transition diagrams

12 / 37

Example 2: Conclusions

Not cost effective: ICER = $500,754 per QALY

The vaccine is expensive and the incidence of shingles is low at thatage

Efficacy of vaccine is very low after 10 to 12 years

Better to use the vaccine in older patients

13 / 37

Example 3: description

Cost effectiveness of alternative treatments for breast cancer

One-year adjuvant trastuzumab (AT) therapy, with or withoutanthracyclines

Evidence comes from clinical trial data; they wanted to model lifespanoutcomes

49 y/o women with early-stage breast cancer

Cycle: 1 month; time-horizon: could not find it in paper, probablyaround 50 years

14 / 37

Example 3: Model

15 / 37

Example 3: Conclusions

AT ICER is $39,982 per QALY

Of course, results sensitivy to medication costs

A straightforward model with a lot of assumptions about parameterinputs

They needed to make assumptions because clinical trials are shortterm

16 / 37

Extending Markov models: time-dependency

So far we have only dealt with Markov models that have the sametransitions probabilities each cycle

The transition probabilities did not depend on the time spent in onestate or just time (i.e. cycle)

Patients getting older would not have an accompanying increase inthe probability of dying (but this doesn’t mean that it matters;remember, always a comparison...)

(This is often called competing risks)

17 / 37

Probabilities vary according to time in model

Some transition probabilities change as people get old

In other words, transition probabilities can be a function of cycle

In the HIV example, the probability of state D (death) should behigher in latter cycles

Straightforward to implement

18 / 37

Probabilities vary according to time in model

Two absorbing states: death due to HIV and death due to aging(non-HIV)Could have kept one death state while increasing the probability ofdying

19 / 37

Probabilities vary according to time in a state

Different situation: transition probabilities depend on time in state

Example: Probability of dying increases according to how long aperson has had AIDS

This is a harder situation to handle with the type of Markov modelswe have covered

We do not have a way to keep track of people moving from state tostate (in the HIV example, we can only keep track of people in stateA)

We need to relax the Markov assumption (memoryless property)

20 / 37

Relaxing Markov assumption

Example (Chapter 2 of Briggs et al, 2006): probability of dying ofcancer after a cancer recurrence

Patients can have a local or regional recurrence and then remission

But after remission, the probability of death should depend onwhether the recurrence was local or regional and the time in remission

We need to find a way to keep track of time in remission

21 / 37

Partial transition diagram

No memory of time and the type of remission once in remission

22 / 37

Relaxing Markov assumption

We can follow a standard trick: adding states

Remember that we “store” costs and benefits in health states and wecan add as many states as we want (although it increases thecomplexity of the model)

Instead of one remission state, we could have remission states foreach type of recurrence

23 / 37

Adding remission states

Now the transition probability from remission to death could bedifferent depending on the type of recurrence

Costs and benefits can also be different24 / 37

We now need to solve the time issue

We have made remission dependent of the type of recurrence

But we wanted to also consider the time in remission because itaffects the probability of dying

We follow the same trick: add more states that reflect time

25 / 37

Health states that reflect time

26 / 37

Why does it work?

Patients cannot stay in the new remission states for more than onecycle (no arrow to the same health state). These type of health statesare called tunnel states

We have added memory to the Markov model. We now know thatpatients in, say, “Remission after LR, Year 2” have been cancer freefor two years (no other way to get to this state)

We now can make the probability of death different for each tunnelstate, which means making the probability different according totime in state

Of course we could add more tunnel states. The only cost is theadded complexity of programming the model

Knowledge of the disease (and data availability) guides choices

Note that the time in a tunnel state is guided by cycle length

27 / 37

Tunnel states

Tunnel states have a very good name; think of them as tunnels willtolls

A tunnel is the only way to go from state A to B

They impose costs and benefits that last only one cycle

If you are in B, that means that you came from A one cycle ago

Markov models with some sort of memory are sometimes calledsemi-Markov processes or models

28 / 37

Big picture

The extensions we have covered could be called “games you can playby adding health states”

They are relatively easy ways to make Markov models more realistic

The methods we have covered so far are extensively used

But for many types of problems, we need still need different tools

29 / 37

Patient-level simulations

Patient-level simulations (or microsimulations) model the transitionsof an individual patient rather than a cohort

Very useful to make transition probabilities dependent on patienthistory, not just cycle time

We could take into account that the probability of an event dependson age, sex, disease severity, time with the disease, and so on

These models are all about memory, so we do not have to worryabout the Markov assumption

Some of these models take into account how people interact witheach other (key aspect of infections diseases)

30 / 37

Discrete event simulation (DES)

Origins in operations research (design and optimization of industrialprocesses). As a result, they use different terminology

Like Markov models, events happen in time (discrete periods, likecycles)

It models entities (e.g. patients, hospitals) that have attributes (e.g.age, sex, disease history)

Attributes can change in the model and they determine how entitiesinteract with environment and events

Events are “things” that could happen to an entity in anenvironment (e.g. death, infection, MI, stroke)

Costs and benefits can be added and DES models capture timespent in events

31 / 37

Example

Cost effectiveness of improving ambulance and thrombolysis responsetimes after myocardial infarction (Chase et al, 2006)

“Interventions: Improving the ambulance response time to 75% ofcalls reached within 8 minutes and the hospital arrival to thrombolysistime interval (door-to-needle time) to 75% receiving it within 30minutes and 20 minutes, compared to best estimates of responsetimes in the mid-1990s”

32 / 37

DES model

33 / 37

Disadvantages

It may not be obvious but more realism is not always better.Remember Einstein:

Everything should be made as simple as possible, but not simpler

More realism means a lot more data is needed

They easily turn into “black boxes” (a note on model calibration)

How do we get transition probabilities that depend on age, sex,disease history?

Many times the type of model follows the data. Given the datawe have, what is the best model we can possibly build?

Sensitivity analyses are more difficult (specially probabilistic sensitivityanalyses)

Computational burden could be a problem

34 / 37

Infectious diseases and vaccinations (static versus dynamicmodels)

Microsimulation models have a clear advantage when consideringinfectious diseases and vaccinations studies

The likelihood of contracting an infection depends on the number ofpeople already infected

Similarly, vaccines can be effective even if not all people arevaccinated (herd immunity)

In static models, the rate of infection is fixed. In dynamic models,rate of infection is not fixed and may depend on the number of peoplealready infected (i.e. possible to model epidemics)

No easy way to add these features in Markov models

Discrete event simulations are useful in these situations; agent-basedmodels are a variation

35 / 37

More on types of models

Adapted from Hay (2004)

A note on Net Logo, Arena, and @Risk

36 / 37

Summary

Markov models are extensively used in CEA

Easy to make transition probabilities depend on time (cycle)

Relatively easy to add more health states to incorporate memory intoMarkov models –but it gets complicated with too many health states

In certain problems, we need to make probabilities conditional oncharacteristics and history of patients

Microsimulation models are needed for infectious diseases andvaccination problems

Field is quickly evolving (classification and naming of models willbecome clearer)

37 / 37