Markov Models II

Brennan Spiegel, MD, MSHS

VA Greater Los Angeles Healthcare SystemDavid Geffen School of Medicine at UCLA

UCLA School of Public HealthCURE Digestive Diseases Research Center

UCLA/VA Center for Outcomes Research and Education (CORE)

HS 249T Spring 2008

Topics• More on Markov models versus decision trees

• More examples of Markov models

• Calculating annual transition probabilities– Time independent (Markov chains)– Time dependent (Markov processes)

• Temporary and tunnel states

• Half-cycle corrections

Disadvantages of Traditional Decision Trees

• Limited to one-way progression without opportunity to “go back”

• Can become unwieldy in short order

• Difficult to capture the dynamic path of moving between health states over time

• Often fails to accurately reflect clinical reality

Markov Models• Allow dynamic movement between

relevant health states

• Allow enhanced flexibility to better emulate clinical reality

• Acknowledge that different people follow different paths through health and disease

Example Markov Model

Inadomi et al. Ann Int Med 2003

Markov ModelAlive

No BarrettAlive

Barrett

DeadBarrett

DeadNo Barrett

Year 0

AliveNo Barrett

AliveBarrett

DeadNo Barrett

DeadBarrett

Markov Model

DeadNo Barrett

AliveNo Barrett

AliveBarrett

DeadBarrett

Markov Model

Year 1

DeadNo Barrett

AliveNo Barrett

AliveBarrett

DeadBarrett

Markov Model

DeadNo Barrett

AliveNo Barrett

AliveBarrett

DeadBarrett

Markov Model

DeadNo Barrett

AliveNo Barrett

AliveBarrett

DeadBarrett

Markov Model

End

DeadNo Barrett

AliveNo Barrett

AliveBarrett

DeadBarrett

Markov Model

Decision Trees and Markov Models may Co-Exist

• Both provide different types of information

• Information from both is not mutually exclusive

• Markov model can be “tacked” onto end of a traditional decision tree

No Therapy

Chronic HBV

Inteferon

Lamivudine

Adefovir Salvage

Adefovir

Virological ResponseNormal Lifespan

No Response

No Cirrhosis

CirrhosisMarkov Model

Normal Lifespan

No Cirrhosis

CirrhosisMarkov Model

Normal Lifespan

Response

No ResponseStart Adefovir

No Resistance

Resistance

Con’t Lamivudine

Chronic HBV

Chronic HBV on Treatment

Virological Response

Markov Model #1

Virological Resistance

Virological Relapse

Uncomplicated Cirrhosis

To Cirrhosis Markov Model

Uncomplicated Cirrhosis

Complicated Cirrhosis

Liver Transplant Death

HepatocellularCarcinoma

Markov Model #2

No GI or CV Complications

Dyspepsia

Myocardial Infarction

Post Myocardial Infarction

Death

Post GI Bleed

GI Bleed

Clinical Response

Hepatocellular Cancer

Liver Transplantation

Death

START

Sub-Clinical HE

Overt HE

Non-HE Complication

Annual Probability Estimates

Annual Probability EstimateCirrhosis in HBeAg(-) 4.0% Cirrhosis in HBeAg(+) 2.2% Chronic HBV liver cancer 1.0% Cirrhosis liver cancer 2.1% Compensated cirrhosis decompensated 3.3% Decompensated cirrhosis liver transplant 25% Liver cancer liver transplant 30% Death in compensated cirrhosis 4.4% Death in decompensated cirrhosis 30% Death in liver cancer 43%

Converting Data Into Annual Probability Estimates

Cannot simply divide long-term data by number of years

Example:If 5-year risk of an event is 40%, then annual risk does not amount to:

40 5

= 8%

Converting Data Into Annual Probability Estimates

General rule for converting long-term data into annual probabilities:

1-(1-x)Y = Probability at Y Years

Example of Converting Long Term Data into Annual Probability

If probability of bleed at 5 years = 0.40, then the annual probability = x, as follows:

1- (1-x)5 = 0.40 (1-x)5 = 1 – 0.40(1-x)5 = 0.60

x = 0.097

(1-x) = 0.902

… or 9.7%

Example of Converting Long Term Data into Annual Probability

Check for errors by back calculating using the inverse equation:

1-(1-annual probability)Y = probability at Y years

1-(1- 0.097)5 = 0.40 1-(0.903)5 = 0.40

0.40 = 0.40

Markov Cycle Converter  Forward Calculator  

Enter Percentage to be Converted 40

   

Enter Number of Cycles 5

   

Cycle Probability= 0.09711955

      

Backwards Calculator  

Enter Cycle Probability for Conversion 0.097

   

Enter Number of Cycles 5

   

Converted Probability= 0.39960267

Converted Percentage= 39.96026709

Steps to Combining Time-Independent Transition Probabilities

Step 1 Collect and abstract relevant studies

Step 2 Select common cycle length

Step 3 Convert all studies to common cycle length units

Step 4 Calculate common cycle transition probabilities

Step 5 Combine common cycle probabilities

StudyStudy

DurationNumber of 12 Mo

CyclesEnd

PercentageCalculated 12-Month

Probability

Jones 6 months 0.5 12% 0.23

James 12 months 1 19% 0.19

Johnson 18 months 1.5 22% 0.15

Marshall 3 months 0.25 8% 0.28

Example

Mean = 21.3% / 12-month cycle

Many Probabilities are Time Dependent

• Time independence is usually a simplifying assumption

• Progress though many systems in health care (biological, organizational, psychosocial, etc) are erratic and non-linear

• May need to account for time-dependent transitional probabilities using:– Tables– Tunnels

Using Tables for Time-Dependent Probabilities

• Tables allow transition probabilities to vary cycle-by-cycle

• Allow greater precision for processes that are non-linear

0.058

0.057

0.056

0.055

0.054

0.053

0.052

0.051

ProbabilityCycle

0.028

0.037

0.046

0.055

0.064

0.073

0.082

0.11

ProbabilityCycle

Time Independent Time Dependent

Time Independent: Linear Curve

Time Dependent: Non-linear Diminishing Returns

Time Dependent: Non-linear Accelerating Returns

Cycle

Prob

abili

ty

• Some events can interfere with otherwise orderly Markov chains

• Can get “stuck in a rut” that removes subjects from the usual flow of events– e.g. developing cancer

• Tunnel states add flexibility to Markov models:– Model getting “stuck in the rut”– Compartmentalize processes into component states– Can model various “recovery states” from the “rut”– Can incorporate time-dependent transitions

Using Tunnels States

Example Prior to Tunnel State

Example With Tunnel State

Half-Cycle Corrections• In “real life,” events can occur anytime during a

given cycle – it is usually a random event

• The default setting for Markov models is for events to occur at the exact end of each cycle

• Yet the default setting can lead to errors in the calculation of average values– Will tend to overestimate benefits (e.g. life

expectancy) by about half of a cycle

Rationale for Half-Cycle Corrections

“In whatever cycle a ‘member’ of the cohort analysis dies, they have already received a full cycle’s worth of state reward, at the beginning of the cycle. In reality, however, deaths will occur halfway through a cycle on average. So, someone that dies during a cycle should lose half of the reward they received at the beginning of the cycle.”

- TreeAge Pro Manual, p476

0 1 2 3 4

Cycle

Prop

ortio

n A

live

1.0

0.8

0.6

0.4

0.2

0.0

AUC=2

0 1 2 3 4

Prop

ortio

n A

live

1.0

0.8

0.6

0.4

0.2

0.0

Cycle

AUC=2.5

0 1 2 3 4

Prop

ortio

n A

live

1.0

0.8

0.6

0.4

0.2

0.0

Cycle

AUC=2.0ish