A Markov Decision Model for Determining Optimal

Outpatient Scheduling

Jonathan PatrickTelfer School of Management

University of Ottawa

Motivation The unwarranted skeptic and the uncritical

enthusiast Outpatient clinics in Canada receiving strong

encouragement to switch to open access Basic operations research would claim that

there is a cost to providing same day access Does the benefit outweigh the costs?

Trade-off Any schedule needs to balance system-

related benefits/costs - revenue, overtime, idle time,… versus patient related benefits – access, continuity of care,….

Available levers include the decision as to how many new requests to serve today and how many requests to book in advance into each day.

Scheduling Decisions

Day

1

Day

2

Day

3

Day

4

Day

5

New Demand

Day

2

Day

3Day

1

Literature Plenty of evidence that overbooking is

advantageous in the presence of no-shows (work by Lawley et al and by Lawrence et al)

Also evidence that a two day booking window outperforms open access (work by Liu et al and by Lawrence and Chen)

Old trade-off between tractability of the model and complexity

Model Aims To create a model that

• Incorporates a show rate that is dependent on the appointment lead time

• Gives managers the ability to determine • the number of new requests to serve today• The number of requests to book into each future

day (called the Advanced Booking Policy – ABP)• Allows the policy to depend on the current

booking slate and demand.

Markov Decision Process Model Decision Epochs

• Made once a day after today’s demand has arrived but before any appointments

State• Current ABP (w), queue size (x) and demand (y)

Actions• How many of today’s demand to serve today (b)• Whether to change the current ABP (a)

Markov Decision Process Model Transitions

• Stochastic element is new demand

• New queue size is equal to current queue size (x) minus today’s slate (x w) plus any new demand not serviced today (y-b)

• New demand represented by random variable D.

Markov Decision Process Model Costs/Rewards

• System Related: revenue, overtime, idle time• Patient Related: lead time• For switching the ABP

Bellman Equation

Used a discounted (but with a discount rate of 0.99), infinite horizon model to avoid arbitrary terminal rewards

Can be solved to optimality

Assumptions/Limitations Advance bookings are done on a FCFS basis Today’s demand arrives before any booking

decisions need to be made Service times are deterministic Show rate dependent on size of queue at time

of service instead of at time of booking Immediate changes to ABP may mean that

previous bookings need to be shifted Does not account for fact that some bookings

have to be booked in advance

Clinic Types Considered

5,5,10,0:#9 Clinic

1,5,10,0:#8 Clinic

0,5,10,0:#7 Clinic

5,5,10,20 :#6 Clinic

1,5,10,20 :#5 Clinic

0,5,10,20 :#4 Clinic

5,0,10,20 :#3 Clinic

1,0,10,20 :#2 Clinic

0,0,10,20 :#1 Clinic

LTITOTR

LTITOTR

LTITOTR

LTITOTR

LTITOTR

LTITOTR

LTITOTR

LTITOTR

LTITOTR

ffff

ffff

ffff

ffff

ffff

ffff

ffff

ffff

ffff

Six Scenarios for each Clinic Type1. Base scenario

• Demand equal to capacity• Show rate based on research by Gallucci• All requests can be serviced the same day

2. Demand > Capacity3. Demand < Capacity4. Some requests must be booked in advance5. Same day bookings given a show probability of 16. Show probability with a steeper decline

Performance Results Clinics #1,2,3:

• OA and MDP policy result in almost identical profits• Same day access ranges from 89% to 100% (max lead time 1 day)

Clinics #4,5,6:• MDP slightly outperforms OA (by less than 2%)• Same day access ranges from 84% to 100% (max lead time 2 days)

Clinics #7,8,9:• MDP vastly outperforms OA in all scenarios (by as much as 70%)• Same day access ranges from 28% to 98% (max lead time 4 days)

For all clinics, MDP provides a significant reduction in throughput variation and peak workload

Optimal Policy (base scenario, w=11, x=0)

Day

1

Day

1

123456789

1019

18

161514131211

17

20

0

,5

,10

,0

LT

IT

OT

R

f

f

f

f

Optimal Policy (base scenario, w=11, x=0)

Day

1

Day

2

123456789

10

19

1816

15

1413

1211

17

20

5

,5

,10

,0

LT

IT

OT

R

f

f

f

f

Performance Trends MDP performed best when demand was high (e.g. when

demand > capacity and when same day show rate was guaranteed).

MDP approaches OA as the lead time cost increases

Presence of revenue makes OA much more attractive

Maximum booking window in any scenario tested was 4 days

MDP manages to perform as well even when revenue is present by sacrificing some throughput in order to reduce overtime and idle time costs.

Conclusion Model provides a booking policy that takes into account

no-shows and reacts to the congestion in the system Simulation results suggest that it achieves better results

(same or higher objective, more predictable throughput) than open access with minimal cost to the patient in terms of lead times

Enhancements to the model certainly possible including the inclusion of stochastic services times, the transition to a continuous time setting, the possibility of a multi-doctor clinic….

Currently in discussion with local clinic to build enhanced model and test it.

Thank You!

Optimal Policy (base scenario, w=11, x=0)5,5,10,0 LTITOTR ffff

Number of New Requests Given Same Day Servicey 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 01 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 02 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 03 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 04 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 05 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 06 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 07 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 08 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 09 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 011 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 012 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 013 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 014 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 015 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 016 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 017 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 018 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 019 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 020 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1

Optimal Policy (base scenario, w=11, x=0)0,5,10,0 LTITOTR ffff

Number of New Requests Given Same Day Servicey '0' '1' '2' '3' '4' '5' '6' '7' '8' '9' '10' '11'

0 1 0 0 0 0 0 0 0 0 0 0 01 0 1 0 0 0 0 0 0 0 0 0 02 0 0 1 0 0 0 0 0 0 0 0 03 0 0 0 1 0 0 0 0 0 0 0 04 0 0 0 0 1 0 0 0 0 0 0 05 0 0 0 0 0 1 0 0 0 0 0 06 0 0 0 0 0 0 1 0 0 0 0 07 0 0 0 0 0 0 0 1 0 0 0 08 0 0 0 0 0 0 0 0 1 0 0 09 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 011 0 0 0 0 0 0 0 0 0 0 1 012 0 0 0 0 0 0 0 0 0 0 1 013 0 0 0 0 0 0 0 0 0 0 1 014 0 0 0 0 0 0 0 0 0 0 1 015 0 0 0 0 0 0 0 0 0 0 1 016 0 0 0 0 0 0 0 0 0 0 1 017 0 0 0 0 0 0 0 0 0 0 1 018 0 0 0 0 0 0 0 0 0 0 1 019 0 0 0 0 0 0 0 0 0 0 0 120 0 0 0 0 0 0 0 0 0 0 0 1

Scenario Policy

Lead Time Costs

Average Daily Cost/Profit Appointment Lead Times

TH OT IT Actual

Percent diff from OA 0 1 2 3 4

Show Rate with Same Day = 100%

OA 100.0% 12.5% 12.5% -18.75

MDP0 91.6% 1.0% 9.4% -5.67 69.8% 63.89% 34.76% 1.34% 0.01% 0.00%1 93.4% 1.8% 8.4% -8.92 52.4% 71.51% 28.19% 0.30% 0.00% 0.00%5 97.1% 6.1% 9.0% -16.93 9.7% 87.37% 12.63% 0.00% 0.00% 0.00%

Increased Demand

(Demand = 12)

OA 88.0% 15.7% 10.1% -20.81

MDP0 78.0% 2.9% 9.2% -7.50 64.0% 27.76% 44.56% 23.06% 4.37% 0.25%1 83.9% 6.9% 6.2% -14.29 31.3% 64.82% 34.43% 0.76% 0.00% 0.00%5 87.8% 14.7% 9.4% -20.67 0.7% 97.91% 2.09% 0.00% 0.00% 0.00%

Base Case

OA 88.0% 6.9% 18.9% -16.34

MDP0 84.1% 0.7% 16.5% -8.92 45.4% 66.44% 32.29% 1.26% 0.01% 0.00%1 85.9% 1.7% 15.8% -11.45 26.8% 81.52% 18.36% 0.12% 0.00% 0.00%5 87.4% 4.5% 17.1% -15.64 -59.5% 94.89% 5.11% 0.00% 0.00% 0.00%

Steep Decline

OA 88.0% 6.9% 18.9% -16.34

MDP0 82.7% 0.8% 18.1% -9.81 40.0% 78.01% 21.83% 0.16% 0.00% 0.00%1 84.3% 1.5% 17.2% -11.60 29.0% 85.06% 14.92% 0.03% 0.00% 0.00%5 86.6% 4.0% 17.4% -15.51 5.1% 94.35% 5.65% 0.00% 0.00% 0.00%

Advanaced Bookings

OA 84.6% 5.6% 21.0% -16.16

MDP0 81.5% 0.6% 19.1% -10.11 37.4% 43.64% 52.99% 3.32% 0.05% 0.00%1 82.9% 1.4% 18.5% -12.03 25.5% 55.14% 44.36% 0.50% 0.00% 0.00%5 84.2% 3.7% 19.6% -13.89 14.0% 65.98% 34.01% 0.00% 0.00% 0.00%

Demand = 8

OA 0 88.0% 2.1% 31.7% -17.89

MDP0 86.9% 0.0% 30.5% -15.28 14.6% 90.39% 9.60% 0.01% 0.00% 0.00%1 87.2% 0.2% 30.4% -15.93 11.0% 92.63% 7.37% 0.00% 0.00% 0.00%5 87.6% 0.8% 30.7% -17.32 3.2% 97.09% 2.91% 0.00% 0.00% 0.00%

Scenario Policy

Lead Time Costs

Average Daily Cost/Profit Appointment Lead Times

TH OT IT Actual

% diff from OA 0 1 2 3 4

Increased Demand

(Demand = 12)

OA 88.0% 15.7% 10.2% 190.33

MDP0 86.2% 10.3% 6.8% 193.21 1.5% 83.88% 16.12% 0.00% 0.00% 0.00%1 87.1% 12.3% 7.8% 191.70 0.7% 91.33% 8.67% 0.00% 0.00% 0.00%5 88.0% 15.7% 10.2% 190.33 0.0% 100.00% 0.00% 0.00% 0.00% 0.00%

Show Rate with Same Day = 100%

OA 100.0% 12.5% 12.5% 181.25

MDP0 97.0% 6.0% 9.0% 183.44 1.2% 87.10% 12.90% 0.00% 0.00% 0.00%1 98.1% 8.1% 10.0% 182.38 0.6% 91.96% 8.04% 0.00% 0.00% 0.00%5 100.0% 12.5% 12.5% 181.25 0.0% 100.00% 0.00% 0.00% 0.00% 0.00%

Base Case

OA 88.0% 6.9% 18.9% 159.63

MDP0 86.6% 2.6% 16.0% 162.49 1.8% 87.43% 12.56% 0.02% 0.00% 0.00%1 86.9% 3.5% 16.5% 161.29 1.0% 91.54% 8.46% 0.00% 0.00% 0.00%5 87.8% 6.2% 18.3% 159.61 0.0% 98.64% 1.36% 0.00% 0.00% 0.00%

Show Rate with Steep Decline

OA 88.0% 6.9% 18.9% 159.63

MDP0 86.6% 4.0% 17.4% 160.46 0.5% 94.35% 5.65% 0.00% 0.00% 0.00%1 87.0% 4.7% 17.7% 160.11 0.3% 95.98% 4.02% 0.00% 0.00% 0.00%5 88.0% 6.9% 18.9% 159.63 0.0% 100.00% 0.00% 0.00% 0.00% 0.00%

Advanaced Bookings

OA 84.6% 5.6% 21.0% 153.03

MDP0 83.3% 1.9% 18.6% 155.46 1.6% 61.19% 34.48% 4.19% 0.14% 0.00%1 83.8% 2.8% 19.0% 154.69 1.1% 63.14% 36.82% 0.05% 0.00% 0.00%5 84.4% 4.8% 20.4% 153.05 0.0% 68.61% 31.39% 0.00% 0.00% 0.00%

Decreased Demand

(Demand =8)

OA 0 88.0% 2.1% 31.7% 122.90

MDP0 87.5% 0.5% 30.5% 124.21 1.1% 95.87% 4.13% 0.00% 0.00% 0.00%1 87.6% 0.6% 30.5% 123.95 0.9% 96.27% 3.73% 0.00% 0.00% 0.00%5 87.8% 1.3% 31.0% 123.16 0.2% 98.52% 1.48% 0.00% 0.00% 0.00%

Scenario Policy

Lead Time Costs

Average Daily Cost/Profit Appointment Lead Times

TH OT IT Actual

Percent diff from OA 0 1 2

Increased Demand (Demand = 12)

OA 88.0% 15.7% 10.2% 195.40

MDP0 86.7% 11.4% 7.4% 196.69 0.7% 88.59% 11.41% 0.00%1 87.5% 13.7% 8.7% 195.69 0.1% 95.46% 4.54% 0.00%5 88.0% 15.7% 10.2% 195.40 0.0% 100.00% 0.00% 0.00%

Show Rate with Same Day = 100%

OA 100.0% 12.5% 12.5% 187.50

MDP0 98.2% 8.3% 10.1% 188.06 0.3% 92.43% 7.57% 0.00%1 99.1% 10.1% 11.1% 187.58 0.0% 95.82% 4.18% 0.00%5 100.0% 12.5% 12.5% 187.50 0.0% 100.00% 0.00% 0.00%

Base Case

OA 88.0% 6.9% 18.9% 169.08

MDP0 86.8% 3.1% 16.3% 170.51 0.8% 89.82% 10.18% 0.00%1 88.0% 6.9% 18.9% 169.08 0.0% 100.00% 0.00% 0.00%5 88.0% 6.9% 18.9% 169.08 0.0% 100.00% 0.00% 0.00%

Show Rate with Steep Decline

OA 88.0% 6.9% 18.9% 169.08

MDP0 87.2% 4.9% 17.8% 169.39 0.2% 96.48% 3.52% 0.00%1 87.8% 6.4% 18.6% 169.11 0.0% 99.11% 0.89% 0.00%5 88.0% 6.9% 18.9% 169.08 0.0% 100.00% 0.00% 0.00%

Advanaced Bookings

OA 84.6% 5.6% 21.0% 160.60 70.00% 30.00% 0.00%

MDP0 83.7% 2.6% 18.9% 164.80 2.6% 62.27% 37.64% 0.09%1 84.0% 3.4% 19.4% 161.16 0.3% 65.12% 34.87% 0.01%5 84.6% 5.6% 21.0% 160.60 0.0% 70.00% 30.00% 0.00%

Decreaded Demand (Demand = 8)

OA 0 88.0% 2.1% 31.7% 138.73 100.00% 0.00% 0.00%

MDP0 87.5% 0.5% 30.5% 139.45 0.5% 96.04% 3.96% 0.00%1 87.6% 0.7% 30.7% 139.12 0.3% 96.74% 3.26% 0.00%5 88.0% 1.9% 31.5% 138.79 0.0% 99.78% 0.22% 0.00%