the linear programming approach to approximate dynamic...

The Linear Programming Approach toApproximate Dynamic Programming

Daniela Pucci de Farias(joint work with Ben Van Roy)

Massachusetts Institute of Technology

http://www.mit.edu/∼pucci – p. 1/29

OutlineMarkov decision processesApproximate Dynamic ProgrammingApproximate linear programmingPerformance and Error AnalysisConstraint Sampling


Markov Decision Processes(finite) state space S

(finite) action sets Ax

costs ga(x)

transition probabilities Pa(x, y)discount factor αMinimize E

[∑∞

t=0 αtga(t)(x(t))

]


Markov Decision Processes(finite) state space S(finite) action sets Ax

costs ga(x)


[∑∞

t=0 αtga(t)(x(t))

]



costs ga(x)

transition probabilities Pa(x, y)

discount factor αMinimize E

[∑∞

t=0 αtga(t)(x(t))

]



costs ga(x)

transition probabilities Pa(x, y)discount factor α

Minimize E[∑∞

t=0 αtga(t)(x(t))

]



costs ga(x)


[∑∞

t=0 αtga(t)(x(t))

]


Tetris

x ∈ S: wall configuration and current piecea ∈ Ax: Piece placement

Pa(x, ·): Distribution of next piece

ga(x): number of rows eliminated


ExamplesScheduling/routing in queueing networksDynamic resource allocationAsset allocation/risk managementPower management in devices


Dynamic ProgrammingBellman’s equation

J(x) = mina∈Ax

E [ga(x) + αJ(y)]

Value iteration, policy iteration, linear programmingObtain an optimal policy

u∗(x) ∈ argmina∈Ax

E [ga(x) + αJ∗(y)]

The curse of dimensionality



J(x) = mina∈Ax

E [ga(x) + αJ(y)]

Value iteration, policy iteration, linear programming

Obtain an optimal policy


E [ga(x) + αJ∗(y)]




J(x) = mina∈Ax

E [ga(x) + αJ(y)]

Value iteration, policy iteration, linear programmingObtain an optimal policy


E [ga(x) + αJ∗(y)]



OutlineMarkov decision processesApproximate Dynamic ProgrammingApproximate linear programmingPerformance and error analysisConstraint Sampling


Value Function Approximation

Approximate J∗ ≈ Jr, for some r ∈ <K

Generate a policy

u(x) ∈ argmina∈Ax

E[

ga(x) + αJr(y)]

Linearly parameterized approximators

Jr(x) = (Φr)(x) =K∑

k=1

r(k)φk(x)φ1

φ2

φ3

J~

Design a function approximator Jr

Compute parameters r ∈ <K so that Jr ≈ J∗


Tetris

22 features / basis functionsColumn heightsDifferences between heights of consecutive columnsMaximum heightNumber of holesConstant function


Approximate DP: ExamplesAmerican options pricing(Longstaff & Schwartz, 2001, Tsitsiklis & Van Roy, 2001)

Job-shop scheduling(Zhang & Dietterich, 1996)

Elevator scheduling(Crites & Barto, 1996)

Backgammon(Tesauro,1995)


LP Formulation of DP

maxJ∑

x

c(x)J(x)

s.t. ga(x) + α∑

y

Pa(x, y)J(y) ≥ J(x), ∀x, ∀a

J ≤ J∗ for all feasible JLP solution is J∗ for all c > 0

one variable per stateone constraint per state-action pair



maxJ∑

x

c(x)J(x)

s.t. ga(x) + α∑

y

Pa(x, y)J(y) ≥ J(x), ∀x, ∀a

J ≤ J∗ for all feasible J

LP solution is J∗ for all c > 0




maxJ∑

x

c(x)J(x)

s.t. ga(x) + α∑

y

Pa(x, y)J(y) ≥ J(x), ∀x, ∀a

J ≤ J∗ for all feasible JLP solution is J∗ for all c > 0



Approximate Linear Programming

maxJ∑

x

c(x)J(x)

s.t. ga(x) + α∑

y

Pa(x, y)J(y) ≥ J(x), ∀x, ∀a

Idea: Consider only solutions J = Φrone variable per basis functionone constraint per state-action pair⇒ efficient constraint sampling



maxJ∑

x

c(x)J(x)

s.t. ga(x) + α∑

y

Pa(x, y)J(y) ≥ J(x), ∀x, ∀a

Idea: Consider only solutions J = Φr

one variable per basis functionone constraint per state-action pair⇒ efficient constraint sampling



maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y

Pa(x, y)(Φr)(y) ≥ (Φr)(x), ∀x, ∀a

Idea: Consider only solutions J = Φr

one variable per basis functionone constraint per state-action pair⇒ efficient constraint sampling



maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y

Pa(x, y)(Φr)(y) ≥ (Φr)(x), ∀x, ∀a

Idea: Consider only solutions J = Φrone variable per basis functionone constraint per state-action pair

⇒ efficient constraint sampling



maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y

Pa(x, y)(Φr)(y) ≥ (Φr)(x), ∀x, ∀a

Idea: Consider only solutions J = Φrone variable per basis functionone constraint per state-action pair⇒ efficient constraint sampling


Some Historyearly work

Schweitzer and Seidmann (1985)Trick and Zin (1993,1997)Gordon (1995)

analytical and computational toolMorrison and Kumar (1999)Paschalidis and Tsitsiklis (2000)Adelman (2002)

more extensive analysis and implementation in large problemsSchuurmans and Patrascu (2001)de Farias and Van Roy (2001,2002)Guestrin et al. (2002)Poupart et al. (2002)


Theory on Value Function ApproximationGoals

Understand what algorithms are doingFigure out which variations work and whenReduce trial and errorImprove performance

Quality of ultimate approximation limited by choice of ΦWill my algorithm A compute weights r that make good use ofmy basis functions Φ?“Competitive” bound

If Φr can come within ε of J∗,then algorithm A will compute r such that1. Φr is within O(ε) of J∗

2. the greedy policy u is O(ε)–optimal




Quality of ultimate approximation limited by choice of Φ

Will my algorithm A compute weights r that make good use ofmy basis functions Φ?“Competitive” bound






Quality of ultimate approximation limited by choice of ΦWill my algorithm A compute weights r that make good use ofmy basis functions Φ?

“Competitive” boundIf Φr can come within ε of J∗,then algorithm A will compute r such that1. Φr is within O(ε) of J∗





Quality of ultimate approximation limited by choice of ΦWill my algorithm A compute weights r that make good use ofmy basis functions Φ?“Competitive” bound




Notation‖J‖∞ = maxx |J(x)|

weighted norms:

‖J‖1,ν =∑

x

ν(x)|J(x)|, ‖x‖∞,ν = maxx

ν(x)|J(x)|


Graphical Interpretation of Approximate LP

J*

J(1)

J(2)

TJ J>

Even with arbitrarily small ‖J∗ − Φr∗‖∞,we can have arbitrarily large ‖J ∗ − Φr‖ (or infeasibility!)



J*

J = ΦrJ(1)

J(2)

TJ J>




J*

J = Φr

Φr~

Φr*

J(1)

J(2)

TJ J>



Error and performance boundsSimple bound: If Φv = e for some v,

‖J∗ − Φr‖1,c ≤2

1− α‖J∗ − Φr∗‖∞

Limitations:state-relevance weights?maximum norm to assess architecture

“Lyapunov function” V > 0:

αmaxaE[V (y)|x, a] ≤ βV (x)

Theorem: If Φv is a “Lyapunov function” for some v,

‖J∗ − Φr‖1,c ≤2cTΦv

1− β‖J∗ − Φr∗‖∞,1/Φv


Error Bound InsightsError proportional to best in architecture

‖J∗ − Φr∗‖∞,1/V = maxx

|J∗(x)− (Φr∗)(x)|

V (x)

V (x) large in rarely visited states⇒ good scalingpropertiesFor multiclass queueing networks, error uniformlybounded on• size of the state space• dimension of the state space

Performance bound:

‖Ju − J∗‖1,πu≤

1

1− α‖J∗ − Φr‖1,πu

We have bound on ‖J∗ − Φr‖1,c



‖J∗ − Φr∗‖∞,1/V = maxx

|J∗(x)− (Φr∗)(x)|

V (x)

V (x) large in rarely visited states⇒ good scalingproperties

For multiclass queueing networks, error uniformlybounded on• size of the state space• dimension of the state space

Performance bound:

‖Ju − J∗‖1,πu≤

1

1− α‖J∗ − Φr‖1,πu




‖J∗ − Φr∗‖∞,1/V = maxx

|J∗(x)− (Φr∗)(x)|

V (x)


Performance bound:

‖Ju − J∗‖1,πu≤

1

1− α‖J∗ − Φr‖1,πu




‖J∗ − Φr∗‖∞,1/V = maxx

|J∗(x)− (Φr∗)(x)|

V (x)


Performance bound:

‖Ju − J∗‖1,πu≤

1

1− α‖J∗ − Φr‖1,πu

We have bound on ‖J∗ − Φr‖1,chttp://www.mit.edu/∼pucci – p. 20/29

Example: 8-dimensional queueing network

Minimize total number of jobs in the system

Linear and quadratic basis functionsState-relevance weights with exponential decayAverage cost:

ALP 136.67LBFS 153.82FIFO 163.63LONGEST 168.66



Minimize total number of jobs in the systemLinear and quadratic basis functionsState-relevance weights with exponential decay

Average cost:ALP 136.67LBFS 153.82FIFO 163.63LONGEST 168.66



Minimize total number of jobs in the systemLinear and quadratic basis functionsState-relevance weights with exponential decayAverage cost:

ALP 136.67LBFS 153.82FIFO 163.63LONGEST 168.66


TetrisComparison against reported results

Algorithm Average Score Time

variation on TD (Bertsekas and Ioffe) 3500 many hours

variation on policy gradient (Kakade) 6000 days

ALP (Farias* and Van Roy) 5000 hours

* not me!

Remarks:3 minutes to solve the approximate LP, rest of the timespent on simulationsolution is very sensitive to c


Constraint Sampling in the Approximate LPone constraint per state-action pair

many constraints in low-dimensional space⇒ redundancyProblem-specific approaches in the literature:

Grötschel and Holland (1991)Morrison and Kumar (1999)Guestrin et al. (2002)Schuurmans and Patrascu (2002)

Generic approach? Complexity bounds?


Constraint Sampling in the Approximate LPone constraint per state-action pairmany constraints in low-dimensional space⇒ redundancy

Problem-specific approaches in the literature:Grötschel and Holland (1991)Morrison and Kumar (1999)Guestrin et al. (2002)Schuurmans and Patrascu (2002)



Constraint Sampling in the Approximate LPone constraint per state-action pairmany constraints in low-dimensional space⇒ redundancyProblem-specific approaches in the literature:

Grötschel and Holland (1991)Morrison and Kumar (1999)Guestrin et al. (2002)Schuurmans and Patrascu (2002)



The Reduced LP

maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y

Pa(x, y)(Φr)(y) ≥ (Φr)(x), ∀x, ∀a

N contains i.i.d. state-action pairsB is a bounding box

Theorem: With ideal sampling distribution, if

|N | = poly

(

p, |A|,1

1− α,1

ε, log

1

δ, θN ,V

)

then with probability at least 1− δ,‖J∗ − Φr‖1,c ≤ ‖J

∗ − Φr‖1,c + ε‖J∗‖1,c.


The Reduced LP

maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y

Pa(x, y)(Φr)(y) ≥ (Phir)(x), ∀(x, a) ∈ N

r ∈ B

N contains i.i.d. state-action pairs

B is a bounding box


|N | = poly

(

p, |A|,1

1− α,1

ε, log

1

δ, θN ,V

)


∗ − Φr‖1,c + ε‖J∗‖1,c.


The Reduced LP

maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y


r ∈ B



|N | = poly

(

p, |A|,1

1− α,1

ε, log

1

δ, θN ,V

)


∗ − Φr‖1,c + ε‖J∗‖1,c.


The Reduced LP

maxr∑

x

c(x)(Φr)(x)

s.t. ga(x) + α∑

y


r ∈ B



|N | = poly

(

p, |A|,1

1− α,1

ε, log

1

δ, θN ,V

)


∗ − Φr‖1,c + ε‖J∗‖1,c.http://www.mit.edu/∼pucci – p. 25/29

Remarks on Constraint SamplingSample complexity is

polynomial in number of basis functionsindependent of dimensions of the state space

linear on maximum number of actions per state |A|but can do with log|A|

“ideal” distribution“Bounding set” N



polynomial in number of basis functionsindependent of dimensions of the state spacelinear on maximum number of actions per state |A|

but can do with log|A|




polynomial in number of basis functionsindependent of dimensions of the state spacelinear on maximum number of actions per state |A|but can do with log|A|



Intuition for constraint sampling

Air + bi ≥ 0, i ∈ I, r ∈ <p

well-approximated with poly(p) constraints

constraint characterized by vector [Ai bi] ∈ <p+1

for feasibility, assume w.l.g. ‖[Ai bi]‖ = 1


In Short...Approximate dynamic programming: central ideas and issues

Approximate linear programming: analysis, performance anderror bounds

first approximation error bounds for arbitrary basisfunctions and decisionsuniform bounds for multiclass queueing networks

Forthcoming:analysis of case α ↑ 1• Lyapunov function argument breaks down• state-relevance weights c disappearrelaxation of Lyapunov function argumentnew variant of approximate LPimproved error bounds


In Short...Approximate dynamic programming: central ideas and issuesApproximate linear programming: analysis, performance anderror bounds

first approximation error bounds for arbitrary basisfunctions and decisionsuniform bounds for multiclass queueing networks

Forthcoming:analysis of case α ↑ 1• Lyapunov function argument breaks down• state-relevance weights c disappearrelaxation of Lyapunov function argumentnew variant of approximate LPimproved error bounds


Future WorkChoice of state-relevance weights c

Address norm discrepancy between error bound andperformance bound

Adaptive selection of basis functionsOnline versions of the algorithm

Robustness to model uncertaintyIncremental solution of the LPLearning the Q function instead of the value function

Issues on constraint samplingSpecific applications: how far can we push guarantees?




Adaptive selection of basis functions

Online versions of the algorithmRobustness to model uncertaintyIncremental solution of the LPLearning the Q function instead of the value function







Issues on constraint sampling

Specific applications: how far can we push guarantees?


the linear programming approach to approximate dynamic...

Documents