1

An Empirical Dynamic Programming Algorithm forContinuous MDPs

William B. Haskell, Rahul Jain, Hiteshi Sharma and Pengqian Yu

Abstract—We propose universal randomized functionapproximation-based empirical value iteration (EVI) algorithmsfor Markov decision processes. The ‘empirical’ nature comesfrom each iteration being done empirically from samplesavailable from simulations of the next state. This makes theBellman operator a random operator. A parametric and anon-parametric method for function approximation using aparametric function space and the Reproducing Kernal HilbertSpace (RKHS) respectively are then combined with EVI. Bothfunction spaces have the universal function approximationproperty. Basis functions are picked randomly. Convergenceanalysis is done using a random operator framework withtechniques from the theory of stochastic dominance. Finitetime sample complexity bounds are derived for both universalapproximate dynamic programming algorithms. Numericalexperiments support the versatility and effectiveness of thisapproach.

Index Terms—Continuous state space MDPs; Dynamic pro-gramming; Reinforcement Learning.

I. INTRODUCTION

There exist a wide variety of approximate dynamic pro-gramming (DP) [1, Chapter 6], [2] and reinforcement learning(RL) algorithms [3] for finite state space Markov decisionprocesses (MDPs). But many real-world problems of interesthave either a continuous state space, or large enough that it isbest approximated as one. Approximate DP and RL algorithmsdo exist for continuous state space MDPs but choosing whichone to employ is an art form: different techniques (state spaceaggregation and function approximation [4]) and algorithmswork for different problems, and universally applicable al-gorithms are lacking. For example, fitted value iteration [5]is very effective for some problems but requires the choiceof an appropriate basis functions for good approximation. Noguidelines are available on how many basis functions to pickto ensure a certain quality of approximation. The strategy thenemployed is to use a large enough basis but has demandingcomputational complexity.

In this paper, we propose approximate DP algorithms forcontinuous state space MDPs that are universal (approximatingfunction space can provide arbitrarily good approximation forany problem), computationally tractable, simple to implementand yet we have non-asymptotic sample complexity bounds.The first is accomplished by picking functions spaces for

W.B. Haskell and P. Yu are with the Department of Industrial and SystemsEngineering, National University of Singapore.

Rahul Jain, and Hiteshi Sharma are with EE Department, University ofSouthern California. The second author’s work was supported by an ONRYoung Investigator Award #N000141210766.

Manuscript submitted: September 25, 2017

approximation that are dense in the space of continuous func-tions. The second goal is achieved by relying on randomizedselection of basis functions for approximation and also by‘empirical’ dynamic programming [6]. The third is enabledbecause standard python routines can be used for functionfitting and the fourth is by analysis in a random operatorframework which provides non-asymptotic rate of convergenceand sample complexity bounds.

There is a large body of well-known literature on rein-forcement learning and approximate dyamic programming forcontinuous state space MDPs. We discuss the most directlyrelated. In [7], a sampling-based state space aggregationscheme combined with sample average approximation forthe expectation in the Bellman operator is proposed. Undersome regularity assumptions, the approximate value func-tion can be computed at any state and an estimate of theexpected error is given. But the algorithm seems to sufferfrom poor numerical performance. A linear programming-based constraint-sampling approach was introduced in [8].Finite sample error guarantees, with respect to this constraint-sampling distribution are provided but the method suffers fromissues of feasibility. The closest paper to ours is [5] thatdoes function fitting with a given basis and does ‘empirical’value iteration in each step. Unfortunately, it is not a universalmethod as approximation quality depends on the functionbasis picked. Other papers worth noting are [9] that discusseskernel-based value iteration and the bias-variance tradeoff,and [10] that proposed a kernel-based algorithm with randomsampling of the state and action spaces, and proves asymptoticconvergence.

This paper is inspired by the ‘random function’ approachthat uses randomization to (nearly) solve otherwise intractableproblems (see e.g., [11], [12]) and the ‘empirical’ approachthat reduces computational complexity of working with ex-pectations [6]. We propose two algorithms. For the firstparametric approach, we pick a parametric function family.In each iteration a number of functions are picked randomlyfor function fitting by sampling the parameters. For the secondnon-parametric approach, we pick a RKHS for approximation.Both function spaces are dense in the space of continuousfunctions. In each iteration, we sample a few states from thestate space. Empirical value iteration (EVI) is then performedon these states. Each step of EVI involves approximatingthe Bellman operator with an empirical (random) Bellmanoperator by plugging a sample average approximation fromsimulation for the expectation. This is akin to doing stochasticapproximations with step size 1. But an elegant randomoperator framework for convergence analysis using stochastic

arX

iv:1

709.

0750

6v1

[m

ath.

OC

] 2

1 Se

p 20

17

2

dominance was introduced recently in [6] and comes in handyfor us as well.

The main contribution of this paper is development ofrandomized function approximation-based (offline) dynamicprogramming algorithms that are universally applicable (i.e.,do not require appropriate choice of basis functions for goodapproximation). A secondary contribution is further devel-opment of the random operator framework for convergenceanalysis in the Lp−norm that also yields finite time samplecomplexity bounds.

The paper is organized as follows. Section II presentspreliminaries including the continuous state space MDP modeland the empirical dynamic programming framework for finitestate MDPs introduced in [6]. Section III presents two em-pirical value iteration algorithms - first, a randomized para-metric function fitting method, and second, a non-parametricrandomized function fitting in an RKHS space. We also pro-vide statements of main theorems about non-asymptotic errorguarantees. Section IV presents a unified analysis of the twoalgorithms in a random operator framework. Numerical resultsare reported in Section V. Secondary proofs are relegated tothe appendix.

II. PRELIMINARIES

Consider a discrete time discounted MDP given by the5-tuple, (S, A, Q, c, γ). The state space S is a compactsubset of Rd with the Euclidean norm, with correspond-ing Borel σ−algebra B (S). Let F (S) be the space of allB (S)−measurable bounded functions f : S→ R in the supre-mum norm ‖f‖∞ := sups∈S |f (s) |. Moreover, let M (S) bethe space of all probability distributions over S and definethe L2 norm as ‖f‖22, µ :=

(∫S |f (s) |2µ (ds)

)for given

µ ∈ M (S). We assume that the action space A is finite. Thetransition law Q governs the system evolution. For B ∈ B (S),Q (B | s, a) is the probability of next visiting the set B giventhat action a ∈ A is chosen in state s ∈ S. The cost functionc : S×A→ R is a measurable function that depends on state-action pairs. Finally, γ ∈ (0, 1) is the discount factor.

We will denote by Π the class of stationary deterministicMarkov policies: mappings π : S → A which only dependon history through the current state. For a given state s ∈ S,π (s) ∈ A is the action chosen in state s under the policyπ. The state and action at time t are denoted st and at,respectively. Any policy π ∈ Π and initial state s ∈ Sdetermine a probability measure Pπs and a stochastic process(st, at) , t ≥ 0 defined on the canonical measurable spaceof trajectories of state-action pairs. The expectation operatorwith respect to Pπs is denoted Eπs [·].

We will assume that the cost function c satisfies |c (s, a) | ≤cmax < ∞ for all (s, a) ∈ S × A. Under this assump-tion, ‖vπ‖∞ ≤ vmax := cmax/ (1− γ) where vπ is thevalue function for policy π ∈ Π defined as vπ (s) =Eπs [

∑∞t=0 γ

tc (st, at)] , ∀s ∈ S. For later use, we defineF (S; vmax) to be the space of all functions f ∈ F (S) suchthat ‖f‖∞ ≤ vmax.

The optimal value function is v∗ (s) :=infπ∈Π Eπs [

∑∞t=0 γ

tc (st, at)] , ∀s ∈ S. To characterize

the optimal value function, we define the Bellman operatorT : F (S)→ F (S) via

[T v] (s) := mina∈A

c (s, a) + γ EX∼Q(· | s, a) [v (X)]

, ∀s ∈ S.

It is known that the optimal value function v∗ is a fixed pointof T , i.e. T v∗ = v∗. Classical value iteration is based oniterating T to obtain a fixed point, it produces a sequence(vk)k≥0 ⊂ F (S) given by vk+1 = T vk, k ≥ 0. Also, weknow that (vk)k≥0 converges to v∗ geometrically in ‖ · ‖∞.

We are interested in approximating the optimal value func-tion v∗ within a tractable class of approximating functionsF ⊂ F (S). We have the following definitions which we useto measure the approximation power of F with respect to T .We define dp, µ (T f, F) := inff ′∈F ‖f ′ − T f‖p, µ to be theapproximation error for a specific f ; and

dp, µ (T F , F) := supf∈F

dp, µ (T f, F)

to be the inherent Lp Bellman error for the function class F .Similarly, the inherent L∞ Bellman error for an approximatingclass F is

d∞ (T F , F) , supg∈F

inff∈F‖f − T g‖∞.

We often compare F to the Lipschitz continuous functionsLip (L) defined as

f ∈ F (S) : |f (s)− f (s′) | ≤ L ‖s− s′‖, ∀s, s′ ∈ S .

In our case, we say that an approximation class F is universalif d∞ (Lip (L) , F) = 0 for all L ≥ 0.

One of the difficulties of dynamic programming algorithmslike value iteration above is that each iteration of the Bellmanoperator involves computation of an expectation which may beexpensive. Thus, in [13], we proposed replacing the Bellmanoperator with an empirical (or random) Bellman operator,[

Tnv]

(s) := mina∈A

c (s, a) +

γ

n

n∑i=1

[v(Xi)]

,

where Xi are samples of the next state from Q (· | s, a) whichcan be obtained from simulation. Now, we can iterate theempirical Bellman operator,

vk+1 = Tnvk

an algorithm we called Empirical Value Iteration (EVI). Thesequence of iterates vk is a random process. Since T is acontractive operator, its’ iterates converge to its fixed pointv∗. The random operator Tn may be expected to inherit thecontractive property in a probabilistic sense and its’ iteratesconverge to some sort of a probabilitic fixed point. Weintroduce (ε, δ) versions of two such notions introduced in[6].

Definition 1. A function v : S → R is an (ε, δ)-strongprobabilistic fixed point for a sequence of random operatorsTn if there exists an N such that for all n > N ,

P(||Tnv − v|| > ε

)< δ.

3

It is called a strong probabilistic fixed point, if the above istrue for every positive ε and δ.

Definition 2. A function v : S → R is an (ε, δ)-weakprobabilistic fixed point for a sequence of random operatorsTn if there exist N and K such that for all n > N and allk > K,

P(||T knv0 − v|| > ε

)< δ, ∀v0 ∈ F (S) .

It is called a weak probabilistic fixed point, if the above istrue for every positive ε and δ. Note that the stochastic iterativealgorithms such as EVI often find the weak probabilistic fixedpoint of Tn whereas what we are looking for is v∗, thefixed point of T . In [13], it was shown that asymptoticallythe weak probabilistic fixed point of Tn coincides with itsstrong probabilistic fixed points which coincide with the fixedpoint of T under certain fairly weak assumptions and a naturalrelationship between T and Tn

limn→∞

P(||Tnv − Tv|| > ε

)= 0, ∀v ∈ F (S) .

This implies that stochastic iterative algorithms such as EVIwill find approximate fixed points of T with high probability.

III. THE ALGORITHMS AND MAIN RESULTS

When the state space S is very large, or even uncountable,exact dynamic programming methods are not practical, or evenfeasible. Instead, one must use a variety of approximationmethods. In particular, function approximation (or fitting thevalue function with a fixed function basis) is a commontechnique. The idea is to sample a finite set of states fromS, approximate the Bellman update at these states, and thenextend to the rest of S through function fitting similar to[5]. Furthermore, the expectation in the Bellman operator, forexample, is also approximated by taking a number of samplesof the next state. There are two main difficulties with thisapproach: First, the function fitting depends on the functionbasis chosen, making the results problem-dependent. Second,with a large basis (for good approximation), function fittingcan be computationally expensive.

In this paper, we aim to address these issues by firstpicking universal approximating function spaces, and thenusing randomization to pick a smaller basis and thus reducecomputational burden of the function fitting step. We considertwo functional families, one is a parametric family F(Θ)parameterized over parameter space Θ and the other is a non-parametric regularized RKHS. By µ ∈M (S), we will denotea probability distribution from which to sample states in S,and by a F ⊂ F (S; vmax) we will denote a functional familyin which to do value function approximation.

Let us denote by (vk)k≥0 ⊂ F (S; vmax), the iterates of thevalue functions produced by an algorithm and a sample of sizeN ≥ 1 from S is denoted s1:N = (s1, . . . , sN ). The empiricalp−norm of f is defined as ‖f‖pp, µ := 1

N

∑Nn=1 |f (sn) |p

for p ∈ [1, ∞) and as ‖f‖∞, µ := supn=1,..., N |f (sn) | forp =∞, where µ is the empirical measure coresponding to thesamples s1:N .

We will make the following technical assumptions for therest of the paper similar to those made in [5].

Assumption 1. (i) For all (s, a) ∈ S × A, Q (· | s, a) isabsolutely continuous with respect to µ and

Cµ := sup(s, a)∈S×A

‖dQ (· | s, a) /dµ‖∞ <∞.

(ii) Given any sequence of policies πmm≥1, the futurestate distribution ρQπ1 · · ·Qπm is absolutely continuous withrespect to µ,

cρ, µ (m) := supπ1,..., πm

‖d (ρQπ1 · · ·Qπm) /dµ‖∞ <∞,

and Cρ, µ :=∑m≥0 γ

mcρ, µ (m) <∞.

The above assumptions are conditions on transition proba-bilities, the first being a sufficient condition for the second.

A. Random Parametric Basis Function (RPBF) Approximation

We introduce an empirical value iteration algorithm withfunction approximation using random parametrized basis func-tions (EVI+RPBF). It requires a parametric family F builtfrom a set of parameters Θ with probability distributionν and a feature function φ : S × Θ → R (that dependson both states and parameters) with the assumption thatsup(s, θ)∈S×Θ |φ (s; θ) | ≤ 1. This can easily be met in practiceby scaling φ whenever S and Θ are both compact and φ iscontinuous in (s, θ). Let α : Θ→ R be a weight function anddefine F (Θ) :=f (·) =

∫Θ

φ (·; θ)α (θ) dθ : |α (θ) | ≤ C ν (θ) , ∀θ ∈ Θ

.

We note that the condition |α (θ) | ≤ C ν (θ) for allθ ∈ Θ is equivalent to requiring that ‖α‖∞, ν :=supθ∈Θ |α (θ) /ν (θ) | ≤ C where ‖α‖∞, ν is the ν−weightedsupremum norm of α and C is a constant.

The function space F (Θ) may be chosen to have the ‘uni-versal’ function approximation property in the sense that anyLipschitz continuous function can be approximated arbitrarilyclosely in this space as shown in [11]. By [11, Theorem 2],many such choices of F (Θ) are possible and are developedin [11, Section 5]. For example, F (Θ) is universal in thefollowing two cases:• φ (s; θ) = cos (〈ω, s〉+ b) where θ = (ω, b) ∈ Rd+1;

and ν (θ) is given by ω ∼ Normal (0, 2 γ I) and b ∼Uniform [−π, π];

• φ (s; θ) = sign (sk − t) where θ = (t, k) ∈R × 1, . . . , d; and ν (θ) to be given by k ∼Uniform 1, . . . , d and t ∼ Uniform [−a, a].

In this approach, we have a parametric function familyF (Θ) but instead of optimizing over parameters in Θ, werandomly sample them first and then do function fittingwhich involves optimizing over finite weighted combinations∑Jj=1 αjφ (·; θj). Unfortunately, this leads to a non-convex

optimization problem. Hence, instead of optimizing overθ1:J = (θ1, . . . , θJ) and α1:J = (α1, . . . , αJ) jointly, we firstdo randomization over θ1:J and then optimization over α1:J ,as in [12], to bypass the non-convexity inherent in optimizingover θ1:J and α1:J simultaneously.

This approach allows us to deploy rich parametric familieswithout much additional computational cost. Once we draw a

4

random sample θjJj=1 from Θ according to ν, we obtain arandom function space: F

(θ1:J

):=f (·) =

J∑j=1

αjφ (·; θj) : ‖ (α1, . . . , αJ) ‖∞ ≤ C/J

.

Step 1 of such an algorithm (Algorithm 1) involves samplingstates s1:N over which to do value iteration and samplingparameters θ1:J to pick basis functions φ(·; θ) which are usedto do function fitting. Step 2 involves doing an empirical valueiteration over states s1:N by sampling next states (Xsn, a

m )Mm=1

according to the transition kernel Q, and using the currentiterate of the value function vk. Note that fresh (i.i.d.) samplesof the next state are regenerated in each iteration. Step 3involves finding the best fit to vk, the iterate from Step 2,within F

(θ1:J

)wherein randomly sampled parameters θ1:J

specify the basis functions for function fitting and weightsα1:J are optimized, which is a convex optimization problem.

Algorithm 1 EVI with random parameterized basis functions(EVI+RPBF)Input: probability distribution µ on S and ν on Θ;Sample sizes N ≥ 1, M ≥ 1, J ≥ 1; initial seed v0. Fork = 1, . . . ,K

1) Sample (sn)Nn=1 ∼ µN and (θj)Jj=1 ∼ νJ .

2) Compute

vk (sn) = mina∈A

c (sn, a) +

γ

M

M∑m=1

vk (Xsn, am )

,

where (Xsn, am ) ∼ Q (· | sn, a), m = 1, · · · ,M are i.i.d.

3) αk = arg minα1N

∑Nn=1

(∑Jj=1 αjφ (sn; θj)− v (sn)

)2

s.t. ‖ (α1, . . . , αJ) ‖∞ ≤ C/J .vk+1(s) =

∑Jj=1 α

kjφ(s; θj).

4) Increment k ← k + 1 and return to Step 1.

We note that Step 3 of the algorithm can be replaced byanother method for function fitting (as we do in the nextsubsection). The above algorithm differs from Fitted ValueIteration (FVI) algorithm of [5] in how it does functionfitting. FVI does function fitting with a deterministic andgiven set of basis functions which limits its universality whilewe do function fitting in a much larger space which hasa universal function approximation property but are able toreduce computational complexity by exploiting randomization.

In [5, Section 7], it is shown that if the transition kerneland cost are smooth, in the sense that

‖Q (· | s, a)−Q (· | s′, a) ‖TV ≤ LQ‖s− s′‖2 (1)

and|c (s, a)− c (s′, a) | ≤ Lc‖s− s′‖2 (2)

hold for all s, s′ ∈ S and a ∈ A, then the Bellmanoperator T maps bounded functions to Lipschitz continuousfunctions. In particular, if v is uniformly bounded by vmax thenT v is (Lc + γ vmaxLQ)−Lipschitz continuous. Subsequently,the inherent L∞ Bellman error satisfies d∞ (T F , F) ≤

d∞ (Lip (L) , F) since T F ⊂ Lip (L). So, it only remainsto choose an F (Θ) that is dense in Lip (L) in the supremumnorm, for which many examples exist.

We now provide non-asymptotic sample complexity boundsto establish that Algorithm 1 yields an approximately optimalvalue function with high probability. We provide guaranteesfor both the L1 and L2 metrics on the error. Denote

N1(ε, δ′) = 2752v2max log

[40 e (J + 1)

δ(10 e vmax)

J

],

M1(ε, δ′) =

(v2

max

2

)log

[10N |A|

δ′

],

J1(ε, δ′) =

5C

ε

1 +

√2 log

5

δ′

2

,

K∗1 =

⌈ln (Cρ, µε)− ln (2 vmax)

ln γ

⌉, and

µ∗ (p; K∗) = (1− p) p(K∗−1),

where C is the same constant that appears in the definitionof F (Θ) (see [12]) and vmax = vmax/ε. Set δ′ := 1 −(1− δ/2)

1/(K∗1−1).

Theorem 1. Given an ε > 0, and a δ ∈ (0, 1), choose J ≥J1(ε, δ′), N ≥ N1(ε, δ′), M ≥ M1(ε, δ′), Then, for K ≥log (4/ (δ µ∗ (δ;K∗1 ))), we have

‖vK − v∗‖1, ρ ≤ 2Cρ, µ (d1, µ (T F (Θ) , F (Θ)) + 2 ε)

with probability at least 1− δ.

Remarks. That is, if we choose enough samples N of thestates, enough samples M of the next state, and enough ran-dom samples J of the parameter θ, and then for large enoughnumber of iterations K, the L1 error in the value functionis determined by the inherent Bellman error of the functionclass F (Θ). For the function families F(Θ) discussed earlier,the inherent Bellman error is indeed zero, and so the valuefunction will have small L1 error with high probability.Next we give a similar guarantee for L2 error for Algorithm1 by considering approximation in L2, µ (S) . We note thatL2, µ (S) is a Hilbert space and that many powerful functionapproximation results exist for this setting because of thefavorable properties of a Hilbert space.

Denote

N2(ε, δ′) = 2752v4max log

[40 e (J + 1)

δ

(10 e v2

max

)J],

M2(ε, δ′) =

(v2

max

2

)log

[10N |A|

δ′

],

J2(ε, δ′) =

5C

ε

1 +

√2 log

5

δ′

2

, and

K∗2 = 2

ln(C

1/2ρ, µε

)− ln (2 vmax)

ln γ

,where again vmax = vmax/ε. Set δ′ := 1−(1− δ/2)

1/(K∗2−1).

5

Theorem 2. Given an ε > 0, and a δ ∈ (0, 1), choose J ≥J2(ε, δ′), N ≥ N2(ε, δ′), M ≥ M2(ε, δ′), Then, for K ≥log (4/ (δ µ∗ (δ;K∗2 ))), we have

‖vK − v∗‖2, ρ ≤ 2γ1/2C1/2ρ, µ (d2, µ (T F (Θ) , F (Θ)) + 2 ε)

with probability at least 1− δ.

Remarks. Again, note that the above result implies that if wechoose a function families F(Θ) with inherent Bellman errorzero, enough samples N of the states, enough samples M ofthe next state, and enough random samples J of the parameterθ, and then for large enough number of iterations K, the valuefunction will have small L2 error with high probability.

B. Non-parametric Function Approximation in RKHS

We now consider non-parametric function approximationcombined with EVI. We employ a Reproducing Kernel HilbertSpace (RKHS) for function approximation since for suitablychosen kernels, it is dense in the space of continuous functionsand hence has a ‘universal’ function approximation property.In the RKHS setting, we can obtain guarantees directly withrespect to the supremum norm.

We will consider a regularized RKHS setting with a con-tinuous, symmetric and positive semidefinite kernel K :S × S → R and a regularization constant λ > 0. TheRKHS space, HK is defined to be the closure of thelinear span of K(s, ·)s∈S endowed with an inner prod-uct 〈·, ·〉HK . The inner product 〈·, ·〉HK for HK is de-fined such that 〈K (x, ·) , K (y, ·)〉HK = K (x, y) forall x, y ∈ S, i.e., 〈

∑i αiK (xi, ·) ,

∑j βjK (yj , ·)〉HK =∑

i, j αiβjK (xi, yj). Subsequently, the inner product satisfiesthe reproducing property: 〈K (s, ·) , f〉HK = f (s) for alls ∈ S and f ∈ HK . The corresponding RKHS norm isdefined in terms of the inner product ‖f‖HK :=

√〈f, f〉HK .

We assume that our kernel K is bounded so that κ :=sups∈S

√K (s, s) <∞.

To find the best fit f ∈ HK to a function with data(sn, v (sn))Nn=1, we solve the regularized least squaresproblem:

minf∈HK

1

N

N∑n=1

(f (sn)− v (sn))2

+ λ ‖f‖2HK

. (3)

This is a convex optimization problem (the norm squared isconvex), and has a closed form solution by the RepresenterTheorem. In particular, the optimal solution is of the formf (s) =

∑Nn=1 αnK (sn, s) where the weights α1:N =

(α1, . . . , αN ) are the solution to the linear system([K (si, sj)]

Ni, j=1 + λN I

)(αn)

Nn=1 = (v (sn))

Nn=1 . (4)

This yields EVI algorithm with randomized function fitting ina regularized RKHS (EVI+RKHS) displayed as Algorithm 2.

Note that the optimization problem in Step 3 in Algorithm2 is analogous to the optimization problem in Step 3 ofAlgorithm 1 which finds an approximate best fit within thefinite-dimensional space F

(θ1:J

), rather than the entire space

F (Θ), while Problem (3) in Algorithm 2 optimizes over the

entire spaceHK . This difference can be reconciled by the Rep-resenter Theorem, since it states that optimization over HKin Problem (3) is equivalent to optimization over the finite-dimensional space spanned by K (sn, ·) : n = 1, . . . , N.Note that the regularization λ ‖f‖2HK is a requirement of theRepresenter Theorem.

Algorithm 2 EVI with regularized RKHS (EVI+RKHS)Input: probability distribution µ on S;sample sizes N ≥ 1, M ≥ 1; penalty λ;initial seed v0; counter k = 0.For k = 1, . . . ,K

1) Sample snNn=1 ∼ µ.2) Compute

vk (sn) = mina∈A

c (sn, a) +

γ

M

M∑m=1

vk (Xsn, am )

,

whereXsn, aj

Mm=1

∼ Q (· | sn, a) are i.i.d.3) vk+1(·) is given by

arg minf∈HK

1

N

N∑n=1

(f(sn)− v(sn))2 + λ||f ||HK

.

4) Increment k ← k + 1 and return to Step 1.

We define the regression function fM : S→ R via

fM (s) , E

[mina∈A

c (s, a) +

γ

M

M∑m=1

v (Xs, am )

], ∀s ∈ S,

it is the expected value of our empirical estimator of T v. Asexpected, fM → T v as M → ∞. We note that fP is notnecessarily equal to T v by Jensen’s inequality. We requirethe following assumption on fM to continue.

Assumption 2. For every M ≥ 1, fM (s) =∫SK (s, y)α (y)µ (dy) for some α ∈ L2, µ (S).

Regression functions play a key role in statistical learningtheory, Assumption 2 states that the regression function lies inthe span of the kernel K. Additionally, when HK is dense inthe Lipschitz functions, then the inherent Bellman error is zero.For example, K (s, s′) = exp (−γ ‖s− s′‖2), K (s, s′) =1 − 1

a‖s − s′‖1, and K (s, s′) = exp (γ ‖s− s′‖1) are alluniversal kernels.

Denote

N∞(ε, δ′) =

(4CKκ

ε (1− γ)

)6

log

(4

δ′

)2

M∞(ε) =160 v2

max

(ε (1− γ))2 log

(2 |A| γ (8 vmax − ε (1− γ))

ε (1− γ) (2− γ)

)K∗∞ =

⌈ln (ε)− ln (4 vmax)

ln γ

⌉where CK is a constant independent of the dimension of S(see [14] for the details on how CK depends on K) and setδ′ = 1− (1− δ/2)

1/(K∗∞−1).

6

Theorem 3. Suppose Assumption 2 holds. Given any ε > 0and δ ∈ (0, 1), choose an N ≥ N∞(ε, δ′) and an M ≥M∞(ε). Then, for any K ≥ log (4/ (δ µ∗ (δ;K∗∞))),

‖vK − v∗‖∞ ≤ ε

with probability at least 1− δ.

Note that we provide guarantees on L1 and L2 error (can begeneralized to Lp) with the RPBF method and for L∞ errorwith the RKHS-based randomized function fitting method.Getting guarantees for the Lp error with the RKHS methodhas proved quite difficult, as has bounds on the L∞ error withthe RBPF method.

IV. ANALYSIS IN A RANDOM OPERATOR FRAMEWORK

We will analyze Algorithms 1 and 2 in terms of randomoperators since this framework is general enough to encompassmany such algorithms. The reader can see that Step 2 of bothalgorithms involves iteration of the empirical Bellman operatorwhile Step 3 involves a randomized function fitting step whichis done differently and in different spaces in both algorithms.We use random operator notation to write these algorithmsin a compact way, and then derive a clean and to a large-extent unified convergence analysis. The key idea is to use thenotion of stochastic dominance to bound the error process withan easy to analyze “dominating” Markov chain. Then, we caninfer the solution quality of our algorithms via the probabilitydistribution of the dominating Markov chain. This analysisidea refines (and in fact, simplifies) the idea we introduced in[13] for MDPs with finite state and action spaces (where thereis no function fitting) in the supremum norm. In this paper,we develop the technique further, give a stronger convergencerate, account for randomized function approximation, and alsogeneralize the technique to Lp norms.

We introduce a probability space (Ω,B (Ω) , P ) on whichto define random operators, where Ω is a sample space withelements denoted ω ∈ Ω, B (Ω) is the Borel σ−algebraon Ω, and P is a probability distribution on (Ω, B (Ω)). Arandom operator is an operator-valued random variable on(Ω, B (Ω) , P ). We define the first random operator on F (S)as T (v) = (sn, v (sn))

Nn=1 where (sn)Nn=1 is chosen from S

according to a distribution µ ∈M (S) and

v (sn) = mina∈A

c (sn, a) +

γ

M

M∑m=1

v (Xsn, am )

,

n = 1, . . . , N is an approximation of [T v] (sn) for alln = 1, . . . , N . In other words, T maps from v ∈ F (S; vmax)to a randomly generated sample of N input-output pairs(sn, v (sn))

Nn=1 of the function T v. Note that T depends on

sample sizes N and M . Next, we have the function reconstruc-tion operator ΠF which maps the data (sn, v (sn))

Nn=1 to an

element in F . Note that ΠF is not necessarily deterministicsince Algorithms 1 and 2 use randomized function fitting. Wecan now write both algorithms succinctly as

vk+1 = G vk := ΠF T vk, (5)

which can be further written in terms of residual error εk =G vk − T vk as

vk+1 = G vk = T vk + εk. (6)

Iteration of these operators corresponds to repeated samplesfrom (Ω,B(Ω), P ), so we define the space of sequences(Ω∞,B(Ω∞),P) where Ω∞ = ×∞k=0Ω with elements denotedω = (ωk)k≥0, B (Ω∞) = ×∞k=0B (Ω), and P is the probabilitymeasure on (Ω∞, B (Ω∞)) guaranteed by the Kolmogorovextension theorem applied to P .

The random sequences (vk)k≥0 in Algorithms 1 and 2 givenby

vk+1 = ΠF T (ωk) vk

= ΠF T (ωk) ΠF T (ωk−1) · · · ΠF T (ω0) v0,

for all k ≥ 0 is a stochastic process defined on(Ω∞,B(Ω∞),P). We now analyze error propagation over theiterations.

Let us now investigate how the Bellman residual at eachiteration of EVI affects the quality of the resulting policy.There have already been some results which address the errorpropagation both in L∞ and Lp (p ≥ 1) norms [15]. Afteradapting [5, Lemma 3], we obtain the following point-wiseerror bounds on vK − v∗ in terms of the errors εkk≥0.

Lemma 4. For any K ≥ 1, and ε > 0, suppose ‖εk‖p, µ ≤ εfor all k = 0, 1, . . . , K − 1, then

‖vK−v∗‖p, ρ ≤ 2

(1− γK+1

1− γ

) p−1p [

C1/pρ, µε+ γK/p (2 vmax)

].

(7)

where Cρ, µ is the discounted-average concentrability coef-ficient of the future-state distributions as defined in [5, As-sumption A2]. Note that Lemma 4 assumes that ‖εk‖2, µ ≤ εwhich we will show in the subsequent section that it is truewith high probability.

The second inequality is for the supremum norm.

Lemma 5. For any K ≥ 1 and ε > 0, suppose ‖εk‖∞ ≤ εfor all k = 0, 1, . . . , K − 1, then

‖vK − v∗‖∞ ≤ ε/ (1− γ) + γK (2 vmax) . (8)

Inequalities (7) and (8) are the key to analyzing iteration ofEquation (6).

A. Convergence analysis using stochastic dominance

We now provide a (unified) convergence analysis for iter-ation of a sequence of random operators given by (5) and(6). Later, we will show how it can be applied to Algorithms1 and 2. We will use ‖ · ‖ to denote a general norm in thefollowing discussion, since our idea applies to all instances ofp ∈ [1, ∞) and p =∞ simultaneously. The magnitude of theerror in iteration k ≥ 0 is then ‖εk‖. We make the followingkey assumption for a general EVI algorithm.

Assumption 3. For ε > 0, there is a q ∈ (0, 1) such thatPr ‖εk‖ ≤ ε ≥ q for all k ≥ 0.

7

Assumption 3 states that we can find a lower bound onthe probability of the event ‖εk‖ ≤ ε that is independentof k and (vk)k≥0 (but does depend on ε). Equivalently, weare giving a lower bound on the probability of the event‖T vk − G vk‖ ≤ ε

. This is possible for all of the algo-

rithms that we proposed earlier. In particular, we can controlq in Assumption 3 through the sample sizes in each iterationof EVI. Naturally, for a given ε, q increases as the number ofsamples grows.

We first choose ε > 0 and the number of iterations K∗ forour EVI algorithms to reach a desired accuracy (this choiceof K∗ comes from the inequalities (7) and (8)). We calliteration k “good” if the error ‖εk‖ is within our desiredtolerance ε and “bad” when the error is greater than our desiredtolerance. We then construct a stochastic process (Xk)k≥0

on (Ω∞, B (Ω∞) , P) with state space K := 1, 2, . . . , K∗such that

Xk+1 =

max Xk − 1, 1 , if iteration k is "good",K∗, otherwise.

The stochastic process (Xk)k≥0 is easier to analyze than(vk)k≥0 because it is defined on a finite state space, however(Xk)k≥0 is not necessarily a Markov chain.

We next construct a “dominating” Markov chain (Yk)k≥0

to help us analyze the behavior of (Xk)k≥0. We construct(Yk)k≥0 on (K∞, B), the canonical measurable space oftrajectories on K, so Yk : K∞ → R, and we let Q denote theprobability measure of (Yk)k≥0 on (R∞, B). Since (Yk)k≥0

will be a Markov chain by construction, the probability mea-sure Q is completely determined by an initial distribution onR and a transition kernel for (Yk)k≥0. We always initializeY0 = K∗, and then construct the transition kernel as follows

Yk+1 =

max Yk − 1, 1 , w.p. q,K∗, w.p. 1− q,

where q is the probability of a “good” iteration with respect tothe corresponding norm. Note that the (Yk)k≥0 we introducehere is different and has much smaller state space than theone we introduced in [6] leading to stronger convergenceguarantees.

We now describe a stochastic dominance relationship be-tween the two stochastic processes (Xk)k≥0 and (Yk)k≥0.We will establish that (Yk)k≥0 is “larger” than (Xk)k≥0 ina stochastic sense.

Definition 3. Let X and Y be two real-valued randomvariables, then X is stochastically dominated by Y , writtenX ≤st Y , when E [f (X)] ≤ E [f (Y )] for all increas-ing functions f : R → R. Equivalently, X ≤st Y whenPr X ≥ θ ≤ Pr Y ≥ θ for all θ in the support of Y .

Let Fkk≥0 be the filtration on (Ω∞, B (Ω∞) , P) corre-sponding to the evolution of information about (Xk)k≥0, andlet [Xk+1 | Fk] denote the conditional distribution of Xk+1

given the information Fk. We have the following initial resultson the relationship between (Xk)k≥0 and (Yk)k≥0.

The following theorem, our main result for our randomoperator analysis, compares the marginal distributions of

(Xk)k≥0 and (Yk)k≥0 at all times k ≥ 0 when the twostochastic processes (Xk)k≥0 and (Yk)k≥0 start from the samestate.

Theorem 6. (i) Xk ≤st Yk for all k ≥ 0.(ii) Pr Yk ≤ η ≥ Pr Xk ≤ η for any η ∈ R and all

k ≥ 0.

Proof. First we note that, by [6, Lemma A.1], [Yk+1 |Yk = η]is stochastically increasing in η for all k ≥ 0, i.e.[Yk+1 |Yk = η] ≤st [Yk+1 |Yk = η′] for all η ≤ η′. Then, by[13, Lemma A.2], [Xk+1 |Xk = η, Fk] ≤st [Yk+1 |Yk = η]for all η ∈ f and Fk for all k ≥ 0.

(i) Trivially, X0 ≤st Y0 since X0 ≤as Y0. Next, we seethat X1 ≤st Y1 by [13, Lemma A.1]. We prove the generalcase by induction. Suppose Xk ≤st Yk for k ≥ 1, and for thisproof define the random variable

Y (θ) =

max θ − 1, 1 , w.p. q,K∗, w.p. 1− q.

to be the conditional distribution of Yk conditional on θ, asa function of θ. We see that Yk+1 has the same distributionas [Y (θ) | θ = Yk] by definition. Since Y (θ) are stochasti-cally increasing by Lemma [13, Lemma A.1], we see that[Y (θ) | θ = Yk] ≥st [Y (θ) | θ = Xk] by [16, Theorem 1.A.6]and our induction hypothesis. Now, [Y (θ) | θ = Xk] ≥st[Xk+1 |Xk, Fk] by [16, Theorem 1.A.3(d)] and Lemma [13,Lemma A.2] for all histories Fk. It follows that Yk+1 ≥stXk+1 by transitivity of ≥st.

(ii) Follows from part (i) by the definition of ≤st.

By Theorem 6, if XK ≤st YK and we can makePr YK ≤ η large, then we will also obtain a meaningfulbound on Pr XK ≤ η. Following this observation, the nexttwo corollaries are the main mechanisms for our generalsample complexity results for EVI.

Corollary 7. For a given p ∈ [1, ∞), and any ε > 0, andδ ∈ (0, 1), suppose Assumption 3 holds for this ε, and chooseany K∗ ≥ 1. Then for q ≥ (1/2 + δ/2)

1/(K∗−1) and K ≥log(4/((1/2− δ/2) (1− q) qK∗−1

)), we have

‖vK−v∗‖p, ρ ≤ 2

(1− γK∗+1

1− γ

) p−1p [

C1/pρ, µε+ γK

∗/p (2 vmax)]

with probability at least δ.

Proof. Since (Yk)k≥0 is an irreducible Markov chain on afinite state space, its steady state distribution µ = (µ (i))K

∗

i=1

on K exists. By [13, Lemma 4.3], the steady state distributionof (Yk)k≥0 is µ = (µ (i))K

∗

i=1 given by:

µ (1) = qK∗−1

µ (i) = (1− q) qK∗−i, ∀i = 2, . . . ,K∗ − 1,

µ (K∗) = 1− q.

The constant

µmin (q; K∗) := minqK∗−1, (1− q) q(K∗−2), (1− q)

,

for all q ∈ (0, 1) and K∗ ≥ 1, which is the minimum of thesteady state probabilities appears shortly in the Markov chain

8

mixing time bound for (Yk)k≥0. We note that µ∗ (q; K∗) =(1− q) qK∗−1 ≤ µmin (q; K∗) is a simple lower boundfor µmin (q; K∗) (we defined µ∗ (q; K∗) = (1− q) qK∗−1

earlier).Now, recall that ‖µ−ν‖TV = 1

2

∑K∗

η=1 |µ (η)−ν (η) | is thetotal variation distance for probability distributions on K. LetQk be the marginal distribution of Yk for k ≥ 0. By a Markovchain mixing time argument, e.g., [17, Theorem 12.3], we havethat

tmix (δ′) := mink ≥ 0 : ‖Qk − µ‖TV ≤ δ′

≤ log

(1

δ′µmin (q; K∗)

)≤ log

(1

δ′ (1− q) qK∗−1

)for any δ′ ∈ (0, 1). So, for K ≥ log

(1/(δ′ (1− q) qK∗−1

))we have |Pr YK = 1 − µ (1) | =

|Pr YK = 1 − qK∗−1| ≤ 2 ‖QK − µ‖TV ≤ 2 δ′,

where we use µ (1) = qK∗−1. By Theorem 6, Pr XK = 1 ≥

Pr YK = 1 and so

Pr XK = 1 ≥ qK∗−1 − 2 δ′.

Choose q and δ′ to satisfy qK∗−1 = 1/2 + δ/2 and 2 δ′ =

qK∗−1 − δ = 1/2 − δ/2 to get qK

∗−1 − 2 δ′ ≥ δ, and thedesired result follows.

The next result uses the same reasoning for the supremumnorm case.

Corollary 8. Given any ε > 0 and δ ∈ (0, 1), sup-pose Assumption 3 holds for this ε, and choose anyK∗ ≥ 1. For q ≥ (1/2 + δ/2)

1/(K∗−1) and K ≥log(4/((1/2− δ/2) (1− q) qK∗−1

)), we have

Pr‖vK − v∗‖∞ ≤ ε/ (1− γ) + γK

∗(2 vmax)

≥ δ.

The sample complexity results for both EVI algorithms fromSection III follow from Corollaries 7 and 8. This is shown next.

B. Proofs of Theorems 1, 2, and 3

We now apply our random operator framework to both EVIalgorithms. We will see that it is easy to check the conditionsof Corollaries 7 and 8, from which we obtain specific samplecomplexity results.

We can now give the proof of Theorem 1 by using ourMarkov chain technique. Based on Theorem 16 in the ap-pendix, we let p (N, M, J, ε) denote the lower bound on theprobability of the event

‖T v − T v‖1, µ ≤ ε

for ε > 0. We

also note that d1, µ (T v, F (Θ)) ≤ d1, µ (T F (Θ) , F (Θ)) forall v ∈ F (Θ).

Proof. (of Theorem 1) Starting with inequality (7) for p = 1and using the statement of Theorem 16, we have ‖vK−v∗‖1, ρ

≤ 2Cρ, µ (d1, µ (T F (Θ) , F (Θ)) + ε) + 4 vmaxγK

when ‖εk‖1, µ ≤ d1, µ (T F (Θ) , F (Θ)) + ε for all k =0, 1, . . . , K − 1. Choose K∗ such that

4 vmaxγK ≤ 2Cρ, µε⇒ K∗ =

⌈ln (Cρ, µε)− ln (2 vmax)

ln γ

⌉.

Based on Corollary 7, we just need to choose N, M, J

such that p (N, M, J, ε) ≥ (1− δ/2)1/(K∗−1). We then

apply the statement of Theorem 16 with probability 1 −(1− δ/2)

1/(K∗−1).

We now give the proof of Theorem 2 along similar linesbased on Theorem 17. Again, we let p (N, M, J, ε) de-note the a lower bound on the probability of the event‖T v − T v‖2, µ ≤ ε

.

Proof. (of Theorem 2) Starting with inequality (7) for p = 2and using the statement of Theorem 17, we have ‖vK−v∗‖2, ρ

≤ 2

(1

1− γ

)1/2

C1/2ρ, µ (d2, µ (T F (Θ) , F (Θ)) + ε)

+4

(1

1− γ

)1/2

vmaxγK/2,

when ‖εk‖2, µ ≤ d2, µ (T F (Θ) , F (Θ)) + ε for all k =0, 1, . . . , K − 1. We choose K∗ ≥ 1 to satisfy

4

(1

1− γ

)1/2

vmaxγK∗/2 ≤ 2

(1

1− γ

)1/2

C1/2ρ, µε

which implies K∗ = 2

⌈ln(C1/2

ρ, µε)−ln(2 vmax)

ln γ

⌉. Based on

Corollary 7, we just need to choose N, M, J such thatp (N, M, J, ε) ≥ (1− δ/2)

1/(K∗−1). We then apply thestatement of Theorem 16 with p = 1−(1− δ/2)

1/(K∗−1).

We now provide proof of L∞ function fitting in RKHSbased on Theorem 19 in the appendix. For this proof, we letp (N, M, ε) denote a lower bound on the probability of theevent

‖T v − T v‖∞ ≤ ε

.

Proof. (of Theorem 3) By inequality (8), we choose ε andK∗ ≥ 1 such that ε/ (1− γ) ≤ ε/2 and γK

∗(2 vmax) ≤ ε/2

by setting

K∗ ≥⌈

ln (ε)− ln (4 vmax)

ln (γ)

⌉.

Based on Corollary 7, we next choose N and M such thatp (N, M, ε) ≥ (1− δ/2)

1/(K∗−1). We then apply the state-ment of Theorem 19 with error ε (1− γ) /2 and probability1− (1− δ/2)

1/(K∗−1).

V. NUMERICAL EXPERIMENTS

We now present numerical performance of our algorithmby testing it on the benchmark optimal replacement problem[7], [5]. The setting is that a product (such as a car) becomesmore costly to maintain with time/miles, and must be replacedit some point. Here, the state st ∈ R+ represents the accumu-lated utilization of the product. Thus, st = 0 denotes a brandnew durable good. Here, A = 0, 1, so at each time step, t,we can either replace the product (at = 0) or keep it (at = 1).Replacement incurs a cost C while keeping the product has

9

Fig. 1. Relative Error with iterations for various algorithms.

a maintenance cost, c(st), associated with it. The transitionprobabilities are as follows:

q(st+1|st, at) =

λe−λ(st+1−st), if st+1 ≥ st and at = 1

λe−λst+1 , if st+1 ≥ 0 and at = 0

0, otherwise

and the reward function is given by

r(st, at) =

−c(st), if at = 1

−C − c(0), if at = 0

For our computation, we use γ = 0.6, λ = 0.5, C = 30 andc(s) = 4s. The optimal value function and the optimal policycan be computed analytically for this problem. For EVI+RPBF,we use J random parameterized Fourier functions φ(s, θj) =cos(θTj s+ b)Jj=1 with θj ∼ N (0, 0.01) and b ∼ Unif[−π, π].We fix J=5. For EVI+RKHS, we use Gaussian kernel definedas k(x, y) = exp(||x − y||2/(2σ2)) with 1/σ2 = 0.01 andL2 regularization. We fix the regularization coefficient to be10−2. The underlying function space for FVI is polynomialsof degree 4. The results are plotted after 20 iterations.

The error in each iteration for different algorithms withN = 100 states and M = 5 is shown in Figure 1. On Y-axis, it shows the relative error computed as sups∈S |v∗(s)−vk(s)/v∗(s)| with iterations k on the X-axis. It shows thatEVI+RPBF has relative error below 10% after 20 iterations.FVI is close but EVI+RKHS is slower though it may improvewith a higher M . This is also refelected in the actual run-time performance: EVI+RPBF takes 8,705s, FVI 8,654s andEVI+RKHS takes 42,173s to get within 0.1 relative error.

Note that performance of FVI depends on being ableto choose suitable basis functions which for the optimal

Goal EVI+RPBF FVI EVI+RKHS50 5.4m 4.8m 8.7m

100 18.3m 23.7m 32.1m150 36.7m 41.5m 54.3m

TABLE IRUNTIME PERFORMANCE OF VARIOUS ALGORITHMS ON THE CART-POLE

PROBLEM.

replacement problem is easy. For other problems, we mayexpect both EVI algorrithms to perform better. So, we testedthe algorithms on the cart-pole balancing problem, anotherbenchmark problem but for which the optimal value function isunknown. We formulate it as a continuous 2-dimensional statespace with 2 action MDP. The state comprises of the positionof the cart and angle of the pole. The actions are to add a forceof −10N or +10N to the cart, pushing it left or right. We add±50% noise to these actions. System dynamics for this systemare given in [3]. Rewards are zero except for failure state (ifthe position of cart reaches beyond ±2.4, or the pole exceedsan angle of ±12 degrees), it is −1. For our experiments, wechoose N = 100 and M = 1. In case of RPBF, we considerparameterized Fourier basis of the form cos(wT s + b) wherew = [w1, w2], w1, w2 ∼ N (0, 1) and b ∼ Unif[−π, π]. We fixJ = 10 for our EVI+RPBF. For RKHS, we consider Gaussiankernel, K(s1, s2) = exp

(−σ||s1 − s2||2/2

)with σ = 0.01.

We limit each episode to 1000 time steps. We compute theaverage length of the episode for which we are able to balancethe pole without hitting the failure state. This is the goal inTable I. The other columns show run-time needed for thealgorithms to learn to achieve such a goal.

From the table, we can see that EVI+RPBF outperformsFVI and EVI+RKHS. Note that guarantees for FVI are onlyavailable for L2-error and for EVI-RPBF for Lp-error. EVI-RKHS is the only algorithm that can provide guarantees on thesup-norm error. Also note that when for problems for whichthe value functions are not so regular, and good basis functionsdifficult to guess, the EVI+RKHS method is likely to performbetter but as of now we do not have a numerical example todemonstrate this.

VI. CONCLUSION

In this paper, we have introduced universally applicableapproximate dynamic programming algorithms for continuousstate space MDPs. The algorithms introduced are based onusing randomization to break the ‘curse of dimensionality’via the synthesis of the ‘random function approximation’ and‘empirical’ approaches. Our first algorithm is based on arandom parametric function fitting by sampling parameters ineach iteration. The second is based on sampling states whichthen yield a set of basis functions in an RKHS from thekernel. Both function fitting steps involve convex optimizationproblems and can be implemented with standard packages.Both algorithms can be viewed as iteration of a type of randomBellman operator followed by a random projection operator.This situation can be quite difficut to analyze. But viewedas a random operator, following [6], we can construct Markovchains that stochastically dominates the error sequences whichare quite easy to analyze. They yield convergence but also non-asymptotic sample complexity bounds. Numerical experimentson the cart-pole balancing problem suggests good performancein practice. More rigorous numerical analysis will be con-ducted as part of future work.

10

REFERENCES

[1] D. P. Bertsekas, “Dynamic programming and optimal control 3rd edition,volume ii,” Belmont, MA: Athena Scientific, 2011.

[2] W. B. Powell, Approximate Dynamic Programming: Solving the cursesof dimensionality. John Wiley & Sons, 2007, vol. 703.

[3] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.Cambridge Univ Press, 1998, vol. 1, no. 1.

[4] R. A. DeVore, “Nonlinear approximation,” Acta numerica, vol. 7, pp.51–150, 1998.

[5] R. Munos and C. Szepesvári, “Finite-time bounds for fitted valueiteration,” The Journal of Machine Learning Research, vol. 9, pp. 815–857, 2008.

[6] W. B. Haskell, R. Jain, and D. Kalathil, “Empirical dynamic program-ming,” Mathematics of Operations Research, vol. 41, no. 2, pp. 402–429,2016.

[7] J. Rust, “Using randomization to break the curse of dimensionality,”Econometrica: Journal of the Econometric Society, pp. 487–516, 1997.

[8] D. P. De Farias and B. Van Roy, “On constraint sampling in thelinear programming approach to approximate dynamic programming,”Mathematics of operations research, vol. 29, no. 3, pp. 462–478, 2004.

[9] D. Ormoneit and S. Sen, “Kernel-based reinforcement learning,” Ma-chine learning, vol. 49, no. 2-3, pp. 161–178, 2002.

[10] S. Grunewalder, G. Lever, L. Baldassarre, M. Pontil, and A. Gretton,“Modelling transition dynamics in mdps with rkhs embeddings,” arXivpreprint arXiv:1206.4655, 2012.

[11] A. Rahimi and B. Recht, “Uniform approximation of functions withrandom bases,” in Communication, Control, and Computing, 2008 46thAnnual Allerton Conference on. IEEE, 2008, pp. 555–561.

[12] ——, “Weighted sums of random kitchen sinks: Replacing minimizationwith randomization in learning,” in Advances in neural informationprocessing systems, 2009, pp. 1313–1320.

[13] W. B. Haskell, R. Jain, and D. Kalathil, “Empirical dynamic program-ming,” To appear in Mathematics of Operations Research, 2015.

[14] S. Smale and D.-X. Zhou, “Shannon sampling ii: Connections to learningtheory,” Applied and Computational Harmonic Analysis, vol. 19, no. 3,pp. 285–302, 2005.

[15] R. Munos, “Performance bounds in l_p-norm for approximate valueiteration,” SIAM journal on control and optimization, vol. 46, no. 2,pp. 541–561, 2007.

[16] M. Shaked and J. G. Shanthikumar, Stochastic Orders. Springer, 2007.[17] D. A. Levin, Y. Peres, and E. L. Wilmer, Markov Chains and Mixing

Times. American Mathematical Society, 2008.[18] M. Anthony and P. L. Bartlett, Neural network learning: Theoretical

foundations. cambridge university press, 2009.[19] D. Haussler, “Sphere packing numbers for subsets of the boolean n-cube

with bounded vapnik-chervonenkis dimension,” Journal of Combinato-rial Theory, Series A, vol. 69, no. 2, pp. 217–232, 1995.

APPENDIX

A. Supplement for Section III

The following computation shows that T maps boundedfunctions to Lipschitz continuous functions when Q and c areboth Lipschitz continuous in the sense of (1) and (2). Suppose‖v‖∞ ≤ vmax, then T v is Lipschitz continuous with constantLc + γ vmaxLQ. We have

| [T v] (s)− [T v] (s′) |≤ max

a∈A|c (s, a)− c (s′, a) |

+ γ maxa∈A|∫v (y)Q (dy | s, a)−

∫v (y)Q (dy | s′, a) |

≤Lc‖s− s′‖2 + γ vmax maxa∈A

∫|Q (dy | s, a)−Q (dy | s′, a) |

≤ (Lc + γ vmaxLQ) ‖s− s′‖2.

B. Supplement for Section IV

First, we need to adapt [5, Lemma 3] to obtain point-wiseerror bounds on vK−v∗ in terms of the errors εkk≥0. These

bounds are especially useful when analyzing the performanceof EVI with respect to other norms besides the supremumnorm, since T does not have a contractive property withrespect to any other norm.

For any π ∈ Π, we define the operator Qπ : F (S)→ F (S)(which gives the transition mapping as a function of π) via

(Qπv) (s) ,∫Sv (y)Q (dy | s, π (s)) , ∀s ∈ S.

Then we define the operator Tπ : F (S)→ F (S) via

[Tπv] (s) , c (s, π (s))+γ

∫Sv (x)Q (dx | s, π (s)) , ∀s ∈ S.

For later use, we let π∗ ∈ Π be an optimal policy satisfying

π∗ (s) ∈ arg mina∈A

c (s, a) + γ

∫Sv∗ (x)Q (dx | s, a)

,

∀s ∈ S,, i.e., it is greedy with respect to v∗. More generally,a policy π ∈ Π is greedy with respect to v ∈ F (S) if Tπv =T v.

For use throughout this section, we let πk be a greedy policywith respect to vk so that Tπkvk = T vk for all k ≥ 0. Then,for fixed K ≥ 1 we define the operators

AK ,1

2

[(Qπ∗)K

+QπK−1QπK−2 · · ·Qπ0

],

Ak ,1

2

[(Qπ∗)K−k−1

+QπK−1QπK−2 · · ·Qπk+1

],

for k = 0, . . . , K − 1, formed by composition of transitionkernels. We let ~1 be the constant function equal to one on S,

and we define the constant γ =2(1−γK+1)

1−γ for use shortly.We note that AkKk=0 are all linear operators and Ak~1 = ~1for all k = 0, . . . , K.

Lemma 9. For any K ≥ 1,(i) vK − v∗ ≤

∑K−1k=0 γK−k−1

(Qπ∗)K−k−1

εk +

γK(Qπ∗)K

(v0 − v∗).(ii) vK−v∗ ≥

∑K−1k=0 γK−k−1 (QπK−1QπK−2 · · ·Qπk+1) εk+

γK (QπK−1QπK−2 · · ·Qπ0) (v0 − v∗) .(iii) |vK−v∗| ≤ 2

[∑K−1k=0 γK−k−1Ak |εk|+ γKAK (2 vmax)

].

Proof. (i) For any k ≥ 1, we have T vk ≤ Tπ∗vk and Tπ

∗vk−

Tπ∗v∗ = γ Qπ

∗(vk − v∗), so vk+1 − v∗ =

T vk+εk−Tπ∗vk+Tπ

∗vk−Tπ

∗v∗ ≤ γ Qπ

∗(vk − v∗)+εk.

The result then follows by induction.(ii) Similarly, for any k ≥ 1, we have T v∗ ≤ Tπkv∗ and

T vk−Tπkv∗ = Tπkvk−Tπkv∗ = γ Qπk (vk − v∗), so vk+1−v∗ =

T vk + εk − Tπkv∗ + Tπkv∗ − T v∗ ≥ γ Qπk (vk − v∗) + εk.

Again, the result follows by induction.(iii) If f ≤ g ≤ h in F (S), then |g| ≤ |f | + |h|, so

combining parts (i) and (ii) gives

|vK − v∗| ≤ 2

K−1∑k=0

γK−k−1Ak|εk|+ 2 γKAK |v0 − v∗|.

11

Then we note that |v0 − v∗| ≤ 2 vmax.

Now we use Lemma 9 to derive p−norm bounds.

Proof. (of Lemma 4) Using∑Kk=0 γ

k =(1− γK+1

)/ (1− γ), we define the constants

αk =(1− γ) γK−k−1

1− γK+1, ∀k = 0, . . . , K − 1,

αK =(1− γ) γK

1− γK+1,

and we note that∑Kk=0 αk = 1. Then, we obtain |vK − v∗|

≤ γ

[K−1∑k=0

αkAk |εk|+ αKAK (2 vmax)

]from Lemma 9(iii). Next, we compute ‖vK − v∗‖pp, ρ =∫

S|vK (s)− v∗ (s) |pρ (ds)

≤ γp∫S

[K−1∑k=0

αkAk |εk|+ αKAK (2 vmax)~1

]p(s) ρ (ds)

≤ γp∫S

[K−1∑k=0

αkAk |εk|p + αKAK (2 vmax)p~1

](s) ρ (ds) ,

using Jensen’s inequality and convexity of x → |x|p. Now,we have ρAk ≤ cρ, µ (K − k − 1)µ for k = 0, . . . , K− 1 byAssumption 1(ii) and so for all k = 0, . . . , K − 1,∫

S[Ak |εk|p] (s) ρ (ds) ≤ cρ, µ (K − k − 1) ‖εk‖pp, µ.

We arrive at ‖vK − v∗‖pp, ρ

≤ γp

[K−1∑k=0

αkcρ, µ (K − k − 1) ‖εk‖pp, µ + αK (2 vmax)p

]

= 2pγp−1

[K−1∑k=0

γK−k−1cρ, µ (K − k − 1) ‖εk‖pp, µ + γK (2 vmax)p

],

where we use |v0 − v∗|p ≤ (2 vmax)p. Now, by subadditivity

of x→ |x|t for t = 1/p ∈ (0, 1] with p ∈ [1, ∞), assumptionthat ‖εk‖p, µ ≤ ε for all k = 0, 1, . . . , K − 1, and since∑K−1k=0 γK−k−1cρ, µ (K − k − 1) ≤ Cρ, µ by Assumption

1(ii), we see

‖vK−v∗‖p, ρ ≤ 2

(1− γK+1

1− γ

) p−1p [

C1/pρ, µε+ γK/p (2 vmax)

],

which gives the desired result.

Supremum norm error bounds follow more easily fromLemma 9.

Proof. (Proof of Lemma 5) We have

‖vK − v∗‖∞ ≤ max‖K−1∑k=0

γK−k−1(Qπ∗)K−k−1

εk

+γK(Qπ∗)K

(v0 − v∗) ‖∞,

‖K−1∑k=0

γK−k−1 (QπK−1QπK−2 · · ·QπK+1) εk

+γK (QπK−1QπK−2 · · ·Qπ0) (v0 − v∗) ‖∞,

by Lemma 9. Now,

‖K−1∑k=0

γK−k−1(Qπ∗)K−k−1

εk

+γK(Qπ∗)K

(v0 − v∗) ‖∞

≤K−1∑k=0

γK−k−1‖εk‖∞ + γK‖v0 − v∗‖∞

and

‖K−1∑k=0

γK−k−1 (QπK−1QπK−2 · · ·QπK+1) εk

+γK (QπK−1QπK−2 · · ·Qπ0) (v0 − v∗) ‖∞

≤K−1∑k=0

γK−k−1‖εk‖∞ + γK‖v0 − v∗‖∞,

where we use the triangle inequality, the fact that| (Qf) (s) | ≤

∫S |f (y) |Q (dy | s) ≤ ‖f‖∞ for any transition

kernel Q on S and f ∈ F (S), and |v0 − v∗| ≤ 2 vmax. Forany K ≥ 1,

‖vK − v∗‖∞ ≤K−1∑k=0

γK−k−1‖εk‖∞ + γK (2 vmax) . (9)

Follows immediately since∑K−1k=0 γK−k−1ε ≤ ε/ (1− γ) for

all K ≥ 1.

We emphasize that Lemma 5 does not require any assump-tions on the transition probabilities, in contrast to Lemma 4which requires Assumption 1(ii).

C. Supplement for Section IV: Function approximation

We record several pertinent results here on the type offunction reconstruction used in our EVI algorithms. Thefirst lemma is illustrative of approximation results in Hilbertspaces, it gives an O

(1/√J)

convergence rate on the error

from using F(θ1:J

)compared to F (Θ) in L2, µ (S) in prob-

ability.

Lemma 10. [12, Lemma 1] Fix f∗ ∈ F (Θ), for any δ ∈(0, 1) there exists a function f ∈ F

(θ1:J

)such that

‖f∗ − f‖2, µ ≤C√J

(1 +

√2 log

1

δ

)with probability at least 1− δ.

The next result is an easy consequence of [12, Lemma 1]and bounds the error from using F

(θ1:J

)compared to F (Θ)

in L1, µ (S).

12

Lemma 11. Fix f∗ ∈ F (Θ), for any δ ∈ (0, 1) there existsa function f ∈ F

(θ1:J

)such that

‖f − f∗‖1, µ ≤C√J

(1 +

√2 log

1

δ

)with probability at least 1− δ.

Proof. Choose f, g ∈ F (S), then by Jensen’s inequality wehave ‖f − g‖1, µ = Eµ [|f (S)− g (S) |]

= Eµ[(

(f (S)− g (S))2)1/2

]≤√Eµ[(f (S)− g (S))

2].

The desired result then follows by [12, Lemma 1].

Now we consider function approximation in the supremumnorm. Recall the definition of the regression function

fM (s) = E

[mina∈A

c (s, a) +

γ

M

M∑m=1

v (Xs, am )

],

∀s ∈ S. Then we have the following approximation result, forwhich we recall the constant κ := sups∈S

√K (s, s).

Corollary 12. [14, Corollary 5] For any δ ∈ (0, 1),

‖fz, λ − fM‖HK

≤ C κ

(log (4/δ)

2

N

)1/6

for λ =

(log (4/δ)

2

N

)1/3

,

with probability at least 1− δ.

Proof. Uses the fact that for any f ∈ HK , ‖f‖∞ ≤ κ ‖f‖HK .For any s ∈ S, we have |f (s) | = |〈K (s, ·) , f (·)〉HK | andsubsequently

|〈K (s, ·) , f (·)〉HK | ≤ ‖K (s, ·) ‖HK‖f‖HK=√〈K (s, ·) , K (s, ·)〉HK‖f‖HK

=√K (s, s)‖f‖HK

≤ sups∈S

√K (s, s)‖f‖HK ,

where the first inequality is by Cauchy-Schwartz and thesecond is by assumption that K is a bounded kernel.

The preceding result is about the error when approximatingthe regression function fM , but fM generally is not equal toT v. We bound the error between fM and T v as well in thenext subsection.

D. Bellman error

The layout of this subsection is modeled after the argumentsin [5], but with the added consideration of randomized functionfitting. We use the following easy-to-establish fact.

Fact 13. Let X be a given set, and f1 : X → R andf2 : X → R be two real-valued functions on X . Then,

(i) | infx∈X f1 (x) − infx∈X f2 (x) | ≤ supx∈X |f1 (x) −f2 (x) |, and

(ii) | supx∈X f1 (x) − supx∈X f2 (x) | ≤ supx∈X |f1 (x) −f2 (x) |.

For example, Fact 13 can be used to show that T iscontractive in the supremum norm.

The next result is about T , it uses Hoeffding’s inequal-ity to bound the estimation error between v (sn)Nn=1 and[T v] (sn)Nn=1 in probability.

Lemma 14. For any p ∈ [1,∞], f, v ∈ F (S; vmax), andε > 0,

Pr |‖f − T v‖p, µ − ‖f − v‖p, µ| > ε

≤ 2N |A| exp

(−2M ε2

v2max

).

Proof. First we have |‖f − T v‖p, µ − ‖f − v‖p, µ| ≤ ‖T v −v‖p, µ by the reverse triangle inequality. Then, for any s ∈ Swe have |[T v] (s)− v (s)| =

maxa∈A|c (s, a) + γ

∫Sv (x)Q (dx | s, a)

−

c (s, a) +

γ

M

M∑m=1

v (Xs, am )

|

≤ γ maxa∈A

∣∣∣∣∣∫Sv (x)Q (dx | s, a)− 1

M

M∑m=1

v (Xs, am )

∣∣∣∣∣by Fact 13. We may also take v (s) ∈ [0, vmax] for all s ∈S by assumption on the cost function, so by the Hoeffdinginequality and the union bound we obtain

Pr

maxn=1,..., N

|[T v] (sn)− v (sn)| ≥ ε

≤ 2N |A| exp

(−2M ε2

vmax2

)and thus

Pr ‖T v − v‖p, µ ≥ ε

= Pr(

1N

∑Nn=1 |[T v] (sn)− v (sn)|p

)1/p

≥ ε

≤ Pr maxn=1,..., N |[T v] (sn)− v (sn)| ≥ ε ,

which gives the desired result.

To continue, we introduce the following additional notationcorresponding to a set of functions F ⊂ F (S):• F

(s1:N

), (f (s1) , . . . , f (sN )) : f ∈ F;

• N(ε, F

(s1:N

))is the ε−covering number of F

(s1:N

)with respect to the 1−norm on RN .

The next lemma uniformly bounds the estimation error be-tween the true expectation and the empirical expectation overthe set F

(θ1:J

)(in the following statement, e is Euler’s

number).

Lemma 15. For any ε > 0 and N ≥ 1,

Pr

sup

f∈F(θ1:J )

∣∣∣∣∣ 1

N

N∑n=1

f (Sn)− Eµ [f (S)]

∣∣∣∣∣ > ε

≤ 8 e (J + 1)

(2 e vmax

ε

)Jexp

(−N ε2

128 v2max

).

Proof. For any F ⊂ F (S; vmax), ε > 0, and N ≥ 1, we have

13

Pr

sup

f∈F(θ1:J )

∣∣∣∣∣ 1

N

N∑n=1

f (Sn)− Eµ [f (S)]

∣∣∣∣∣ > ε

≤ 8E[N(ε/8, F

(θ1:J

) (s1:N

))]exp

(−N ε2

128 v2max

).

It remains to bound E[N(ε/8, F

(θ1:J

) (s1:N

))]. We note

that F(θ1:J

)is a subset off (·) =

J∑j=1

αjφ (·; θj) : (α1, . . . , αJ) ∈ RJ ,

which is a vector space with dimension J . By [18, Corollary11.5], the pseudo-dimension of F

(θ1:J

)is bounded above by

J . Furthermore,

N(ε, F

(θ1:J

) (s1:N

))≤ e (J + 1)

(2 e vmax

ε

)Jby [19, Corollary 3] which gives the desired result.

To continue, we let v′ = v′ (v, N, M, J, µ, ν) denote the(random) output of one iteration of EVI applied to v ∈ F (S)as a function of the parameters N, M, J ≥ 1 and theprobability distributions µ and ν. The next theorem boundsthe error between T v and v′ in one iteration of EVI withrespect to L1, µ (S), it is a direct adaptation of [5, Lemma 1]modified to account for the randomized function fitting andthe effective function space being F

(θ1:J

).

Theorem 16. Choose v ∈ F (S; vmax), ε > 0, and

δ ∈ (0, 1). Also choose J ≥[

5Cε

(1 +

√2 log 5

δ

)]2,

N ≥ 2752v2max log

[40 e(J+1)

δ (10 e vmax)J], and M ≥(

v2max

2 ε2

)log[

10N |A|δ

]. Then, for

v′ = v′ (v, N, M, J, µ, ν) we have ‖v′ − T v‖1, µ ≤d1, µ (T v, F (Θ)) + ε with probability at least 1− δ.

Proof. Let ε′ > 0 be arbitrary and choose f∗ ∈ F (Θ) suchthat ‖f∗ − T v‖1, µ ≤ inff∈F(Θ) ‖f − T v‖1, µ + ε′. Then,choose f ∈ F

(θ1:J

)such that ‖f−T v‖1, µ ≤ ‖f∗−T v‖1, µ+

ε/5 with probability at least 1−δ/5 by Lemma 11 by choosingJ ≥ 1 to satisfy

C√J

(1 +

√2 log

1

(δ/5)

)≤ ε

5

⇒ J ≥

[(5C

ε

)(1 +

√2 log

5

δ

)]2

.

Now consider the inequalities:

‖v′ − T v‖1, µ ≤‖v′ − T v‖1, µ + ε/5 (10)≤‖v′ − v‖1, µ + 2 ε/5 (11)

≤‖f − v‖1, µ + 2 ε/5 (12)

≤‖f − T v‖1, µ + 3 ε/5 (13)

≤‖f − T v‖1, µ + 4 ε/5 (14)≤‖f∗ − T v‖1, µ + ε (15)≤ d1, µ (T v, F (Θ)) + ε+ ε′. (16)

First, note that inequality (12) is immediate since ‖v′ −v‖1, µ ≤ ‖f−v‖1, µ for all f ∈ F

(θ1:J

)by the choice of v′ as

the minimizer in Step 3 of Algorithm 1. Second, inequalities(10) and (14) follow from Lemma 15 by choosing N ≥ 1 tosatisfy

8 e (J + 1)

(2 e vmax

ε/5

)Jexp

(−N (ε/5)

2

128 v2max

)≤ δ

5

⇒ N ≥ 2752v2max log

[40 e (J + 1)

δ(10 e vmax)

J

].

Third, inequality (16) follows from the choice of f∗ ∈ F .Finally, inequalities (11) and (13) follow from Lemma 14 bychoosing M ≥ 1 to satisfy

2N |A| exp

(−2M ε2

vmax2

)≤ δ

5

⇒M ≥(v2

max

2 ε2

)log

[10N |A|

δ

].

Since ε′ was arbitrary, the desired result then follows by theunion bound.

Using similar steps as Theorem 16, the next theorem boundsthe error in one iteration of EVI with respect to L2, µ (S).

Theorem 17. Choose v ∈ F (S; vmax), ε > 0, and

δ ∈ (0, 1). Also choose J ≥[

5Cε

(1 +

√2 log 5

δ

)]2,

N ≥ 2752v4max log

[40 e(J+1)

δ (10 e vmax)J], and M ≥(

v2max

2 ε2

)log[

10N |A|δ

]. Then, for

v′ = v′ (v, N, M, J, µ, ν) we have ‖v′ − T v‖2, µ ≤d2, µ (T v, F (Θ)) + ε with probability at least 1− δ.

In the next lemma we show that we can make the biasbetween the regression function fM and the Bellman updateT v arbitrarily small uniformly over s ∈ S through the choiceof M ≥ 1.

Lemma 18. For any ε > 0 and M ≥ 1,

‖fM − T v‖∞ ≤ γ[ε+ 2 |A| exp

(−2M ε2

vmax2

)(vmax − ε)

].

14

Proof. For any s ∈ S, we compute

|fM (s)− [T v] (s) |

≤E[|mina∈A

c (s, a) +

γ

M

M∑m=1

v (Xs, am )

−min

a∈A

c (s, a) + γ EX∼Q(· | s, a) [v (X)]

|]

≤ γ E

[maxa∈A| 1

M

M∑m=1

v (Xs, am )− EX∼Q(· | s, a) [v (X)] |

]

≤ γ[ε+ 2 |A| exp

(−2M ε2

vmax2

)(vmax − ε)

],

where the second inequality follows from Fact 13 and the thirdis by the Hoeffding inequality.

We make use of the following RKHS function fitting resultfor the one step Bellman error in the supremum norm.

Theorem 19. Fix v ∈ F (S; vmax), ε > 0, and δ ∈(0, 1). Also choose N ≥

(2CKκε

)6log (4/δ)

2 and M ≥v2max

2(ε/4)2log(

4 |A| γ(vmax−ε/4)(4−2 γ)ε

), where CK is a constant inde-

pendent of the dimension of S. Then for

fλ , arg minf∈HK

1

N

N∑n=1

(f (Sn)− Yn)2

+ λ ‖f‖2HK

,

we have ‖fλ − T v‖∞ ≤ ε with probability at least 1− δ.

Proof. By the triangle inequality, ‖fλ − T v‖∞ ≤ ‖fλ −fM‖∞ + ‖fM − T v‖∞. We choose N ≥ 1 to satisfy

CKκ

(log (4/δ)

2

N

)1/6

≤ ε

2⇒ N ≥

(2CKκ

ε

)6

log (4/δ)2,

so that ‖fλ − fM‖∞ ≤ ε/2 with probability at least 1− δ by[14, Corollary 5] and the fact that ‖f‖∞ ≤ κ ‖f‖HK . Then,we choose M ≥ 1 to satisfy

γ

[ε/4 + 2 |A| exp

(−2M (ε/4)

2

v2max

)(vmax − ε/4)

]≤ ε

2

⇒M ≥ v2max

2 (ε/4)2 log

(4 |A| γ (vmax − ε/4)

(4− 2 γ) ε

),

so that ‖fM − T v‖∞ ≤ ε/2 by Lemma 18.

William Haskell received his B.S. degree in Mathematics and M.S. degreein Econometrics from the University of Massachusetts Amherst in 2006, andhis Ph.D. degree in Operation Research from the University of CaliforniaBerkeley in 2011. Currently, he is an assistant professor in the Department ofIndustrial Systems Engineering and Management at the National Universityof Singapore. His research interests include convex optimization, risk-awaredecision making, and dynamic programming.

Rahul Jain is an associate professor and the K. C. Dahlberg Early CareerChair in the EE department at the University of Southern California. Hereceived his PhD in EECS and an MA in Statistics from the University ofCalifornia, Berkeley, his B.Tech from IIT Kanpur. He is winner of numerousawards including the NSF CAREER award, an IBM Faculty award and theONR Young Investi gator award. His research interests span stochastic sys-tems, statistical learning, queueing systems and game theory with applicationsin communication networks, power systems, transportation and healthcare.

Hiteshi Sharma is a Ph.D. student at USC with particular interests inapproximate dynamic programming, reinforcement learning and stochasticoptimization. Prior to enrolling at USC, she was at IIT Bombay where sheworked on spectrum sharing in cognitive radio networks.

Pengqian Yu was born in Yunnan, China, in 1989. He received the B.S.degree in mechanical design, manufacturing and automation from the Collegeof Mechanical Engineering, Chongqing University, Chongqing, China, in2012, and the Ph.D. degree in the Department of Mechanical Engineeringat the National University of Singapore (NUS), in 2016. He is currentlya Postdoctoral Research Fellow of the Department of Industrial SystemsEngineering and Management at NUS.