Economics 2010c: Lecture 8Brownian Motion and Continuous Time

Dynamic Programming

David Laibson

9/25/2014

Outline: Continuous Time Dynamic Programming

1. Continuous time random walks: Wiener Process

2. Ito’s Lemma

3. Continuous time Bellman Equation

1 Brownian Motion

Consider a continuous time world ∈ [0∞)

Imagine that every ∆ intervals, a process () either goes up or down:

∆ ≡ (+∆)− () =

(+ with prob − with prob ≡ 1−

)

Here ∆ is a fixed interval of time. Later we will let ∆→ 0

(∆) = + (−) = (− )

[(∆)2] = 2 + 2 = 2

(∆) = [∆−∆]2 = [(∆)2]− [∆]2 = 42

• Time span of length implies = ∆ steps in ∆ So ()− (0) isa binomial random variable.

• For example, suppose = 5 and = = 12

Pr [()− (0) = −5] =³50

´05 = 003

Pr [()− (0) = −3] =³51

´14 = 016

Pr [()− (0) = −1] =³52

´23 = 031

Pr [()− (0) = +1] =³53

´32 = 031

Pr [()− (0) = +3] =³54

´41 = 016

Pr [()− (0) = +5] =³55

´50 = 003

• Generally, the probability that

()− (0) = ()() + (− )(−)

is³

´−

• ()− (0) is a binomial random variable with

[()− (0)] = (− ) = (− )∆

[()− (0)] = 42 = 42∆

We will now vary ∆ Begin by letting

= √∆

=1

2

∙1 +

√∆¸

The following five implications follow:

= 1− =1

2

∙1−

√∆¸

(− ) =

√∆

[()− (0)] =

√∆

√∆∆ =

[()− (0)] = 4µ1

4

¶Ã1−

µ

¶2∆

!2∆

∆−→ 2 (as ∆→ 0)

0 10 20 30 40 50 60 70 80 90 100-10

-8

-6

-4

-2

0

2

4

6

8

10Brownian Motion (∆t = 1, σ = 1, α = 0)

Time (t)

x(t)

0 10 20 30 40 50 60 70 80 90 100-10

-8

-6

-4

-2

0

2

4

6

8

10Brownian Motion (∆t = .1, σ = 1, α = 0)

Time (t)

x(t)

0 10 20 30 40 50 60 70 80 90 100-10

-8

-6

-4

-2

0

2

4

6

8

10Brownian Motion (∆t = .01, σ = 1, α = 0)

Time (t)

x(t)

0 10 20 30 40 50 60 70 80 90 100-10

-8

-6

-4

-2

0

2

4

6

8

10Brownian Motion (∆t = .00001, σ = 1, α = 0)

Time (t)

x(t)

1. Vertical movements proportional to√∆ (not ∆)

2. [()− (0)]→ ( 2) since Binomial →Normal

3. “Length of curve during 1 time period” = 1∆

√∆ = 1√

∆→∞

4. ∆∆ =

±√∆

∆ = ±√∆→ ±∞. Time derivative,

³

´, doesn’t exist.

5. (∆)∆ =

22

∆ = . So we write () = .

6. (∆)∆ =

4³14

´³1−()

2∆´2∆

∆ → 2. So we write () = 2.

When we let ∆ converge to zero, the limiting process is called a continuoustime random walk with (instantaneous) drift and (instantaneous) variance2 We generated this continuous-time stochastic process by building it up asa limit case. We could have also just defined the process directly.

Definition 1.1 If a continuous time stochastic process, () is a WienerProcess, then (0)− () satisfies the following conditions:1. (0)− () ∼ (0 0 − )

2. If ≤ 0 ≤ 00 ≤ 000,

h((0)− ())((000)− (00))

i= 0

You can also think of the two condition as:1. (0)− () =

√0 − where ∼ (0 1)

2. non-overlapping increments of are independent

Summary:

• A Wiener process is a continuous time random walk with zero drift andunit variance.

• () has the Markov property: the current value of the process is a suffi-cient statistic for the distribution of future values.

• () ∼ ((0) ) so the variance of () rises linearly with

Generalization: Let () be a Wiener Process. Let () be another continuoustime stochastic process such that,

lim∆→0

∆

∆= ( ) i.e. () = ( )

lim∆→0

∆

∆= ( )2 i.e. () = ( )2

We summarize these properties by writing:

= ( )+ ( )

This is called an Ito Process. Important examples:

• = + (random walk with drift and variance 2)

• = + (geometric random walk with proportional drift and proportional variance 2)

2 Ito’s Lemma

Our goal: work with functions that take an Ito Process as an argument.

• Suppose that the price of oil follows an Ito Process:

= ( )+ ( )

• The value of an oil well will depend on the price of oil and time: ( )

• We would like to be able to write the stochastic process that describes theevolution of :

= ̂( )+ ̂( )

which we will call the total differential of

Theorem 2.1 (Ito’s Lemma) Let () be a Wiener Process. Let () be anIto Process with = ( )+ ( ) Let = ( ) then

=

+

+

1

2

2

2( )2

=

"

+

( ) +

1

2

2

2( )2

#+

( )

Proof: Using a Taylor expansion:

=

+

1

2

2

2()2 +

+

1

2

2

2()2 +

2

+

Any deterministic term of order ()32 or higher is small relative to terms oforder Any stochastic term of order or higher is small relative to terms oforder

√ So,

()2 =

= ( )()2 + ( ) =

()2 = ( )2()2 + = ( )2+

Combining these results, we have our key result:

=

+

+

1

2

2

2( )2

¥

2.1 Intuition:

• Assume ( ) = 0 (no drift in the Ito Process (): () = 0).

• Assume that () = 0 (holding fixed, doesn’t depend on ).

• But, ( ) = 1222

( )2 6= 0

• If is concave (convex), is expected to fall (rise) due to variation in

• For Ito Processes, ()2 behaves like ( )2 so the effect of concavity(convexity) is of order and can not be ignored when calculating thedifferential of

• () = log(), so 0 = 1 and

00 = − 12

• = +

=

"

+

( ) +

1

2

2

2( )2

#+

( )

=∙0 +

1

+

1

2

µ− 1

2

¶22

¸+

1

=∙− 1

22¸+

• falls below due to concavity of

3 Continuous time Bellman Equation

Let ( ) = instantaneous payoff function, where is state variable, iscontrol variable and is time. Let 0 = +∆ and 0 = +∆ So

( ) = max

n( )∆+ (1 + ∆)−1 (0 0)

o(1 + ∆) ( ) = max

n(1 + ∆)( )∆+ (0 0)

o ( )∆ = max

n(1 + ∆)( )∆+ (0 0)− ( )

oMultiply out and let ∆→ 0 Terms of order ()2 = 0

( ) = max{( )+( )} (*)

Now substitute in for ( ) using Ito’s Lemma:

=

"

+

+

1

2

2

22#+

where = ( ), = ( ) and = ( ) + ( )

Since, = 0 we have,

( ) =

"

+

+

1

2

2

22#

Substituting this expression into equation (*), we get

( ) = max

(( )+

"

+

+

1

2

2

22#

)which is a partial differential equation (in and ).

Outline: Continuous Time Dynamic Programming

1. Continuous time random walks: Wiener Process

2. Ito’s Lemma

3. Continuous time Bellman Equation