Lecture 4: Hamilton-Jacobi-Bellman Equations,
Stochastic Differential Equations
ECO 521: Advanced Macroeconomics I
Benjamin Moll
Princeton University
Fall 2012
Outline
(1) Hamilton-Jacobi-Bellman equations in deterministic settings
(with derivation)
(2) Numerical solution: finite difference method
(3) Stochastic differential equations
Hamilton-Jacobi-Bellman Equation: Some “History”
William Hamilton Carl Jacobi Richard Bellman
• Aside: why called “dynamic programming”?
• Bellman: “Try thinking of some combination that will possibly
give it a pejorative meaning. It’s impossible. Thus, I thought
dynamic programming was a good name. It was something
not even a Congressman could object to. So I used it as an
umbrella for my activities.” http://www.ingre.unimore.it/or/corsi/vecchi_
corsi/complementiro/materialedidattico/originidp.pdf
Hamilton-Jacobi-Bellman Equations
• Recall the generic deterministic optimal control problem from
Lecture 1:
∫ ∞
V (x0 ) = max
∞
e −ρt h (x (t) , u (t)) dt
u(t)t=0 0
subject to the law of motion for the state
ẋ (t) = g (x (t) , u (t)) and u (t) ∈ U
for t ≥ 0, x(0) = x0 given.
• ρ ≥ 0: discount rate
• x ∈ X ⊆ Rm : state vector
• u ∈ U ⊆ Rn : control vector
• h : X × U → R: instantaneous return function
Example: Neoclassical Growth Model
∫ ∞
V (k0 ) = max
∞
e −ρt U(c(t))dt
c(t)t=0 0
subject to
k̇ (t) = F (k(t)) − δk(t) − c(t)
for t ≥ 0, k(0) = k0 given.
• Here the state is x = k and the control u = c
• h(x, u) = U(u)
• g (x, u) = F (x) − δx − u
Generic HJB Equation
• The value function of the generic optimal control problem
satisfies the Hamilton-Jacobi-Bellman equation
ρV (x) = max h(x, u) + V ′ (x) · g (x, u)
u∈U
• In the case with more than one state variable m > 1,
V ′ (x) ∈ Rm is the gradient of the value function.
Example: Neoclassical Growth Model
• “cookbook” implies:
ρV (k) = max U(c) + V ′ (k)[F (k) − δk − c]
c
• Proceed by taking first-order conditions etc
U ′ (c) = V ′ (k)
Derivation from Discrete-time Bellman
• Here: derivation for neoclassical growth model.
• Extra class notes: generic derivation.
• Time periods of length ∆
• discount factor
β(∆) = e −ρ∆
• Note that lim∆→0 β(∆) = 1 and lim∆→∞ β(∆) = 0.
• Discrete-time Bellman equation:
V (kt ) = max ∆U(ct ) + e −ρ∆ V (kt+∆ ) s.t.
ct
kt+∆ = ∆[F (kt ) − δkt − ct ] + kt
Derivation from Discrete-time Bellman
• For small ∆ (will take ∆ → 0), e −ρ∆ = 1 − ρ∆
V (kt ) = max ∆U(ct ) + (1 − ρ∆)V (kt+∆ )
ct
• Subtract (1 − ρ∆)V (kt ) from both sides
ρ∆V (kt ) = max ∆U(ct ) + (1 − ∆ρ)[V (kt+∆ ) − V (kt )]
ct
• Divide by ∆ and manipulate last term
V (kt+∆ ) − V (kt ) kt+∆ − kt
ρV (kt ) = max U(ct ) + (1 − ∆ρ)
ct kt+∆ − kt ∆
Take ∆ → 0
ρV (kt ) = max U(ct ) + V ′ (kt )k̇t
ct
Connection Between HJB Equation and Hamiltonian
• Hamiltonian
H(x, u, λ) = h(x, u) + λg (x, u)
• Bellman
ρV (x) = max h(x, u) + V ′ (x)g (x, u)
u∈U
• Connection: λ(t) = V ′ (x(t)), i.e. co-state = shadow value
• Bellman can be written as
ρV (x) = max H(x, u, V ′ (x))
u∈U
• Hence the “Hamilton” in Hamilton-Jacobi-Bellman
• Can show: playing around with FOC and envelope condition
gives conditions for optimum from Lecture 1.
Numerical Solution: Finite Difference Method
• Example: Neoclassical Growth Model
ρV (k) = max U(c) + V ′ (k)[F (k) − δk − c]
c
• Functional forms
c 1−σ
U(c) = , F (k) = k α
1−σ
• See material at
http://www.princeton.edu/~moll/HACTproject.htm
particularly
• http://www.princeton.edu/~moll/HACTproject/HACT_Additional_Codes.pdf
• Code 1: http://www.princeton.edu/~moll/HACTproject/HJB_NGM.m
• Code 2: http://www.princeton.edu/~moll/HACTproject/HJB_NGM_implicit.m
Diffusion Processes
• A diffusion is simply a continuous-time Markov process (with
continuous sample paths, i.e. no jumps)
• Simplest possible diffusion: standard Brownian motion
(sometimes also called “Wiener process”)
• Definition: a standard Brownian motion is a stochastic
process W which satisfies
√
W (t + ∆t) − W (t) = εt ∆t, εt ∼ N(0, 1), W (0) = 0
• Not hard to see
W (t) ∼ N(0, t)
• Continuous time analogue of a discrete time random walk:
Wt+1 = Wt + εt , εt ∼ N(0, 1)
Standard Brownian Motion
• Note: mean zero, E(W (t)) = 0...
• ... but blows up Var (W (t)) = t.
Brownian Motion
• Can be generalized
x(t) = x(0) + µt + σW (t)
• Since E(W (t)) = 0 and Var (W (t)) = t
E[x(t) − x(0)] = µt, Var [x(t) − x(0)] = σ 2 t
• This is called a Brownian motion with drift µ and variance σ 2
• Can write this in differential form as
dx(t) = µdt + σdW (t)
√
where dW (t) ≡ lim∆t→0 εt ∆t, with εt ∼ N(0, 1)
• This is called a stochastic differential equation
• Analogue of stochastic difference equation:
xt+1 = µt + xt + σεt , εt ∼ N(0, 1)
Further Generalizations: Diffusion Processes
• Can be generalized further (suppressing dependence of x and
W on t)
dx = µ(x)dt + σ(x)dW
where µ and σ are any non-linear etc etc functions.
• This is called a “diffusion process”
• µ(·) is called the drift and σ(·) the diffusion.
• all results can be extended to the case where they depend on
t, µ(x, t), σ(x, t) but abstract from this for now.
• The amazing thing about diffusion processes: by choosing
functions µ and σ, you can get pretty much any
stochastic process you want (except jumps)
Example 1: Ornstein-Uhlenbeck Process
• Brownian motion dx = µdt + σdW is not stationary (random
walk). But the following process is
dx = θ(x̄ − x)dt + σdW
• Analogue of AR(1) process, autocorrelation e −θ ≈ 1 − θ
xt+1 = θx̄ + (1 − θ)xt + σεt
• That is, we just choose
µ(x) = θ(x̄ − x)
and we get a nice stationary process!
• This is called an “Ornstein-Uhlenbeck process”
Ornstein-Uhlenbeck Process
• Can show: stationary distribution is N(x̄, σ 2 /(2θ))
Example 2: “Moll Process”
• Design a process that stays in the interval [0, 1] and
mean-reverts around 1/2
µ(x) = θ (1/2 − x) , σ(x) = σx(1 − x)
dx = θ (1/2 − x) dt + σx(1 − x)dW
• Note: diffusion goes to zero at boundaries σ(0) = σ(1) = 0 &
mean-reverts ⇒ always stay in [0, 1]
Other Examples
• Geometric Brownian motion:
dx = µxdt + σxdW
x ∈ [0, ∞), no stationary distribution:
log x(t) ∼ N((µ − σ 2 /2)t, σ 2 t).
• Feller square root process (finance: “Cox-Ingersoll-Ross”)
√
dx = θ(x̄ − x)dt + σ xdW
x ∈ [0, ∞), stationary distribution is Gamma(γ, 1/β), i.e.
f∞ (x) ∝ e −βx x γ−1 , β = 2θx̄/σ 2 , γ = 2θx̄/σ 2
• Other processes in Wong (1964), “The Construction of a
Class of Stationary Markoff Processes.”
Next Time
(1) Hamilton-Jacobi-Bellman equations in stochastic settings
(without derivation)
(2) Ito’s Lemma
(3) Kolmogorov Forward Equations
(4) Application: Power laws (Gabaix, 2009)
