Deep Learning-Based Algorithms for

High-Dimensional PDEs and Control Problems

Weinan E

Department of Mathematics and PACM,

Princeton University
Joint work with Jiequn Han and Arnulf Jentzen
October 9, 2019

1 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

2 / 38
Outline
Deep learning-based algorithms for
• stochastic control problems in high dimension
Jiequn Han and Weinan E, ”Deep learning approximation for stochastic control
problems”, NIPS Workshop on Deep Reinforcement Learning (2016)

• high dimensional nonlinear PDEs, including the HJB equation, based on

stochastic control formulation
Weinan E, Jiequn Han and Arnulf Jentzen, ”Deep learning-based numerical
methods for high-dimensional parabolic partial differential equations and
backward stochastic differential equations”, Communications in Mathematics
and Statistics (2017)
Jiequn Han, Arnulf Jentzen and Weinan E, ”Solving high-dimensional partial
differential equations using deep learning”, Proceedings of the National
Academy of Sciences (2018)

3 / 38
Motivating Examples

• The Hamilton-Jacobi-Bellman equation in stochastic control

1  T
  
 
∇v ·
 
vt + max
a 
Tr σσ (Hessx v)  + b + f = 0,
2
 


v(xt) = max
a
{f (xt, a) + γ Ev(xt+1)} .

• The Black-Scholes equation for pricing financial derivatives,

1
vt + ∆v + r∇v · x − rv + f = 0.
2
• Reinforcement learning (model-based)

E
∞
 

max γ trt .
X

π t=0

4 / 38
Curse of Dimensionality

• The dimension can be easily large in practice.

Equation Dimension
Black-Scholes equation # of underlying financial assets
HJB equation the same as the state space

• A key computational challenge is the curse of dimensionality: the complexity is

exponential in dimension d for finite difference/element method – usually
unavailable for d ≥ 4.

5 / 38
Related Work in High-dimensional Case

• Linear parabolic PDEs: Monte Carlo methods based on the Feynman-Kac

formula

• Semilinear parabolic PDEs:

1. branching diffusion approach (Henry-Labordère 2012, Henry-Labordère et

al. 2014)

2. multilevel Picard approximation (E and Jentzen et al. 2015)

• Hamilton-Jacobi PDEs: using Hopf formula and fast convex/nonconvex

optimization methods (Darbon & Osher 2016, Chow et al. 2017)

• Deep reinforcement learning

6 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

7 / 38
Linear Parabolic PDE and Feynman-Kac Formula

∂u 1  T 

(t, x) + Tr σσ (t, x)(Hessxu)(t, x) + ∇u(t, x) · µ(t, x) + f (x, t) = 0.

∂t 2
Terminal condition u(T, x) = g(x).

Let
dXt = µ(t, Xt) dt + σ(t, Xt) dWt,
Feynman-Kac formula:

u(t, x) = E[g(XT ) +
Z
T
t f (s, Xs)ds|Xt = x].

Compute the solution of PDE using Monte Carlo, overcoming the curse of
dimensionality.

8 / 38
Semilinear Parabolic PDE

∂u 1  T 

(t, x) + Tr σσ (t, x)(Hessxu)(t, x) + ∇u(t, x) · µ(t, x)

∂t 2
T
 
+ f t, x, u(t, x), σ (t, x)∇u(t, x) = 0.

Terminal condition u(T, x) = g(x).

Let Z
t Z
t
Xt = ξ + 0 µ(s, Xs ) ds + 0 σ(s, Xs ) dWs .
Itô’s lemma:

u(t, Xt) − u(0, X0)

Z
t  T

= − 0 f s, Xs, u(s, Xs), σ (s, Xs) ∇u(s, Xs) ds
t
+ 0 [∇u(s, Xs)]T σ(s, Xs) dWs.
Z

9 / 38
Connection between PDE and BSDE

• BSDEs give a nonlinear Feynman-Kac representation of some nonlinear

parabolic PDEs. (Pardoux & Peng 1992, El Karoui et al. 1997, etc).

• Consider the following BSDE


t
Z Z
t
Xt = ξ + 0 µ(s, Xs) ds + 0 σ(s, Xs) dWs,





Z
T Z
T T
Yt = g(XT ) + f (s, X , Y , Z ) ds − (Z ) dWs,



s s s s

t t

The solution is an (unique) adapted process {(Xt, Yt, Zt)}t∈[0,T ] with values in
Rd × R × Rd .

• This is the Pontryagin maximum principle for stochastic control.

• This is also the method of characteristics for nonlinear parabolic PDEs.

10 / 38
Reformulating the PDE problem

• Connection between BSDE and PDE

Yt = u(t, Xt) and Zt = σ T(t, Xt) ∇u(t, Xt).

• In other words, consider the following variational problem

inf
Y0 ,{Zt }0≤t≤T
E|g(XT ) − YT |2,
Z
t Z
t
s.t. Xt = ξ + 0 µ(s, X s ) ds + 0 Σ(s, Xs ) dWs ,
Z
t Z
t T
Yt = Y0 − 0 h(s, X s , Ys , Z s ) ds + 0 (Zs ) dWs .

The unique minimizer is the solution to the PDE.

11 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

12 / 38
Deep BSDE Method
• Key step: approximate the unknown functions

X0 7→ u(0, Y0) and Xt 7→ σ T(t, Xt) ∇u(t, Xt)

by feedforward neural networks ψ and φ.

• work with variational formulation, discretize time using Euler scheme on a grid
0 = t0 < t1 < . . . < tN = T :

inf N −1 E|g(XT ) − YT |2,

ψ0 ,{φn }n=0
s.t. X0 = ξ, Y0 = ψ0(ξ),
Xtn+1 = Xti + µ(tn, Xtn )∆t + σ(tn, Xtn )∆Wn,
Ztn = φn(Xtn ),
Ytn+1 = Ytn − f (tn, Xtn , Ytn , Ztn )∆t + (Ztn )T∆Wn.

• there is a subnetwork at each time tj

• Observation: we can stack all the subnetworks together to form a deep neural
network (DNN) as a whole
13 / 38
Network Architecture

Figure: Network architecture for solving parabolic PDEs. Each column corresponds to a subnetwork
at time t = tn. The whole network has (H + 1)(N − 1) layers in total that involve free parameters
to be optimized simultaneously.

14 / 38
Optimization

• This network takes the paths {Xtn }0≤n≤N and {Wtn }0≤n≤N as the input data
and gives the final output, denoted by û({Xtn }0≤n≤N , {Wtn }0≤n≤N ), as an
approximation to u(tN , XtN ).

• The error in the matching of given terminal condition defines the expected loss
function
l(θ) = E g(XtN ) − û {Xtn }0≤n≤N , {Wtn }0≤n≤N .
 
 2

• The paths can be simulated easily. Therefore the commonly used SGD
algorithm fits this problem well.

• We call the introduced methodology deep BSDE method since we use the
BSDE and DNN as essential tools.

15 / 38
Why such deep networks can be trained?

Intuition: there are skip connections between different subnetworks

u(tn+1, Xtn+1 ) − u(tn, Xtn )
≈ − f tn, Xtn , u(tn, Xtn ), φn(Xtn ) ∆tn + (φn(Xtn ))T∆Wn
 

resemble residual networks (fully connected deep NN are unstable!)

16 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

17 / 38
Implementation

• Each subnetwork has 4 layers, with 1 input layer (d-dimensional), 2 hidden

layers (both d + 10-dimensional), and 1 output layer (d-dimensional).

• Choose the rectifier function (ReLU) as the activation function and optimize
with Adam method.

• The means and the standard deviations of the relative errors are approximated
by 5 independent runs of the algorithm with different random seeds.

• Implement in Tensorflow and reported examples are all run on a Macbook Pro.

• Github: https://github.com/frankhan91/DeepBSDE

18 / 38
LQG (linear quadratic Gaussian) Example for
d=100
√ √
dXt = 2 λ mt dt + 2 dWt,
Cost functional: J({mt}0≤t≤T ) = E 0 kmtk2 dt + g(XT ) .
2
RT
 

HJB equation:
∂u
+ ∆u − λk∇uk22 = 0
∂t 
√

1 
u(t, x) = − lnE exp − λg(x + 2WT −t).
  


λ
4.7
Deep BSDE Solver
4.6 Monte Carlo
4.5

u(0,0,...,0)
4.4

4.3

4.2

4.1

4.0
0 10 20 30 40 50
lambda

Figure: Left: Relative error of the deep BSDE method for u(t=0, x=(0, . . . , 0)) when λ = 1, which achieves
0.17% in a runtime of 330 seconds. Right: Optimal cost u(t=0, x=(0, . . . , 0)) against different λ.
19 / 38
Black-Scholes Equation with Default Risk

• The classical Black-Scholes model can and should be augmented by some

important factors in real markets, including defaultable securities, transactions
costs, uncertainties in the model parameters, etc.

• Ideally the pricing models should take into account the whole basket of
financial derivative underlyings, resulting in high-dimensional nonlinear PDEs.

• To test the deep BSDE method, we study a special case of the recursive
valuation model with default risk (Duffie et al. 1996, Bender et al. 2015).

20 / 38
Black-Scholes Equation with Default Risk

• Consider the fair price of a European claim based on 100 underlying assets
conditional on no default having occurred yet.

• The underlying asset price moves as a geometric Brownian motion and the
possible default is modeled by the first jump time of a Poisson process.

• The claim value is modeled by a parabolic PDE with the nonlinear function
T
 
f t, x, u(t, x), σ (t, x)∇u(t, x)
= − (1 − δ) Q(u(t, x)) u(t, x) − R u(t, x).

21 / 38
Black-Scholes Equation with Default Risk
The unknown “exact” solution at t = 0 x = (100, . . . , 100) is computed by the
multilevel Picard method.

Figure: Approximation of u(t=0, x=(100, . . . , 100)) against number of iteration steps. The deep
BSDE method achieves a relative error of size 0.46% in a runtime of 617 seconds.

22 / 38
Allen-Cahn Equation
The Allen-Cahn equation is a reaction-diffusion equation for the modeling of phase
separation and transition in physics. Here we consider a typical Allen-Cahn equation
with the “double-well potential” in 100-dimensional space:
∂u
(t, x) = ∆u(t, x) + u(t, x) − [u(t, x)]3 ,
∂t
with initial condition u(0, x) = g(x).

0.30

0.25

u(t,0,...,0)
0.20

0.15

0.10

0.05

0.00
0.00 0.05 0.10 0.15 0.20 0.25 0.30

Figure: Left: relative error of the deep BSDE method for u(t=0.3, x=(0, . . . , 0)), which achieves 0.30% in a
runtime of 647 seconds. Right: time evolution of u(t, x=(0, . . . , 0)) for t ∈ [0, 0.3], computed by means of the deep
BSDE method.
23 / 38
An Example with Oscillating Explicit Solution

We consider an example studied for the numerical methods of PDE in literature

(Gobet & Turkedjiev 2017). We set d = 100 instead of d = 2.

The PDE is constructed artificially in a form

∂u 1  
2 
(t, x) + ∆u(t, x) + min1, u(t, x) − u∗(t, x)  = 0,
 

∂t 2
in which u∗(t, x) is the explicit oscillating solution
∗ Pd λ2 d(t−T )
   
u (t, x) = κ + sin λ i=1 xi exp 2 .

24 / 38
Ablation Study
Number of layers† 29 58 87 116 145
Mean of relative error 2.29% 0.90% 0.60% 0.56% 0.53%
Std. of relative error 0.0026 0.0016 0.0017 0.0017 0.0014
Table: The mean and standard deviation (std.) of the relative error for the above PDE, obtained by
the deep BSDE method with different number of hidden layers. † We only count the layers that have
free parameters to be optimized.

Nonlinear BS LQG Allen-Cahn

ReLU 0.46% (0.0008) 0.17% (0.0004) 0.30% (0.0021)
Tanh 0.44% (0.0006) 0.17% (0.0005) 0.28% (0.0024)
Sigmoid 0.46% (0.0004) 0.19% (0.0008) 0.38% (0.0026)
Softplus 0.45% (0.0007) 0.17% (0.0004) 0.18% (0.0017)
Table: The mean and standard deviation (in parenthesis) of relative error obtained by the deep
BSDE method with different activation functions, for the nonlinear Black-Scholes equation, the
Hamilton-Jacobi-Bellman equation, and the Allen-Cahn equation.
25 / 38
References and Follow-up Works

• References:
I Han, Jentzen, and E, Solving high-dimensional partial differential equations using deep
learning, Proceedings of the National Academy of Sciences (2018)

I E, Han, and Jentzen, Deep learning-based numerical methods for high-dimensional

parabolic partial differential equations and backward stochastic differential equations,
Communications in Mathematics and Statistics (2017)

• Follow-up works:
I Beck et al. 2017 (deep 2BSDE method), Henry-Labordère 2017 (deep primal-dual for
BSDEs), Fujii et al. 2017 (deep BSDE with asymptotic expansion), Becker et al. 2018
(deep optimal stopping), Raissi 2018, Beck et al. 2018, Chan-Wai-Nam et al. 2018, Huré
et al. 2019

26 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

27 / 38
Formulation of Stochastic Control
Model dynamics:
st+1 = st + bt(st, at) + ξt+1,
st is state, at is control, ξt is randomness. Consider objective:
 −1
TX 
min
T −1
E ct(st, at(st)) + cT (sT ) | s0 ,
{at }t=0 t=0

We look for a feedback control:

at = at(st).
• Neural network approximation:

at(st) ≈ at(st|θt),

Solve directly the approximate optimization problem

 −1
TX
min E −1
ct(st, at(st|θt)) + cT (sT )},
{θt }Tt=0 t=0

rather than dynamic programming principle.

28 / 38
Network Architecture

Figure: Network architecture for solving stochastic control in discrete time. The whole network has
(N + 1)T layers in total that involve free parameters to be optimized simultaneously. Each column
(except ξt) corresponds to a sub-network at t.

29 / 38
Example in Optimal Execution of Portfolios
The goal is to minimize the expected cost for trading multiple stocks over a fixed time horizon:
−1
TX
min
T −1
E pTt at,
{at }t=0 t=0

T −1
subject to t=0 at = ā ∈ Rn. The execution price is influenced by the amount we buy at each time
P

and the stochastic market conditions.

60 Time horizon Time horizon

7.5

relative error for the controls

T=20 T=20
relative trading cost

T=25 T=25
T=30 T=30
40 5.0

20 2.5

analytic optimal cost

0 0.0
4000 8000 12000 4000 8000 12000

iteration iteration

Figure: Learning curves of relative trading cost (left) and relative error for the controls (right). The
space of control function is R23 → R10.

30 / 38
Example in Energy Storage with a Single Device

The goal is to maximize revenues from an energy storage device and a renewable
wind energy source while satisfying stochastic electricity demand.

benchmark reward
of lookup table
1.00

0.99

relative reward
Demand 0.98

0.97
Time horizon

0.96 T=10
T=15

Spot Market Storage Wind Source 0 10000 20000 30000 40000 50000

iteration

Figure: Left: network diagram of energy; Right: learning curves of relative reward. The space of
control function is R4 → R5 with multiple equality and inequality constrains.

31 / 38
Example in Energy Storage with Multiple Devices
The setting is similar to the above but now there are multiple devices, in which we
do not find any other available solution for comparison.

1.1

reward relative to the case n=50

1.0

0.9

0.8
Number of devices

0.7 n=30
n=40
n=50
0.6
10000 20000 30000 40000 50000

iteration

Figure: Relative reward to the case n = 50 (with controls satisfying constraints strictly). The space
of control function is Rn+2 → R3n for n = 30, 40, 50, with multiple equality and inequality
constrains.

32 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

33 / 38
Theorem (A Posteriori Estimates (Han and Long))
Under some assumptions, there exists a constant C, independent of h, d, and m,
such that for sufficiently small h,

sup (E|Xt − + E|Yt − E|Zt − Ẑtπ |2 dt

T
X̂tπ |2 Ŷtπ |2)
Z
+ 0
t∈[0,T ]
≤C[h + E|g(XTπ ) − YTπ |2],

where X̂tπ = Xtπi , Ŷtπ = Ytπi , Ẑtπ = Ztπi for t ∈ [ti, ti+1).

34 / 38
Theorem (Upper Bound of Optimal Loss (Han and Long))
Under some assumptions, there exists a constant C, independent of h , d and m,
such that for sufficiently small h,
E|g(XTπ ) − YTπ |2
−1
≤ C h + E|Y0 − µ0 (ξ)| + E|E[Z̃ti |Xti , Yti ] − φi (Xti , Yti )| h ,
( NX )
π 2 π π π π π 2
i=0

where Z̃ti = h−1E[ tii+1 Zt dt|Fti ]. If b and σ are independent of Y , the term
Rt

E[Z̃ti |Xtπi , Ytπi ] can be replaced with E[Z̃ti |Xtπi ].

Remark
Similar bounds can be derived for the stochastic control problem (in both
continuous time and discrete time)

35 / 38
Table of Contents

1. Background

2. BSDE Formulation of Parabolic PDE

3. Deep BSDE Method

4. Numerical Examples of High-Dimensional PDEs

5. Stochastic Control in Discrete Time

6. Convergence of the Deep BSDE Method

7. Summary

36 / 38
Summary

• Deep learning are providing us powerful tools to overcome the curse of

dimensionality in high-dimensional parabolic PDEs and control problems.

• For general nonlinear parabolic PDEs, the deep BSDE method reformulate it
into a variational problem based on BSDEs and approximate the unknown
gradients by neural networks.

• Similar methodology can be applied to solve model based stochastic control

problems, in which the optimal policies are approximated by neural networks.

• Numerical results validate the proposed algorithm in high dimensions, in terms

of both accuracy and speed.

• This opens up new possibilities in various disciplines, including economics,

finance, operational research, and physics.

37 / 38
DL for Other High-Dimensional Problems

• Moment closure for kinetic equations: a machine learning based hydrodynamic

model with uniform accuracy1

• Solving many-electron Schrödinger equation using deep neural networks2

• Large-scale simulation of molecular dynamics: Deep Potential3 model provides

potential energy, forces, or even coarse-grained models based on neural
networks with quantum accuracy

Thank you for your attention!

1
J. Han, C. Ma, Z. Ma, W. E, PNAS (2019)
2
J. Han, L. Zhang, and W. E, JCP, 399, 108929 (2019)
3
J. Han, L. Zhang, R. Car, and W. E, CiCP, 23, 629–639 (2018); L. Zhang, J. Han, H. Wang, R. Car, and W. E,
PRL, 120(10), 143001 (2018)
38 / 38