Numerical Methods II

One-Step Methods for ODEs

Aleksandar Donev
Courant Institute, NYU1
[email protected]

1 MATH-GA.2020-001 / CSCI-GA.2421-001, Spring 2019

Feb 12th, 2019

A. Donev (Courant Institute) ODEs 2/12/2019 1 / 35

Outline

1 Initial Value Problems

2 One-step methods for ODEs

3 Convergence (after LeVeque)

4 MATLAB ode suite

A. Donev (Courant Institute) ODEs 2/12/2019 2 / 35

 Initial Value Problems

Initial Value Problems

A. Donev (Courant Institute) ODEs 2/12/2019 3 / 35

 Initial Value Problems

Initial Value Problems

We want to numerically approximate the solution to the ordinary
differential equation
dx
= x 0 (t) = ẋ(t) = f (x(t), t),
dt
with initial condition x(t = 0) = x(0) = x0 .
This means that we want to generate an approximation to the
trajectory x(t), for example, a sequence x(tk = k∆t) for
k = 1, 2, . . . , N = T /∆t, where ∆t is the time step used to
discretize time.
If f is independent of t we call the system autonomous.
Note that second-order equations can be written as a system of
first-order equations:
(
d 2x ẋ(t) = v (t)
2
= ẍ(t) = f [x(t), t] ≡
dt v̇ (t) = f [x(t), t]
A. Donev (Courant Institute) ODEs 2/12/2019 4 / 35
 Initial Value Problems

Relation to Numerical Integration

If f is independent of x then the problem is equivalent to numerical

integration Z t
x(t) = x0 + f (s)ds.
0
More generally, we cannot compute the integral because it depends
on the unknown answer x(t):
Z t
x(t) = x0 + f (x(s), s) ds.
0

Numerical methods are based on approximations of f (x(s), s) into

the “future” based on knowledge of x(t) in the “past” and “present”.

A. Donev (Courant Institute) ODEs 2/12/2019 5 / 35

 Initial Value Problems

Convergence

Consider a trajectory numerically discretized as a sequence that

approximates the exact solution at a discrete set of points:

x (k) ≈ x(tk = k∆t), k = 1, . . . , T /∆t.

A method is said to converge with order p > 0, or to have order of

accuracy p, if for any finite T for which the ODE has a solution,
 
 (k)
x − x(k∆t) = O(∆t p ) for all 0 ≤ k ≤ T /∆t.


All methods are recursions that compute a new x (k+1) from previous
x (k) by evaluating f (x) several times. For example, one-step
methods have the form
 
x (k+1) = G x (k) , ∆t; f .

A. Donev (Courant Institute) ODEs 2/12/2019 6 / 35

 Initial Value Problems

Consistency

The local trunction error (LTE) of a method is the amount by which

the exact solution does not satisfy the numerical scheme at the end of
the time step if started from the correct solution x (k) = x (k∆t):

ek = x [(k + 1) ∆t] − G [x(k∆t), ∆t; f ] ,

A method is consistent with order q > 1 if |ek | = O(∆t q ).

The global truncation error is the actual error
 
 (k)
Et=k∆t = x − x(t = k∆t).


Numerical analysis question: Can the global error be bounded by

O (∆t p ) if the local one is O(∆t q )?

A. Donev (Courant Institute) ODEs 2/12/2019 7 / 35

 Initial Value Problems

Propagation of errors

Crude estimate: If one makes an error O(∆t q ) at each time step, the
global error after T /∆t time steps can become on the order of
 

 (k)

q T
= O ∆t q−1 = O(∆t p ),


x − x(k∆t) = O ∆t ·

∆t

and we must have p = q − 1 > 0 for convergence.

This result is often the right one, but it has a hidden assumption that
errors made at previous steps do not grow but rather stay of the same
order so they can be added.
In practice, errors made in previous time steps will either grow or
shrink with time. If they grow “too fast” we are in trouble.
So we arrive for the first time at a recurring theme: Convergence
requires stability in addition to consistency. What does stability
mean?
A. Donev (Courant Institute) ODEs 2/12/2019 8 / 35
 One-step methods for ODEs

One-step methods for ODEs

A. Donev (Courant Institute) ODEs 2/12/2019 9 / 35

 One-step methods for ODEs

Euler’s Method

Assume that we have our approximation x (k) and want to move by

one time step:
Z (k+1)∆t
(k+1) (k)
x ≈x + f (x(s), s) ds.
k∆t

The simplest possible thing is to use a piecewise constant

approximation:
f (x(s), s) ≈ f (x (k) ) = f (k) ,
which gives the forward Euler method

x (k+1) = x (k) + f (k) ∆t.

This method requires only one function evaluation per time step.

A. Donev (Courant Institute) ODEs 2/12/2019 10 / 35

 One-step methods for ODEs

Euler’s Method

Scheme: x (k+1) − x (k) − f (k) ∆t = 0

The local trunction error is easy to find using a Taylor series

expansion:

ek = x [(k + 1) ∆t] − x (k∆t) − f [x (k∆t)] ∆t =

x 00 (ξ) 2
= x [(k + 1) ∆t] − x (k∆t) − x 0 (k∆t) ∆t =
 
∆t ,
2
for some k∆t ≤ ξ ≤ (k + 1) ∆t.
Therefore the LTE is O(∆t 2 ), q = 2.
The global truncation error, however, is of order O(∆t), p = q + 1,
so this is a first-order accurate method.

A. Donev (Courant Institute) ODEs 2/12/2019 11 / 35

 One-step methods for ODEs

Backward Euler

Z (k+1)∆t
(k+1) (k)
x ≈x + f [x(s), s] ds.
k∆t

How about we use a piecewise constant-approximation, but based on

the end-point:
f [x(s), s] ≈ f (x (k+1) ) = f (k+1) ,
which gives the first-order backward Euler method

x (k+1) = x (k) + f (x (k+1) )∆t.

This implicit method requires solving a non-linear equation at

every time step, which is expensive and hard.
We will understand why implicit methods are needed next class.

A. Donev (Courant Institute) ODEs 2/12/2019 12 / 35

 One-step methods for ODEs

Runge-Kutta Methods

Runge-Kutta methods are a powerful class of one-step methods

similar to Euler’s method, but more accurate.
As an example, consider using a trapezoidal rule to approximate the
integral
Z (k+1)∆t
∆t
x (k) + f [x(s), s] ds ≈ x (k) + [f (k∆t) + f ((k + 1) ∆t)] ,
k∆t 2

∆t h  (k) (k)   i
x (k+1) = x (k) + f x , t + f x (k+1) , t (k+1)
2
which requires solving a nonlinear equation for x (k+1) .
This is the simplest implicit Runge-Kutta method, usually called
the implicit trapezoidal method.
The local truncation error is O(∆t 3 ), so the global error is
second-order accurate O(∆t 2 ).
A. Donev (Courant Institute) ODEs 2/12/2019 13 / 35
 One-step methods for ODEs

LTE: θ-method (1.4 in Iserles)



yn+1 = yn + h θf (tn , yn ) + (1 − θ) f tn+1 , yn+1

θ = 1 is Forward Euler, θ = 0 is Backward Euler, θ = 1/2 is Implicit

Trapezoidal

A. Donev (Courant Institute) ODEs 2/12/2019 14 / 35

 One-step methods for ODEs

Midpoint/Trapezoidal Methods
Schemes that treat beginning and end of time step in a
symmetric fashion will lead to a cancellation of first-order error
terms in Taylor series and will thus be second order (Lesson: second
order is easy).
In addition to trapezoidal one can do implicit midpoint scheme:
!
(k+1) (k) ∆t x (k) + x (k+1) (k) ∆t
x =x + f , t +
2 2 2
Observe this is the same as trapezoidal for linear problems (why?).
In an explicit method, we would approximate x ? ≈ x (k+1) first using
Euler’s method, to get the simplest explicit Runge-Kutta method,
usually called Heun’s or explicit trapezoidal method
 
x (k+1,?) =x (k) + f x (k) , t (k) ∆t
∆t h  (k) (k)   i
x (k+1) =x (k) + f x , t + f x (k+1,?) , t (k+1) .
2
A. Donev (Courant Institute) ODEs 2/12/2019 15 / 35
 One-step methods for ODEs

Explicit Midpoint

Explicit midpoint rule

1
  ∆t
x (k+ 2 ,?) =x (k) + f x (k) , t (k)
 2 
(k+1) (k) ∆t ( 1
k+ 2 ,?) (k) ∆t
x =x + f x , t + .
2 2

Explicit midpoint/trapezoidal are a representative of a powerful class

of second-order methods called predictor-corrector methods:
Euler (forward or backward) method is the predictor, and then
(implicit or explicit) trapezoidal/midpoint method is the corrector.
One can also consider these as examples of multi-stage one-step
methods: the predictor is the first stage, the corrector the second.

A. Donev (Courant Institute) ODEs 2/12/2019 16 / 35

 One-step methods for ODEs

LTE: explicit midpoint (LeVeque)

A. Donev (Courant Institute) ODEs 2/12/2019 17 / 35

 One-step methods for ODEs

Higher-Order Runge-Kutta Methods

The idea in RK methods is to evaluate the function f (x, t) several
times and then take a time-step based on an average of the values.
In practice, this is done by performing the calculation in stages:
Calculate an intermediate approximation x ? , evaluate f (x ? ), and go
to the next stage.
The most celebrated Runge-Kutta methods is a four-stage
fourth-order accurate RK4 method based on Simpson’s rule for the
integral:
Z (k+1)∆t
(k)
x + f [x(s), s] ds
k∆t
∆t h i
≈ x (k) + f (x (k) ) + 4f (x (k+1/2) ) + f (x (k+1) )
6
∆t h i
= x (k) + f (k) + 4f (k+1/2) + f (k+1) ,
6
and we approximate 4f (k+1/2) = 2f (k+1/2;1) + 2f (k+1/2;2) .
A. Donev (Courant Institute) ODEs 2/12/2019 18 / 35
 One-step methods for ODEs

RK4 Method

  ∆t (k)
f (k) = f x (k) , x (k+1/2;1) , = x (k) + f
2
 
f (k+1/2;1) = f x (k+1/2;1) , t (k) + ∆t/2
∆t (k+1/2;1)
x (k+1/2;2) = x (k) + f
2
 
f (k+1/2;2) = f x (k+1/2;2) , t (k) + ∆t/2
x (k+1;1) = x (k) + ∆t f (k+1/2;2)

 
f (k+1) = f x (k+1;1) , t (k) + ∆t
∆t h (k) i
x (k+1) =x (k) + f + 2f (k+1/2;1) + 2f (k+1/2;2) + f (k+1)
6
A. Donev (Courant Institute) ODEs 2/12/2019 19 / 35
 One-step methods for ODEs

Intro to multistep Methods

Z (k+1)∆t
(k+1) (k)
x ≈x + f [x(s), s] ds.
k∆t

Euler’s method was based on a piecewise constant approximation

(extrapolation) of f (s) ≡ f [x(s), s].
If we instead integrate the linear extrapolation
  f x (k) , t (k)
− f x (k−1) , t (k−1)

(k) (k)
f (s) ≈ f x , t + (s − tk ),
∆t
we get the second-order two-step Adams-Bashforth method

(k+1) (k) ∆t h  (k) (k)  

(k−1) (k−1)
i
x =x + 3f x , t −f x ,t .
2
This is an example of a multi-step method, which requires keeping
previous values of f .
A. Donev (Courant Institute) ODEs 2/12/2019 20 / 35
 Convergence (after LeVeque)

Convergence (after LeVeque)

A. Donev (Courant Institute) ODEs 2/12/2019 21 / 35

 Convergence (after LeVeque)

Zero Stability
We must also examine how perturbations grow with time: error
propagation.
A method is called zero stable if for all sufficiently small but finite
∆t, introducing perturbations at each step (e.g., roundoff errors,
errors in evaluating f ) with magnitude less than some small  perturbs
the solution by at most O().
This simply means that errors do not increase but rather decrease
from step to step, as we saw with roundoff errors in the first
homework.
A central theorem in numerical methods for differential equations is
variants of the Lax equivalence theorem:
Any consistent method is convergent if and only if it is zero stable, or
consistency + (zero) stability = convergence.
We will show now that one-step methods are zero-stable if f is
well-behaved (Lipschitz continuous w.r.t. second argument).
A. Donev (Courant Institute) ODEs 2/12/2019 22 / 35
 Convergence (after LeVeque)

Lipschitz Constants

Let us consider a system of ODEs

dx
= f (x, t), x(t = 0) = x0 .
dt
Standard theory for ODEs shows that the solution exists and is unique
over some finite time interval if the r.h.s. is Lipschitz continuous in
x a neighborhood of the initial condition:

kf (x, t) − f (x? , t)k ≤ L kx − x? k,

for all {x, x? } within some neighborhood of x0 over some finite

interval t ≥ 0.
For differentiable functions we can take

∂f
L = max
∂x .

A. Donev (Courant Institute) ODEs 2/12/2019 23 / 35

 Convergence (after LeVeque)

Convergence
Denote our numerical approximation with time step size τ :
x(k) ≈ x (k∆t) .
A method is convergent if applying it to any system of ODEs where
f is Lipshitz over a finite time interval T > 0 during which the ODE
has a solution, the numerical approximation converges to that
solution,
lim x(N=T /∆t) = x (T ).
∆t→0
Convergence is a statement about a limit, and does not imply a
method will give reasonable answers for finite ∆t > 0.
For that we will later introduce absolute stability.
Note that we haven’t given a precise definition to zero stability
because in some sense it is defined as: the extra conditions that are
needed to make a consistent method convergent.
For multistep methods, covered later, we will figure out an explicit
condition.
A. Donev (Courant Institute) ODEs 2/12/2019 24 / 35
 Convergence (after LeVeque)

Convergence of One Step Methods

Let us prove that all consistent one-step methods are convergent

(i.e., they are zero stable).
 
x(k+1) = x(k) + ∆tΨ x(k) , t = k∆t, ∆t; f

The method is consistent and we assume that Ψ is continuous in t

and ∆t, and Lipshitz in x with constant e
L,

Ψ (x, t, 0; f) = f (x, t) ,

For example, for explicit midpoint rule,

 

(k)
 ∆t ∆t
Ψ x , t = k∆t, ∆t; f = f x + f (x, t) , t +
2 2

A. Donev (Courant Institute) ODEs 2/12/2019 25 / 35

 Convergence (after LeVeque)

Convergence of One Step Methods

Now use the Lipshitz continuity to bound the error growth:

kΨ (x, t, ∆t) − Ψ (x? , t, ∆t)k

   
∆t ? ∆t ?

≤ L x +
f (x, t) − x + f (x , t)
2 2
L∆t
≤ L kx − x? k + kf (x, t) − f (x? , t)k
  2
L∆t
≤ 1+ L kx − x? k
2
= L̃ kx − x? k .

A. Donev (Courant Institute) ODEs 2/12/2019 26 / 35

 Convergence (after LeVeque)

Error growth factor

Let us now define a local truncation error in the way done in
LeVeque’s book, which is better for analysis:
x ((k + 1) ∆t) − x (k∆t)
e(k) = − Ψ (x (k∆t) , k∆t, ∆t) ,
∆t

x ((k + 1) ∆t) = x (k∆t) + ∆tΨ (x (k∆t) , k∆t, ∆t) + ∆te(k)

 
x(k+1) = x(k) + ∆tΨ x(k) , t = k∆t, ∆t
Subtracting the two we get the error recursion relation for the global
error E(k) = x (k∆t) − x(k) ,
  
E(k+1) = E(k) + ∆t Ψ (x (k∆t) , . . . ) − Ψ x(k) , . . . + ∆te(k)

(k+1) (k) (k) (k)
E ≤
E + ∆t L
e E + ∆t e
A. Donev (Courant Institute) ODEs 2/12/2019 27 / 35
 Convergence (after LeVeque)

Convergence

This is the typical relationship we will see many times

 
(k+1)
E ≤ 1 + ∆t e L E(k) + ∆t e(k)

Quite generally, we get recursions of the form:

global error(k+1) <= ampl factor*global error(k)+local error(k)
The recurrence relationship for the error has the explicit solution
 k k 
X k−m
(k) (0) (m−1)
E ≤ 1 + ∆t L E + ∆t
e 1 + ∆t L
e e .
m=1

We can take E(0) = 0 if we know the initial condition “exactly”,

leaving us to bound
 k
1 + ∆t e
L

A. Donev (Courant Institute) ODEs 2/12/2019 28 / 35

 Convergence (after LeVeque)

Error bound
Very generally we will bound error growth factors by exponentials
(here simple scalars but more generally matrix powers and matrix
exponentials):
   k
L ≤ e ∆t L ⇒
1 + ∆t e L ≤ e k∆t L ≤ e T L .
1 + ∆t e
e e e

k  k
!
X k−m X
(k) (m−1)
≤ ∆te T L
(m−1)
E ≤ ∆t 1 + ∆t L e e
e e

m=1 m=1

(k)
E ≤ Te T L max e(m−1) .
e
1≤m<k
This now proves that if the local error (defined in LeVeque’s way!) is
of O (∆t p ) then so is the global error.
The factor Te T L is a constant for the purpose of zero stability as we
e

are taking the limit ∆t → 0, but in practice it is extremely important

as it controls how small ∆t has to be for the method to be useful...
A. Donev (Courant Institute) ODEs 2/12/2019 29 / 35
 MATLAB ode suite

MATLAB ode suite

A. Donev (Courant Institute) ODEs 2/12/2019 30 / 35

 MATLAB ode suite

In MATLAB

In MATLAB, there are several functions whose names begin with

[t, x] = ode(f , [t0 , te ], x0 , odeset(. . . )).

ode23 is a second-order adaptive explicit Runge-Kutta method,

while ode45 is a fourth-order version (try it first).
ode23tb is a second-order implicit RK method.
ode113 is a variable-order explicit multi-step method that can
provide very high accuracy.
ode15s is a variable-order implicit multi-step method.
For implicit methods the Jacobian can be provided using the odeset
routine – very important!

A. Donev (Courant Institute) ODEs 2/12/2019 31 / 35

 MATLAB ode suite

Rigid body motion

f u n c t i o n dy = r i g i d ( t , y )
dy = z e r o s ( 3 , 1 ) ; % a column v e c t o r
dy ( 1 ) = y ( 2 ) ∗ y ( 3 ) ;
dy ( 2 ) = −y ( 1 ) ∗ y ( 3 ) ;
dy ( 3 ) = −0.51 ∗ y ( 1 ) ∗ y ( 2 ) ;
%−−−−−−−−−−−

o p t s = o d e s e t ( ’ R e l T o l ’ , 1 e −3, ’ AbsTol ’ , [ 1 e−4 1 e−4 1 e − 5 ]

[ T , Y ] = ode45 ( @ r i g i d , [ 0 1 2 ] , [ 0 1 1 ] , o p t s ) ;

p l o t (T , Y ( : , 1 ) , ’ o−−r ’ , T , Y ( : , 2 ) , ’ s−−b ’ , T , Y ( : , 3 ) , ’ d−−g ’

x l a b e l ( ’ t ’ ) ; y l a b e l ( ’ y ’ ) ; t i t l e ( ’ R e l T o l=1e − 3 ’ ) ;

A. Donev (Courant Institute) ODEs 2/12/2019 32 / 35

 MATLAB ode suite

van der Pol equation

r =10; % Try r =100

f = @( t , y ) [ y ( 2 ) ; r ∗ ( 1 − y ( 1 ) ˆ 2 ) ∗ y ( 2 ) − y ( 1 ) ] ;

figure (2); clf

[ T , Y ] = ode45 ( f , [ 0 3∗ r ] , [ 2 1 ] ) ;
p l o t (T , Y ( : , 1 ) , ’ o−−r ’ , T , Y ( : , 2 ) / r , ’ o−−b ’ )
t i t l e ( [ ’ ode45 ( e x p l i c i t ) n s t e p s = ’ , i n t 2 s t r ( s i z e (T , 1 ) ) ]

figure (3); clf

[ T , Y ] = o d e 1 5 s ( f , [ 0 3∗ r ] , [ 2 0 ] ) ;
p l o t (T , Y ( : , 1 ) , ’ o−−b ’ , T , Y ( : , 2 ) / r , ’ o−−r ’ )
t i t l e ( [ ’ o d e 1 5 s ( i m p l i c i t ) n s t e p s = ’ , i n t 2 s t r ( s i z e (T , 1 ) )

A. Donev (Courant Institute) ODEs 2/12/2019 33 / 35

 MATLAB ode suite

Stiff van der Pol system (r = 10)

ode45 (explicit) nsteps=877 ode15s (implicit) nsteps=326

2.5 2.5

2 2

1.5 1.5

1 1

0.5 0.5

0 0

−0.5 −0.5

−1 −1

−1.5 −1.5

−2 −2

−2.5 −2.5
0 5 10 15 20 25 30 0 5 10 15 20 25 30

A. Donev (Courant Institute) ODEs 2/12/2019 34 / 35

 MATLAB ode suite

Conclusions/Summary

Time stepping methods for ODEs are convergent if and only if they
are consistent and zero-stable.
All one-step methods are zero-stable, therefore, there are generic
methods that work for any (finite-dimensional) system of ODEs
(not true of PDEs).
We distinguish methods based on their order of accuracy and on
whether they are explicit (forward Euler, Heun, RK4,
Adams-Bashforth), or implicit (backward Euler, Crank-Nicolson), and
whether they are adaptive.
Runge-Kutta methods require more evaluations of f but are more
robust, especially if adaptive (e.g., they can deal with sharp changes
in f ). Generally the recommended first-try (ode45 or ode23 in
MATLAB).

A. Donev (Courant Institute) ODEs 2/12/2019 35 / 35