Parameter Estimation for ODE.
Shyan-Shiou Chen
Ph. D., Applied Mathematics,
Department of Mathematics,
National Taiwan Normal University
August 12, 2024
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 1 / 24
Outline
Outline
1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm
2 fminsearch in Matlab
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 2 / 24
lsqcurvefit in Matlab
lsqcurvefit 內含的演算法
1 Gauss-Newton Method (高斯-牛頓法)
2 Levenberg-Marquardt Method (LM 法)
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 3 / 24
lsqcurvefit in Matlab Gradient Descent Method
Outline
1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm
2 fminsearch in Matlab
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 4 / 24
lsqcurvefit in Matlab Gradient Descent Method
Gradient Descent Method
xk+1 = xk − α∇f (xk ) ,
Example
Consider f (x, y) = x2 + y 2 /4. Here, x = (x, y).
2 2 2 2
Gradient Descent on f(x,y) = x + y /4 Gradient Descent on f(x,y) = x + y /4
3 3
2 2
1 1
0 0
y
-1 y -1
Contour Contour
-2 Iteration Points -2 Iteration Points
Gradient Directions Gradient Directions
-3 -3
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
x x
(a) (b)
Figure: Matlab Code. (EX0050GradientDescent2dwithGraph.m)
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 5 / 24
lsqcurvefit in Matlab Newton’s Method for finding zeros
Outline
1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm
2 fminsearch in Matlab
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 6 / 24
lsqcurvefit in Matlab Newton’s Method for finding zeros
Newton Method for f (x) = 0 (cont.)
1 Also known as the Newton-Raphson method.
2 A fundamental algorithm in numerical analysis for finding the
roots of a real-valued function.
3 This method iteratively generates increasingly accurate
approximations to the roots (or zeros) of a function.
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 7 / 24
lsqcurvefit in Matlab Newton’s Method for finding zeros
Newton Method for f (x) = 0 (cont.)
f (x0 )
x1 = x0 −
f ′ (x0 )
Geometric Interpretation Geometrically, the point (x1 , 0) represents the
x -intercept of the tangent to the graph of f at (x0 , f (x0 )). In other
words, x1 is the root of the linear approximation of f at the initial guess
x0 . This approximation process is iterated as follows:
f (xn )
xn+1 = xn −
f ′ (xn )
This process continues until the difference between successive
approximations is sufficiently small. The number of correct digits in
the approximation typically doubles with each iteration, showcasing
the quadratic convergence of the method.
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 8 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Outline
1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm
2 fminsearch in Matlab
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 9 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Newton’s method in optimization (cont.)
Given a twice differentiable function f : R → R, we seek to solve the
optimization problem
min f (x)
x∈R
Newton’s method attempts to solve this problem by constructing a se-
quence {xk } from an Newton’s method uses curvature information (i.e.
the second derivative) to take a more direct route. initial guess (starting
point) x0 ∈ R that converges towards a minimizer x∗ of f by using a se-
quence of second-order Taylor approximations of f around the iterates.
The second-order Taylor expansion of f around xk is
1
f (xk + t) ≈ f (xk ) + f ′ (xk ) t + f ′′ (xk ) t2
2
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 10 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Newton’s method in optimization (cont.)
The next iterate xk+1 is defined so as to minimize this quadratic ap-
proximation in t, and setting xk+1 = xk + t. If the second derivative is
positive, the quadratic approximation is a convex function of t, and its
minimum can be found by setting the derivative to zero. Since
( )
d 1 ′′
0= f (xk ) + f (xk ) t + f (xk ) t = f ′ (xk ) + f ′′ (xk ) t
′ 2
dt 2
the minimum is achieved for
f ′ (xk )
t=− .
f ′′ (xk )
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 11 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Newton’s method in optimization (cont.)
Putting everything together, Newton’s method performs the iteration
f ′ (xk )
xk+1 = xk + t = xk −
f ′′ (xk )
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 12 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Parameter Estimation by Newton’s method (cont.)
1 Objective Function: The goal is to minimize the objective
function, which is typically the sum of squared residuals:
∑
n
S(θ) = [yi − f (xi , θ)]2
i=1
where yi are the observed data, f (xi , θ) is the model prediction,
and θ represents the parameters to be estimated.
2 Gradient (First Derivative): The gradient of the objective function
with respect to the parameters θ is given by:
∂S(θ)
∇S(θ) =
∂θ
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 13 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Parameter Estimation by Newton’s method (cont.)
Example
∑
Let θ be a parameter. S(θ) = 2i=1 [yi − f (xi , θ)]2 . Find that ∇S(θ).
Let ri = yi − f (xi , θ). Find J(θ) such that ∇S(θ) = −J(θ) · (r1 , r2 )t .
3 Hessian Matrix (Second Derivative): The Hessian matrix H is the
matrix of second-order partial derivatives of the objective function
with respect to the parameters:
∂ 2 S(θ)
H(θ) =
∂θ2
The Hessian captures the curvature of the objective function and
provides additional information that can be used to update the
parameters more effectively.
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 14 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Parameter Estimation by Newton’s method (cont.)
Example
[( )2 ]
∂ 2 S(θ) ∑2 ∂f (xi ,θ) 2 (x ,θ)
Show that ∂θ 2
=2 i=1 ∂θ − ri ∂ f∂θ 2
i
.
4 Parameter Update Rule: The Newton method updates the
parameters using the following rule:
θk+1 = θk − H (θk )−1 ∇S (θk )
where θk is the current estimate of the parameters, H (θk ) is the
Hessian matrix at θk , and ∇S (θk ) is the gradient at θk .
5 Iteration and Convergence: The method iteratively updates the
parameter estimates until convergence is achieved, which typically
means that the change in parameters or the gradient is below a
specified tolerance.
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 15 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Gauss-Newton Method (cont.)
Hessian Approximation: The Hessian matrix H(θ), which would
normally involve second derivatives, is approximated in the
Gauss-Newton method as:
H(θ) ≈ J(θ)T J(θ)
Parameter Update Rule: The parameters are updated using the
following rule:
θk+1 = θk + ∆θ
where ∆θ is computed as:
[ ]−1
∆θ = J (θk )T J (θk ) J (θk )T r (θk )
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 16 / 24
lsqcurvefit in Matlab Newton’s method in optimization
Gauss-Newton Method (cont.)
∂f (x1 ,θ) ∂f (x1 ,θ) ∂f (x1 ,θ)
∂θ1 ∂θ2 ... ∂θm
∂f (x2 ,θ) ∂f (x2 ,θ) ∂f (x2 ,θ)
...
J(θ) =
∂θ1
..
∂θ2
.. ..
∂θm
..
. . . .
∂f (xn ,θ) ∂f (xn ,θ) ∂f (xn ,θ)
∂θ1 ∂θ2 ... ∂θm
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 17 / 24
lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm
Outline
1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm
2 fminsearch in Matlab
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 18 / 24
lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm
The Levenberg-Marquardt Algorithm
Parameter Update Rule: The LM algorithm updates the parameters
using:
[ ]−1
θk+1 = θk − J (θk )T J (θk ) + λI J (θk )T r (θk )
Here, λ is the damping factor that controls the step size and direction.
When λ is large, the algorithm behaves more like gradient descent,
ensuring stability. When λ is small, it behaves more like the
Gauss-Newton method, promoting faster convergence.
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 19 / 24
lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm
Examples for Newton Method (cont.)
Example
Consider a simple curve-fitting problem where we want to fit a model
y = aebx to some data points (xi , yi ) The residuals are given by
ri (a, b) = yi − aebxi
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 20 / 24
lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm
Examples for Newton Method (cont.)
Newton Method Fitting: y = a * exp(b * x)
6.5
Observed Data
6 Fitted Model
5.5
y 4.5
3.5
2.5
2
1 1.5 2 2.5 3 3.5 4 4.5 5
x
Figure: (Code: EX0052NewtonMethod01.m)
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 21 / 24
fminsearch in Matlab
fminsearch in Matlab (Animation: Adobe only)
f (x, y) = 100(y − x2 )2 + (1 − x)2
Figure: Nelder-Mead Simplex Method Optimization Process (fminsearch.m)
(Adobe only.)
S-S Chen (Dep. of Math., NTNU)
Parameter Estimation August 12, 2024 22 / 24
fminsearch in Matlab
Compare three different methods
Comparison of Parameter Estimation Methods
250
Observed Data
lsqcurvefit
fminsearch
200 fminunc
150
y
100
50
0
1 2 3 4 5 6
x
Figure: Compare three methods,
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 23 / 24
fminsearch in Matlab
Thank You for Your Attention!
S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 24 / 24
