Numerical Optimization in MATLAB
Andrii Parkhomenko
Universitat Autònoma de Barcelona and Barcelona GSE
Spring 2017
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 0 / 24
Optimization
Typical problem in economics: find an optimum
Case 1:
max f (x) = 0.5x − 0.02x 2
x
Easy: just solve for x that satisfies f 0 (x) = 0
Case 2:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3
Can’t solve analytically ⇒ need to use numerical methods
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 1 / 24
Numerical Optimization
Continuous differentiable objective function: derivative-based methods
Arbitrary objective function: derivative-free methods
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 2 / 24
Numerical Optimization in MATLAB
These slides cover three functions contained in MATLAB’s Optimization
Toolbox:
fminunc: unconstrained optimization, derivative-based
fmincon: constrained optimization, derivative-based
fminsearch: unconstrained optimization, derivative-free
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 3 / 24
fminunc
Finds parameters that minimize a given function
Syntax:
x = fminunc(objfun,x0)
I x0: starting guess
I objfun: objective function specified in a different script
I x: solution
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 4 / 24
fminunc
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3
0.6
0.4
0.2
−0.2
f(x)
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 5 / 24
fminunc
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3
Step 1. Main code
x0 = 1;
x = fminunc(@objfun,x0)
Step 2. Function
function y = objfun(x)
y = 0.5*log(x) - 0.02*x^2 + (1+x)^(-5/3);
y = -y;
end
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 6 / 24
fminunc
fminunc uses the BFGS Quasi-Newton method:
xk+1 = xk + αk pk
1 Compute direction pk
pk = −Bk−1 ∇fk
Bk sk sk0 Bk yk yk0
Bk+1 = Bk − +
sk0 Bk sk yk0 sk
where sk ≡ xk+1 − xk and yk ≡ fk+1 − fk
2 Compute stepsize αk . Use line search to approximately solve
min f (xk + αk pk )
αk
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 7 / 24
fminunc
Introduce options:
options = optimset(’Opt1’,Opt1Val,’Opt2’,Opt2Val,...);
x = fminunc(@objfun,x0, options)
Important algorithm options:
Display:
optimset(’Display’, ’off’)
Tolerance level:
optimset(’TolFun’, 1e-6)
optimset(’TolX’, 1e-6)
Maximum number of iterations or evaluations
optimset(’MaxIter’, 400)
optimset(’MaxFunEvals’, 100)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 8 / 24
fminunc
Caution:
Objective function must be continuous
Local minima and flat regions are big issues
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 9 / 24
fminunc
Example 2. Discontinuous objective function:
0.5 ln x − 0.02x 2 + 1 5 , 0<x ≤4
(
(1+x) 3
max f (x) =
x 0.5 ln x − 0.02x 2 + 1 5 + 1, 4<x
(1+x) 3
1.5
0.5
f(x)
−0.5
−1
−1.5
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 10 / 24
fminunc
Example 3. Objective function with flat regions:
0.5 ln x − 0.02x 2 + 1 5 , 0 < x ≤ 4
(
max f (x) = (1+x) 3
x 0.441546, 4<x
0.6
0.4
0.2
−0.2
f(x)
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 11 / 24
fminunc
Example 4. Objective function with local maxima:
max f (x) = sin(3x) + x − 0.25x 2
x
1.5
1
f(x)
0.5
−0.5
−1
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 12 / 24
fminunc
To obtain more details on the workings of fminunc:
Read documentation: Help → Type ’fminunc’
Read the code: Type ’open fminunc’
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 13 / 24
fmincon
Finds parameters that minimize a given function, subject to a
constraint
Same algorithm as fminunc
Syntax:
x = fmincon(objfun,x0,A,b,Aeq,beq,lb,ub,nlcon,options)
I x0: starting guess
I objfun: objective function specified in a different script
I A, b: linear inequality constraint: Ax ≤ b
I Aeq, beq: linear equality constraint: Ax = b
I lb, ub: lower and upper bounds: x ∈ [lb, ub]
I nlcon: nonlinear constraint
I x: solution
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 14 / 24
fmincon
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5 s.t. x ≤ a
x (1 + x) 3
0.6
0.4
0.2
−0.2
f(x)
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 15 / 24
fmincon
Sometimes it’s better to reformulate a constrained problem as an
unconstrained
I Can use a formulation with Lagrange multipliers
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 16 / 24
fminsearch
Finds parameters that minimize a given function
Derivative-free algorithm (Nelder-Mead)
Syntax:
x = fminsearch(objfun,x0,options)
I x0: starting guess
I objfun: objective function specified in a different script
I x: solution
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 17 / 24
fminsearch
Nelder-Mead algorithm:
1 select 2 points: x1 and x2
2 order s.t. x̂ = x1 and x̄ = x2 if f (x̂) ≥ f (x̄); and x̂ = x2 and x̄ = x1
otherwise
3 reflection: x R = x̂ + (x̂ − x̄)
4 if f (x R ) > f (x̂) and f (x R ) > f (x̄), expansion: x E = x̂ + α(x R − x̂), α > 1
if f (x R ) ≤ f (x̂) and f (x R ) > f (x̄), update x̄ = x R and go to step 2
if f (x R ) ≤ f (x̂) and f (x R ) ≤ f (x̄), contraction: x C = x̂ + β(x̄ − x̂), β < 21
5 if f (x E ) > f (x R ), update x̄ = x E and go to step 2
if f (x E ) < f (x R ), update x̄ = x R and go to step 2
6 if f (x C ) > f (x̄), update x̄ = x C and go to step 2
if f (x C ) < f (x̄), shrink: replace x̄ = x̂ + γ(x̄ − x̂) and go to step 2, γ < 1
Iterate until convergence: x̂ ≈ x̄ and/or f (x̂) ≈ f (x̄)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 18 / 24
fminsearch
Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3
0.6
0.4
0.2
−0.2
f(x)
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 19 / 24
fminsearch
Example 2. Discontinuous objective function:
0.5 ln x − 0.02x 2 + 1 5 , 0<x ≤4
(
(1+x) 3
max f (x) =
x 0.5 ln x − 0.02x 2 + 1 5 + 1, 4<x
(1+x) 3
1.5
0.5
f(x)
−0.5
−1
−1.5
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 20 / 24
fminsearch
Example 3. Objective function with flat regions:
0.5 ln x − 0.02x 2 + 1 5 , 0 < x ≤ 4
(
max f (x) = (1+x) 3
x 0.441546, 4<x
0.6
0.4
0.2
−0.2
f(x)
−0.4
−0.6
−0.8
−1
−1.2
−1.4
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 21 / 24
fminsearch
Example 4. Objective function with local maxima:
max f (x) = sin(3x) + x − 0.25x 2
x
1.5
1
f(x)
0.5
−0.5
−1
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 22 / 24
fminsearch
Example 5. Objective function with many discontinuities:
max f (x) = bxc + 0.2x − 0.04x 2
x
5
f(x)
0
0 5 10 15 20
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 23 / 24
fminunc vs fminsearch
Rule of thumb: use fminunc for continuous differentiable functions,
fminsearch otherwise
No option guarantees that you find a (global) optimum
For complicated cases:
I understand the problem well
I try different algorithms
I try different starting points
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 24 / 24
