Numerical Methods for
Engineers
EE0903301
Lecture 9: Optimization
Dr. Yanal Faouri
Optimization
• From a mathematical perspective, optimization deals with finding the maxima and minima of a function that
depends on one or more variables.
• The goal is to determine the values of the variables that yield maxima or minima for the function. These can then
be substituted back into the function to compute its optimal values.
• Optimization is similar in spirit to the root-location methods in that both involve guessing and searching for a
point on a function, but it involves searching for the function’s extreme points instead of the location where the
function equals zero.
• The optimums are the points where the curve is
flat. ➔ where f ′(x) is equal to zero.
• The second derivative, f ″(x), indicates whether the
optimum is a minimum or a maximum:
• if f ″(x) < 0, the point is a maximum;
• if f ″(x) > 0, the point is a minimum.
Example: Determining the Optimum Analytically
by Root Location
• An object like a bungee jumper can be projected upward at a specified velocity. If it is subject to linear drag, its
altitude as a function of time can be computed as
• where z0 = the initial altitude (m), m = mass (kg), c = a linear drag coefficient (kg/s), υ0 = initial velocity (m/s), and
t = time (s).
• Problem Statement.
Determine the time and magnitude of the peak
elevation based on the equation. Use the
following parameter values for your calculation: g
= 9.81 m/s2, z0 = 100 m, υ0 = 55 m/s, m = 80 kg,
and c = 15 kg/s.
Example: Determining the Optimum Analytically
by Root Location
• Solution. The equation can be differentiated to give
• Set the derivative equal to zero and solve for t;
• Substitute in the original equation
• Notice that the second derivative is negative, so the function exhibit a maximum.
One- and Two-Dimensional Optimization
• The process of finding a maximum versus finding a
minimum is essentially identical because the same value
x* both minimizes f(x) and maximizes −f(x), as shown for
a one-dimensional function in Fig. a.
• Fig. b displays two-dimensional optimization. Note that
this figure can be taken to represent either a
maximization (contours increase in elevation up to the
maximum like a mountain) or a minimization (contours
decrease in elevation down to the minimum like a valley).
One-Dimensional Optimization
• What are the techniques to find the minimum or maximum of a function of a single variable f(x)?
• Recall that root location was complicated by the fact that several roots can occur for a single function. Similarly,
both local and global optima can occur in optimization.
• A global optimum represents the very best solution. A local optimum, though not the very best, is better than its
immediate neighbors. Cases that include local optima are called multimodal. In such cases, we will almost always
be interested in finding the global optimum.
• Just as in root location, optimization in one dimension can be divided into bracketing and open methods. The
golden-section search is an example of a bracketing method that is very similar in spirit to the bisection method
for root location and it is based on the golden ratio.
Golden-Section Search
• It is a simple, general-purpose method for determining the optimum of a single-variable function.
• The method starts by determining an interval that should contain a single minimum, and hence is called
unimodal. Then two intermediate function values are required to detect whether a minimum occurred.
• For the golden section search, the two intermediate points are chosen according to the golden ratio:
• x1 = xl + d
• x2 = xu − d
• d = (ϕ − 1)(xu − xl)
• The function is evaluated at these two interior points. Two results can occur:
1. If, as in Fig. a, f(x1) < f(x2), then f(x1) is the minimum, and the domain of x to the left of x2, from xl to x2, can be
eliminated because it does not contain the minimum. For this case, x2 becomes the new xl for the next round.
2. If f(x2) < f(x1), then f(x2) is the minimum and the domain of x to the right of x1, from x1 to xu would be eliminated.
For this case, x1 becomes the new xu for the next round.
Golden-Section Search
• The real benefit from the use of the golden ratio. Because the original x1 and x2 were
chosen using the golden ratio, we do not have to recalculate all the function values for
the next iteration. For example, for the case illustrated in Fig. a, the old x1 becomes the
new x2. This means that we already have the value for the new f(x2), since it is the same
as the function value at the old x1.
• To complete the algorithm, we need only determine the new x1. This is done with d
which is based on the new values of xl and xu. A similar approach would be used for the
alternate case where the optimum fell in the left subinterval. For this case, the new x2
would be computed.
• As the iterations are repeated, the interval containing the extremum is reduced rapidly.
In fact, each round the interval is reduced by a factor of ϕ − 1 (about 61.8%).
• That means that after 10 rounds, the interval is shrunk to about 0.61810 or 0.008 or
0.8% of its initial length.
• After 20 rounds, it is about 0.0066%. This is not quite as good as the reduction
achieved with bisection (50%), but optimization is a harder problem than root location.
Example: Golden-Section Search
• Problem Statement.
• Use the golden-section search to find the minimum within the interval from xl = 0 to xu = 4 of
• Solution.
• First, the golden ratio is used to create the two interior points:
• d = 0.61803(4 − 0) = 2.4721
• x1 = 0 + 2.4721 = 2.4721 and x2 = 4 − 2.4721 = 1.5279
• The function can be evaluated at the interior points:
Example: Golden-Section Search
• Because f(x2) < f(x1), our best estimate of the minimum at this point is that it is located at x = 1.5279 with a value
of f(x) = –1.7647.
• In addition, the minimum is known to be in the interval defined by xl, x2 and x1.
• Thus, for the next iteration, the lower bound remains xl = 0, and x1 becomes the upper bound, that is, xu =
2.4721.
• In addition, the former x2 value becomes the new x1, that is, x1 = 1.5279. In addition, we do not have to
recalculate f(x1), it was determined on the previous iteration as f(1.5279) = –1.7647.
• All that remains is to compute the new value of d and x2:
• d = 0.61803(2.4721 − 0) = 1.5279
• x2 = 2.4721 − 1.5279 = 0.9443
• The function evaluation at x2 is f(0.9943) = −1.5310. Since this value is greater than the function value
at x1, the minimum is f(1.5279) = −1.7647, and it is in the interval prescribed by x2, x1, and xu.
Example: Golden-Section Search
• The process can be repeated, with the results tabulated here:
• Note that the current minimum is highlighted for every iteration. After the eighth iteration, the minimum occurs
at x = 1.4427 with a function value of −1.7755. Thus, the result is converging on the true value of −1.7757 at x =
1.4276.
Example: Golden-Section Search
• An upper bound for golden-section search can be derived as follows: Once an iteration is complete, the optimum
will either fall in one of two intervals. If the optimum function value is at x2, it will be in the lower interval (xl, x2,
x1).
• If the optimum function value is at x1, it will be in the upper interval (x2, x1, xu). Because the interior points are
symmetrical, either case can be used to define the error.
• Looking at the upper interval (x2, x1, xu), if the true value were at the far left, the maximum distance from the
estimate would be Δ𝑥𝑎 . If the true value were at the far right, the maximum distance from the estimate would
be Δ𝑥𝑏 .
• Therefore, this case would represent the maximum error which can
then be normalized to the optimal value for that iteration xopt to yield
the basis for terminating the iterations
An M-file to determine the minimum of a function
with the golden-section search: goldmin
function [x,fx,ea,iter] = goldmin(f,xl,xu,es,maxit,varargin) d = (phi-1)*(xu-xl);
% goldmin: minimization golden section search x1 = xl + d; x2 = xu-d;
% [x,fx,ea,iter] = goldmin(f,xl,xu,es,maxit,p1,p2,...): f1 = f(x1,varargin{:}); f2 = f(x2,varargin{:});
% uses golden section search to find the minimum of f while (1)
% input: xint = xu-xl;
% f = name of function if f1 < f2
% xl, xu = lower and upper guesses xopt = x1; xl = x2; x2 = x1; f2 = f1;
% es = desired relative error (default = 0.0001%) x1 = xl + (phi-1)*(xu-xl); f1 = f(x1,varargin{:});
% maxit = maximum allowable iterations (default = 50) else
% p1,p2,... = additional parameters used by f xopt = x2; xu = x1; x1 = x2; f1 = f2;
% output: x2 = xu-(phi-1)*(xu-xl); f2 = f(x2,varargin{:});
% x = location of minimum end
% fx = minimum function value iter = iter + 1;
% ea = approximate relative error (%) if xopt ~= 0, ea = (2-phi) * abs(xint / xopt) *
% iter = number of iterations 100;end
if nargin < 3,error('at least 3 input arguments required'),end if ea <= es || iter >= maxit,break,end
if nargin < 4||isempty(es), es = 0.0001;end end
if nargin < 5||isempty(maxit), maxit = 50;end x = xopt; fx = f(xopt,varargin{:});
phi = (1+sqrt(5))/2; iter = 0;
Testing: goldmin
>> g = 9.81;v0 = 55;m = 80;c = 15;z0 = 100;
>> z = @(t)-(z0 + m/c*(v0 + m*g/c)*(1 - exp(-c/m*t)) -m*g/c*t);
>> [xmin,fmin,ea] = goldmin(z,0,8)
➔ xmin = 3.8317
➔ fmin = −192.8609
➔ea = 6.9356e−005
• Notice fmin is negative which corresponds to a maximum height of 192.8609.
Golden-Section Search
• Why it is necessary to reduce function evaluations of the golden-section search? Of course, for solving a single
optimization, the speed savings would be negligible.
• There are two important contexts where minimizing the number of function evaluations can be important. These
are:
1. Many evaluations. There are cases where the golden-section search algorithm may be a part of a much larger
calculation. In such cases, it may be called many times. Therefore, keeping function evaluations to a minimum
could pay great dividends for such cases.
2. Time-consuming evaluation. For academic reasons, we use simple functions in most of our examples. You should
understand that a function can be very complex and time-consuming to evaluate. For example, optimization can be
used to estimate the parameters of a model consisting of a system of differential equations. For such cases, the
“function” involves time-consuming model integration. Any method that minimizes such evaluations would be
advantageous.
Parabolic Interpolation
• Parabolic interpolation takes advantage of the fact that a second-order polynomial often provides a good
approximation to the shape of f(x) near an optimum.
• Just as there is only one straight line connecting two points, there is only one parabola connecting three points.
• Then it can be differentiated and set the result equal to zero, and solve for an estimate of the optimal x.
• It can be shown through some algebraic manipulations that the result is
• Where x1, x2, and x3 are the initial guesses, and x4 is the value of x that corresponds to the optimum value of the
parabolic fit to the guesses.
Example: Parabolic Interpolation
• Problem Statement.
• Use parabolic interpolation to approximate the minimum of
• with initial guesses of x1 = 0, x2 = 1, and x3 = 4.
• Solution. The function values at the three guesses can be evaluated:
• x1 = 0 ➔ f(x1) = 0
• x2 = 1 ➔ f(x2) = −1.5829
• x3 = 4 ➔ f(x3) = 3.1136
• Substitute into the parabolic interpolation equation:
• Which has a function value of f(1.5055) = −1.7691.
Example: Parabolic Interpolation
• Next, a strategy similar to the golden-section search can be employed to determine which point should be
discarded. Because the function value for the new point is lower than for the intermediate point (x2) and the
new x value is to the right of the intermediate point, the lower guess (x1) is discarded. Therefore, for the next
iteration:
• x1 = 1 ➔ f(x1) = −1.5829
• x2 = 1.5055 ➔ f(x2) = −1.7691
• x3 = 4 ➔ f(x3) = 3.1136
• Which can be substituted into parabolic interpolation equation to give
• Which has a function value of f(1.4903) = −1.7714
Example: Parabolic Interpolation
• The process can be repeated, with the results tabulated here:
• Thus, within five iterations, the result is converging rapidly on the true value of −1.7757 at x = 1.4276.
MATLAB Function: fminbnd
• MATLAB fminbnd function combines the slow, dependable golden-section search with the faster, but possibly
unreliable, parabolic interpolation. It first attempts parabolic interpolation and keeps applying it as long as
acceptable results are obtained. If not, it uses the golden-section search to get matters in hand.
• A simple expression of its syntax is
>> [xmin, fval] = fminbnd(function,x1,x2)
• where x and fval are the location and value of the minimum, function is the name of the function being
evaluated, and x1 and x2 are the bounds of the interval being searched.
• MATLAB fminbnd to solve the example in slide 3.
>> g = 9.81;v0 = 55;m = 80;c = 15;z0 = 100; Try:
>> options = optimset('display','iter');
>> z = @(t) −(z0 + m/c*(v0 + m*g/c)*(1 − exp(−c/m*t)) − m*g/c*t);
>> fminbnd (z,0,8,options)
>> [x,f] = fminbnd(z,0,8)
➔ x = 3.8317
➔ f = −192.8609
Example: Visualizing a Two-Dimensional
Function
• Problem Statement. Use MATLAB’s graphical capabilities to display the following function and visually
estimate its minimum in the range –2 ≤ x1 ≤ 0 and 0 ≤ x2 ≤ 3:
• Solution. The following script generates contour and mesh plots of the function:
x = linspace(−2,0,40);y = linspace(0,3,40);
[X,Y] = meshgrid(x,y);
Z = 2 + X - Y + 2 * X.^2 + 2 * X.* Y + Y.^2;
subplot(1,2,1);
cs = contour(X,Y,Z);clabel(cs);
xlabel('x_1');ylabel('x_2');
title('(a) Contour plot');grid;
subplot(1,2,2);
cs = surfc(X,Y,Z);
zmin = floor(min(Z));
zmax = ceil(max(Z));
xlabel('x_1');ylabel('x_2');zlabel('f(x_1,x_2)'); both plots indicate that function has a minimum value of
title ('(b) Mesh plot'); about f(x1, x2) = 0 to 1 located at about x1 = −1 and x2 = 1.5.
MATLAB Function: fminsearch
• Standard MATLAB has a function fminsearch that can be used to determine the minimum of a
multidimensional function. It is based on the Nelder-Mead method, which is a direct-search method that
uses only function values (does not require derivatives) and handles non-smooth objective functions.
• A simple expression of its syntax is
>> [xmin, fval] = fminsearch (function,x0)
• where xmin and fval are the location and value of the minimum, function is the name of the function being
evaluated, and x0 is the initial guess. Note that x0 can be a scalar, vector, or a matrix.
• MATLAB session that uses fminsearch to determine minimum for the function we just graphed in the
previous example:
>> f = @ (x) 2 + x (1) - x (2) + 2*x (1)^2 + 2*x (1)*x (2) + x (2)^2;
>> [x,fval] = fminsearch (f,[-0.5,0.5])
➔ x = −1.0000 1.5000
➔ fval = 0.7500
