Equality Constraint

Equality and Inequality Constraints

Algorithms

Chapter 8
Constrained Optimization

School of Electrical & Computer Engineering, AAiT

Addis Ababa University.

Class notes on: Mathematical Foundations in Engineering,

ECEG 6209

1 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Outline

1 Equality Constraint
Solution by Direct Substitution
Solution by the Method of Lagrange Multipliers

2 Equality and Inequality Constraints

3 Algorithms

2 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Constrained optimization
This lecture considers constrained optimization
Constrained optimization

min f (x )
x ∈Rn
subject to :
ci (x ) = 0, i = 1, . . . , ne
dj (x ) ≤ 0, j = 1, . . . , ni
Equality constraint functions: ci : Rn → R
Inequality constraint functions: dj : Rn → R
Feasible set: D = {x ∈ Rn | ci (x ) = 0, dj (x ) ≤ 0, i =
1, . . . , ne , j = 1, . . . , ni }
We continue to assume all functions are twice-continuously
differentiable.
3 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Solution methods

In this section we consider the optimization of functions

subjected to equality constraints:

min f (x )
x ∈Rn
subject to :
ci (x ) = 0, i = 1, . . . , ne

ne ≤ n; otherwise (if ne > n), the problem becomes

overdefined and, in general, there will be no solution.
There are several methods available for the solution of this
problem. We consider the methods of:
direct substitution, and
Lagrange multipliers.

4 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Solution by direct substitution

Theoretically it is possible to solve simultaneously the ne

equality constraints and express them in terms of the
remaining n − ne variables.
Substitute the expressions in the objective function
The new objective function is not subjected to any constraint,
and we have unconstrained optimization problem!
The method appears to be simple in theory, but not
convenient from a practical point of view.
The reason: the constraint equations will be nonlinear for
most of practical problems.

5 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Solution by direct substitution..

Example: Find the dimensions of a box of largest volume that can

be inscribed in a sphere of unit radius.
Solution:
Let the origin of the Cartesian coordinate system x1 , x2 , x3 be
at the center of the sphere
the sides of the box are 2x1 , 2x2 , and 2x3 .
The volume of the box is given by f (x1 , x2 , x3 ) = 8x1 x2 x3
The corners of the box lie on the surface of the sphere of unit
radius, hence x12 + x22 + x32 = 1.
The problem is therefore:

max f (x1 , x2 , x3 ) = 8x1 x2 x3 s.t. x12 + x22 + x32 = 1

6 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Solution by direct substitution..

Use the equality constraint to eliminate any one of the design

variables from the objective function.
q
If we choose to eliminate x3 , then x3 = 1 − x12 − x22
The problem becomes:
q
max f (x1 , x2 ) = 8x1 x2 1 − x12 − x22

which can be maximized as an unconstrained function in two

variables.

7 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Solution by direct substitution..

√ √
∇f = 0 ⇒ x1∗ = x2∗ = 1/ 3 and hence x3∗ = 1/ 3.
√ √
Therefore at (1/ 3, 1/ 3), f becomes

8
fmax = √
3 3
.
To find whether the solution found corresponds to a maximum
or a minimum, compute the Hessian H(x1∗ , x2∗ ) of f :
√ √ !
−32/√3 −16/√3
H=
−16/ 3 −32/ 3

The Hessian matrix of f is negative definite at (x1∗ , x2∗ ),

Hence the point corresponds to the maximum of f .J
8 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Lagrange Multipliers

Suppose we have a problem:

max f = 5 − (x1 − 2)2 − 2(x2 − 1)2 s.t. x1 + 4x2 = 3

If we ignore the constraint, we get the solution x1 = 2; x2 = 1,

which is too large for the constrain.
Let us penalize ourselves λ for making the constraint too big.
We end up with a function called the Lagrangian:

L(x1 , x2 , λ) = 5 − (x1 − 2)2 − 2(x2 − 1)2 + λ(3 − x1 − 4x2 )

9 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Lagrange Multipliers..

The main idea is to adjust λ so that we use exactly the right

amount of the resource.
λ = 0 leads to (2, 1),
λ = 1 leads to (3/2, 0) which uses too little of the resource,
λ = 2/3 gives (5/3, 1/3) and the constraint is satisfied exactly.
We now explore this idea more formally. Given:

max(min)f (x ) s.t. ci (x ) = 0, i = 1, . . . , ne

a solution can be found using the Lagrangian:

ne
X
L(x , λ) = f (x ) + λi ci (x )
i=1

Each λi gives the price associated with constraint i.

10 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Lagrange Multipliers..

Lagrange Multipliers
Theorem: Assume x ∗ = (x1∗ , x2∗ , . . . , xn∗ ) maximizes or minimizes
f (x ) subject to ci (x ) = 0, for i = 1, . . . , ne . Then there exists a
vector λ∗ = (λ∗1 , . . . , λ∗ne ) such that ∇L(x ∗ , λ∗ ) = 0. i.e.,

∂L(x ∗ , λ∗ ) ∂L(x ∗ , λ∗ )
= ··· = =0
∂x1 ∂xn
and
∂L(x ∗ , λ∗ ) ∂L(x ∗ , λ∗ )
= ··· = =0
∂λ1 ∂λne

11 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Lagrange Multipliers..
Example:
min f = 2x12 + x22 s.t. x1 + x2 = 1
Solution:

L(x1 , x2 , λ) = 2x12 + x22 + λ1 (1 − x1 − x2 )

∂L(x ∗ , λ∗ )
∇L = 0 ⇒ = 4x1∗ − λ∗1 = 0
∂x1
∂L(x ∗ , λ∗ )
= 2x2∗ − λ∗1 = 0
∂x2
∂L(x ∗ , λ∗ )
= 1 − x1∗ − x2∗ = 0
∂λ
which gives x1∗ = 1/3, x2∗ = 2/3 and λ = 4/3, with function value
2/3.
12 / 19
 Equality Constraint
Solution by Direct Substitution
Equality and Inequality Constraints
Solution by the Method of Lagrange Multipliers
Algorithms

Lagrange Multipliers..

!
4 0
Since f (x1 , x2 ) is convex (its Hessian matrix H(x1 , x2 ) = is
0 2
positive definite) and c(x1 , x2 ) = x1 + x2 is a linear function, the
above solution minimizes f (x1 , x2 ) subject to the constraint.J
Exercise: Suppose we have a refinery that must ship finished
goods to some storage tanks. Suppose further that there are
two pipelines, A and B, to do the shipping. The cost of
shipping x units on A is ax 2 ; the cost of shipping y units on B
is by 2 , where a > 0 and b > 0 are given. How can we ship Q
abQ 2
units while minimizing cost? Ans. min cost
a+b

13 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Equality and inequality constraints

Consider:

min f (x )
x ∈Rn
subject to :
ci (x ) = 0, i = 1, . . . , ne
dj (x ) ≤ 0, j = 1, . . . , ni

The inequality constraints can be transformed to equality

constraints by adding nonnegative slack variables, sj2 , as

dj (x ) + sj2 = 0, j = 1, . . . , ni

where the values of the slack variables are yet unknown.

14 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Equality and inequality constraints..

The problem now becomes:

min f (x )
x ∈Rn
subject to :
ci (x ) = 0, i = 1, . . . , ne
dj (x ) + sj2 = 0, j = 1, . . . , ni

The Lagrangian of the problem becomes:

ne
X ni
X
L(x , λ, µ, s) = f (x ) + λi ci (x ) + µj (dj (x ) + sj2 )
i=1 j=1

15 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Equality and inequality constraints..

The optimality condition is obtained for ∇L(x , λ, µ, s) = 0;

i.e.,
∇x L = 0, ∇λ L = 0, ∇µ L = 0, ∇s L = 0
or,
Krush-Kuhn-Tucker (KKT) conditions

ne
X ni
X
∇f (x ) + λi ∇ci (x ) + µj ∇dj (x ) = 0
i=1 j=1

ci (x ) = 0
dj + sj2 = 0
µj s j = 0 (switching conditions)

16 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Equality and inequality constraints..

In general, to solve these equations, we begin with the

switching conditions and note that either µj = 0 or sj = 0.
Based on the various possibilities of the switching conditions,
we come up with one or more candidate solutions.
Example:
max f = x 3 − 3x s.t. x −1≤0
x

Solution: the Lagrangian is:

L = x 3 − 3x + µ(x − 1 + s 2 )

17 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Equality and inequality constraints..

The resulting KKT conditions (∇L = 0) evaluate as:

3x 2 − 3 + µ = 0,
x − 2 + s 2 = 0,
2µs = 0

From the switching condition, µ = 0 or s = 0,

If µ = 0 then x = 1, s = 1 or x = −1, s = 3 which leads to
f (1) = −2, f (−1) = 2.
If s = 0 then x = 2 which leads to f (2) = 2. So the
maximum value is f = 2 which occurs for x = −1, 2. J

18 / 19
 Equality Constraint
Equality and Inequality Constraints
Algorithms

Constrained optimization algorithms

Linear programming (LP)

Simplex method: created by Dantzig in 1947. Birth of the
modern era in optimization
Interior-point methods
Nonlinear programming (NLP)
Penalty methods
Sequential quadratic programming methods
Interior-point methods
Almost all these methods rely strongly on line-search and
trust region methodologies for unconstrained optimization.

19 / 19