Fundamentals of Optimization
Lecture 2
Xavier Cabezas
FCNM-ESPOL
December 21, 2020
Outline
Convex sets
Topological properties of convex sets
Separation and support of convex sets
Convex sets 2
Definition of convex set
Let C be a subset of Rn . C is said a convex set if
λx1 + (1 − λ)x2 ∈ C, ∀x1 , x2 ∈ C, ∀λ ∈ [0, 1]
.
x2
x2
x1
x1
(a) A convex set. (b) A nonconvex set.
Convex sets 3
Convex combination
A convex combination is any point x ∈ Rn of the form
x = λ 1 x1 + λ 2 x2 + . . . + λ k xk ,
with λ1 + λ2 + . . . + λk = 1 and λi ≥ 0, ∀i = 1, 2, . . . , k,
x1 , x2 , . . . , xk ∈ Rn and λ1 , λ2 , . . . , λk ∈ R.
The set of all convex combinations of points of a subset S of Rn is called
convex hull and is denoted by conv S.
x4 x3
x5
x1 x2
(a) S = { x1 , x2 , x3 , x4 , x5 } (b) S is uncountable.
Convex sets 4
Examples of convex sets
An hyperplane H is a set of the form { x | a0 x = b }, where a ∈ Rn ,
a 6= 0 and b ∈ R.
• If x ∈ H, then a0 x = b. Hence a0 x − a0 x = b − b for all point in
x ∈ H. That is, H can be written in reference to a fixed point,
a0 (x − x) = 0.
a0 x = b
Figure: A hyperplane H.
Convex sets 5
Examples of convex sets
An halfspace (H − , H + ) associated with a hyperplane, is a set of the
form { x | a0 x { ≤, ≥ } b }, where a ∈ Rn , a 6= 0 and b ∈ R.
• Similar to H, H − (H + ) can be written in reference to a fixed point,
a0 (x − x) ≤ 0 (a0 (x − x) ≥ 0).
a0 x ≤ b
Figure: A negative halfspace H − .
Convex sets 6
Examples of convex sets
A polyhedral set is a nonempty subset P of Rn of the form
P = { x | a0i x ≤ bi , i = 1, . . . , r } ,
where ai are vectors and bi are scalars.
• A polyhedral set is also called polyhedron.
• The feasible region of a linear programming problem is a polyhedron.
Convex sets 7
Examples of convex sets
A cone is a subset C of Rn such that ax ∈ C, for all a ∈ R and x ∈ C.
• A cone (nonnegative) combination is any point x ∈ Rn of the
form
x = λ1 x1 + λ2 x2 + . . . + λk xk ,
with λi ≥ 0, ∀i = 1, 2, . . . , k,
x1 , x2 , . . . , xk ∈ Rn and λ1 , λ2 , . . . , λk ∈ R.
• A convex cone (finitely generated), is a set of the form
k
X
C = {x | x = λi xi , λi ≥ 0, i = 1, . . . , k }
i=1
.
Convex sets 8
Intersection and sum of vectors of a convex set
Theorem
Let C1 and C2 be convex sets in Rn . Then
1. C1 ∩ C2 is convex.
2. C1 + C2 = { x1 + x2 | x1 ∈ C1 , x2 ∈ C2 } is convex.
3. C1 − C2 = { x1 − x2 | x1 ∈ C1 , x2 ∈ C2 } is convex.
Proof.
1. Taking x1 and x2 of C1 ∩ C2 , it is easy to note that
λx1 + (1 − λ)x2 is also in .C1 ∩ C2 , since that x1 , x2 ∈ C1 and
x1 , x2 ∈ C2 , and both sets are convex.
2. Let x11 + x12 and x21 + x22 be elements of C1 + C2 . The
expression λ(x11 + x12 ) + (1 − λ)(x21 + x22 ) can be written as
(λx11 + (1 − λ)x21 ) + (λx12 + (1 − λ)x22 ).
3. Similar to 2.
Convex sets 9
Caratheodory theorem
Theorem
Let S be a subset of Rn . Every element of conv S can be represented as
a convex combination of no more than n + 1 elements of S.
• The theorem is equivalent to:
(n+1 n+1
)
X X
conv S = λ i xi | λi = 1, λi ≥ 0, xi ∈ S, i = 1, . . . , n + 1
i=1 i=1
Proof.
In the lecture.
Convex sets 10
Outline
Convex sets
Topological properties of convex sets
Separation and support of convex sets
Topological properties of convex sets 11
Closure and interior of a convex sets
Theorem
Let C be a convex set in Rn with a nonempty interior. Let x1 ∈ cl C and
x2 ∈ int C. Then λx1 + (1 − λ)x2 ∈ int C for each λ ∈ (0, 1).
Proof.
In the lecture, but we need the following figure:
C x2
y z2
x1 z
z1
Topological properties of convex sets 12
Closure and interior of a convex sets
Corollary
If C is a convex set, then int C is convex.
Corollary
If C is a convex set with int C 6= ∅, then cl C is convex.
Corollary
If C is a convex set with int C 6= ∅, then cl int C = cl C.
Corollary
If C is a convex set with int C 6= ∅, then int cl C = int C.
Topological properties of convex sets 13
Outline
Convex sets
Topological properties of convex sets
Separation and support of convex sets
Separation and support of convex sets 14
Projection theorem
Theorem
Let C be a closed convex in Rn and y ∈ / C. Then there exist a unique
point x̄ ∈ S with minimum distance from y. Furthermore, x̄ is such a
point if and only if (y − x̄)0 (x − x̄) ≤ 0 for all x ∈ C.
The vector x̄ is called the projection of y on C and is sometimes denoted
by [x]+ , i.e.,
x̄ = [x]+ = argmin ky − xk
x∈C
Proof.
In the lecture
Separation and support of convex sets 15
Projection theorem
C
x̄ x
(y − x̄)0 (x − x̄) ≤ 0
Separation and support of convex sets 16
Hyperplanes of separation
Let C1 and C2 be nonempty sets in Rn and Let H = { x | a0 x = b }, then
H is said to separate C1 and C2 if a0 x ≥ b for each x ∈ C1 and a0 x ≤ b
for each x ∈ C2 .
• If in addition C1 ∪ C2 6⊂ H, then H properly separates C1 and C2 .
• If a0 x > b for each x ∈ C1 and a0 x < b for each x ∈ C2 , then H
strictly separates C1 and C2
• If a0 x ≥ b + ε for each x ∈ C1 and a0 x ≤ b for each x ∈ C2 , where
ε > 0, then H strongly separates C1 and C2 .
Separation and support of convex sets 17
Hyperplanes of separation
C1
C2
H
C1
C2
H
(a) Improper separation. (b) Proper separation.
C1 C1
H H
C2 C2
(c) Strict separation. (d) Strong separation.
Separation and support of convex sets 18
Fundamental separation theorem
Theorem
Let C be a nonempty closed convex set in Rn and y ∈ / C. Then there
exists a nonzero vector a and a scalar b such that a0 y > b and a0 x ≤ b for
each x ∈ C.
Proof.
• By the projection theorem there exists an unique minimizing point
x̄ ∈ C such that (y − x̄)0 (x − x̄) ≤ 0 for each x ∈ C. Let
a = y − x̄ 6= 0 and b = a0 x̄, we obtain a0 x ≤ b.
• ky − x̄k2 = (y − x̄)0 (y − x̄) = a0 (y − x̄) > 0, then a0 y > b.
Corollary
Let C be a nonempty closed convex set in Rn . Then C is the intersection
of all halfspaces containing C.
Separation and support of convex sets 19
Theorem of Farkas
Theorem
Let A ∈ Rm×n and c ∈ Rn . Then exactly one of the following systems
has a solution:
1. Ax ≤ 0 and c0 x > 0.
2. A0 y = c and y ≥ 0.
Proof.
If system 2 has a solution, A0 y = c for some y > 0. If there is an x such
that Ax ≤ 0, then c0 x = (A0 y)0 x = y 0 Ax ≤ 0, i.e., the system 1 has no
solution.
The second part uses the fundamental separation theorem. That will be a
homework (easy to follow from the literature) as it is relevant particularly
in linear programming.
Separation and support of convex sets 20
Supporting hyperplane theorem
Let S be a nonempty subset of Rn and x̄ ∈ ∂S.
H = { x | a0 (x − x̄) = 0 } is called a supporting hyperplane of S at x̄ if
either S ⊂ H + or S ⊂ H − .
Theorem
If C ∈ Rn is a nonempty convex set and x̄ is a point that does not
belong to int C, there exists a nonzero vector a, such that a0 (x − x̄) ≤ 0
for all x ∈ C.
a H
C
x̄ x
Separation and support of convex sets 21
Proving the supporting hyperplane theorem
Proof.
If x̄ ∈
/ cl C, then the fundamental separation theorem can be applied.
Therefore, we assume that x̄ ∈ ∂C. Then there is a sequence
{ yk } ∈
/ cl C such that yk → x̄.
Corresponding to each yk there is a ak = pk /kpk k such that a0k yk > a0k x
for each x ∈ cl C (fundamental separation theorem).
Since kak k = 1, The sequence { ak } is bounded and, as we already
know, it has a convergent subsequence { ak }K with limit a.
Consider a0k yk > a0k x for all ak belonging to this subsequence and take
the limit as k → ∞, we obtain a0 x ≤ a0 x̄, i.e., a0 (x − x̄) ≤ 0.
Separation and support of convex sets 22
Separating hyperplane theorem
Theorem
If C1 ∈ Rn and C2 ∈ Rn are two nonempty and disjoint convex sets,
there exists a hyperplane that separates them, i.e., a nonzero vector a
such that a0 x1 ≤ a0 x2 for all x1 ∈ C1 and x2 ∈ C2 .
C1
H
C2
Proof.
Let C = { x | x = x1 − x2 , x1 ∈ C1 , x2 ∈ C2 }. Note that 0 ∈/ C, since
C1 ∩ C2 = ∅. Since the supporting hyperplane theorem there is a vector
a 6= 0 such that a0 (x − 0) ≤ 0 for all x ∈ C. Then a0 x1 ≤ a0 x2 .
Separation and support of convex sets 23
