Convex Optimization

Stephen Boyd Lieven Vandenberghe

Revised slides by Stephen Boyd, Lieven Vandenberghe, and Parth Nobel

2. Convex sets
 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.1

 Affine set

line through x1 , x2 : all points of form x = 𝜃x1 + (1 − 𝜃)x2 , with 𝜃 ∈ R

𝜃 = 1.2 x1
𝜃=1
𝜃 = 0.6
x2
𝜃=0
𝜃 = −0.2

affine set: contains the line through any two distinct points in the set

example: solution set of linear equations {x | Ax = b}

(conversely, every affine set can be expressed as solution set of system of linear equations)

Convex Optimization Boyd and Vandenberghe 2.2

 Convex set

line segment between x1 and x2 : all points of form x = 𝜃x1 + (1 − 𝜃)x2 , with 0 ≤ 𝜃 ≤ 1

convex set: contains line segment between any two points in the set

x1 , x2 ∈ C, 0≤𝜃≤1 =⇒ 𝜃x1 + (1 − 𝜃)x2 ∈ C

examples (one convex, two nonconvex sets)

Convex Optimization Boyd and Vandenberghe 2.3

 Convex combination and convex hull

convex combination of x1 ,. . . , xk : any point x of the form

x = 𝜃 1 x1 + 𝜃 2 x2 + · · · + 𝜃 k xk

with 𝜃 1 + · · · + 𝜃 k = 1, 𝜃 i ≥ 0

convex hull conv S: set of all convex combinations of points in S

Convex Optimization Boyd and Vandenberghe 2.4

 Convex cone
conic (nonnegative) combination of x1 and x2 : any point of the form

x = 𝜃 1 x1 + 𝜃 2 x2

with 𝜃 1 ≥ 0, 𝜃 2 ≥ 0

x1

x2
0

convex cone: set that contains all conic combinations of points in the set

Convex Optimization Boyd and Vandenberghe 2.5

 Hyperplanes and halfspaces
a

hyperplane: set of the form {x | aT x = b}, with a ≠ 0 x0

x
aT x = b
a

x0 aT x ≥ b
halfspace: set of the form {x | aT x ≤ b}, with a ≠ 0
aT x ≤ b

▶ a is the normal vector

▶ hyperplanes are affine and convex; halfspaces are convex

Convex Optimization Boyd and Vandenberghe 2.6

 Euclidean balls and ellipsoids

(Euclidean) ball with center xc and radius r:

B(xc , r) = {x | ∥x − xc ∥ 2 ≤ r} = {xc + ru | ∥u∥ 2 ≤ 1}

ellipsoid: set of the form

{x | (x − xc ) T P−1 (x − xc ) ≤ 1}
with P ∈ Sn++ (i.e., P symmetric positive definite)

xc

another representation: {xc + Au | ∥u∥ 2 ≤ 1} with A square and nonsingular

Convex Optimization Boyd and Vandenberghe 2.7

 Norm balls and norm cones
▶ norm: a function ∥ · ∥ that satisfies
– ∥x∥ ≥ 0; ∥x∥ = 0 if and only if x = 0
– ∥tx∥ = |t| ∥x∥ for t ∈ R
– ∥x + y∥ ≤ ∥x∥ + ∥y∥
▶ notation: ∥ · ∥ is general (unspecified) norm; ∥ · ∥ symb is particular norm
▶ norm ball with center xc and radius r: {x | ∥x − xc ∥ ≤ r}
▶ norm cone: {(x, t) | ∥x∥ ≤ t}
▶ norm balls and cones are convex

Euclidean norm cone

0.5

t
n+1
{(x, t) | ∥x∥ 2 ≤ t} ⊂ R
0
1
1
is called second-order cone 0
0
x2 −1 −1
x1
Convex Optimization Boyd and Vandenberghe 2.8
 Polyhedra

▶ polyhedron is solution set of finitely many linear inequalities and equalities

{x | Ax ⪯ b, Cx = d}

(A ∈ Rm×n , C ∈ Rp×n , ⪯ is componentwise inequality)

▶ intersection of finite number of halfspaces and hyperplanes
▶ example with no equality constraints; aTi are rows of A
a1 a2

P
a5
a3

a4

Convex Optimization Boyd and Vandenberghe 2.9

 Positive semidefinite cone
notation:
▶ Sn is set of symmetric n × n matrices
▶ Sn+ = {X ∈ Sn | X ⪰ 0}: positive semidefinite (symmetric) n × n matrices

X ∈ Sn+ ⇐⇒ zT Xz ≥ 0 for all z

▶ Sn+ is a convex cone, the positive semidefinite cone

▶ Sn++ = {X ∈ Sn | X ≻ 0}: positive definite (symmetric) n × n matrices

 
x y 0.5

z
example: ∈ S2+
y z
0
1
1
0 0.5
y −1 0 x
Convex Optimization Boyd and Vandenberghe 2.10
 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.11

 Showing a set is convex

methods for establishing convexity of a set C

1. apply definition: show x1 , x2 ∈ C, 0 ≤ 𝜃 ≤ 1 =⇒ 𝜃x1 + (1 − 𝜃)x2 ∈ C

– recommended only for very simple sets

2. use convex functions (next lecture)

3. show that C is obtained from simple convex sets (hyperplanes, halfspaces, norm balls, . . . )
by operations that preserve convexity
– intersection
– affine mapping
– perspective mapping
– linear-fractional mapping

you’ll mostly use methods 2 and 3

Convex Optimization Boyd and Vandenberghe 2.12

 Intersection
▶ the intersection of (any number of) convex sets is convex
▶ example:
– S = {x ∈ Rm | |p(t)| ≤ 1 for |t| ≤ 𝜋/3}, with p(t) = x1 cos t + · · · + xm cos mt
Ñ
– write S = |t| ≤ 𝜋/3 {x | |p(t)| ≤ 1}, i.e., an intersection of (convex) slabs

▶ picture for m = 2:
2

1
1
0
p(t)

x2
0
−1

−1

−2
0 2 𝜋/3
𝜋/3
t
𝜋 −2 −1
x01 1 2

Convex Optimization Boyd and Vandenberghe 2.13

 Affine mappings

▶ suppose f : Rn → Rm is affine, i.e., f (x) = Ax + b with A ∈ Rm×n , b ∈ Rm

▶ the image of a convex set under f is convex

S ⊆ Rn convex =⇒ f (S) = {f (x) | x ∈ S} convex

▶ the inverse image f −1 (C) of a convex set under f is convex

C ⊆ Rm convex =⇒ f −1 (C) = {x ∈ Rn | f (x) ∈ C} convex

Convex Optimization Boyd and Vandenberghe 2.14

 Examples

▶ scaling, translation: aS + b = {ax + b | x ∈ S}, a, b ∈ R

▶ projection onto some coordinates: {x | (x, y) ∈ S}
▶ if S ⊆ Rn is convex and c ∈ Rn , cT S = {cT x | x ∈ S} is an interval
▶ solution set of linear matrix inequality {x | x1 A1 + · · · + xm Am ⪯ B} with Ai , B ∈ Sp
▶ hyperbolic cone {x | xT Px ≤ (cT x) 2 , cT x ≥ 0} with P ∈ Sn+

Convex Optimization Boyd and Vandenberghe 2.15

 Perspective and linear-fractional function

▶ perspective function P : Rn+1 → Rn :

P(x, t) = x/t, dom P = {(x, t) | t > 0}

▶ images and inverse images of convex sets under perspective are convex

▶ linear-fractional function f : Rn → Rm :

Ax + b
f (x) = , dom f = {x | cT x + d > 0}
cT x + d
▶ images and inverse images of convex sets under linear-fractional functions are convex

Convex Optimization Boyd and Vandenberghe 2.16

 Linear-fractional function example

1
f (x) = x
x1 + x2 + 1

1 1
x2

x2
0 C 0 f (C)

−1 −1
−1 0 1 −1 0 1
x1 x1

Convex Optimization Boyd and Vandenberghe 2.17

 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.18

 Proper cones

a convex cone K ⊆ Rn is a proper cone if

▶ K is closed (contains its boundary)
▶ K is solid (has nonempty interior)
▶ K is pointed (contains no line)

examples
▶ nonnegative orthant K = Rn+ = {x ∈ Rn | xi ≥ 0, i = 1, . . . , n}
▶ positive semidefinite cone K = Sn+
▶ nonnegative polynomials on [0, 1] :

K = {x ∈ Rn | x1 + x2 t + x3 t2 + · · · + xn tn−1 ≥ 0 for t ∈ [0, 1]}

Convex Optimization Boyd and Vandenberghe 2.19

 Generalized inequality

▶ (nonstrict and strict) generalized inequality defined by a proper cone K :

x ⪯K y ⇐⇒ y − x ∈ K, x ≺K y ⇐⇒ y − x ∈ int K
▶ examples
– componentwise inequality (K = Rn+ ): x ⪯Rn y ⇐⇒ xi ≤ yi , i = 1, . . . , n
+
– matrix inequality (K = Sn+ ): X ⪯Sn Y ⇐⇒ Y − X positive semidefinite
+
these two types are so common that we drop the subscript in ⪯K

▶ many properties of ⪯K are similar to ≤ on R, e.g.,

x ⪯K y, u ⪯K v =⇒ x + u ⪯K y + v

Convex Optimization Boyd and Vandenberghe 2.20

 Outline

Some standard convex sets

Operations that preserve convexity

Generalized inequalities

Separating and supporting hyperplanes

Convex Optimization Boyd and Vandenberghe 2.21

 Separating hyperplane theorem

▶ if C and D are nonempty disjoint (i.e., C ∩ D = ∅) convex sets, there exist a ≠ 0, b s.t.

aT x ≤ b for x ∈ C, aT x ≥ b for x ∈ D

aT x ≥ b aT x ≤ b

D
C

▶ the hyperplane {x | aT x = b} separates C and D

▶ strict separation requires additional assumptions (e.g., C is closed, D is a singleton)

Convex Optimization Boyd and Vandenberghe 2.22

 Supporting hyperplane theorem

▶ suppose x0 is a boundary point of set C ⊂ Rn

▶ supporting hyperplane to C at x0 has form {x | aT x = aT x0 }, where a ≠ 0 and aT x ≤ aT x0
for all x ∈ C

a
x
C 0

▶ supporting hyperplane theorem: if C is convex, then there exists a supporting

hyperplane at every boundary point of C

Convex Optimization Boyd and Vandenberghe 2.23