Convex set

line segment between x1 and x2: all points

x = ✓x1 + (1 ✓)x2

with 0  ✓  1

convex set: contains line segment between any two points in the set

x1, x2 2 C, 0✓1 =) ✓x1 + (1 ✓)x2 2 C

examples (one convex, two nonconvex sets)

Convex sets 2–3

 Convex combination and convex hull

convex combination of x1,. . . , xk : any point x of the form

x = ✓1 x1 + ✓2 x2 + · · · + ✓k xk

with ✓1 + · · · + ✓k = 1, ✓i 0

convex hull conv S: set of all convex combinations of points in S

Convex sets 2–4

 Convex cone

conic (nonnegative) combination of x1 and x2: any point of the form

x = ✓1 x1 + ✓2 x2

with ✓1 0, ✓2 0

x1

x2
0

convex cone: set that contains all conic combinations of points in the set

Convex sets 2–5

 Hyperplanes and halfspaces
hyperplane: set of the form {x | aT x = b} (a 6= 0)
a

x0
x
aT x = b

halfspace: set of the form {x | aT x  b} (a 6= 0)

a

x0 aT x b

aT x  b

• a is the normal vector

• hyperplanes are affine and convex; halfspaces are convex

Convex sets 2–6

 Euclidean balls and ellipsoids

(Euclidean) ball with center xc and radius r:

B(xc, r) = {x | kx xck2  r} = {xc + ru | kuk2  1}

ellipsoid: set of the form

{x | (x xc ) T P 1
(x xc)  1}

with P 2 Sn++ (i.e., P symmetric positive definite)

xc

other representation: {xc + Au | kuk2  1} with A square and nonsingular

Convex sets 2–7

 Convex Functions
Definition
f : Rn ! R is convex if dom f is a convex set and

f (✓x + (1 ✓)y)  ✓f (x) + (1 ✓)f (y)

for all x, y 2 dom f , 0  ✓  1

(y, f (y))
(x, f (x))

• f is concave if f is convex
• f is strictly convex if dom f is convex and

f (✓x + (1 ✓)y) < ✓f (x) + (1 ✓)f (y)

for x, y 2 dom f , x 6= y, 0 < ✓ < 1

Convex functions 3–2

 Examples on R

convex:
• affine: ax + b on R, for any a, b 2 R
• exponential: eax, for any a 2 R
• powers: x↵ on R++, for ↵ 1 or ↵  0
• powers of absolute value: |x|p on R, for p 1
• negative entropy: x log x on R++

concave:
• affine: ax + b on R, for any a, b 2 R
• powers: x↵ on R++, for 0  ↵  1
• logarithm: log x on R++

Convex functions 3–3

 Examples on Rn and Rm⇥n
affine functions are convex and concave; all norms are convex
examples on Rn
• affine function f (x) = aT x + b
Pn
• norms: kxkp = ( i=1 |xi|p)1/p for p 1; kxk1 = maxk |xk |

examples on Rm⇥n (m ⇥ n matrices)

• affine function
m X
X n
f (X) = tr(AT X) + b = Aij Xij + b
i=1 j=1

• spectral (maximum singular value) norm

T
f (X) = kXk2 = max (X) =( max (X X))1/2

Convex functions 3–4

 First-order condition

f is di↵erentiable if dom f is open and the gradient

✓ ◆
@f (x) @f (x) @f (x)
rf (x) = , ,...,
@x1 @x2 @xn

exists at each x 2 dom f

1st-order condition: di↵erentiable f with convex domain is convex i↵

f (y) f (x) + rf (x)T (y x) for all x, y 2 dom f

f (y)
f (x) + rf (x)T (y x)

(x, f (x))

first-order approximation of f is global underestimator

Convex functions 3–7

 Second-order conditions

f is twice di↵erentiable if dom f is open and the Hessian r2f (x) 2 Sn,

2 @ 2f (x)
r f (x)ij = , i, j = 1, . . . , n,
@xi@xj

exists at each x 2 dom f

2nd-order conditions: for twice di↵erentiable f with convex domain

• f is convex if and only if

r2f (x) ⌫ 0 for all x 2 dom f

• if r2f (x) 0 for all x 2 dom f , then f is strictly convex

Convex functions 3–8

 Examples
quadratic function: f (x) = (1/2)xT P x + q T x + r (with P 2 Sn)

rf (x) = P x + q, r2f (x) = P

convex if P ⌫ 0
least-squares objective: f (x) = kAx bk22

rf (x) = 2AT (Ax b), r2f (x) = 2AT A

convex (for any A)

quadratic-over-linear: f (x, y) = x2/y 2

f (x, y)
  T 1
2 y y
r2f (x, y) = 3 ⌫0
y x x 0
2 2
1 0
convex for y > 0 y 0 2 x

Convex functions 3–9

 Jensen’s inequality

basic inequality: if f is convex, then for 0  ✓  1,

f (✓x + (1 ✓)y)  ✓f (x) + (1 ✓)f (y)

extension: if f is convex, then

f (E z)  E f (z)

for any random variable z

basic inequality is special case with discrete distribution

prob(z = x) = ✓, prob(z = y) = 1 ✓

Convex functions 3–12

 Operations that preserve convexity

practical methods for establishing convexity of a function

1. verify definition (often simplified by restricting to a line)

2. for twice di↵erentiable functions, show r2f (x) ⌫ 0

3. show that f is obtained from simple convex functions by operations

that preserve convexity
• nonnegative weighted sum
• composition with affine function
• pointwise maximum and supremum
• composition
• minimization
• perspective

Convex functions 3–13

Positive weighted sum & composition with affine function

nonnegative multiple: ↵f is convex if f is convex, ↵ 0

sum: f1 + f2 convex if f1, f2 convex (extends to infinite sums, integrals)

composition with affine function: f (Ax + b) is convex if f is convex

examples

• log barrier for linear inequalities

m
X
f (x) = log(bi aTi x), dom f = {x | aTi x < bi, i = 1, . . . , m}
i=1

• (any) norm of affine function: f (x) = kAx + bk

Convex functions 3–14

 Pointwise maximum

if f1, . . . , fm are convex, then f (x) = max{f1(x), . . . , fm(x)} is convex

examples

• piecewise-linear function: f (x) = maxi=1,...,m(aTi x + bi) is convex

• sum of r largest components of x 2 Rn:

f (x) = x[1] + x[2] + · · · + x[r]

is convex (x[i] is ith largest component of x)

proof:
f (x) = max{xi1 + xi2 + · · · + xir | 1  i1 < i2 < · · · < ir  n}

Convex functions 3–15

 Composition with scalar functions

composition of g : Rn ! R and h : R ! R:

f (x) = h(g(x))

g convex, h convex, h̃ nondecreasing

f is convex if
g concave, h convex, h̃ nonincreasing

• proof (for n = 1, di↵erentiable g, h)

f 00(x) = h00(g(x))g 0(x)2 + h0(g(x))g 00(x)

• note: monotonicity must hold for extended-value extension h̃

examples
• exp g(x) is convex if g is convex
• 1/g(x) is convex if g is concave and positive

Convex functions 3–17