Prerequisites for Convex Optimization

Maxima and Minima

Prerequisites for Convex Optimization 1 / 42

Maxima and Minima

Definitions
A local maximum of a function f is a point x = c where f (c) is
greater than all nearby values:

f (c) ≥ f (x) for all x near c

A local minimum of f is a point x = d where f (d) is less than all
nearby values:
f (d) ≤ f (x) for all x near d
A global (absolute) maximum is the highest value on the entire
domain.
A global (absolute) minimum is the lowest value on the entire
domain.

Prerequisites for Convex Optimization 2 / 42

Local Maxima and Minima

f (x)

Local Max

x
c d

Local Min

Local maxima are points where f (x) changes from increasing to

decreasing.
Local minima are points where f (x) changes from decreasing to
increasing.

Prerequisites for Convex Optimization 3 / 42

Global Maxima and Minima

f (x)
Global Max

Global Min
x
a c

Global extrema are the absolute highest or lowest values on the entire
domain.
They may occur at local extrema or at domain boundaries.

Prerequisites for Convex Optimization 4 / 42

Methods to Compute Maxima and Minima

Critical Points
Maxima and minima occur where the first derivative is zero or
undefined:
f ′ (x) = 0 or f ′ (x) does not exist
Points satisfying this are called critical points.

Prerequisites for Convex Optimization 5 / 42

First Derivative Test

Procedure
Find critical points by solving f ′ (x) = 0.
Analyze the sign of f ′ (x) around the critical points:
If f ′ (x) changes from positive to negative, f has a local maximum.
If f ′ (x) changes from negative to positive, f has a local minimum.
If no sign change, then no local extremum at that point.

Saddle point
A saddle point is a point on the surface of a function where the slope (all
derivatives in orthogonal directions) is zero, making it a critical point, but
the point is not a local maximum or a local minimum.

Prerequisites for Convex Optimization 6 / 42

Second Derivative Test

Procedure
Find critical points by solving f ′ (x) = 0.
Evaluate the second derivative f ′′ (x) at each critical point:
If f ′′ (x) > 0, the function has a local minimum.
If f ′′ (x) < 0, the function has a local maximum.
If f ′′ (x) = 0, the test is inconclusive.

Prerequisites for Convex Optimization 7 / 42

Example: Find Extrema of f (x) = x 3 − 3x 2 + 4

Step 1: First derivative

f ′ (x) = 3x 2 − 6x

Step 2: Critical points

Solve f ′ (x) = 0:

3x 2 − 6x = 0 =⇒ 3x(x − 2) = 0 =⇒ x = 0, x = 2

Step 3: Second derivative

f ′′ (x) = 6x − 6
f ′′ (0) = −6 < 0 =⇒ local maximum at x = 0
f ′′ (2) = 6 > 0 =⇒ local minimum at x = 2

Prerequisites for Convex Optimization 8 / 42

Graphical Representation of Extrema

f (x)
Max (0,4)

x
Min0(2,0) 2

The red dot marks the local maximum at x = 0.

The blue dot marks the local minimum at x = 2.
Prerequisites for Convex Optimization 9 / 42
Questions: Critical Points, Maxima and Minima

1 Find the critical points of the function f (x) = x 3 − 3x 2 + 4.

2 Determine the nature (local max, local min, or saddle point) of the
critical points of
f (x) = x 4 − 4x 3 + 6x 2 .
3 For the function f (x, y ) = x 2 + y 2 − 4x + 6y , find the critical
point(s) and classify them.
4 Find the critical points of
2
f (x) = e −x

and determine whether they are maxima or minima.

5 Given f (x) = x 3 − 6x 2 + 9x + 2, find the critical points and
determine the intervals where f is increasing and decreasing.

Prerequisites for Convex Optimization 10 / 42

Critical Points and Classification

Example 1: Find critical points of f (x) = x 3 − 3x 2 + 4.

f ′ (x) = 3x 2 − 6x = 3x(x − 2) =⇒ x = 0, 2

Example 2: For f (x) = x 4 − 4x 3 + 6x 2 ,

f ′ (x) = 4x 3 −12x 2 +12x = 4x(x 2 −3x +3), only real critical point x = 0,

f ′′ (0) = 12 > 0 =⇒ x = 0 is local minimum.

Prerequisites for Convex Optimization 11 / 42

Critical Points and Classification

Example 3: f (x, y ) = x 2 + y 2 − 4x + 6y Critical point at (2, −3) since

fx = 2x − 4 = 0, fy = 2y + 6 = 0.
2
Example 4: For f (x) = e −x , critical point at x = 0 since
2
f ′ (x) = −2xe −x = 0 ⇒ x = 0.

f ′′ (0) = −2 < 0 =⇒ x = 0 is local maximum.

Example 5: f (x) = x 3 − 6x 2 + 9x + 2

f ′ (x) = 3(x − 1)(x − 3), x = 1, 3 critical points.

Increasing on (−∞, 1), decreasing on (1, 3), increasing on (3, ∞). Local
max at x = 1, local min at x = 3.

Prerequisites for Convex Optimization 12 / 42

How to Find Global Maximum and Minimum of a Function
Step 1: Identify the domain
Step 2: Find critical points Compute f ′ (x) and solve f ′ (x) = 0 to find
candidates.
Step 3: Evaluate endpoints If domain is closed or bounded, evaluate f
at endpoints.
Step 4: Determine global maxima/minima Compare function values at
critical points and endpoints;
max f (x) and min f (x)
occur at these points.

Additional Notes: - Use second derivative test or first derivative test to

classify critical points as local max/min. - For functions not bounded,
global max or min may not exist.

Example: To find global max/min of f (x) = x 3 − 3x 2 + 4 on [−1, 3], find

critical points, evaluate f at critical points and at −1, 3, then compare
values. Prerequisites for Convex Optimization 13 / 42
Problem Statement

Find the global maximum and global minimum of the function

f (x) = x 3 − 3x 2 + 4

on the closed interval [−1, 3].

Approach:
1 Find critical points by setting f ′ (x) = 0.
2 Evaluate f at critical points within the interval.
3 Evaluate f at the endpoints −1 and 3.
4 Compare all values to determine global max and min.

Prerequisites for Convex Optimization 14 / 42

Finding Critical Points

Compute the first derivative:

f ′ (x) = 3x 2 − 6x = 3x(x − 2).

Set derivative equal to zero to find critical points:

3x(x − 2) = 0 =⇒ x = 0 or x = 2.

Both points x = 0 and x = 2 lie inside the interval [−1, 3], so consider
both for evaluation.

Prerequisites for Convex Optimization 15 / 42

Evaluating f (x) at Critical Points and Endpoints

Compute values:

f (−1) = (−1)3 − 3(−1)2 + 4 = −1 − 3 + 4 = 0,

f (0) = 03 − 3(0)2 + 4 = 4,
f (2) = 23 − 3(2)2 + 4 = 8 − 12 + 4 = 0,
f (3) = 33 − 3(3)2 + 4 = 27 − 27 + 4 = 4.
Values at points:

f (−1) = 0, f (0) = 4, f (2) = 0, f (3) = 4.

Prerequisites for Convex Optimization 16 / 42

Conclusion: Global Maximum and Minimum

From the evaluated values:

The global maximum is 4 at x = 0 and x = 3.
The global minimum is 0 at x = −1 and x = 2.

Summary: To find global extrema on a closed interval, evaluate the

function at critical points within the interval and at the endpoints, then
pick the highest and lowest values.

Prerequisites for Convex Optimization 17 / 42

Find Global Max/Min of f (x) = x 3 − 3x 2 + 4

On the interval [−1.5, 3.5], find the global maximum and minimum of

f (x) = x 3 − 3x 2 + 4.

Compute derivative:

f ′ (x) = 3x 2 − 6x = 3x(x − 2).

Set derivative to zero to find critical points:

x = 0, x = 2.

Prerequisites for Convex Optimization 18 / 42

Evaluate f (x) at Critical Points and Endpoints

Calculate values:

f (−1.5) = −6.125, f (0) = 4, f (2) = 0, f (3.5) = 10.125.

These correspond to function values at the endpoints and critical points

within the interval.

Prerequisites for Convex Optimization 19 / 42

Conclusion: Extrema of f (x) = x 3 − 3x 2 + 4 on [−1.5, 3.5]

Local maxima: f (0) = 4 and f (3.5) = 10.125 are points where the
function reaches local highs.
Local minimum: f (2) = 0 is a local low point.
Global maximum: The highest value on the interval is 10.125 at
x = 3.5.
Global minimum: The lowest value on the interval is −6.125 at
x = −1.5.

Summary: To find global extrema, evaluate function values at local

extrema and endpoints, then determine the absolute highest and lowest.

Prerequisites for Convex Optimization 20 / 42

Definition of Affine Set

A set C ⊆ Rn is affine if for any two points x1 , x2 ∈ C and any scalar

θ ∈ R, the point
θx1 + (1 − θ)x2 ∈ C
That is, C contains the entire line passing through any two points in it.

More generally, for points x1 , . . . , xk ∈ C , any affine combination of the

form
Xk
θ1 x1 + · · · + θk xk , with θi = 1
i=1

also lies in C .

Prerequisites for Convex Optimization 21 / 42

Properties and Examples

An affine set contains every line through any pair of its points.
Every affine set is a translation of a linear subspace.
Examples include:
Lines and planes in Rn
Hyperplanes (like a flat surface in 3D space)
Solutions sets of linear equations

Prerequisites for Convex Optimization 22 / 42

Convex Set

Definition
A set C in a vector space is convex if for any two points x, y ∈ C , the line
segment connecting them lies entirely within C . Formally,

If x, y ∈ C , then λx + (1 − λ)y ∈ C ∀λ ∈ [0, 1].

x
Convex Set

Affine sets differ from convex sets in that they contain entire lines,
not just line segments.
Prerequisites for Convex Optimization 23 / 42
Convex Hull

The convex hull of a set of points S ⊆ Rn is the smallest convex

set containing S.
Intuitively, imagine stretching a rubber band around the outermost
points in S; when released, it snaps tightly around the points forming
the convex hull.
Formally, the convex hull is the set of all convex combinations of
points in S:
( k k
)
X X
θi xi | xi ∈ S, θi ≥ 0, θi = 1, k ∈ N
i=1 i=1

Prerequisites for Convex Optimization 24 / 42

Convex Hull

Prerequisites for Convex Optimization 25 / 42

Applications of Convex Hull

Used in computational geometry algorithms

Image processing: shape analysis and object recognition
Robotics: collision detection and path planning
Geographic information systems (GIS) to define bounding regions

Prerequisites for Convex Optimization 26 / 42

Non-Convex Set

Definition
A set S is non-convex if there exist points x, y ∈ S such that the line
segment connecting them is not entirely contained in S. Formally,

∃x, y ∈ S, ∃λ ∈ (0, 1) such that λx + (1 − λ)y ∈

/ S.

Outside set

Non-Convex Set
y

The blue line segment crosses outside the shaded red region.
This demonstrates the set is non-convex.

Prerequisites for Convex Optimization 27 / 42

Example: Convex Set

Convex Set (Disk in R2 )

The set C = {(x, y ) ∈ R2 | x 2 + y 2 ≤ 1} is convex.

The line segment between any two points within the disk stays
entirely inside the blue region.

Prerequisites for Convex Optimization 28 / 42

Example: Non-Convex Set
Non-Convex Set (Crescent Shape)
Let S be a ”crescent” shape formed by subtracting a smaller disk from a
bigger one:

S = {(x, y ) | x 2 + y 2 ≤ 22 } \ {(x, y ) | (x − 1)2 + y 2 < 12 }

The line segment from P to Q exits the region, illustrating

non-convexity.
Prerequisites for Convex Optimization 29 / 42
Hyperplanes in Rn

A hyperplane is a flat affine subspace of dimension n − 1 in Rn .

It can be defined as the set of points x ∈ Rn satisfying a linear
equation:
{x ∈ Rn | a⊤ x = b}
where a ∈ Rn is a non-zero vector (normal vector), and b ∈ R is a
scalar.
Geometrically, a hyperplane divides the space into two halves.

Prerequisites for Convex Optimization 30 / 42

Halfspaces

A halfspace is one of the two parts into which a hyperplane divides

Rn .
The two halfspaces defined by a hyperplane a⊤ x = b are:

H+ = {x ∈ Rn | a⊤ x ≤ b}, H− = {x ∈ Rn | a⊤ x ≥ b}

Halfspaces are convex sets.

In optimization, halfspaces represent linear inequality constraints.

Prerequisites for Convex Optimization 31 / 42

Convex Function
Definition
A function f : R → R is convex if for all x, y in its domain and for any
λ ∈ [0, 1]:
f (λx + (1 − λ)y ) ≤ λf (x) + (1 − λ)f (y ).
Intuitively, the graph of f lies below the straight line connecting any two
points on it.

(y , f (y ))
f (x)

f ( x+y
2 )
(x, f (x)) x

The black point on the curve lies below the green point on the chord,
illustrating convexity.
Prerequisites for Convex Optimization 32 / 42
Non-Convex Function
Definition
A function f is non-convex if it violates the convexity condition for some
x, y and λ ∈ [0, 1], i.e.,

f (λx + (1 − λ)y ) > λf (x) + (1 − λ)f (y ).

Intuitively, the graph of f lies above the line segment joining some two
points on it.

f (x)

(x, f (x)) Chord midpoint

x
(y , f (y ))

The black point on the curve lies above the green point on the chord,
illustrating non-convexity.
Prerequisites for Convex Optimization 33 / 42
Methods to Check Convexity of a Function

Definition Method: Check if for all x, y and λ ∈ [0, 1]:

f (λx + (1 − λ)y ) ≤ λf (x) + (1 − λ)f (y ).

First Derivative Test (for differentiable functions): The function

f is convex if and only if its domain is convex and

f (y ) ≥ f (x) + ∇f (x)T (y − x), ∀x, y .

Second Derivative Test (for twice differentiable functions): The

function f is convex if and only if its Hessian matrix satisfies

∇2 f (x) ⪰ 0, (positive semidefinite) ∀x.

Graphical Check: The graph of a convex function lies below the

chord connecting any two points.

Prerequisites for Convex Optimization 34 / 42

First Derivative Test for Convexity

Definition: A differentiable function f on interval I is convex if for all

x, y ∈ I ,
f (y ) ≥ f (x) + f ′ (x)(y − x).
This inequality means the function always lies above its tangent lines.

Geometric Interpretation:
For any point x, the tangent line at x is a supporting line that
touches but never dips below the curve.
The slope f ′ (x) is monotonically non-decreasing for convex functions.

Prerequisites for Convex Optimization 35 / 42

Example: Applying the First Derivative Test

Check convexity of f (x) = e x :

Derivative:
f ′ (x) = e x > 0 ∀x ∈ R.
Tangent line at x:

L(y ) = f (x) + f ′ (x)(y − x) = e x + e x (y − x) = e x (1 + y − x).

Check inequality:
?
e y ≥ e x (1 + y − x).
Conclusion: f (x) = e x is convex on R.

Prerequisites for Convex Optimization 36 / 42

Second Derivative Test for Convexity

For twice differentiable functions, the second derivative test states:

f is convex ⇐⇒ ∇2 f (x) ≥ 0 ∀x
For functions of one variable:

f is convex ⇐⇒ f ′′ (x) ≥ 0 ∀x
This means the function’s curvature is always upward or flat.

Prerequisites for Convex Optimization 37 / 42

Example: Check f (x) = e x Using Second Derivative

Step 1: Compute first derivative:

f ′ (x) = e x

Step 2: Compute second derivative:

f ′′ (x) = e x

Step 3: Since
f ′′ (x) = e x > 0 ∀x ∈ R,
the function is convex.

Conclusion: The exponential function f (x) = e x is convex on R as its

second derivative is always positive.

Prerequisites for Convex Optimization 38 / 42

Examples of Convex Functions
f (x) = x 2 (Upward-opening parabola)
f (x) = e x (Exponential function)
f (x) = 2x + 1 (Straight line)
f (x)

ex

x2

2x + 1

Prerequisites for Convex Optimization 39 / 42

Examples of Non-Convex Functions
f (x) = −x 2 (Downward parabola)
f (x) = sin(x) (Sine wave)
f (x) = x 3 − 3x (Cubic function)
f (x)

sin(x)

x
−x 2

x 3 − 3x

Prerequisites for Convex Optimization 40 / 42

Importance of Convex Functions in Optimization

Convex functions guarantee global optimality because any local

minimum is also a global minimum.
This property eliminates local traps, making optimization problems
easier to solve.
The well-defined curvature allows efficient algorithms such as
gradient descent and Newton’s method to converge quickly.
Convex optimization problems are mathematically tractable and
widely used in machine learning, economics, and engineering.

Prerequisites for Convex Optimization 41 / 42

Advantages of Convexity in Optimization Algorithms

Strong duality holds for convex problems, enabling powerful dual

optimization techniques.
Smoothness and Lipschitz continuity of gradients often hold,
improving convergence rates.
Convex problems admit polynomial-time algorithms, ensuring
scalability.
Robustness to noise and approximate computations due to convex
structure.

Prerequisites for Convex Optimization 42 / 42