Chapter 4: Optimization for Data Science

Duality

TANN Chantara

Department of Applied Mathematics and Statistics

Institute of Technology of Cambodia

October 8, 2022

TANN Chantara (ITC) Convex Functions October 8, 2022 1 / 18

Table of Contents

1 The Lagrange Duel Function

2 The Dual Problem

3 Dual Cones

4 Generalized Inequalities

5 Summary

TANN Chantara (ITC) Convex Functions October 8, 2022 1 / 18

 The Lagrange Duel Function

Lagrangian Duality


minx∈Rn

 f0 (x)
P: s.t. fi (x) ≤ 0, ∀i = 1, ..., m

hi (x) = 0, ∀i = 1, ..., p



x∈R decision variable

f0 : Rn →R objective function
fi : Rn →R inequality constraint functions
hi : Rn →R equality constraint functions
Assume for simplicity that function values cannot be ∞

TANN Chantara (ITC) Convex Functions October 8, 2022 2 / 18

 The Lagrange Duel Function

Lagrangian

Definition: The Lagrangian L of the problem P is defined as the

function L : Rn × Rn+ × Rp → R
m
X p
X
L(x, λ, y) = f0 (x) + λi fi (x) + µi hi (x)
i=1 i=1

Weighted sum of objective and constraint functions.

λi is the Lagrange multiplier corresponding to fi (x) ≤ 0.
µi is the Lagrange multiplier corresponding to hi (x) = 0.
The Lagrangian is concave (affine) in (λ, µ) for any fixed x.
If P is convex optimization problem, the Lagrangian is convex in x for any
fixed (λ, µ), i.e., L is saddle function.

TANN Chantara (ITC) Convex Functions October 8, 2022 3 / 18

 The Lagrange Duel Function

Lagrangian Cont’d
The Lagrangian allows us to reexpress the optimization problem P as a
min-max problem. Indeed, define
f(x) = sup
λ∈Rm
+ ,µ∈R
p

and note that

(
f0 (x), if fi (x) ≤ 0∀i and hi (x) = 0∀i
f(x) =
∞ otherwise.
Thus, we obtain

Lagrangian Reformulation:

inf P = infn f(x) = infn sup L(x, λ, µ)

x∈R x∈R λ∈Rm ,µ∈Rp
+

TANN Chantara (ITC) Convex Functions October 8, 2022 4 / 18

 The Dual Problem

The Dual Problem

Below we refer to P as the primal problem. Using the Lagrangian we can

introduced a dual problem with objective function.

g(λ, µ) = ∞x∈Rn L(x, λ, µ)

Dual Problem:
D: sup g(λ, µ)
λ∈Rm
+ ,µ∈R
p

By construction, D is equivalent to a min-max problem.

sup D = sup g(λ, µ) = sup infn L(x, λ, µ)

λ∈Rm p λ∈Rm p x∈R
+ ,µ∈R + ,µ∈R

TANN Chantara (ITC) Convex Functions October 8, 2022 5 / 18

 The Dual Problem

Weak Duality
Proposition: g(λ, µ) ≥ f(x), ∀x ∈ Rn , λ ∈ Rm
+, µ ∈ R
p

Proof: By definition we have

g(λ, µ) = infn L(x̄, λ, µ) ≤ L(x, λ, µ) ≤ sup L(x, λ̄, µ̄) = f(x)
x̄∈R λ̄,µ̄

Corollary (Weak Duality):

sup D ≤ inf P

Thus if D is unbounded (sup D = ∞), then P must be infeasible. If P is

unbounded (inf P = −∞), then D must be infeasible.
TANN Chantara (ITC) Convex Functions October 8, 2022 6 / 18
 The Dual Problem

Significance of Dual Solutions

Note: Every feasible solution of P (D) provides an upper (lower) bound

on both inf P and sup D.

Assume x̂ is a (feasible) candidate solution for P. Its quality is quantified

by f0 (x̂) − inf P. However, inf P is unknown!

A dual feasible solution (λ, µ) provides a proof or certificate that

inf P ≥ g(λ, µ) ⇒ f0 (x̂) − inf P ≤ f0 (x̂) − g(λ, µ)

Thus, if f0 (x̂) − g(λ, µ) ≤ ε, then x̂ is an ε-optimal solution.

TANN Chantara (ITC) Convex Functions October 8, 2022 7 / 18

 The Dual Problem

Strong Duality
Definition: ∆ = inf P − sup P is called the duality gap. By weak duality,
we have that ∆ ≥ 0. If ∆ = 0, we say that strong duality holds.


minx∈Rn

 f0 (x)
P: s.t. fi (x) ≤ 0, ∀i = 1, ..., m

hi (x) = 0, ∀i = 1, ..., p



Strong Duality
does not hold in general.
always holds if P is a convex problem satisfying a constraint
qualification.
Slater’s constraint qualification holds if there exist xS with
fi (xS ) < 0, ∀i = 1, ..., m
hi (xS ) = 0, ∀i = 1, ..., p
TANN Chantara (ITC) Convex Functions October 8, 2022 8 / 18
 The Dual Problem

Least squares problem

(
minx∈Rn xT x
Primal Problem
s.t. Ax = b
Lagrangian L(x, µ) = xT x + µT (Ax − b)

Dual Objective (µ) = minx∈Rn L(x, µ)

⇒ ∇x L(x, µ) = 2x + AT µ = 0
1
⇒ x = − AT µ
2
1
⇒ g(µ) = − µT AAT µ − bT µ
4
1
Dual Problem maxµ∈Rm − µT AAT µ − bT µ
4

TANN Chantara (ITC) Convex Functions October 8, 2022 9 / 18

 The Dual Problem

Standard form Linear Program

(
minx≥0 cT x
Primal problem
s.t. Ax = b
Lagrangian L(x, λ, µ) = cT x − λT x + µT (Ax − b)

Dual objective g(λ, µ) = minx L(x, λ, µ)

(
−bT µ, c − λ + AT µ = 0
=
−∞ else
(
Dual Problem maxµ,λ≥0 −bT µ
s.t. c − λ + AT µ = 0
(
maxµ bT µ
⇐⇒
s.t. AT µ ≤ c

TANN Chantara (ITC) Convex Functions October 8, 2022 10 / 18

 The Dual Problem

Quadratic Program

(
minx xT Px
Primal Problem (P ≻ 0)
s.t. Ax ≤ b
Lagrangian T T
L(x, λ) = x Px + λ (Ax − b)

Dual Objective g(λ, µ) = minx L(x, λ, µ)

1
= − λT AP−1 AT λ − bT λ
4
1
Dual Problem maxλ≥0 − λT AP−1 AT λ − bT λ
4

TANN Chantara (ITC) Convex Functions October 8, 2022 11 / 18

 Dual Cones

Dual Cones

Definition: If K is a cone, then the set

K∗ = {y ∈ Rn : xT y ≥ 0, ∀x ∈ K}

is the dual cone of K.

Note that K∗ is a cone by construction.

Example: (Sn+ ) = {Y ∈ Sn : tr(XT Y) ≥ 0, ∀X ∈ Sn+ }

TANN Chantara (ITC) Convex Functions October 8, 2022 12 / 18

 Dual Cones

Dual Cones Cont’d

Properties of the dual cone:
K∗ is closed and convex.
K2 ⊂ K1 =⇒ K∗1 ⊆ K∗2 (the smaller K, the larger K∗ ).
K∗∗ = cl(conv(K)), the smallest convex closed superset of K.
If a convex cone K is proper, then K∗ is proper and K∗∗ = K

Definition: A cone K is called self-dual, if K∗ = K.

Example of self-dual cones: Rn+ , the second-order cone Sn+ .
TANN Chantara (ITC) Convex Functions October 8, 2022 13 / 18
 Dual Cones

Problems with Generalized Inequality


minx∈Rn

 f0 (x)
P: s.t. fi (x) ⪯Ki 0, ∀i = 1, ..., m

hi (x) = 0, ∀i = 1, ..., p



Where Ki ⊂ Rri is a proper convex cone.

Assign a Lagrange multiplier λi ∈ K∗i to fi (x) ⪯Ki 0.
Assign a Lagrange multiplier µi ∈ R to hi (x) = 0.
The Lagrangian L : Rn × K∗1 × K∗m × Rp → R is defined as
m
X p
X
L(x, λ, µ) = f0 (x) + λT
i fi (x) + µi hi (x)
i=1 i=1

TANN Chantara (ITC) Convex Functions October 8, 2022 14 / 18

 Generalized Inequalities

Problems with Generalized Inequality Cont’d

P : infn f(x) = sup L(x, λ, µ)

x∈R λ∈K∗ ,µ∈Rp

D: sup g(λ, µ) g(λ, µ) = infn L(x, λ, µ)

λ∈K∗ ,µ∈Rp x∈R

Weal duality holds because

fi (x) ∈ Ki ⇒ ∀λi ∈ K∗i : λTi fi (x) ≤ 0
fi (x) ̸∈ Ki ⇒ ∀λi ∈ K∗i : λT
i fi (x) > 0

Strong duality holds for covex problems satisfy a constraint

qualification.
Slater’s constraint qualification holds if there exist xS with
fi (xS ) ≺ 0, ∀i = 1, ..., m
hi (xS ) = 0, ∀i = 1, ..., p

TANN Chantara (ITC) Convex Functions October 8, 2022 15 / 18

 Generalized Inequalities

SOCP Duality

Primal Program:
(
minx∈Rn f Tx
P:
s.t. ||Ai x − bi ||2 ≤ cT
i x + di , i = 1, ..., m

Dual Program:
 Pm
− T
maxVi ,µ∈Rm i=1 (bi vi + di µi )


Pm T
D: s.t. i=1 (Ai vi + ci µi ) = f

||Vi || ≤ µi , i = 1, ..., m

s.t.

Derivation: Keep as Homework.

TANN Chantara (ITC) Convex Functions October 8, 2022 16 / 18

 Generalized Inequalities

SPD Duality

Primal Program:
(
minx∈Rn cT x
s.t. F1 x1 + · · · + Fn xx ⪯ G

Dual Program:
(
min∧⪰0 −tr(∧T G)
s.t. tr(∧T Fi ) = −ci , ∀i = 1, ..., n

Derivation: Homework to proof.

TANN Chantara (ITC) Convex Functions October 8, 2022 17 / 18

 Summary

Summary

Lagrangian: weighted sum of objective and constraints; saddle

function for convex problems; used to construct primal objective
(partial maximum w.r.t. Lagrange multipliers) and dual objective
(partial minimum w.r.t. decision variables).
Duality: sup D never exceeds inf P (weak duality); sup D equals inf P
for convex problems satisfying a constraint qualification (strong
duality); all primal (dual) feasible solutions offer upper (lower) bounds
on both inf P and sup D.
Explicit dual problems: The dual of an LP/QP/SOCP/SDP is also
an LP/QP/SOCP/SDP, respectively.
Dual cone: a proper convex cone coincides with the dual of its dual
(bidual) cone; the nonnegative orthant, the second-order cone and the
positive semidefinite cone are self-dual.

TANN Chantara (ITC) Convex Functions October 8, 2022 18 / 18