Introduction to
Convex Optimization
EE/CS/EST 135
Feb 12, 2018
Outline
• Motivation
• Recap of Linear Algebra and Real Analysis
• Convex Set
• Convex Function
• Convex Program
Outline
• Motivation
• Recap of Linear Algebra and Real Analysis
• Convex Set
• Convex Function
• Convex Program
Motivation
Motivation
• Many practical problems can be modeled as
optimization problems:
• Optimal Power Flow (OPF)
• EV Charging Scheduling
Motivation
• Many practical problems can be modeled as
optimization problems:
• Optimal Power Flow (OPF)
• EV Charging Scheduling
• Convex program
• is a convex set
• is a convex function
Motivation
• Many practical problems can be modeled as
optimization problems:
• Optimal Power Flow (OPF)
• EV Charging Scheduling
• Convex programs have good properties
• Certificate of global optimality
• Efficient algorithms exist
• Powerful modeling capability
Outline
• Motivation
• Recap of Linear Algebra and Real Analysis
• Convex Set
• Convex Function
• Convex Program
Recap of Linear Algebra
• Euclidean space
• Vectors: Matrices:
• Transpose:
• Rank:
• Trace:
Recap of Linear Algebra
• Euclidean space
• Vectors: Matrices:
• Inner product:
• Norm:
• Orthonormal basis:
Recap of Linear Algebra
• Euclidean space
• Vectors: Matrices:
• Standard inner product:
• Standard norm:
• Orthonormal basis:
Recap of Linear Algebra
• Real symmetric matrices
Recap of Linear Algebra
• Real symmetric matrices
• Eigenvalue decomposition for
•
• forms an orthonormal basis
•
Recap of Linear Algebra
• is a real linear space with
• Inner product:
• Frobenius norm:
Recap of Linear Algebra
• Positive semidefinite (PSD) matrices
• Notation:
Recap of Linear Algebra
• Positive semidefinite (PSD) matrices
• Notation:
• A real symmetric matrix is PSD iff one of the
following holds:
• The eigenvalues of are all nonnegative
• for some matrix
Recap of Linear Algebra
• A real symmetric matrix is PSD iff one of the
following holds:
• The eigenvalues of are all nonnegative
• for some matrix
Recap of Linear Algebra
• A real symmetric matrix is PSD iff one of the
following holds:
• The eigenvalues of are all nonnegative
• for some matrix
• Corollary: A real symmetric matrix is equal to
for some iff
Recap of Linear Algebra
• Complex linear space
• Complex transpose:
• Hermitian matrix:
• PSD matrix:
Recap of Linear Algebra
• Complex linear space
• Complex transpose:
• Hermitian matrix:
• PSD matrix:
• is PSD iff
Recap of Real Analysis
Recap of Real Analysis
• Open ball
Recap of Real Analysis
• Open ball
• Interior point
• Boundary point
Recap of Real Analysis
• Interior & Boundary
Recap of Real Analysis
• Interior & Boundary
• Open set:
• Closed set:
Recap of Real Analysis
• Interior & Boundary
• Open set:
• Closed set:
• Complement of an open (closed) set is closed (open)
•
Recap of Real Analysis
• Interior & Boundary
• Open set:
• Closed set:
• Bounded set:
Recap of Real Analysis
• Compact set
Recap of Real Analysis
• Compact set
• (Definition) Any open cover has a finite subcover
Recap of Real Analysis
• Compact set
• (Definition) Any open cover has a finite subcover
• (Heine - Borel) If , then
Recap of Real Analysis
• Compact set
• (Definition) Any open cover has a finite subcover
• (Heine - Borel) If , then
• Extreme Value Theorem
Suppose is compact and is continuous.
Then there exist such that
Recap of Real Analysis
Recap of Real Analysis
open,
Recap of Real Analysis
open,
• Gradient:
Recap of Real Analysis
open,
• Gradient:
• Hessian:
Recap of Real Analysis
open,
• Gradient:
• Hessian:
Outline
• Motivation
• Recap of Linear Algebra and Real Analysis
• Convex Set
• Convex Function
• Convex Program
Convex Set
• Line segment:
• is called convex if
convex set nonconvex set
Convex Set
• Convex hull
• The union of all line segments
• The smallest convex set containing
nonconvex set convex hull
Examples of Convex Sets
Examples of Convex Sets
• Hyperplanes
• Halfspaces
Examples of Convex Sets
• Hyperplanes
• Halfspaces
• Open and closed balls
Separating Hyperplanes
Separating Hyperplanes
convex
Separating Hyperplanes
convex
• Separating Hyperplane Theorem
If and , then there exists
a nonzero and such that
for any and
Separating Hyperplanes
convex
• Separating Hyperplane Theorem
If and , then there exists
a nonzero and such that
for any and
In other words,
Convex Cones
Convex Cones
• is called a cone if for any and any
positive scalar , ones has
Convex Cones
• is called a cone if for any and any
positive scalar , ones has
• Examples of convex cones:
• Nonnegative orthant
Convex Cones
• is called a cone if for any and any
positive scalar , ones has
• Examples of convex cones:
• Nonnegative orthant
• Second-order cone
Convex Cones
• is called a cone if for any and any
positive scalar , ones has
• Examples of convex cones:
• Nonnegative orthant
• Second-order cone
• PSD cone
Convex Cones
• Nonnegative orthant
• Second-order cone
• PSD cone
Convex Cones 1. Closed
2. Non-empty interior
• Nonnegative orthant
3. Pointed
• Second-order cone
• PSD cone 4. Self-dual
Convex Cones 1. Closed
2. Non-empty interior
• Nonnegative orthant
3. Pointed
• Second-order cone
• PSD cone 4. Self-dual
1+2+3 Possible to define a partial order:
partial
order
Convex Cones 1. Closed
2. Non-empty interior
• Nonnegative orthant
3. Pointed
• Second-order cone
• PSD cone 4. Self-dual
1+2+3 Possible to define a partial order:
Self-dual:
Operations that Preserve Convexity
Operations that Preserve Convexity
• Intersection
Operations that Preserve Convexity
• Intersection
• Affine transformation
• is convex if is convex
• is convex if is convex
Operations that Preserve Convexity
• Intersection
• Affine transformation
• is convex if is convex
• is convex if is convex
• Cartesian product, Minkowski sum, etc.
More Examples of Convex Sets
More Examples of Convex Sets
• Convex polytopes
More Examples of Convex Sets
• Convex polytopes
• Solutions of linear matrix inequalities
Outline
• Motivation
• Recap of Linear Algebra and Real Analysis
• Convex Set
• Convex Function
• Convex Program
Convex Function
• A function with a convex domain
is called convex if
for all and
Convex Function
• A function with a convex domain
is called convex if
for all and
• … is called strictly convex if
for all with and
Convex Function
• A function with a convex domain
is called convex if
is convex.
Sublevel Set
Sublevel Set
Sublevel Set
•
Sublevel Set
•
• Sublevel sets are convex if is convex.
First-order Condition
First-order Condition
• A differentiable function with a convex
domain is convex iff
for all
First-order Condition
• A differentiable function with a convex
domain is convex iff
for all
• The tangent plane
is the hyperplane separating and
First-order Condition
• A differentiable function with a convex
domain is convex iff
for all
• … is strictly convex iff
for all with
Second-order Condition
Second-order Condition
• A twice differentiable function with a
convex domain is convex iff
for all
Operations that Preserve Convexity
• Positive weighted sum
• Pointwise supremum of a family of convex functions
• Composition with affine functions
Examples of Convex Functions
•
•
•
•
Outline
• Motivation
• Recap of Linear Algebra and Real Analysis
• Convex Set
• Convex Function
• Convex Program
Convex Program
• Convex program: is convex and is convex.
Convex Program
• Convex program: is convex and is convex.
• Any local optimum is a global optimum:
Suppose and there exists such that
for all with . Then
Conic Program
Conic Program
Conic Program
• : Linear program (LP)
• : Second-order cone program (SOCP)
• : Semidefinite program (SDP)
Duality Theory
Duality Theory
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
• Because is self-dual, we have
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
• Because is self-dual, we have
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
• Because is self-dual, we have
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
• Because is self-dual, we have
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
• Because is self-dual, we have
Duality Theory
• For a conic program with
being self-dual, the Lagrangian is
where
• Weak duality:
Duality Theory
• Weak duality:
Duality Theory
• Strong duality:
Duality Theory
• Strong duality:
• For a convex conic program with being self-dual,
strong duality holds under Slater’s condition
Duality Theory
• Strong duality:
• For a convex conic program with being self-dual,
strong duality holds under Slater’s condition
Duality Theory
• Strong duality:
• Primal feasible points produce upper bounds of
optimal value.
Dual feasible points produce lower bounds of
optimal value.
Duality Theory
• Strong duality:
• Certificate of optimality: If is primal feasible and
is dual feasible, and
then is an optimal solution.
KKT Conditions
KKT Conditions
•
primal feasibility
dual feasibility
complementary slackness
KKT Conditions
•
primal feasibility
dual feasibility
complementary slackness
• Necessary and sufficient conditions for optimality
(under Slater’s condition)
Linear Program
Linear Program
•
Linear Program
•
• Scheduling for EV charging
Semidefinite Program
Semidefinite Program
•
QCQP and SDP Relaxation
QCQP and SDP Relaxation
• QCQP: quadratically constrained quadratic program
QCQP and SDP Relaxation
• QCQP: quadratically constrained quadratic program
• Could be non-convex if some is not PSD.
• Generally NP-hard.
QCQP and SDP Relaxation
• QCQP: quadratically constrained quadratic program
QCQP and SDP Relaxation
• QCQP: quadratically constrained quadratic program
QCQP and SDP Relaxation
• QCQP: quadratically constrained quadratic program
QCQP and SDP Relaxation
• QCQP: quadratically constrained quadratic program
• Semidefinite relaxation of QCQP
Algorithms
Algorithms
• Unconstrained optimization ( )
• Gradient descent & its variants
• Newton & quasi-Newton method
Algorithms
• Unconstrained optimization ( )
• Gradient descent & its variants
• Newton & quasi-Newton method
• Constrained optimization
• Projected gradient descent & its variants
• Dual ascent & its variants
• Simplex method for LP
• Interior point method
• Distributed algorithms
Software
• Solvers
• SDPT3, Sedumi (LP+SOCP+SDP, MATLAB), CVXOPT (Python)
• IPOPT (nonlinear opt, local solution)
• Gurobi (LP+SOCP+…), Mosek (LP+SOCP+SDP+…)
• Interfaces and modelling tools
• CVX, YALMIP (MATLAB)
• CVXPY (Python)
References
• Linear algebra:
• G. Strang. Linear Algebra and Its Applications.
• P. D. Lax. Linear Algebra and Its Applications.
• Real analysis:
• W. Rudin. Principles of Mathematical Analysis.
• N. L. Carothers. Real Analysis.
• Convex Optimization:
• S. Boyd and L. Vandenberghe. Convex Optimization.
• A. Ben-Tal and A. Nemirovski. Lectures on Modern Convex
Optimization.
• Numerical Methods:
• J. Nocedal and S. J. Wright. Numerical Optimization.
Backup Slides
Projection onto Closed Convex Sets
closed
• Projection onto a closed set
• Points in that are closest to
• Projection onto a closed convex set
• (Motzkin)
• If is convex, then
Projection onto Closed Convex Sets
closed and convex
• Projection onto a closed convex set
