Convex Optimization
Instructor: Angelia Nedich
August 26, 2008
Lecture 1
Outline
What is the Course About
Who Cares and Why
Course Objective
Convex Optimization History
New Interest in the Topic
Formal Introduction
Convex Optimization 1
Lecture 1
What is the Course About?
A special class of optimization (includes Linear Programming)
Convex Optimization 2
Lecture 1
Who Cares and Why?
Who?
Anyone using or interested in computational aspects of optimization
Why?
To understand the underlying basic terminology, principles, and
methodology (to efficiently use the existing software tools)
To develop ability to modify tools when needed
To develop ability to design new algorithms or improve the efficiency
of the existing ones
Convex Optimization 3
Lecture 1
Course Objective
The goal of this course is to provide you with working knowledge of
convex optimization
In particular, to provide you with skills and knowledge to
Recognize convex problems
Model problems as convex
Solve the problems
Convex Optimization 4
Lecture 1
Convex Optimization History
Convexity Theory and Analysis have being studied for a long time, mostly
by mathematicians
Until late 1980s:
Algorithmic development focused mainly on solving Linear Problems
Simplex Algorithm for linear programming (Dantzig, 1947)
Ellipsoid Method (Shor, 1970)
Interior-Point Methods for linear programming (Karmarkar, 1984)
Applications in operations research and few in engineering
Since late 1980s: A new interest in Convex Optimization emerges
Convex Optimization 5
Lecture 1
New Interest in the Topic
Recent developments stimulated a new interest in Convex Optimization
The recognition that Interior-Point Methods can efficiently solve certain
classes of convex problems, including semi-definite programs and second-
order cone programs, almost as easily as linear programs
The new technologies and their applications created a need for new
models (convex models are often suitable)
Convex Problems are now prevalent in practice
Automatic Control Systems
Estimation, Signal and Image Processing
Communication and Data Networks
Data Analysis and Modeling
Statistics and Finance
Convex Optimization 6
Lecture 1
Formal Introduction
Mathematical Formulation of Optimization
Some Examples
Solving Optimization Problems
Least-Squares
Linear Optimization
Convex Optimization
Practical Example
Ongoing Research in Convex Optimization
Convex Optimization 7
Lecture 1
Mathematical Formulation of Optimization Problem
minimize f (x)
subject to gi(x) 0, i = 1, . . . , m
xX
Vector x = (x1, . . . , xn) represents optimization (decision) variables
Function f : Rn R is an objective function
Functions gi : Rn R, i = 1, . . . , m are constraint functions (represent
inequality constraints)
Set X Rn is a constraint set
Optimal value: The smallest value of f among all vectors that satisfy the
set and the inequality constraints
Optimal solution: A vector that achieves the optimal value of f and
satisfies all the constraints
Convex Optimization 8
Lecture 1
Some Examples
Communication Networks
Variables: communication rates for users
Constraints: link capacities
Objective: user cost
Portfolio Optimization
Variables: amounts invested in different assets
Constraints: available budget, maximum/minimum investment per asset,
minimum return, time constraints
Objective: overall risk or return variance
Data Fitting
Variables: model parameters
Constraints: prior information, parameter limits
Objective: measure of misfit or prediction error
Convex Optimization 9
Lecture 1
Solving Optimization Problems
General Optimization Problem
Very difficult to solve
Existing methods involve trade offs between time and accuracy, eg.,
very long computation time, or finding a sub-optimal solution
Exceptions: Certain problem classes can be solved efficiently and reliably
Least-Squares Problems
Linear Programming Problems
Some classes of Convex Optimization Problems
Convex Optimization 10
Lecture 1
Least-Squares
minimize kAx bk2
Solving Least-Squares Problems
Analytical solution: x = (AT A)1AT b
Reliable and efficient algorithms and software
A mature technology
Computation time proportional to n2k (A Rkn); less if structured
Using Least-Squares
Least-squares problems are easy to recognize
In regression analysis, optimal control, parameter estimation
A few standard techniques increase its flexibility in applications (eg.,
including weights, regularization terms)
Convex Optimization 11
Lecture 1
Linear Programming
minimize c0x
subject to a0ix bi, 1im
Solving Linear Programs
No analytical solution
Reliable and efficient algorithms and software
A mature technology
Convex Optimization 12
Lecture 1
Using Linear Programming
Not as easy to recognize as least-squares problems (linear formulation
possible but not always obvious)
A few standard tricks used to convert problems into linear programs
(eg., problems involving maximum norm, piecewise-linear functions)
Convex Optimization 13
Lecture 1
Convex Optimization Problems
minimize f (x)
subject to gi(x) 0, i = 1, . . . , m
xX
Objective and constraint functions are convex
Constraint set is convex
Includes least-squares problems and linear programs as special cases
Solving Convex Optimization Problems
No analytical solution
Reliable and efficient algorithms for some classes
Computation time (roughly) proportional to max{n3, n2m, G}, where
G is a cost of evaluating gis and their first and second derivatives
Almost a technology
Using Convex Optimization
Often difficult to recognize
Many tricks for transforming problems into convex form
Many practical problems can be modeled as convex optimization
Convex Optimization 14
Lecture 1
Practical Example
Image Reconstruction in PET-scan [Ben-Tal, 2005]
Maximum Likelihood Model results in convex optimization
m
X n
X
min
0
yi ln pij xj
x0, e x1
i=1 j=1
x is a decision vector
y models measured data (by PET detectors)
pij probabilities modeling detections of emitted positrons
Convex Optimization 15
Lecture 1
Ongoing Research in Convex Optimization and Beyond
Distributed computations for large scale (nonsmooth) convex problems
Approximation schemes with rate and error estimates
Extending the methodology to non-convex problems
Convex Optimization 16
