Numerical geometry of non-rigid shapes Numerical Optimization
Numerical Optimization
Alexander Bronstein, Michael Bronstein 2008 All rights reserved. Web: tosca.cs.technion.ac.il
Numerical geometry of non-rigid shapes Numerical Optimization
Slowest
Longest
Shortest
Maximal
Minimal
Fastest Largest
Smallest
Common denominator: optimization problems
Numerical geometry of non-rigid shapes Numerical Optimization
Optimization problems
Generic unconstrained minimization problem
where Vector space A solution The value is the minimum is the search space
is a cost (or objective) function is the minimizer of
Numerical geometry of non-rigid shapes Numerical Optimization
Local vs. global minimum
Find minimum by analyzing the local behavior of the cost function
Local minimum
Global minimum
Numerical geometry of non-rigid shapes Numerical Optimization
Local vs. global in real life
False summit 8,030 m
Main summit 8,047 m
Broad Peak (K3), 12th highest mountain on Earth
Numerical geometry of non-rigid shapes Numerical Optimization
Convex functions
A function defined on a convex set is called convex if
for any
and
For convex function local minimum = global minimum
Convex
Non-convex
Numerical geometry of non-rigid shapes Numerical Optimization
One-dimensional optimality conditions
Point Approximate a function around as a parabola using Taylor expansion . is the local minimizer of a -function if
guarantees the minimum at
guarantees the parabola is convex
Numerical geometry of non-rigid shapes Numerical Optimization
Gradient
In multidimensional case, linearization of the function according to Taylor
gives a multidimensional analogy of the derivative.
The function
, denoted as
, is called the gradient of
In one-dimensional case, it reduces to standard definition of derivative
Numerical geometry of non-rigid shapes Numerical Optimization
Gradient
In Euclidean space ( ), can be represented in standard basis
in the following way:
i-th place
which gives
Numerical geometry of non-rigid shapes Numerical Optimization
10
Example 1: gradient of a matrix function
Given product Compute the gradient of the function an matrix where is (space of real matrices) with standard inner
For square matrices
Numerical geometry of non-rigid shapes Numerical Optimization
11
Example 2: gradient of a matrix function
Compute the gradient of the function an matrix where is
Numerical geometry of non-rigid shapes Numerical Optimization
12
Hessian
Linearization of the gradient
gives a multidimensional analogy of the secondorder derivative.
The function
is called the Hessian of
, denoted as
Ludwig Otto Hesse (1811-1874)
In the standard basis, Hessian is a symmetric matrix of mixed second-order
derivatives
Numerical geometry of non-rigid shapes Numerical Optimization
13
Optimality conditions, bis
Point matrix (denoted Approximate a function around . for all ) as a parabola using Taylor expansion , i.e., the Hessian is a positive definite is the local minimizer of a -function if
guarantees the minimum at
guarantees the parabola is convex
Numerical geometry of non-rigid shapes Numerical Optimization
14
Optimization algorithms
Descent direction Step size
Numerical geometry of non-rigid shapes Numerical Optimization
15
Generic optimization algorithm
Start with some Determine descent direction
Choose step size
such that
Update iterate
Until convergence
Increment iteration counter Solution Descent direction Step size Stopping criterion
Numerical geometry of non-rigid shapes Numerical Optimization
16
Stopping criteria
Near local minimum, (or equivalently )
Stop when gradient norm becomes small
Stop when step size becomes small
Stop when relative objective change becomes small
Numerical geometry of non-rigid shapes Numerical Optimization
17
Line search
Optimal step size can be found by solving a one-dimensional optimization problem
One-dimensional optimization algorithms for finding the optimal step size are generically called exact line search
Numerical geometry of non-rigid shapes Numerical Optimization
18
Armijo [ar-mi-xo] rule
The function sufficiently decreases if Armijo rule (Larry Armijo, 1966): start with multiplying by some and decrease it by
until the function sufficiently decreases
Numerical geometry of non-rigid shapes Numerical Optimization
19
Descent direction
How to descend in the fastest way? Go in the direction in which the height lines are the densest
Devils Tower
Topographic map
Numerical geometry of non-rigid shapes Numerical Optimization
20
Steepest descent
Directional derivative: how much changes in the direction (negative for a descent direction)
Find a unit-length direction minimizing directional
derivative
Numerical geometry of non-rigid shapes Numerical Optimization
21
Steepest descent
L2 norm
L1 norm
Normalized steepest descent
Coordinate descent (coordinate axis in which descent is maximal)
Numerical geometry of non-rigid shapes Numerical Optimization
22
Steepest descent algorithm
Start with some Compute steepest descent direction
Choose step size using line search
Until convergence
Update iterate
Increment iteration counter
Numerical geometry of non-rigid shapes Numerical Optimization
23
MATLAB
intermezzo
Steepest descent
Numerical geometry of non-rigid shapes Numerical Optimization
24
Condition number
Condition number is the ratio of maximal and minimal eigenvalues of the Hessian
1
,
1
0.5
0.5
-0.5
-0.5
-1 -1
-0.5
0.5
-1 -1
-0.5
0.5
Problem with large condition number is called ill-conditioned Steepest descent convergence rate is slow for ill-conditioned problems
Numerical geometry of non-rigid shapes Numerical Optimization
25
Q-norm
Change of coordinates
Q-norm
L2 norm
Function Gradient Descent direction
Numerical geometry of non-rigid shapes Numerical Optimization
26
Preconditioning
Using Q-norm for steepest descent can be regarded as a change of coordinates, called preconditioning Preconditioner should be chosen to improve the condition number of
the Hessian in the proximity of the solution In system of coordinates, the Hessian at the solution is
(a dream)
Numerical geometry of non-rigid shapes Numerical Optimization
27
Newton method as optimal preconditioner
Best theoretically possible preconditioner direction , giving descent
Ideal condition number
Problem: the solution
is unknown in advance
Newton direction: use Hessian as a preconditioner at each iteration
Numerical geometry of non-rigid shapes Numerical Optimization
28
Another derivation of the Newton method
Approximate the function as a quadratic function using second-order Taylor expansion
(quadratic function in
Close to solution the function looks like a quadratic function; the Newton method converges fast
Numerical geometry of non-rigid shapes Numerical Optimization
29
Newton method
Start with some Compute Newton direction
Choose step size using line search
Until convergence
Update iterate
Increment iteration counter
Numerical geometry of non-rigid shapes Numerical Optimization
30
Frozen Hessian
Observation: close to the optimum, the Hessian does not change significantly Reduce the number of Hessian inversions by keeping the Hessian from previous iterations and update it once in a few iterations Such a method is called Newton with frozen Hessian
Numerical geometry of non-rigid shapes Numerical Optimization
31
Cholesky factorization
Decompose the Hessian
where
is a lower triangular matrix
Solve the Newton system Andre Louis Cholesky (1875-1918)
in two steps Forward substitution
Backward substitution
Complexity: , better than straightforward matrix inversion
Numerical geometry of non-rigid shapes Numerical Optimization
32
Truncated Newton
Solve the Newton system approximately
A few iterations of conjugate gradients or other algorithm for the solution of linear systems can be used Such a method is called truncated or inexact Newton
Numerical geometry of non-rigid shapes Numerical Optimization
33
Non-convex optimization
Using convex optimization methods with non-convex functions does not guarantee global convergence! There is no theoretical guaranteed global optimization, just heuristics
Local minimum
Global minimum
Good initialization
Multiresolution
Numerical geometry of non-rigid shapes Numerical Optimization
34
Iterative majorization
Construct a majorizing function . Majorizing inequality: for all satisfying
is convex or easier to optimize w.r.t.
Numerical geometry of non-rigid shapes Numerical Optimization
35
Iterative majorization
Start with some Find such that
Update iterate
Until convergence
Increment iteration counter Solution
Numerical geometry of non-rigid shapes Numerical Optimization
36
Constrained optimization
MINEFIELD CLOSED ZONE
Numerical geometry of non-rigid shapes Numerical Optimization
37
Constrained optimization problems
Generic constrained minimization problem
where
are inequality constraints
are equality constraints in which the constraints hold is called
A subset of the search space feasible set A point
belonging to the feasible set is called a feasible solution may be infeasible!
A minimizer of the problem
Numerical geometry of non-rigid shapes Numerical Optimization
38
An example
Equality constraint Inequality constraint
Feasible set
Inequality constraint
A point
is active at point
if
, inactive otherwise
and of
is regular if the gradients of equality constraints are linearly independent
active inequality constraints
Numerical geometry of non-rigid shapes Numerical Optimization
39
Lagrange multipliers
Main idea to solve constrained problems: arrange the objective and constraints into a single function
and minimize it as an unconstrained problem is called Lagrangian and are called Lagrange multipliers
Numerical geometry of non-rigid shapes Numerical Optimization
40
KKT conditions
If is a regular point and a local minimum, there exist Lagrange multipliers and Known as Karush-Kuhn-Tucker conditions Necessary but not sufficient! such that for all such that inactive constraints and for all
for active constraints and zero for
Numerical geometry of non-rigid shapes Numerical Optimization
41
KKT conditions
Sufficient conditions:
If the objective
is convex, the inequality constraints
are affine, and and for all
are convex
and the equality constraints then for all such that inactive constraints
for active constraints and zero for
is the solution of the constrained problem (global constrained
minimizer)
Numerical geometry of non-rigid shapes Numerical Optimization
42
Geometric interpretation
Consider a simpler problem: Equality constraint
The gradient of objective and constraint must line up at the solution
Numerical geometry of non-rigid shapes Numerical Optimization
43
Penalty methods
Define a penalty aggregate
where
and
are parametric penalty functions
For larger values of the parameter
is stronger
, the penalty on the constraint violation
Numerical geometry of non-rigid shapes Numerical Optimization
44
Penalty methods
Inequality penalty
Equality penalty
Numerical geometry of non-rigid shapes Numerical Optimization
45
Penalty methods
Start with some Find and initial value of
by solving an unconstrained optimization problem initialized with
Until convergence
Set
Set Update
Solution
