Convex Optimization

Instructor: Angelia Nedich

August 26, 2008

 Lecture 1

Outline

What is the Course About

Who Cares and Why

Course Objective

Convex Optimization History

New Interest in the Topic

Formal Introduction

Convex Optimization 1
 Lecture 1

What is the Course About?

A special class of optimization (includes Linear Programming)

Convex Optimization 2
 Lecture 1

Who Cares and Why?

Who?
Anyone using or interested in computational aspects of optimization

Why?
To understand the underlying basic terminology, principles, and
methodology (to efficiently use the existing software tools)
To develop ability to modify tools when needed
To develop ability to design new algorithms or improve the efficiency
of the existing ones

Convex Optimization 3
 Lecture 1

Course Objective

The goal of this course is to provide you with working knowledge of

convex optimization

In particular, to provide you with skills and knowledge to

Recognize convex problems
Model problems as convex
Solve the problems

Convex Optimization 4
 Lecture 1

Convex Optimization History

Convexity Theory and Analysis have being studied for a long time, mostly
by mathematicians

Until late 1980s:

Algorithmic development focused mainly on solving Linear Problems
 Simplex Algorithm for linear programming (Dantzig, 1947)
 Ellipsoid Method (Shor, 1970)

 Interior-Point Methods for linear programming (Karmarkar, 1984)

Applications in operations research and few in engineering

Since late 1980s: A new interest in Convex Optimization emerges

Convex Optimization 5
 Lecture 1

New Interest in the Topic

Recent developments stimulated a new interest in Convex Optimization

The recognition that Interior-Point Methods can efficiently solve certain
classes of convex problems, including semi-definite programs and second-
order cone programs, almost as easily as linear programs
The new technologies and their applications created a need for new
models (convex models are often suitable)
Convex Problems are now prevalent in practice
Automatic Control Systems
Estimation, Signal and Image Processing
Communication and Data Networks
Data Analysis and Modeling
Statistics and Finance

Convex Optimization 6
 Lecture 1

Formal Introduction

Mathematical Formulation of Optimization

Some Examples

Solving Optimization Problems

Least-Squares
Linear Optimization
Convex Optimization

Practical Example

Ongoing Research in Convex Optimization

Convex Optimization 7
 Lecture 1

Mathematical Formulation of Optimization Problem

minimize f (x)
subject to gi(x) 0, i = 1, . . . , m
xX
Vector x = (x1, . . . , xn) represents optimization (decision) variables

Function f : Rn R is an objective function

Functions gi : Rn R, i = 1, . . . , m are constraint functions (represent

inequality constraints)
Set X Rn is a constraint set
Optimal value: The smallest value of f among all vectors that satisfy the
set and the inequality constraints
Optimal solution: A vector that achieves the optimal value of f and
satisfies all the constraints

Convex Optimization 8
 Lecture 1

Some Examples
Communication Networks
Variables: communication rates for users
Constraints: link capacities
Objective: user cost
Portfolio Optimization
Variables: amounts invested in different assets
Constraints: available budget, maximum/minimum investment per asset,
minimum return, time constraints
Objective: overall risk or return variance
Data Fitting
Variables: model parameters
Constraints: prior information, parameter limits
Objective: measure of misfit or prediction error

Convex Optimization 9
 Lecture 1

Solving Optimization Problems

General Optimization Problem

Very difficult to solve
Existing methods involve trade offs between time and accuracy, eg.,
very long computation time, or finding a sub-optimal solution
Exceptions: Certain problem classes can be solved efficiently and reliably
Least-Squares Problems
Linear Programming Problems
Some classes of Convex Optimization Problems

Convex Optimization 10
 Lecture 1

Least-Squares

minimize kAx bk2

Solving Least-Squares Problems
Analytical solution: x = (AT A)1AT b
Reliable and efficient algorithms and software
A mature technology
Computation time proportional to n2k (A Rkn); less if structured
Using Least-Squares
Least-squares problems are easy to recognize
In regression analysis, optimal control, parameter estimation
A few standard techniques increase its flexibility in applications (eg.,
including weights, regularization terms)

Convex Optimization 11
 Lecture 1

Linear Programming

minimize c0x
subject to a0ix bi, 1im
Solving Linear Programs
No analytical solution
Reliable and efficient algorithms and software
A mature technology

Convex Optimization 12
 Lecture 1

Using Linear Programming

Not as easy to recognize as least-squares problems (linear formulation
possible but not always obvious)
A few standard tricks used to convert problems into linear programs
(eg., problems involving maximum norm, piecewise-linear functions)

Convex Optimization 13
 Lecture 1

Convex Optimization Problems

minimize f (x)
subject to gi(x) 0, i = 1, . . . , m
xX
Objective and constraint functions are convex
Constraint set is convex
Includes least-squares problems and linear programs as special cases
Solving Convex Optimization Problems
No analytical solution
Reliable and efficient algorithms for some classes
Computation time (roughly) proportional to max{n3, n2m, G}, where
G is a cost of evaluating gis and their first and second derivatives
Almost a technology
Using Convex Optimization
Often difficult to recognize
Many tricks for transforming problems into convex form
Many practical problems can be modeled as convex optimization

Convex Optimization 14
 Lecture 1

Practical Example
Image Reconstruction in PET-scan [Ben-Tal, 2005]

Maximum Likelihood Model results in convex optimization

m
X n
X
min
0
yi ln pij xj
x0, e x1
i=1 j=1

x is a decision vector
y models measured data (by PET detectors)
pij probabilities modeling detections of emitted positrons

Convex Optimization 15
 Lecture 1

Ongoing Research in Convex Optimization and Beyond

Distributed computations for large scale (nonsmooth) convex problems

Approximation schemes with rate and error estimates

Extending the methodology to non-convex problems

Convex Optimization 16