Convex Optimization M2

Lecture 1

A. d’Aspremont. Convex Optimization M2. 1/49

Today

 Convex optimization: introduction

 Course organization and other gory details...
 Convex sets, basic definitions.

A. d’Aspremont. Convex Optimization M2. 2/49

 Convex Optimization

A. d’Aspremont. Convex Optimization M2. 3/49

Convex Optimization

 How do we identify easy and hard problems?

 Convexity: why is it so important?
 Modeling: how do we recognize easy problems in real applications?
 Algorithms: how do we solve these problems in practice?

A. d’Aspremont. Convex Optimization M2. 4/49

Least squares (LS)

minimize kAx − bk22

A ∈ Rm×n, b ∈ Rm are parameters; x ∈ Rn is variable

 Complete theory (existence & uniqueness, sensitivity analysis . . . )

 Several algorithms compute (global) solution reliably
 We can solve dense problems with n = 1000 vbles, m = 10000 terms
 By exploiting structure (e.g., sparsity) can solve far larger problems

. . . LS is a (widely used) technology

A. d’Aspremont. Convex Optimization M2. 5/49

Linear program (LP)

minimize cT x
subject to aTi x ≤ bi, i = 1, . . . , m

c, ai ∈ Rn are parameters; x ∈ Rn is variable

 Nearly complete theory

(existence & uniqueness, sensitivity analysis . . . )
 Several algorithms compute (global) solution reliably
 Can solve dense problems with n = 1000 vbles, m = 10000 constraints
 By exploiting structure (e.g., sparsity) can solve far larger problems

. . . LP is a (widely used) technology

A. d’Aspremont. Convex Optimization M2. 6/49

Quadratic program (QP)

minimize kF x − gk22
subject to aTi x ≤ bi, i = 1, . . . , m

 Combination of LS & LP
 Same story . . . QP is a technology
 Reliability: Programmed on chips to solve real-time problems
 Classic application: portfolio optimization

A. d’Aspremont. Convex Optimization M2. 7/49

The bad news

 LS, LP, and QP are exceptions

 Most optimization problems, even some very simple looking ones, are
intractable
 The objective of this class is to show you how to recognize the nice ones. . .
 Many, many applications across all fields. . .

A. d’Aspremont. Convex Optimization M2. 8/49

Polynomial minimization

minimize p(x)
p is polynomial of degree d; x ∈ Rn is variable

 Except for special cases (e.g., d = 2) this is a very difficult problem

 Even sparse problems with size n = 20, d = 10 are essentially intractable
 All algorithms known to solve this problem require effort exponential in n

A. d’Aspremont. Convex Optimization M2. 9/49

What makes a problem easy or hard?

Classical view:

 linear is easy
 nonlinear is hard(er)

A. d’Aspremont. Convex Optimization M2. 10/49

What makes a problem easy or hard?

Emerging (and correct) view:

. . . the great watershed in optimization isn’t between linearity and

nonlinearity, but convexity and nonconvexity.

— R. Rockafellar, SIAM Review 1993

A. d’Aspremont. Convex Optimization M2. 11/49

Convex optimization

minimize f0(x)
subject to f1(x) ≤ 0, . . . , fm(x) ≤ 0

x ∈ Rn is optimization variable; fi : Rn → R are convex:

fi(λx + (1 − λ)y) ≤ λfi(x) + (1 − λ)fi(y)

for all x, y, 0 ≤ λ ≤ 1

 includes LS, LP, QP, and many others

 like LS, LP, and QP, convex problems are fundamentally tractable

A. d’Aspremont. Convex Optimization M2. 12/49

Example: Stochastic LP

Consider the following stochastic LP:

minimize cT x
subject to Prob(aTi x ≤ bi) ≥ η, i = 1, . . . , m

coefficient vectors ai IID, N (ai, Σi); η is required reliability

 for fixed x, aTi x is N (aTi x, xT Σix)

 so for η = 50%, stochastic LP reduces to LP

minimize cT x
subject to aTi x ≤ bi, i = 1, . . . , m

and so is easily solved

 what about other values of η, e.g., η = 10%? η = 90%?

A. d’Aspremont. Convex Optimization M2. 13/49

Hint

{x | Prob(aTi x ≤ bi) ≥ η, i = 1, . . . , m}

η = 10% η = 50% η = 90%

A. d’Aspremont. Convex Optimization M2. 14/49

Convexity again

stochastic LP with reliability η = 90% is convex, and very easily solved

stochastic LP with reliability η = 10% is not convex, and extremely difficult

moral: very difficult and very easy problems can look quite similar
(to the untrained eye)

A. d’Aspremont. Convex Optimization M2. 15/49

Convex Optimization

A brief history. . .

 The field is about 50 years old.

 Starts with the work of Von Neumann, Kuhn and Tucker, etc
 Explodes in the 60’s with the advent of “relatively” cheap and efficient
computers. . .
 Key to all this: fast linear algebra
 Some of the theory developed before computers even existed. . .

A. d’Aspremont. Convex Optimization M2. 16/49

Convex optimization: history

 Convexity =⇒ low complexity:

”... In fact the great watershed in optimization isn’t between linearity and
nonlinearity, but convexity and nonconvexity.” T. Rockafellar.

 True: Nemirovskii and Yudin [1979].

 Very true: Karmarkar [1984].

 Seriously true: convex programming, Nesterov and Nemirovskii [1994].

A. d’Aspremont. Convex Optimization M2. 17/49

Standard convex complexity analysis

 All convex minimization problems with: a first order oracle (returning f (x) and
a subgradient) can be solved in polynomial time in size and number of
precision digits.

 Proved using the ellipsoid method by Nemirovskii and Yudin [1979].

 Very slow convergence in practice.

A. d’Aspremont. Convex Optimization M2. 18/49

Linear Programming

 Simplex algorithm by Dantzig (1949): exponential worst-case complexity, very

efficient in most cases.

 Khachiyan [1979] then used the ellipsoid method to show the polynomial
complexity of LP.

 Karmarkar [1984] describes the first efficient polynomial time algorithm for LP,
using interior point methods.

A. d’Aspremont. Convex Optimization M2. 19/49

From LP to structured convex programs

 Nesterov and Nemirovskii [1994] show that the interior point methods used for
LPs can be applied to a larger class of structured convex problems.

 The self-concordance analysis that they introduce extends the polynomial

time complexity proof for LPs.

 Most operations that preserve convexity also preserve self-concordance.

 The complexity of a certain number of elementary problems can be directly

extended to a much wider class.

A. d’Aspremont. Convex Optimization M2. 20/49

Symmetric cone programs

 An important particular case: linear programming on symmetric cones

minimize cT x
subject to Ax − b ∈ K

 These include the LP, second-order (Lorentz) and semidefinite cone:

LP: {x ∈ Rn : x ≥ 0}
Second order: {(x, y) ∈ Rn × R : kxk ≤ y}
Semidefinite: {X ∈ Sn : X  0}

 Again, the class of problems that can be represented using these cones is
extremely vast.

A. d’Aspremont. Convex Optimization M2. 21/49

 Course Organization

A. d’Aspremont. Convex Optimization M2. 22/49

Course Plan

 Convex analysis & modeling

 Duality
 Algorithms: interior point methods, first order methods.
 Applications

A. d’Aspremont. Convex Optimization M2. 23/49

Grading

Course website with lecture notes, homework, etc.

http://di.ens.fr/~aspremon/OptConvexeM2.html

 A few homeworks, will be posted online.

Email your homeworks to dm.daspremont@gmail.com

you will get an automatic reply to your message if it has been received.
 A final exam.

A. d’Aspremont. Convex Optimization M2. 24/49

Short blurb

 Contact info on http://di.ens.fr/~aspremon

 Email: aspremon@ens.fr
 Dual PhDs: Ecole Polytechnique & Stanford University
 Interests: Optimization, machine learning, statistics & finance.

A. d’Aspremont. Convex Optimization M2. 25/49

References

 All lecture notes will be posted online

 Textbook: Convex Optimization by Lieven Vandenberghe and Stephen Boyd,
available online at:

http://www.stanford.edu/~boyd/cvxbook/

 See also Ben-Tal and Nemirovski [2001], “Lectures On Modern Convex

Optimization: Analysis, Algorithms, And Engineering Applications”, SIAM.

http://www2.isye.gatech.edu/~nemirovs/

 Nesterov [2003], “Introductory Lectures on Convex Optimization”, Springer.

 Nesterov and Nemirovskii [1994], “Interior Point Polynomial Algorithms in
Convex Programming”, SIAM.

A. d’Aspremont. Convex Optimization M2. 26/49

 Convex Sets

A. d’Aspremont. Convex Optimization M2. 27/49

Convex Sets

 affine and convex sets

 some important examples
 operations that preserve convexity
 generalized inequalities
 separating and supporting hyperplanes
 dual cones and generalized inequalities

A. d’Aspremont. Convex Optimization M2. 28/49

Affine set

line through x1, x2: all points

x = θx1 + (1 − θ)x2 (θ ∈ R)

θ = 1.2 x1
θ=1
θ = 0.6
x2
θ=0
θ = −0.2

affine set: contains the line through any two distinct points in the set

example: solution set of linear equations {x | Ax = b}

A. d’Aspremont. Convex Optimization M2. 29/49

Convex set

line segment between x1 and x2: all points

x = θx1 + (1 − θ)x2

with 0 ≤ θ ≤ 1

convex set: contains line segment between any two points in the set

x1, x2 ∈ C, 0≤θ≤1 =⇒ θx1 + (1 − θ)x2 ∈ C

examples (one convex, two nonconvex sets)

A. d’Aspremont. Convex Optimization M2. 30/49

Convex combination and convex hull

convex combination of x1,. . . , xk : any point x of the form

x = θ 1 x 1 + θ 2 x 2 + · · · + θk x k

with θ1 + · · · + θk = 1, θi ≥ 0

convex hull CoS: set of all convex combinations of points in S

A. d’Aspremont. Convex Optimization M2. 31/49

Convex cone

conic (nonnegative) combination of x1 and x2: any point of the form

x = θ 1 x 1 + θ 2 x2

with θ1 ≥ 0, θ2 ≥ 0

x1

x2
0

convex cone: set that contains all conic combinations of points in the set

A. d’Aspremont. Convex Optimization M2. 32/49

Hyperplanes and halfspaces

hyperplane: set of the form {x | aT x = b} (a 6= 0)

a

x0
x
aT x = b

halfspace: set of the form {x | aT x ≤ b} (a 6= 0)

a

x0 aT x ≥ b

aT x ≤ b

 a is the normal vector

 hyperplanes are affine and convex; halfspaces are convex

A. d’Aspremont. Convex Optimization M2. 33/49

Euclidean balls and ellipsoids

(Euclidean) ball with center xc and radius r:

B(xc, r) = {x | kx − xck2 ≤ r} = {xc + ru | kuk2 ≤ 1}

ellipsoid: set of the form

{x | (x − xc)T P −1(x − xc) ≤ 1}

with P ∈ Sn++ (i.e., P symmetric positive definite)

xc

other representation: {xc + Au | kuk2 ≤ 1} with A square and nonsingular

A. d’Aspremont. Convex Optimization M2. 34/49

Norm balls and norm cones

norm: a function k · k that satisfies

 kxk ≥ 0; kxk = 0 if and only if x = 0

 ktxk = |t| kxk for t ∈ R
 kx + yk ≤ kxk + kyk

notation: k · k is general (unspecified) norm; k · ksymb is particular norm

norm ball with center xc and radius r: {x | kx − xck ≤ r}

norm cone: {(x, t) | kxk ≤ t} 0.5

Euclidean norm cone is called second- t 0

order cone 1
1
0 0
x2 −1 −1
x1

norm balls and cones are convex

A. d’Aspremont. Convex Optimization M2. 35/49

Polyhedra

solution set of finitely many linear inequalities and equalities

Ax  b, Cx = d

(A ∈ Rm×n, C ∈ Rp×n,  is componentwise inequality)

a1 a2

P
a5
a3

a4

polyhedron is intersection of finite number of halfspaces and hyperplanes

A. d’Aspremont. Convex Optimization M2. 36/49

Positive semidefinite cone

notation:
 Sn is set of symmetric n × n matrices
 Sn+ = {X ∈ Sn | X  0}: positive semidefinite n × n matrices

X ∈ Sn+ ⇐⇒ z T Xz ≥ 0 for all z

Sn+ is a convex cone

 Sn++ = {X ∈ Sn | X  0}: positive definite n × n matrices

  0.5
x y
z

example: ∈ S2+
y z
0
1
1
0
0.5
y −1 0 x

A. d’Aspremont. Convex Optimization M2. 37/49

Operations that preserve convexity

practical methods for establishing convexity of a set C

1. apply definition

x1, x2 ∈ C, 0≤θ≤1 =⇒ θx1 + (1 − θ)x2 ∈ C

2. show that C is obtained from simple convex sets (hyperplanes, halfspaces,

norm balls, . . . ) by operations that preserve convexity
 intersection
 affine functions
 perspective function
 linear-fractional functions

A. d’Aspremont. Convex Optimization M2. 38/49

Intersection

the intersection of (any number of) convex sets is convex

example:
S = {x ∈ Rm | |p(t)| ≤ 1 for |t| ≤ π/3}

where p(t) = x1 cos t + x2 cos 2t + · · · + xm cos mt

for m = 2:
2

1
1
p(t)

0
S
x2
0
−1

−1

−2
0 π/3 2π/3 π −2 −1 0 1 2
t x1

A. d’Aspremont. Convex Optimization M2. 39/49

Affine function

suppose f : Rn → Rm is affine (f (x) = Ax + b with A ∈ Rm×n, b ∈ Rm)

 the image of a convex set under f is convex

S ⊆ Rn convex =⇒ f (S) = {f (x) | x ∈ S} convex

 the inverse image f −1(C) of a convex set under f is convex

C ⊆ Rm convex =⇒ f −1(C) = {x ∈ Rn | f (x) ∈ C} convex

examples

 scaling, translation, projection

 solution set of linear matrix inequality {x | x1A1 + · · · + xmAm  B}
(with Ai, B ∈ Sp)
 hyperbolic cone {x | xT P x ≤ (cT x)2, cT x ≥ 0} (with P ∈ Sn+)

A. d’Aspremont. Convex Optimization M2. 40/49

Perspective and linear-fractional function

perspective function P : Rn+1 → Rn:

P (x, t) = x/t, dom P = {(x, t) | t > 0}

images and inverse images of convex sets under perspective are convex

linear-fractional function f : Rn → Rm:

Ax + b
f (x) = T
, dom f = {x | cT x + d > 0}
c x+d

images and inverse images of convex sets under linear-fractional functions are
convex

A. d’Aspremont. Convex Optimization M2. 41/49

example of a linear-fractional function

1
f (x) = x
x 1 + x2 + 1

1 1
x2

x2
0 C 0
f (C)

−1 −1
−1 0 1 −1 0 1
x1 x1

A. d’Aspremont. Convex Optimization M2. 42/49

Generalized inequalities

a convex cone K ⊆ Rn is a proper cone if

 K is closed (contains its boundary)

 K is solid (has nonempty interior)
 K is pointed (contains no line)

examples
 nonnegative orthant K = Rn+ = {x ∈ Rn | xi ≥ 0, i = 1, . . . , n}
 positive semidefinite cone K = Sn+
 nonnegative polynomials on [0, 1]:

K = {x ∈ Rn | x1 + x2t + x3t2 + · · · + xntn−1 ≥ 0 for t ∈ [0, 1]}

A. d’Aspremont. Convex Optimization M2. 43/49

generalized inequality defined by a proper cone K:

x K y ⇐⇒ y − x ∈ K, x ≺K y ⇐⇒ y − x ∈ int K

examples
 componentwise inequality (K = Rn+)

x Rn+ y ⇐⇒ x i ≤ yi , i = 1, . . . , n

 matrix inequality (K = Sn+)

X Sn+ Y ⇐⇒ Y − X positive semidefinite

these two types are so common that we drop the subscript in K

properties: many properties of K are similar to ≤ on R, e.g.,

x K y, u K v =⇒ x + u K y + v

A. d’Aspremont. Convex Optimization M2. 44/49

Minimum and minimal elements

K is not in general a linear ordering : we can have x 6K y and y 6K x

x ∈ S is the minimum element of S with respect to K if

y∈S =⇒ x K y

x ∈ S is a minimal element of S with respect to K if

y ∈ S, y K x =⇒ y=x

example (K = R2+)
S2
S1 x2
x1 is the minimum element of S1
x2 is a minimal element of S2 x1

A. d’Aspremont. Convex Optimization M2. 45/49

Separating hyperplane theorem

if C and D are disjoint convex sets, then there exists a 6= 0, b such that

aT x ≤ b for x ∈ C, aT x ≥ b for x ∈ D

aT x ≥ b aT x ≤ b

D
C

the hyperplane {x | aT x = b} separates C and D

strict separation requires additional assumptions (e.g., C is closed, D is a

singleton)

A. d’Aspremont. Convex Optimization M2. 46/49

Supporting hyperplane theorem

supporting hyperplane to set C at boundary point x0:

{x | aT x = aT x0}

where a 6= 0 and aT x ≤ aT x0 for all x ∈ C

x0
C

supporting hyperplane theorem: if C is convex, then there exists a supporting

hyperplane at every boundary point of C

A. d’Aspremont. Convex Optimization M2. 47/49

Dual cones and generalized inequalities

dual cone of a cone K:

K ∗ = {y | y T x ≥ 0 for all x ∈ K}

examples

 K = Rn+: K ∗ = Rn+
 K = Sn+: K ∗ = Sn+
 K = {(x, t) | kxk2 ≤ t}: K ∗ = {(x, t) | kxk2 ≤ t}
 K = {(x, t) | kxk1 ≤ t}: K ∗ = {(x, t) | kxk∞ ≤ t}

first three examples are self-dual cones

dual cones of proper cones are proper, hence define generalized inequalities:

y K ∗ 0 ⇐⇒ y T x ≥ 0 for all x K 0

A. d’Aspremont. Convex Optimization M2. 48/49

Minimum and minimal elements via dual inequalities

minimum element w.r.t. K

x is minimum element of S iff for all
λ K ∗ 0, x is the unique minimizer of S
λT z over S
x

minimal element w.r.t. K

 if x minimizes λT z over S for some λ K ∗ 0, then x is minimal

λ1

x1
S

λ2
x2

 if x is a minimal element of a convex set S, then there exists a nonzero

λ K ∗ 0 such that x minimizes λT z over S

A. d’Aspremont. Convex Optimization M2. 49/49

*

References

A. Ben-Tal and A. Nemirovski. Lectures on modern convex optimization : analysis, algorithms, and engineering applications. MPS-SIAM
series on optimization. Society for Industrial and Applied Mathematics : Mathematical Programming Society, Philadelphia, PA, 2001.
N. K. Karmarkar. A new polynomial-time algorithm for linear programming. Combinatorica, 4:373–395, 1984.
L. G. Khachiyan. A polynomial algorithm in linear programming (in Russian). Doklady Akademiia Nauk SSSR, 224:1093–1096, 1979.
A. Nemirovskii and D. Yudin. Problem complexity and method efficiency in optimization. Nauka (published in English by John Wiley,
Chichester, 1983), 1979.
Y. Nesterov. Introductory Lectures on Convex Optimization. Springer, 2003.
Y. Nesterov and A. Nemirovskii. Interior-point polynomial algorithms in convex programming. Society for Industrial and Applied
Mathematics, Philadelphia, 1994.

A. d’Aspremont. Convex Optimization M2. 50/49