ET4350: Applied Convex Optimization

Delft University of Technology

1
 Course Information

▶ Instructors:
- Prof. Geert Leus
- Prof. Borbala Hunyadi (Bori)
- MSc. Alberto Natali
▶ Class schedule (watch out for any changes on TU Delft roosters):
- Wednesdays between 10.45-12.30
- Fridays between 10.45-12.30

2
 Course Information

I Book(s) are freely available online

– Stephen Boyd and Lieven Vandenberghe, ”Convex Optimization”,
Cambridge University Press, 2004.
– Slides/lecture notes for subgradient methods.

I Assessment
– Open-book written exam.
– Compulsory lab assignment worth 1 EC (20%); report and short
presentation. Enroll via Brightspace.

I Course information:
– http://ens.ewi.tudelft.nl/Education/courses/ee4530/index.php

3
3
 Mathematical optimization

(mathematical) optimization problem

minimize f0(x)
subject to fi(x) ≤ bi, i = 1, . . . , m

• x = (x1, . . . , xn): optimization variables

• f0 : Rn → R: objective function

• fi : Rn → R, i = 1, . . . , m: constraint functions

optimal solution x⋆ has smallest value of f0 among all vectors that

satisfy the constraints

4
 θ

Array processing
Phased-array antenna beamforming

• omnidirectional antenna elements at positions (x1, y1),

(xi, yi)

• unit plane wave incident from angle θ induces in ith ele

θ
ej(xi cos θ+yi sin θ−ωt)
√
(j = −1, frequency ω, wavelength 2π)

•• demodulate
omnidirectional antenna
to get elements
output ej(xi cosatθ+y
positions
i sin θ) ∈(x
C1, y1), . . . , (xn, yn)

•• linearly
unit plane wave incident
combine from angle
with complex
j(xi cos θ+yi sin θ−ωt)
θ induces
weights wi: in ith element a signal
e
50◦
√ n |y(θ)|
(j = −1, frequency ω, wavelength ! 2π) ❅

y(θ) = wiej(xi cos θ+yi sin θ) ❅

❘
❅
θtar = 30◦

i=1
10◦
sidelobe
15 level
❅
❘
❅

• y(θ) is (complex) antenna array gain pattern

• |y(θ)| gives sensitivity of array as function of incident angle θ

• depends on design variables Re w, Im w

(called antenna array weights or shading coefficients)
18

design problem: choose w to achieve desired gain pattern

5
 Sidelobe level minimization
Array processing
make |y(θ)|level
Sidelobe for |θ − θtar | > α
small minimization

make |y(θ)| small for(θ|θ : target

tar− θtar | >direction;
α 2α: beamwidth)

via2α:
(θtar : target direction; beamwidth) (discretize angles)
least-squares
50◦
2
!
|y(θ)| via least-squares (discretize angles) minimize i |y(θi )|
❅
θtar = 30◦
❅
❘
❅ subject to y(θtar ) = 1
2
!
10◦ minimize i |y(θi )|
sidelobe level
❅
❘
❅ (sum subject
is over angles
to y(θoutside
tar ) = 1beam)

(sum is over angles outside beam)

least-squares problem with two (real) linear equality constraints

least-squares problem with two (real) linear equality constraints

18

50◦
|y(θ)|
minimize sidelobe level (discretize angles) ❅
❅ θtar = 30◦ 17
❘
❅

10◦
minimize maxi |y(θi)| sidelobe level
❅
❘
❅
subject to y(θtar ) = 1

(max over angles outside beam)

can be cast as SOCP

6
20
Machine learning
Linear discrimination
separate two sets of points {x1, . . . , xN }, {y1, . . . , yM } by a hyperplane:

aT xi + b > 0, i = 1, . . . , N, aT yi + b < 0, i = 1, . . . , M

homogeneous in a, b, hence equivalent to

aT xi + b ≥ 1, i = 1, . . . , N, aT yi + b ≤ −1, i = 1, . . . , M

a set of linear inequalities in a, b

Geometric problems 8–8 7

 Robust linear discrimina
Machine learning

Robust(Euclidean)
linear discrimination
distance between hyperplanes

8–9
(1)
H1 = {z | aT z + b = 1}
H2 = {z | aT z + b = −1}
(Euclidean) distance between hyperplanes

H1 = {z | aT z + isb = 1} 1, H2) = 2/∥a∥2

dist(H

aT yi + b ≤ −1, i = 1, . . . , M
H2 = {z | aT z + b = −1}

aT xi + b ≥ 1, i = 1, . . . , N
st linear discrimination

oints by maximum margin, to separate two sets of points by maximum margin

is dist(H1, H2) = 2/∥a∥2
minimize (1/2)∥a∥2
subject to aT xi + b ≥ 1, i = 1,
to separate two sets of points by maximum margin, aT yi + b ≤ −1, i =
een hyperplanes

a QP in a, b
(1/2)∥a∥2
+ b = −1}

minimize (1/2)∥a∥
(after squaring
2 objective) a QP in a, b
+ b = 1}

T
subject to a xi + b ≥ 1, i = 1, . . . , N (1)
T
aGeometric b ≤ −1, i = 1, . . . , M
yi +problems
8
2
 Examples
portfolio optimization
• variables: amounts invested in different assets
• constraints: budget, max./min. investment per asset, minimum return
• objective: overall risk or return variance

device sizing in electronic circuits

• variables: device widths and lengths
• constraints: manufacturing limits, timing requirements, maximum area
• objective: power consumption

data fitting
• variables: model parameters
• constraints: prior information, parameter limits
• objective: measure of misfit or prediction error
9
 Solving optimization problems

general optimization problem

• very difficult to solve

• methods involve some compromise, e.g., very long computation time, or
not always finding the solution

exceptions: certain problem classes can be solved efficiently and reliably

• least-squares problems
• linear programming problems
• convex optimization problems

10
 Least-squares

minimize ∥Ax − b∥22

solving least-squares problems

• analytical solution: x⋆ = (AT A)−1AT b

• reliable and efficient algorithms and software
• computation time proportional to n2k (A ∈ Rk×n); less if structured
• a mature technology

using least-squares

• least-squares problems are easy to recognize

• a few standard techniques increase flexibility (e.g., including weights,
adding regularization terms)
11
 Linear programming

minimize cT x
subject to aTi x ≤ bi, i = 1, . . . , m

solving linear programs

• no analytical formula for solution
• reliable and efficient algorithms and software
• computation time proportional to n2m if m ≥ n; less with structure
• a mature technology

using linear programming

• not as easy to recognize as least-squares problems
• a few standard tricks used to convert problems into linear programs
(e.g., problems involving ℓ1- or ℓ∞-norms, piecewise-linear functions)
12
 Convex optimization problem

minimize f0(x)
subject to fi(x) ≤ bi, i = 1, . . . , m

• objective and constraint functions are convex:

fi(αx + βy) ≤ αfi(x) + βfi(y)

if α + β = 1, α ≥ 0, β ≥ 0

• includes least-squares problems and linear programs as special cases

13
 The case of a convex cost function

Local minima: x⋆ is an unconstrained local minimum of f0 : Rn !→ R if

is no worse than its neighbors.

f0 (xThe
⋆
) ≤ case
f0 (x),of a∀x
convex
∈ Rn cost
with function
∥x − x⋆ ∥ < ϵ

for ϵ > 0.
Local minima: x⋆ is an unconstrained local minimum of f0 : Rn !→ R if
is no worse than its neighbors.
n
Global minima: xf0⋆(xis⋆ )an unconstrainedn local minimum
⋆
≤ f0 (x), ∀x ∈ R with ∥x − x ∥ < ϵ of f 0 : R !→ R if
is no worse than all other vectors.
for ϵ > 0.
f0 (x⋆ ) ≤ f0 (x), ∀x ∈ Rn .
Global minima: x⋆ is an unconstrained local minimum of f0 : Rn !→ R if
it is no worse than all other vectors.
When the function is convex every local minimum is also global.
f0 (x⋆ ) ≤ f0 (x), ∀x ∈ Rn .

When the function is convex every local minimum is also global. 14 7

solving convex optimization problems

• no analytical solution
• reliable and efficient algorithms
• computation time (roughly) proportional to max{n3, n2m, F }, where F
is cost of evaluating fi’s and their first and second derivatives
• almost a technology

using convex optimization

• often difficult to recognize

• many tricks for transforming problems into convex form
• surprisingly many problems can be solved via convex optimization

15
 Example

m lamps illuminating n (small, flat) patches

lamp power pj

rkj
θkj

illumination Ik

intensity Ik at patch k depends linearly on lamp powers pj :

m
!
−2
Ik = akj pj , akj = rkj max{cos θkj , 0}
j=1

problem: achieve desired illumination Ides with bounded lamp powers

minimize maxk=1,...,n | log Ik − log Ides|

subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m
16
how to solve?
1. use uniform power: pj = p, vary p
2. use least-squares:
!n
minimize k=1 (Ik − Ides)2

round pj if pj > pmax or pj < 0

3. use weighted least-squares:
!n 2
!m 2
minimize k=1 (I k − I des ) + j=1 w j (p j − p max /2)

iteratively adjust weights wj until 0 ≤ pj ≤ pmax

4. use linear programming:
minimize maxk=1,...,n |Ik − Ides|
subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m

which can be solved via linear programming

of course these are approximate (suboptimal) ‘solutions’

17
5. use convex optimization: problem is equivalent to

minimize f0(p) = maxk=1,...,n h(Ik /Ides)

subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m

with h(u) = max{u, 1/u}

5

3
h(u)

0
0 1 2 3 4
u

f0 is convex because maximum of convex functions is convex

exact solution obtained with effort ≈ modest factor × least-squares effort

18
additional constraints: does adding 1 or 2 below complicate the problem?

1. no more than half of total power is in any 10 lamps

2. no more than half of the lamps are on (pj > 0)

• answer: with (1), still easy to solve; with (2), extremely difficult
• moral: (untrained) intuition doesn’t always work; without the proper
background very easy problems can appear quite similar to very difficult
problems

19
 Course goals and topics

Goals
1. recognize and formulate problems (such as the illumination problem,
classification, etc.) as convex optimization problems
2. Use optimization tools (CVX, YALMIP, etc.) as a part the lab
assignment.
3. characterize optimal solution (optimal power distribution), give
limits of performance, etc.
Topics
1. Background and optimization basics;
2. Convex sets and functions;
3. Canonical convex optimization problems (LP, QP, SDP);
4. Second-order methods (unconstrained and constrained optimization);
5. First-order methods (gradient, subgradient);

5
20
 1 Project 1: Change Detection in Time Series Model
Context
In statistical signal processing, change point detection tries to identify time
instances when the probability distribution of a stochastic process or time
Time Signal
series changes. The problem in this assignment AR concerns both detecting
Coefficients
4
whether or not a change has occurred, or whether several changes might
1
3
have occurred, and identifying the times0.8
of any such changes.
2 This exercise consists of two parts:0.6 (a) formulate the step detection
1 problem as a suitable convex optimization
0.4 problem; and (b) implement the
0.2
0 change detector. In a group of 2 students,
0
make a short report (4-5 pages;
y

−1 pdf file) containing the required Matlab scripts, plots, and answers. Also,
−0.2
−2 prepare a short presentation to explain your results and defend your choices.
−0.4
−0.6
−3
−0.8
−4

−5
Dataset explanation −1

0 50 100 150 200 250 300 50 100 150 200 250 300
t t
Consider the following scalar autoregressive (AR) time-series model

y(t + 3) = a(t)y(t + 2) + b(t)y(t + 1) + c(t)y(t) + v(t); v(t) s N (0, 0.52 ).

The assumption
assumption: a(t), b(t),isand
that
c(t)the
are AR coefficients
piecewise are piecewise constant and change
constant, change
infrequently.
infrequently Given y(t), t = 1, . . . , T, the problem is to estimate a(t), b(t),
 Project 2: Linear Support Vector Machines

10

8
Decision Boundary

6 Class A

−2 Class B

−4

−6
0 0.5 1 1.5 2 2.5 3 3.5 4
 Project 3: Multidimensional Scaling for Localization.
assumption: a(t), b(t), and c(t) are piecewise constant, cha
53.5
infrequently
53 X Real Positions
Estimated Positions A R M V G
assumption: a(t), b(t), and c(t) are piecewise
⎛ constant, change ⎞
52.5
infrequently A 0 71 146 177 127
R ⎜
⎜ 71 0 136 104 159 ⎟
M⎜
⎟
⎜ 146 136 0 208 258
52 ⎟
V ⎝ 177 ⎞
⎟
⎛ 104 208 0 279 ⎠
0 71 146 177 G 127 159 258 279
51.5
0
⎜ 71 0 136 104 159 ⎟
51 ⎜ ⎟
⎜ 146 136 0
2 208 258 ⎟
⎝ 177 104 208tij ∝0∥xi 279
⎜ ∥2
− xj ⎟
⎠
50.5
T
X) − 2XT X + diag(XT X)1T
3 4 5 6 7 8
= 1diag(X
127 159 258T 279 0

t2ij ∝ ∥xi − xj ∥2
T = 1diag(XT X) − 2XT X + diag(XT X)1T
 ymbols. This is basically an inverse problem with a finite-alphabet
nt. Project 4: MIMO Detection
exercise consists of two parts:Dataset explanation
(a) formulate the MIMO detection
as a suitable convex optimization problem;
Consider a genericand (b) Mimplement
N -input -output modelthe
receiver. In a group of 2 students, make a short report (4-5 pages;
yc = Hc sc + vc .
containing the required Matlab scripts, plots, and answers. Also,
M is the received vector, H 2 CM ⇥
a short presentation to explain Here, c 2 C and
youryresults defend your choices.
c
nel, sc 2 C is the transmitted symbol vector, and v
N

1
white Re(s)
Gaussian noise vector. In this application exam
transmitted symbols follow a quaternary phase-shift-k
set explanation0.5

0 lation; i.e., the entries of sc belong to the finite-alpha

−0.5 dataset MIMODetection.mat in the course webpage
r a generic N -input M -output model
−1

−1.5
0 5 10 15 20 25 30 35 40

Im(s)
yc = Hc sc + v
1 c.
0.5

c 2 CM is the received vector, Hc 2 CM ⇥N is the MIMO chan-

0

−0.5

2 CN is the transmitted symbol vector, and vc 2 CM is an additive

−1

aussian noise vector. In this application example we assume that the

−1.5
0 5 10 15 20 25 30 35 40

tted symbols follow a quaternary phase-shift-keying (QPSK) constel-

e., the entries of sc belong to the finite-alphabet set {±1 ± j}. The
 Project 5: Compressed Sensing

FFT

Time-domain signal Frequency-domain signal

(Positivity, sparsity?)
Sampling Mask

Recover time-domain signal

Subsampled frequency-domain signal