MATH3161 Lecture 1

Topic and contents
School of Mathematics and Statistics

MATH3161/MATH5165 Optimization
Prof Jeya Jeyakumar
Topic 01 Optimization What is it?

1 Convex sets
Convex sets
Intersections
Extreme points
Convex combinations
Separating Hyperplane Theorem
2 Convex and concave functions
Convex functions
Concave functions
3 Global solutions
Convex programming problem
Concave programming problem
Linear programming
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 1 / 18
Optimization Modelling
Modelling
Denition (Optimization/Optimisation)
Optimization is nding the best way of doing something, subject to any
restrictions on what you can do.
Mathematical optimization is also known as
Mathematical (linear/nonlinear) programming
Optimal control
Meaning of best = concept of ordering
Business/Economics:
Maximize: prot, return, utility. . .
Minimize: cost, risk, . . .
Science/Biology/Chemistry/Physics:
Maximize: product, reaction rate, . . .
Minimize: energy, heat, . . .
Engineering
Maximize: strength, production, . . .
Minimize: cost, materials, time, . . .
Optimization Background
Background
Mathematical model
Variables
Objective
Constraints
Characterising optima Optimality conditions
What is an optimum/optimizer?
Formulae for identifying/characterizing optima
Derivatives I Newton (1642 1726), G W Leibniz (1646 1711)
Analytic methods for nding optima
Fermat (1601 1665), L Euler (1707 1783), Lagrange (1736 1813)
Finding optima = Numerical methods
Newtons method: I Newton (1642 1726), C F Gauss (1777 1855)
Linear programming: L Kantorovich (1912 1986), G B Dantzig (1914 2005)
Computer algorithms Linear algebra
Convexity R T Rockafellar (1938 )
Duality J von Neumann (1903 1957)
Maximum principle L S Pontryagin (1908-1988)
Optimization Variables
Variables
Decision variables: what can you change
Finite dimensional x R
n
, Number of variables n
Column vector
x =
_
_
x
1
x
2
.
.
.
x
n
_
_
Univariate: n = 1, x R
Multivariate: n = 2, 3, 4, . . . up to millions
Mathematical background: column vectors, symmetric matrices, matrix
operations
Discrete optimization: x
i
{0, 1}, x
i
N, x
i
Z
Matrix of variables X R
mm
=x R
m
2
Innite dimensional: the control
A function u C([a, b])
A function u L
1
([0, T])
Optimization Objective
Objective
A mathematical function of the variables quantifying the idea of best.
Finite dimensional: variables x R
n
, Objective function f : R
n
R
Linear f(x) = g
T
x, g R
n
xed
Ane f(x) = g
T
x + f
0
, g R
n
, f
0
R xed
Quadratic f(x) =
1
2
x
T
Gx + g
T
x + f
0
, G R
nn
, g R
n
, f
0
R
Innite dimensional: f : C([0, T] R, variables u C([0, T])
f(u) =
_
T
0
u(t)dt
Co-domain of objective function must be ordered (total order)
If , , R, then
and =
and =
Either or
If u, v R
n
, n 2,
u v v u 0 (v
i
u
i
) 0, i = 1, . . . , n
Optimization Constraints
Constraints
Constraints: Describe restrictions on the allowable values of variables
Constraint structure for variables x R
n
Equality constraints c
i
(x) = 0, i = 1, . . . , m
E
Inequality constraints c
i
(x) 0, i = m
E
+ 1, . . . , m
Feasible region R
n
= {x R
n
: c
i
(x) = 0, i = 1, . . . , m
E
;
c
i
(x) 0, i = m
E
+ 1, . . . , m}
Unconstrained problem = R
n
Standard formulation c
i
(x) 0 c
i
(x) 0
Algebraic structure of constraints
Simple bounds: x u
i
x
i
u
i
, i = 1, . . . , n
Linear constraints c
i
(x) = a
T
i
x b
i
Nonlinear constraints
Optimization Constraints
Constraint representation
Example (Constraint representation)
What feasible regions do the following constraints represent?
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
= r
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
r
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
r
x
2
= x
2
1
x
2
x
2
1
x
2
1
1 = 0
Optimization Standard formulation
Standard formulation
Denition (Standard formulation)
The standard formulation of a continuous nite dimensional optimization is
Minimize f(x)
x R
n
subject to c
i
(x) = 0, i = 1, . . . , m
E
;
c
i
(x) 0, i = m
E
+ 1, . . . , m
Conversions
Maximize to minimize: max

f(x) = min
f(x)
Constraint right-hand-side: c
i
(x) = b
i
c
i
(x) b
i
= 0
Less than or equal to inequalities: c
i
(x) 0 c
i
(x) 0
Strict inequality: c
i
(x) < 0 c
i
(x) + 0, some > 0
A simplied farm planning problem
Example (Farm Planning)
Farmer Jack has 100 acres to devote to wheat and corn and wishes to plan
his planting to maximize the expected revenue. Jack has only $800 in
capital to apply to planting the crops, and it costs $5 to plant an acre of
wheat and $10 for an acre of corn. Their other activities leave the Jack
family only 150 days of labour to devote to the crops. Two days will be
required for each acre of wheat and one day for an acre of corn. If past
experience indicates a return of $40 from each acre of wheat and $30 from
each acre of corn, how many acres of each should be planted to maximize
his revenue?
Pose this as an optimization problem in standard form.
Post Oce Parcel problem
Example (Post Oce Parcel Problem)
At one time the post oce regulations were that
the length plus the girth of a parcel must not exceed 1.8 metres.
What is the parcel of largest volume that could be sent through the post?
Pose this as an optimization problem in standard form. Assume that
The parcel has rectangular sides
The length of the parcel is the longest edge
The girth is the distance around the parcel perpendicular to the
length. For a rectangular box, girth is 2(height + depth).
x
1
x
2
x
3
Standard formulation Example 2
Example (Standard formulation)
Maximize x
2
1
(x
2
1)
2
(x
2
3)
2
x
2
/2 on the set
= {x R
2
: x
2
x
2
1
, x
1
1, x
2
2 + x
1
, x
2
4 x
1
}
Write this problem in standard form and plot the feasible region .
What is the feasible region if the rst constraint becomes x
2
= x
2
1
?
2 1.5 1 0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
3.5
4
Mathematical background Vector norms
Measures of size - norms
If R then its magnitude (or absolute value) || is
|| =
_
if 0;
if 0.
In R
n
there are several possible denitions norms denoted using || ||.
Denition (Vector norm)
A vector norm on R
n
is a function ||.|| from R
n
to R such that
1
x 0 for all x R
n
and x = 0 x = 0.
2
x + y x +y for all x, y R
n
. (Triangle inequality)
3
x = || x for all R, x R
n
.
Example (Vector norms)
The most widely used vector norms are
1-norm x
1
=
n
i=1
|x
i
|
2-norm x
2
=
_
n
i=1
|x
i
|
2
_1
2
= (x
T
x)
1
2
-norm or maximum norm x
= max
i=1,...,n
|x
i
|
Mathematical background Optima and optimizers
Local and global minima
Denition (Global minimum)
A point x
is a global minimizer of f(x) over R

n
f(x
) f(x) for all x . The global minimum is f(x
).
Denition (Strict global minimum)
A point x
is a strict global minimizer of f(x) over R

n
f(x
) < f(x) for all x , x = x
.
Denition (Local minimum)
A point x
is a local minimizer of f(x) over R

n
there exists a > 0 such that f(x
) f(x) for all x with

x x
. Then f(x
) is a local minimum.
Denition (Strict local minimum)
A point x
is a strict local minimizer of f(x) over R

n
) < f(x) for all x with

0 < x x
.
Local and global maxima
Denition (Global maximum)
A point x
is a global maximizer of f(x) over R

n
f(x
) f(x) for all x . The global maximum is f(x
).
Denition (Strict global maximum)
A point x
is a strict global maximizer of f(x) over R

n
f(x
) > f(x) for all x , x = x
.
Denition (Local maximum)
A point x
is a local maximizer of f(x) over R

n
) f(x) for all x with

x x
. Then f(x
) is a local maximum.
Denition (Strict local maximum)
A point x
is a strict local maximizer of f(x) over R

n
) > f(x) for all x with

0 < x x
.
Local and global minima and maxima
Example (Local and global minima and maxima)
= [0, 5], f(x) =
_
_
(x 1)
2
if x 0.5;
0.25 if 0.5 < x 1;
1.25 (x 2)
2
if 1 < x 2.5;
x 1.5 if 2.5 < x 3;
1.5 0.25 sin((x 3)) if 3 < x.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Local and global extrema
x
f(x)
Local and global minima and maxima
Solution (Local and global minima and maxima)
Consider f(x) on the interval = [0, 5] and the points
0, 0.5, 2, 2.5, 3, 3.5, 4.5, 5
x
(a)
= 0 is strict local maximizer with f
(a)
= 1;
any point x
(b)
in the interval [0.5, 1] is a local and global minimizer
with f
(b)
= 0.25 (but not strict as adjacent points have the same
function value);
x
(c)
= 2 is a strict local maximizer with f
(c)
= 1.25;
x
(d)
= 2.5 is a strict local minimizer with f
(d)
= 1;
x
(e)
= 3 is a strict local maximizer with f
(e)
= 1.5;
x
(f)
= 3.5 is a strict local minimizer with f
(f)
= 1.25;
x
(g)
= 4.5 is a strict local and global maximizer with f
(g)
= 1.75;
x
(h)
= 5 is a strict local minimizer with f
(h)
= 1.5.
Mathematical background Existence
Existence
Denition (Extrema)
The global/local extrema of f over are all the global/local minima and
all the global/local maxima
Proposition (Existence of global extrema)
Let be a compact set and let f be continuous on . Then the global
extrema of f over exist.
Finite dimensional R
n
is compact is closed and bounded
Example (Existence)
Find the global extrema, if they exist, for the following problems
f(x) = e
x
on = [0, 1]
f(x) = e
x
on = [0, )
f(x) = sin x on = [0, 2)
f(x) = cos x on = [0, )
Mathematical background Relaxation
Relaxation
Proposition (Relaxation)
If f : R
n
R and

then
min
x
f(x) min
x
f(x)
Proof.
Let x

and f(x
) be the global minimizer and minimum of f over

.
As

, x

=x
. Thus min
x
f(x) f(x
).
If you make the feasible region larger then the minimum value of
objective function must not increase
If you make the feasible region larger then the maximum value of
objective function must not decrease
Mathematical background Derivatives
Gradients
Denition (Gradient)
Let f : R
n
R be continuously dierentiable.
The gradient f : R
n
R
n
of f at x is
f(x) =
_
_
f(x)
x1
f(x)
x2
.
.
.
f(x)
xn
_
_
The gradient is a column vector with n elements
The gradient f(x) of f at x is orthogonal to the contour
{x R
n
: f(x) = f(x)}
Hessians
Denition (Hessian)
Let f : R
n
R be continuously dierentiable. The Hessian
2
f : R
n
R
nn
of f at x is
2
f(x) =
_
2
f(x)
x
2
1
2
f(x)
x1x2

2
f(x)
x1xn
2
f(x)
x2x1
2
f(x)
x
2
2

2
f(x)
x2xn
.
.
.
.
.
.
.
.
.
.
.
.
2
f(x)
xnx1
2
f(x)
xnx2

2
f(x)
x
2
n
_
_
The Hessian
2
f(x) is an n by n matrix
If f is twice continuously dierentiable at x then
2
f(x)
x
i
x
j
=

2
f(x)
x
j
x
i
for all i = j
That is the Hessian matrix G =
2
f(x) is symmetric (G
T
= G)
Gradient and Hessian Example
Find the gradient and Hessian of
f(x) = 2x
2
1
3x
2
2
+ 4x
1
x
2
+ 2x
1
+ 6x
2
+ 8.
Gradient and Hessian Exercise
Example (Gradients and Hessians)
Let f(x) = x
2
1
+ (x
2
1)
2
(x
2
3)
2
+ x
2
/2 and
c
1
(x) = x
2
1
+ x
2
,
c
2
(x) = x
1
+ 1,
c
3
(x) = x
1
+ x
2
2,
c
4
(x) = x
1
x
2
+ 4.
For each function f(x), c
i
(x), i = 1, 2, 3, 4
Find the gradient and Hessian
Determine if the function is linear, quadratic or nonlinear
Example 1 Plots
2 1.5 1 0.5 0 0.5 1 1.5 2
0.4
0.1
0.6
1.1
1.6
2.1
2.6
3.1
3.6
x
1
x
2
x
1
2
+ (x
2
1)
2
(x
2
3)
2
+ x
2
/2
21
15
9
6
4 3
2.5
2
1.75 2
2.5
2
1.75
2
3
4
6
9
1.5
0.75
1
0.5
1.25
1.5
Linear and Quadratic functions
Example (Linear and Quadratic functions)
Let f
0
R, g R
n
and G R
nn
, G symmetric, be xed.
Find the gradient f(x) and Hessian
2
f(x) for the
Linear function f(x) = g
T
x + f
0
Quadratic function f(x) =
1
2
x
T
Gx + g
T
x + f
0
Solution
f(x) = g
T
x + f
0
=
n
i=1
g
i
x
i
+ f
0
f(x) = g, does not depend on x
2
f(x) = 0 R
nn
, n by n zero matrix
f(x) =
1
2
x
T
Gx + g
T
x + f
0
=
1
2
n
i=1
n
j=1
x
i
G
ij
x
j
+
n
i=1
g
i
x
i
+ f
0
f(x) = Gx + g
2
f(x) = G, does not depend on x
Mathematical background Positive denite matrices
Positive denite matrices Denition
Denition
A real square matrix A R
nn
is
positive denite x
T
Ax > 0 for all x R
n
, x = 0
positive semi-denite x
T
Ax 0 for all x R
n
negative denite x
T
Ax < 0 for all x R
n
, x = 0
negative semi-denite x
T
Ax 0 for all x R
n
indenite there exist x, y R
n
: x
T
Ax > 0 and y
T
Ay < 0
Generalization of nonnegative (order) to symmetric matrices:
A B AB 0 AB positive semidenite
Theoretical denition, not a practical test
A is negative denite A is positive denite
Positive denite matrices Eigenvalues
A symmetric matrix A R
nn
has n real eigenvalues
i
, i = 1, . . . , n
there exists an orthogonal matrix Q (Q
T
Q = I) such that
A = QDQ
T
where D = diag(
1
, . . . ,
n
) and Q = [v
1
v
2
v
n
]
where v
i
is an eigenvector of A corresponding to eigenvalue
i
Determinant: det (A) =
n
i=1

i
Trace: trace (A) :=
n
i=1
a
ii
=
n
i=1

i
Proposition
nn
is
positive denite
i
> 0 for all i = 1, . . . , n
positive semi-denite
i
0 for all i = 1, . . . , n
negative denite
i
< 0 for all i = 1, . . . , n
negative semi-denite
i
0 for all i = 1, . . . , n
indenite there exist i, j :
i
> 0 and
j
< 0
Positive denite matrices Cholesky factorization
Proposition (Cholesky factorization)
nn
is positive denite the Cholesky
factorization A = R
T
R exists, R a nonsingular upper triangular matrix
Proof.
= Suppose the Cholesky factorization A = R
T
R exists. Then
x
T
Ax = x
T
(R
T
R)x = (Rx)
T
(Rx) = y
T
y =
n
i=1
y
2
i
0
where y = Rx. Moreover, as R is nonsingular,
x
T
Ax = 0 y = 0 Rx = 0 x = 0.
= Text book on Linear Algebra, eg Golub and van Loan
Matlab chol

MATH3161 Lecture 1

Uploaded by

Copyright:

Available Formats

MATH3161 Lecture 1

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MATH3161 Lecture 1

Uploaded by

Copyright:

Available Formats

Topic and contents

School of Mathematics and Statistics

Topic 01 Optimization What is it?

is a global minimizer of f(x) over R

) f(x) for all x . The global minimum is f(x

is a strict global minimizer of f(x) over R

) < f(x) for all x , x = x

is a local minimizer of f(x) over R

there exists a > 0 such that f(x

) f(x) for all x with

is a strict local minimizer of f(x) over R

there exists a > 0 such that f(x

) < f(x) for all x with

is a global maximizer of f(x) over R

) f(x) for all x . The global maximum is f(x

is a strict global maximizer of f(x) over R

) > f(x) for all x , x = x

is a local maximizer of f(x) over R

there exists a > 0 such that f(x

) f(x) for all x with

is a strict local maximizer of f(x) over R

there exists a > 0 such that f(x

) > f(x) for all x with

) be the global minimizer and minimum of f over

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.