Optimization Techniques

Amrita Vishwa Vidyapeetham

September 21, 2020

Outline

Introduction
What is Optimization?
Motivation
Mathematical Aspects
Outline

Introduction
What is Optimization?
Motivation
Mathematical Aspects
Introduction
I Optimization is the act of obtaining the best result under given
circumstances. In design, construction, and maintenance of any
engineering system, engineers have to take many technological and
managerial decisions at several stages. The ultimate goal of all such
decisions is either to minimize the effort required or to maximize
the desired benefit. Since the effort required or the benefit desired
in any practical situation can be expressed as a function of certain
decision variables, optimization can be defined as the process of
finding the conditions that give the maximum or minimum value of
a function.
I Many engineers and researchers in industries and academics face
difficulty in understanding the role of optimization in engineering
design. In most engineering design activities; the design objective
could be simply to minimize the cost of production or to maximize the
efficiency of production. An optimization algorithm is a procedure
which is executed iteratively by comparing various solutions till the
optimum or a satisfactory solution is found. In many industrial design
activities, optimization is achieved indirectly by comparing a few
chosen design solutions and accepting the best solution.
Who??
Fermat and Lagrange found calculus-based formulae for identifying optima,
while Newton and Gauss proposed iterative methods for moving towards an
optimum.
The term “linear programming” for certain optimization cases was due to
George B. Dantzig, although much of the theory had been introduced by
Leonid Kantorovich in 1939. (Programming in this context does not refer to
computer programming, but comes from the use of program by the United
States military to refer to proposed training and logistics schedules, which
were the problems Dantzig studied at that time.) Dantzig published the
Simplex algorithm in 1947, and John von Neumann developed the theory of
duality in the same year.
Who??
Fermat and Lagrange found calculus-based formulae for identifying optima,
while Newton and Gauss proposed iterative methods for moving towards an
optimum.
The term “linear programming” for certain optimization cases was due to
George B. Dantzig, although much of the theory had been introduced by
Leonid Kantorovich in 1939. (Programming in this context does not refer to
computer programming, but comes from the use of program by the United
States military to refer to proposed training and logistics schedules, which
were the problems Dantzig studied at that time.) Dantzig published the
Simplex algorithm in 1947, and John von Neumann developed the theory of
duality in the same year.
George Dantzig Leonid Kantorovich
(1914-2005) (1912-1976)
Outline

Introduction
What is Optimization?
Motivation
Mathematical Aspects
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
I Design of water resources systems for maximum benefit.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
I Design of water resources systems for maximum benefit.
I Optimum design of electrical machinery such as motors, generators,
and transformers.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
I Design of water resources systems for maximum benefit.
I Optimum design of electrical machinery such as motors, generators,
and transformers.
I Optimum design of electrical network.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
I Design of water resources systems for maximum benefit.
I Optimum design of electrical machinery such as motors, generators,
and transformers.
I Optimum design of electrical network.
I Allocation of resources or services among several activities to
maximize the benefit.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
I Design of water resources systems for maximum benefit.
I Optimum design of electrical machinery such as motors, generators,
and transformers.
I Optimum design of electrical network.
I Allocation of resources or services among several activities to
maximize the benefit.
I Optimum design of control systems.
What are the engineering applications of optimization?
Optimization, in its broadest sense, can be applied to solve any engineering
problem. Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
I Design of aircraft and aerospace structures for minimum weight.
I Finding the optimal trajectories of space vehicles.
I Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost.
I Design of water resources systems for maximum benefit.
I Optimum design of electrical machinery such as motors, generators,
and transformers.
I Optimum design of electrical network.
I Allocation of resources or services among several activities to
maximize the benefit.
I Optimum design of control systems.
I Design of optimum pipeline networks for process industries.
Outline

Introduction
What is Optimization?
Motivation
Mathematical Aspects
How to define a maximum or minimum of the function?

B Local minimum : A function of one variable f (x) is said to have a

relative or local minimum at x = x∗ if f (x∗ ) ≤ f (x∗ + h) for all
sufficiently small positive and negative values of h.
How to define a maximum or minimum of the function?

B Local minimum : A function of one variable f (x) is said to have a

relative or local minimum at x = x∗ if f (x∗ ) ≤ f (x∗ + h) for all
sufficiently small positive and negative values of h.
B Local maximum : Similarly, a point x∗ is called a relative or local
maximum if f (x∗ ) ≥ f (x∗ + h) for all values of h sufficiently close to
zero.
How to define a maximum or minimum of the function?

B Local minimum : A function of one variable f (x) is said to have a

relative or local minimum at x = x∗ if f (x∗ ) ≤ f (x∗ + h) for all
sufficiently small positive and negative values of h.
B Local maximum : Similarly, a point x∗ is called a relative or local
maximum if f (x∗ ) ≥ f (x∗ + h) for all values of h sufficiently close to
zero.
B Global minimum : A function f (x) is said to have a global or absolute
minimum at x∗ if f (x∗ ) ≤ f (x) for all x, and not just for all x close to x∗ ,
in the domain over which f (x) is defined.
How to define a maximum or minimum of the function?

B Local minimum : A function of one variable f (x) is said to have a

relative or local minimum at x = x∗ if f (x∗ ) ≤ f (x∗ + h) for all
sufficiently small positive and negative values of h.
B Local maximum : Similarly, a point x∗ is called a relative or local
maximum if f (x∗ ) ≥ f (x∗ + h) for all values of h sufficiently close to
zero.
B Global minimum : A function f (x) is said to have a global or absolute
minimum at x∗ if f (x∗ ) ≤ f (x) for all x, and not just for all x close to x∗ ,
in the domain over which f (x) is defined.
B Global maximum : Similarly, a point x∗ will be a global maximum of
f (x) if f (x∗ ) ≥ f (x) for all x in the domain.
Graphical Representation
Necessary and sufficient conditions:

Necessary condition:

Theorem
If a function f (x) is defined in the interval a ≤ x ≤ b and has a relative
minimum or maximum at x = x∗ , where a < x∗ < b, and if the derivative
df
= f 0 (x) exists as a finite number at x = x∗ , then f 0 (x∗ ) = 0.
dx
Necessary and sufficient conditions:

Necessary condition:

Theorem
If a function f (x) is defined in the interval a ≤ x ≤ b and has a relative
minimum or maximum at x = x∗ , where a < x∗ < b, and if the derivative
df
= f 0 (x) exists as a finite number at x = x∗ , then f 0 (x∗ ) = 0.
dx
Sufficient condition:
Theorem
Let f 0 (x∗ ) = f 00 (x∗ ) = = f (n−1) (x∗ ) = 0, but f (n) (x∗ ) 6= 0. Then f (x∗ ) is (i)
a minimum value of f (x) if f (n) (x∗ ) > 0 and n is even; (ii) a maximum value
of f (x) if f (n) (x∗ ) < 0 and n is even; (iii) neither a maximum nor a minimum
if n is odd.
Examples
Example
1. Check whether f (x) = x3 attains optimum points or not.
Examples
Example
1. Check whether f (x) = x3 attains optimum points or not.

The function f (x) = x3 is shown in the above figure. It is clear that from the
figure that the point x = 0 is an inflection point, since the function value
increases for x ≥ 0 and decreases for x ≤ 0 in the small neighbourhood of
x = 0.
Examples
Example
1. Check whether f (x) = x3 attains optimum points or not.

The function f (x) = x3 is shown in the above figure. It is clear that from the
figure that the point x = 0 is an inflection point, since the function value
increases for x ≥ 0 and decreases for x ≤ 0 in the small neighbourhood of
x = 0.
Since f 0 (x) = 0 at x = 0. Searching for a nonzero higher-order derivative,
we observe that f 00 (x = 0) = 6x|x=0 = 0 and f 000 (x = 0) = 6|x=0 = 6(6= 0).
Thus, the nonzero derivative of order is 3, and since n = 3 is odd. Therefore,
the point x = 0 is an inflection point.
Examples

Example
2. Check whether f (x) = x4 attains optimum points or not.
Examples

Example
2. Check whether f (x) = x4 attains optimum points or not.

B A plot of this function f (x) = x4 is shown in the previous slide.

B The point x = 0 is a minimal point as can been seen from the figure.
B Since f 0 (x = 0) = 4x3 |x=0 = 0, we calculate the higher-order
derivatives in search of non-zero derivative at x = 0:
f 00 (x = 0) = f 000 (x = 0) = 0, but f 0000 (x = 0) = 24 6= 0. Since the value
of the fourth-order derivative is positive, n = 4, which is an even
number.
I Thus the point x = 0 is a local minimum point.
Examples

Example
2. Check whether f (x) = x4 attains optimum points or not.

B A plot of this function f (x) = x4 is shown in the previous slide.

B The point x = 0 is a minimal point as can been seen from the figure.
B Since f 0 (x = 0) = 4x3 |x=0 = 0, we calculate the higher-order
derivatives in search of non-zero derivative at x = 0:
f 00 (x = 0) = f 000 (x = 0) = 0, but f 0000 (x = 0) = 24 6= 0. Since the value
of the fourth-order derivative is positive, n = 4, which is an even
number.
I Thus the point x = 0 is a local minimum point.
Exercise: Check whether the following functions attain optimum points or
not
1
 h(x) = x 3 , x ∈ R.
 g(x) = x2 , x ∈ R.
Examples

Example
3. Determine the maximum and minimum values of the function
f (x) = 12x5 − 45x4 + 40x3 + 5.
Examples

Example
3. Determine the maximum and minimum values of the function
f (x) = 12x5 − 45x4 + 40x3 + 5.

SOLUTION: Since f 0 (x) = 60(x4 − 3x3 + 2x2 ) = 60x2 (x − 1)(x − 2),

f 0 (x) = 0 at x = 0, x = 1, and x = 2. The second derivative is
f 00 (x) = 60(4x3 − 9x2 + 4x).
Examples

Example
3. Determine the maximum and minimum values of the function
f (x) = 12x5 − 45x4 + 40x3 + 5.

SOLUTION: Since f 0 (x) = 60(x4 − 3x3 + 2x2 ) = 60x2 (x − 1)(x − 2),

f 0 (x) = 0 at x = 0, x = 1, and x = 2. The second derivative is
f 00 (x) = 60(4x3 − 9x2 + 4x).
B At x = 1, f 00 (x) = −60 and hence x = 1 is a relative maximum.
Therefore, fmax = f (x = 1) = 12.
Examples

Example
3. Determine the maximum and minimum values of the function
f (x) = 12x5 − 45x4 + 40x3 + 5.

SOLUTION: Since f 0 (x) = 60(x4 − 3x3 + 2x2 ) = 60x2 (x − 1)(x − 2),

f 0 (x) = 0 at x = 0, x = 1, and x = 2. The second derivative is
f 00 (x) = 60(4x3 − 9x2 + 4x).
B At x = 1, f 00 (x) = −60 and hence x = 1 is a relative maximum.
Therefore, fmax = f (x = 1) = 12.
B At x = 2, f 00 (x) = 240 and hence x = 2 is a relative minimum.
Therefore, fmin = f (x = 2) = −11.
Examples

Example
3. Determine the maximum and minimum values of the function
f (x) = 12x5 − 45x4 + 40x3 + 5.

SOLUTION: Since f 0 (x) = 60(x4 − 3x3 + 2x2 ) = 60x2 (x − 1)(x − 2),

f 0 (x) = 0 at x = 0, x = 1, and x = 2. The second derivative is
f 00 (x) = 60(4x3 − 9x2 + 4x).
B At x = 1, f 00 (x) = −60 and hence x = 1 is a relative maximum.
Therefore, fmax = f (x = 1) = 12.
B At x = 2, f 00 (x) = 240 and hence x = 2 is a relative minimum.
Therefore, fmin = f (x = 2) = −11.
B At x = 0, f 00 (x) = 0 and hence we must investigate the next derivative:
f 000 (x) = 60(12x2 − 18x + 4) = 240 at x = 0.
Examples

Example
3. Determine the maximum and minimum values of the function
f (x) = 12x5 − 45x4 + 40x3 + 5.

SOLUTION: Since f 0 (x) = 60(x4 − 3x3 + 2x2 ) = 60x2 (x − 1)(x − 2),

f 0 (x) = 0 at x = 0, x = 1, and x = 2. The second derivative is
f 00 (x) = 60(4x3 − 9x2 + 4x).
B At x = 1, f 00 (x) = −60 and hence x = 1 is a relative maximum.
Therefore, fmax = f (x = 1) = 12.
B At x = 2, f 00 (x) = 240 and hence x = 2 is a relative minimum.
Therefore, fmin = f (x = 2) = −11.
B At x = 0, f 00 (x) = 0 and hence we must investigate the next derivative:
f 000 (x) = 60(12x2 − 18x + 4) = 240 at x = 0.Since f 000 (x) 6= 0 at x = 0,
x = 0 is neither a maximum nor a minimum, and it is an inflection
point.
Exercise:

Identify the optimum points of the following functions. Find the optimum
function values.
F f (x) = x3 − 10x − 2x2 + 10.
F f (x) = x2 − 3x − 20.
F f (x) = x3 − 2x + 10.
F f (x) = 2x4 − x3 + 5x2 − 12x + 1.
F f (x) = x2 + 3x5.
F f (x) = (x − 2)4 .

2 − (x − 1)2 , 0 ≤ x < 3,
F f (x) =
−3 + (x − 3)2 , 3 ≤ x ≤ 6.