Algorithms and Growth of Functions

 Algorithms
 The growth of functions
 Complexity of Algorithms

1
Algorithms
 An algorithm is a finite sequence of
precise instructions for performing a
computation or for solving a problem.

An algorithm
 is defined on specified inputs and
generates an output
 stops after finitely many instructions are
executed.
2
A Recipe is an Algorithm
The set of steps to assemble a Piece of Furniture is
an Algorithm.
Properties of Algorithms

1. Input
2. Output
3. Definiteness
4. Correctness An algorithm
has input
5. Effectiveness values from a
6. Finiteness specified set.

7. Generality
Properties of Algorithms

1. Input
2. Output From each set
of input values,
3. Definiteness an algorithm
4. Correctness An produces
algorithm
output
has input
values
5. Effectiveness values
from
from
a a
6. Finiteness specified
specifiedset.
set.
The output
7. Generality values are the
solution to the
problem.
Properties of Algorithms

1. Input
2. Output From each set
of input values,
3. Definiteness an algorithm
4. Correctness An produces
algorithm
output
The
has steps
input
values
of an
5. Effectiveness values
algorithm
from
from
a must
a
6. Finiteness specified
specified
be defined
set.
set.
The
precisely.
output
7. Generality values are the
solution to the
problem.
Properties of Algorithms

1. Input
2. Output From each set
of input values,
3. Definiteness an algorithm
4. Correctness An produces
algorithm
An algorithm
output
The
has steps
input
values
of an
5. Effectiveness should produce
values
algorithm
from
from
a must
a
the correct
6. Finiteness specified
specified
be defined
set.
set.
output values
Theprecisely.
output
7. Generality for each set of
values are the
input values.
solution to the
problem.
Properties of Algorithms

1. Input
2. Output From each set
ofItinput
mustvalues,
be
3. Definiteness possible
an algorithmto
4. Correctness perform
An produces
algorithm
each
An algorithm
output
The
step
has steps
input
ofvalues
anof an
5. Effectiveness should produce
values
algorithm
algorithm
from
froma must
a
the correct
6. Finiteness exactly
specified
specified
be and
defined
set.
set.
in a
output values
finite
Theprecisely.
amount
output
7. Generality for each set of
values
of time.
are the
input values.
solution to the
problem.
Properties of Algorithms

1. Input
2. Output
From each set of
It must be possible
3. Definiteness Aninput
to
values,should
algorithm
perform each
an
An
algorithm
algorithm
produce produces
the has
desired
4. Correctness step
input
An
of an
outputThe
output
exactly
algorithm
algorithm
values
values
steps
after
produce
from
andthe a
should
from
offinite
an
a a
in acorrect
specified
specified
algorithm
(but set.
perhaps set.
mustThebe
large)
5. Effectiveness finite
output
output
number
amount
defined
each
values
values
time.
of for
precisely.
of
setsteps
arefor
of input
the
any solution to the
inputvalues.
in the set.
6. Finiteness problem.
7. Generality
Properties of Algorithms

1. Input
2. Output
From each set of
It must be possible
3. Definiteness Aninput
to
values,should
algorithm
perform
The each
an
procedure
An
algorithm
algorithm
produce produces
the has
desired
4. Correctness step
input
An
of an
should
outputThe
output
exactly
algorithm
values
produce
for
algorithm
be
values
steps
after
and
all
from
theafrom
should
applicable
offinite
an
a a
in acorrect
problems
specified
specified
algorithm
(but perhapsset.
set.
mustThebeof
large)
5. Effectiveness finiteoutput
the
output
amount
defined
number
each
time.
values
desired
values
of
ofform,
precisely.
steps
for
arefor
the notset offor
justto input
a
any solution in thethe
inputvalues. set.
6. Finiteness particular
problem. set of
input values.
7. Generality
How to express an Algorithm
C code Java code Pseudo-code
int is_prime(int class SpecialInt procedure is_prime(m)
m) { for i: = 2 to m−1 do
{ int m; if m mod i = 0
int i; boolean is_prime() then return(false)
for (i=2; i<m;i++) { endif
if (m % i ==0) for (i=2; i<m; i++) endfor
return 0; if (m % i == 0) return(true)
return 1; return false; end is_prime
} return true;
} }
12
Psuedocode
 Psuedocode is an intermediate between
an English description and an
implementation in a particular language
of an algorithm.
Advantages of using pseudo-code
 Pseudo-code has a structure similar
to most computer languages.
 No need to worry about the precise
syntax.
 Not specific to any particular
computer language.

14
Example
 Example: Write an algorithm that finds
the largest element in a finite sequence
s1, s2, … , sn
procedure find_large(s, n)
large := s1
i := 2
while i ≤ n do
if si > large then large := si endif
i := i + 1
endwhile
return(large)
end find_large
15
Search Algorithms
 Search
 Find a given element in a list. Return the location of the
element in the list (index), or 0 if not found.
 Linear Search
 Compare key (element being searched for) with each element
in the list until a match is found, or the end of the list is
reached.
 Binary Search
 Compare key only with elements in certain locations. Split list
in half at each comparison. Requires list to be sorted.
Linear Search
 Find the location of an element X in an array of possible
unsorted items

17
 Linear Search Exercise

19, 1, 17, 2, 11, 13, 7, 9, 10, 5, 15, 6, 14, 20, 16, 12, 4, 18, 3, 8

 How many comparisons to find:

 17?
 21?
Binary Search
 Find the location of an element X in an
array of sorted items

19
 Binary Search Exercise

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

 How many comparisons to find:

 Find 7
 Find 21
Sort
 Sort
 Put the elements of a list in ascending order
 Bubble Sort
 Compare every element to its neighbor and swap
them if they are out of order. Repeat until list is
sorted.
 Insertion Sort
 For each element of the unsorted portion of the
list, insert it in sorted order in the sorted portion of
the list.
Bubble Sort

22
Bubble Sort Exercise
10, 2, 1, 5, 3
Insertion Sort

24
Insertion Sort Exercise
10, 2, 1, 5, 3
Greedy Algorithms
The goal of an optimization problem is to
maximize or minimize an objective function.

One of the simplest approaches to solving

optimization problems is to select the “best”
choice at each step.
Greedy Change-Making
 Give an algorithm for making n cents change with
quarters, dimes, nickels, and pennies, and using the least
total number of coins.

27
Make Change
69 cents:

56 cents:
The Halting Problem
Is there a procedure that does the following:
Takes as input a program and input to that
program and determines whether that program will
eventually stop when run on that input, for any
program and input

No, there is no such program.

29
Algorithms and Growth of Functions
 Algorithms
 The growth of functions
 Complexity of Algorithms

1
Algorithms
 An algorithm is a finite sequence of
precise instructions for performing a
computation or for solving a problem.

An algorithm
 is defined on specified inputs and
generates an output
 stops after finitely many instructions are
executed.
2
Search Algorithms
 Search
 Find a given element in a list. Return the location of the
element in the list (index), or 0 if not found.
 Linear Search
 Compare key (element being searched for) with each element
in the list until a match is found, or the end of the list is
reached.
 Binary Search
 Compare key only with elements in certain locations. Split list
in half at each comparison. Requires list to be sorted.
Sort
 Sort
 Put the elements of a list in ascending order
 Bubble Sort
 Compare every element to its neighbor and swap
them if they are out of order. Repeat until list is
sorted.
 Insertion Sort
 For each element of the unsorted portion of the
list, insert it in sorted order in the sorted portion of
the list.
Greedy Algorithms
The goal of an optimization problem is to
maximize or minimize an objective function.

One of the simplest approaches to solving

optimization problems is to select the “best”
choice at each step.
Greedy Change-Making
 Give an algorithm for making n cents change with
quarters, dimes, nickels, and pennies, and using the least
total number of coins.

6
Make Change
69 cents:

56 cents:
The Halting Problem
Is there a procedure that does the following:
Takes as input a program and input to that
program and determines whether that program will
eventually stop when run on that input, for any
program and input

No, there is no such program.

8
The Growth of functions
 The time required to solve a problem using a
procedure depends on:
 Number of operations used
 Depends on the size of the input
 Speed of the hardware and software
 Does not depend on the size of the input
 Can be accounted for using a constant multiplier

 The growth of functions refers to the number of

operations used by the function to solve the problem.

9
Complexity of Algorithms
The complexity of an algorithm refers to the
amount of time and space required to execute
the algorithm.

Computing the amount of time and space used

without having the actual program requires
one to focus on the essential features that
affect performance.

10
Analyzing algorithm find_largest
 Time of execution depends on the number
of iterations of the while loop.
 Performance does not generally depend
on the values of the elements.
 How many iterations are executed?
n−1
The time needed is linearly proportional to n.

11
Example
for i := 1 to n do
for j:=1 to n do
si := si + sj

number of iterations executed: n2

time needed: proportional to n2

12
Big-O Notation
 Estimate the growth of a function without
worrying about constant multipliers or smaller
order terms.
 Do not need to worry about hardware or software
used
 Assume that different operations take the same
time.
 Addition is actually much faster than division, but for
the purposes of analysis we assume they take the
same time.
13
Big-O

14
Example

 X2 + 2x + 1 <= x2 + 2x2 + x2 for x >=1

 x2 + 2x2 + x2 = 4x2

 Witness
 C=4
 K=1

15
Example

 Assume n2 is O(n)
 Then  C,k ∀ n>k, n2 <= Cn
 n <= C
 But no constant is bigger than all n
 contradiction

16
Big-𝑂 for Polynomials
Example
 Give a big-O estimate for
 f(x) = 5x2-18x+20

 Solution
 5𝑥2−18𝑥+20≤5𝑥2+20 for 𝑥>0
 5𝑥2+20≤5𝑥2+20𝑥2 for 𝑥>1
 5𝑥2+20𝑥2=25𝑥2≤𝐶𝑔(𝑥) for 𝑥>1
 Let 𝑔(𝑥)=𝑥2
 𝒇(𝒙) is 𝑶(𝒙𝟐 ). 𝑪=𝟐𝟓, 𝒌=𝟏

18
Example
 Give a big-O estimate for the sum of the first n
positive integers

 Solution
 1+2+⋯+𝑛≤𝑛+𝑛+⋯+𝑛=𝑛2
 1+2+⋯+𝑛 is 𝑂(𝑛2 ), 𝐶=1,𝑘=1

19
Example
 Give a big-O estimate for the factorial function f(n)=n!
 Give a big-O estimate for the logarithm of the
factorial function

 Solution
 𝑛!=1⋅2⋅3⋅⋯⋅𝑛≤𝑛⋅𝑛⋅𝑛⋅⋯⋅𝑛=𝑛𝑛
 𝑛! is 𝑂(𝑛𝑛 )

 log(𝑛!)≤log(𝑛𝑛 )=𝑛 log𝑛

 log(𝑛!) is 𝑂(𝑛 log𝑛 )

20
Basic Growth Functions
Factorial O(n!)

Exponential O(nn)

Polynomial O(n2)

Linearithmic O(n log n)

Linear O(n)

Logarithmic O(log n)

Constant O(1)

21
Useful Big-𝑂 Estimates
The Growth of Combinations of Functions
Example
 Give a big-𝑂
estimate for
2
𝑓 𝑛 = 3𝑛log 𝑛! + 𝑛 + 3 log 𝑛

 O(n2 log n)
Big-Ω
Example

 Show that 8𝑥 3 + 5𝑥 2 + 7 is Ω 𝑥 3

 8𝑥3+5𝑥2+7≥8𝑥3 for x > 0

 C=8, k =0
Big-Θ
 Big- Θ (big theta)
 𝑓 𝑛 is 𝑂 𝑔 𝑛 and Ω 𝑔 𝑛
 𝑓 𝑛 is 𝑂 𝑔 𝑛 and 𝑔 𝑛 is 𝑂 𝑓 𝑛
 𝑓 𝑛 is Θ 𝑔 𝑛 ↔ g 𝑛 is Θ 𝑓 𝑛
 ∃𝐶1 , 𝐶2 , 𝑘 ∀𝑛 > 𝑘 𝐶1 𝑔 𝑛 ≤ 𝑓 𝑛 ≤ 𝐶2 𝑔 𝑛
 𝑓 𝑛 is of order 𝑔 𝑛
 𝑓 𝑛 and 𝑔 𝑛 are of the same order

27
Example
 Show that 3x 2 + 8xlog x is Θ x 2

 Big-o
 3x2 +8x log x <= 11x2
 C=11, k =1

 Big-omega
 x2 <= 3x2 + 8x log x

28
Big-Θ for Polynomials
 Let 𝑓 𝑥 = 𝑎𝑛 𝑥 𝑛 + 𝑎𝑛−1 𝑥 𝑛−1 + ⋯ + 𝑎1 𝑥 +
𝑎0 .
 Then, 𝑓 𝑥 is of order 𝑥 𝑛 .
 “𝑓 𝑥 is bounded [above and below] by 𝑔 𝑥 ”
 Example:
 3𝑥 8 + 10𝑥 7 + 221𝑥 2 + 1444 is of order 𝑥 8

29