CSI2110

Data Structures
and Algorithms

1
 Algorithms

Input Algorithm Output

An algorithm is a step-by-step procedure for

solving a problem in a finite amount of time.

Analyze an algorithm = determine its efficiency

Analysis of Algorithms 2
 Analyze an algorithm

Analyze an algorithm = determine its efficiency

Analysis of Algorithms 3
 Efficiency ?

• Execution time …

• Memory …

• Quality of the result ….

• Simplicity ….

Analysis of Algorithms 4
 Running Time

The running time depends on

the input size

It also depends on the input data:

Different inputs can have different
running times

Analysis of Algorithms 5
 Running time

best case
average case
worst case
120

100
Running Time

80

60

40

20

0
1000 2000 3000 4000
Input Size

Analysis of Algorithms 6
 Analysis of Algorithms
T(n)
• Running Time
• Upper Bounds n=4
• Lower Bounds
• Examples
• Mathematical
facts Input Algorithm Output

Analysis of Algorithms 7
 Average Case vs. Worst Case
Running Time of an algorithm

• Finding the average case can be very difficult

• Knowing the worst-case time complexity can be
important. We focus on the worst case running time.
– Easier to analyze
– Crucial to applications such as games and robotics
5 ms worst-case
4 ms

3 ms
}
average-case?
best-case
2 ms

1 ms

Analysis of Algorithms 8
A B C D E F G
Input
 Measuring the Running Time

• How should we measure the running time of

an algorithm?

• Approach 1: Experimental Study

– Write a program which implement the algorithm
– Run the program with all possible input sets of
data of various size and contents
– Use a method (system.currentTimeMillis()) to
measure exact running time

Analysis of Algorithms 9
 Measuring the Running Time
• Approach 1: Experimental Study
• The measurement of the experiment can
be like this… best case
average case
worst case
120

100

Running Time
80

60

40

20

0
Analysis of Algorithms 1000 2000 3000 4000
Input Size
 Beyond Experimental Studies

• Experimental studies have several

limitations:
– need to implement
– limited set of inputs
– hardware and software environments.

Analysis of Algorithms 11
 Theoretical Analysis

• We need a general methodology which:

– is independent of implementation.
• Uses a high-level description of the algorithm
– takes into account all possible inputs.
• Characterizes running time as a function of the input size.
– is independent of the hardware and software
environment.

Analysis of Algorithms 12
 Analysis of Algorithms

• Primitive Operations: Low-level computations independent

from the programming language can be identified in
pseudocode.

• Examples:
– calling a method and returning from a method
– arithmetic operations (e.g. addition)
– comparing two numbers, etc.

• By inspecting the pseudo-code, we can count the number

of primitive operations executed by an algorithm.

13
 Pseudo-code

• Mixture of natural language and high-level programming

concept
– to tell the general idea behind the data structure or algorithm
implementation
• The pseudo-code is an description of algorithm which is
– more structured than ordinary prose, but
– less definite than programming languages

Analysis of Algorithms 14
 Pseudo-code

• Expressions: use standard mathematical symbols

to describe Boolean and numerical expressions
– use ß for allocations (“=“ en Java)
– use = for equal relation (“==“ en Java)
• Method declaration
– Algorithm nom(param1, param2)

Analysis of Algorithms 15
 Pseudo-code

• Programming element:
– decision: if.. Then.. [else..]
– Loop while: while…do
– Loop repeat: repeat … until..
– Loop for: for.. Do
– Vector index: A[i]
• methods:
– call: object method (args)
– return: return value

Analysis of Algorithms 16
 Example:

Find the maximum element of a vector (array)

Algorithm arrayMax(A, n):
input: A vector A containing n entries
output: the maxim element of A

currentMax ¬ A[0]
for i ¬ 1 to n -1 do
if currentMax < A[i] then
currentMax ¬ A[i]
return currentMax
Analysis of Algorithms 17
A
5 13 4 7 6 2 3 8 1 2

currentMax

currentMax ¬ A[0]
for i ¬ 1 to n -1 do
if currentMax < A[i] then
currentMax ¬ A[i]

return currentMax

Analysis of Algorithms 18
A
5 13 4 7 6 2 3 8 1 2

currentMax 5

currentMax ¬ A[0]
for i ¬ 1 to n -1 do
if currentMax < A[i] then
currentMax ¬ A[i]

return currentMax

Analysis of Algorithms 19
A
5 13 4 7 6 2 3 8 1 2

currentMax 5

currentMax ¬ A[0]
for i ¬ 1 to n -1 do
if currentMax < A[i] then
currentMax ¬ A[i]

return currentMax

Analysis of Algorithms 20
 What are the primitive operations to count
A
5 13 4 7 6 2 3 8 1 2
Comparisons
Assignments to
currentMax
currentMax 13

currentMax ¬ A[0]
for i ¬ 1 to n -1 do
if currentMax < A[i] then
currentMax ¬ A[i]

return currentMax

Analysis of Algorithms 21
currentMax ¬ A[0] 1 assignment
for i ¬ 1 to n -1 do
if currentMax < A[i] then
n-1 comparisons
currentMax ¬ A[i]
n-1 assignments (worst case)

return currentMax

5 7 8 10 11 12 14 16 17 20

Analysis of Algorithms 22
 In the best case ?

15 1 12 3 9 7 6 4 2 11

currentMax ¬ A[0] 1 assign

for i ¬ 1 to n -1 do
if currentMax < A[i] then
n-1 comparaisons
currentMax ¬ A[i]
0 assign

return currentMax
Analysis of Algorithms 23
 Summarizing:

Worst Case: Best Case:

n-1 comparisons n-1 comparisons
n assignments 1 assignment

Analysis of Algorithms 24
 Another Example

Looking for the rank of an element in A

(size of A is sizeA)

i¬0 1 assignment
while (A[i] ¹ element)
sizeA checks & assignment
i ¬ i+1
(if we are not lucky)
return i
Worst Case

Analysis of Algorithms 25
 Upper Bound

The “Big-Oh” Notation:

– given functions f(n) and g(n), we say
that
f(n) is O(g(n))
if and only if there are positive constants
c and n0 such that
f(n)≤ c g(n) for n ≥ n0
c • g(n)

f(n)

n0
Analysis of Algorithms 26n
prove that f(n)≤ c g(n) for some n ≥ n0
An Example
f(n) = 60n2 + 5n + 1 g(n) = n2
≤ prove that f(n)≤ c n2

60n2 + 5n2 + n2 for n ≥ 1

= 66n2
c = 66 n0 = 1
f(n) ≤ c n2 n ≥ n0

Analysis of Algorithms 27
On the other hand…

n2 is not O(n) because there is no c and n0 such that:

n2 ≤ cn for n ≥ n0

( no matter how large a c is chosen there is an n big

enough that n2 > c n ) .
n2

Analysis of Algorithms 28
n0 n
 O(1) < O(log n) < O(n) < O(n log n) < O(n2) <
O(n3) < O(2n) … remember !!

n2

log(n)
n0 n

Analysis of Algorithms 29
n0 n
Analysis of Algorithms 30
Analysis of Algorithms 31
Analysis of Algorithms 32
 O(1) < O(log n) < O(n) < O(n log n) < O(n2) <
n
O(n ) < O(2 ) … remember !!
3

Analysis of Algorithms 33
 n= 2 16 256 1024
log log n 0 2 3 3.32
log n 1 4 8 10
n 2 16 256 1024
n log n 2 64 448 10 200
n2 4 256 65 500 1.05 * 106

n3 8 4 100 16 800 800 1.07 * 109

2n 4 35 500 11.7 * 106 1.80 * 10308

Analysis of Algorithms 34
 Asymptotic Notation (cont.)

• Note: Even though it is correct to say

• “7n - 3 is O(n3)”,
• a better statement is
• “7n - 3 is O(n)”, that is,
• one should make the approximation as tight as possible

Analysis of Algorithms 35
Theorem:
If g(n) is O(f(n)) , then for any constant
c >0
g(n) is also O(c f(n))

Theorem:
O(f(n) + g(n)) = O(max(f(n), g(n)))

Ex 1:
2n3 + 3n2 = O (max(2n3, 3n2))
= O(2n3) = O(n3)
Ex 2:
n2 + 3 log n – 7 = O(max(n2, 3 log n – 7))
= O(n2)
Analysis of Algorithms 36
 Simple Big Oh Rule:

Drop lower order terms and constant factors

7n-3 is O(n)

8n2log n + 5n2 + n is O(n2log n)

12n3 + 5000n2 + 2n4 is O(n4)

Analysis of Algorithms 37
 Other Big Oh Rules:

•Use the smallest possible class of functions

–Say “2n is O(n)” instead of “2n is O(n2)”

•Use the simplest expression of the class

–Say “3n + 5 is O(n)” instead of
“3n + 5 is O(3n)”

Analysis of Algorithms 38
 Asymptotic Notation
(terminology)

• Special classes of algorithms: 1E+30

constant: O(1) 1E+28
1E+26
Cubic

logarithmic: O(log n) 1E+24 Quadratic

1E+22
linear: O(n) 1E+20
Linear

quadratic:
1E+18
O(n2)

T (n )
1E+16
cubic: O(n3) 1E+14
1E+12
polynomial: O(nk), k ≥ 1 1E+10
1E+8
exponential: O(an), n > 1 1E+6
1E+4
1E+2
1E+0
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n

Analysis of Algorithms 39
 Asymptotic Analysis and execution
time
• Use the Big-O notation
– to indicate the number of primitive operations
executed according to the entry size
• For example, we say that algorithm arrayMax
has an execution time O(n)
• While comparing the asymptotic execution
times
– O(log n) is better than O(n)
– O(n) is better than O(n2)
– log n << n-2 << n << n log n << n2 << n3 << 2n

Analysis of Algorithms 40
 Asymptotic Analysis and execution
time
• Use the Big-O notation
– to indicate the number of primitive operations
executed according to the entry size
• For example, we say that algorithm arrayMax
has an execution time O(n)
• While comparing the asymptotic execution
times
– O(log n) is better than O(n)
– O(n) is better than O(n2)
– log n << n << n log n << n2 << n3 << 2n

Analysis of Algorithms 41
Example of Asymptotic Analysis

An algorithm for computing prefix averages

The i-th prefix average of an array X is average of

the first (i + 1) elements of X
A[i] = X[0] + X[1] + … + X[i]

Analysis of Algorithms 42
 Example of Asymptotic Analysis

Algorithm prefixAverages1(X, n)

Input array X of n integers

Output array A of prefix averages of X #operations

A ¬ new array of n integers n

for i ¬ 0 to n - 1 do
s ¬ X[0] n
for j ¬ 1 to i do
s ¬ s + X[j] 1 + 2 + …+ (n - 1)
A[i] ¬ s / (i + 1) n
return A 1
43
• The running time of
prefixAverages1 is
O(1 + 2 + …+ n) 7
6
• The sum of the first n
integers is n(n + 1) / 2 5

– There is a simple visual 4

proof of this fact 3
• Thus, algorithm 2
prefixAverages1 runs 1
in O(n2) time 0
1 2 3 4 5 6

Analysis of Algorithms 44
 Another Example

• A better algorithm for computing prefix averages:

Algorithm prefixAverages2(X):
Input: An n-element array X of numbers.
Output: An n -element array A of numbers such that A[i] is the average of
elements X[0], ... , X[i].
Let X be an array of n numbers. # operations
s¬ 0 1
for i ¬ 0 to n do n
s ¬ s + X[i] n
A[i] ¬ s/(i+ 1) n
return array A 1

Analysis of Algorithms O(n) time 45

 Lower Bound
… is big omega …
f(n) is Ω(g(n))
if there exist c > 0 and n0 > 0 such that
f(n) ≥ c • g(n) for all n ≥ n0

(thus, f(n) is Ω(g(n)) iff g(n) is O(f(n)) )

f(n)
c • g(n)

Analysis of Algorithms n0 46
n
 Tight Bound

… is big theta …
g(n) is Q(f(n))
<===>
if g(n) Î O(f(n))
AND
f(n) Î O(g(n))

is an element of (set membership)

Analysis of Algorithms 47
Mathematical notation instead of “is”
An Example
We have seen that
f(n) = 60n2 + 5n + 1 is O(n2)

but 60n2 + 5n + 1 ≥ 60n2 for n ≥ 1

So: with c = 60 and n0 = 1
f(n) ≥ c • n2 for all n ≥ 1 f(n) is Ω (n2)

Therefore:
f(n) is O(n2)
AND
f(n) is Ω(n2)

f(n) is Q(n2)
Analysis of Algorithms 48
 Intuition for Asymptotic
Notation
Big-Oh
– f(n) is O(g(n)) if f(n) is asymptotically
less than or equal to g(n)
big-Omega
– f(n) is W(g(n)) if f(n) is asymptotically
greater than or equal to g(n)
big-Theta
– f(n) is Q(g(n)) if f(n) is asymptotically
equal to g(n)

Analysis of Algorithms 49
 Math You Need to Review
Logarithms and Exponents

properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba= logxa/logxb
properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab
Analysis of Algorithms 50
 More Math to Review

• Floor: ëxû = the largest integer ≤ x ë2.3û = 2

• Ceiling: éxù = the smallest integer ≥ x é2.3ù = 3
• Summations:
– General definition:
t
∑ f(i) = f(s) + f(s+1) + f(s+2) + .. + f(t)
i=s

– where f is a function, s is the starting index, and t is the

ending index
Analysis of Algorithms 51
 More Math to Review

• Arithmetic Progression : f(i) = i a

n
S = ∑ id = 0 + d + 2d + … + nd
i=0
= nd+(n-1)d+(n-2)d + … + 0

2S = nd + nd + nd + …+ nd
= (n+1) nd
S = d/2 n(n+1)

Analysis of Algorithms
for d=1, S = 1/2 n(n+1) 52
 More Math to Review

• Geometric Sum : f(i) = ai

• The geometric progressions have an exponential growth
n
S = ∑ ri = 1 + r + r2 + … + rn
i=0
rS = r + r2 + … + rn + rn+1
rS - S = (r-1)S = rn+1 - 1
S = (rn+1-1)/(r-1)
If r=2, S = (2n+1-1)
Analysis of Algorithms 53
Analysis of Algorithms 54