ALGORITHM ANALYSIS AND DESIGN

AAD CSE SRM-AP 1

Syllabus
Unit-I: Introduction
 Algorithmic thinking & motivation
 Reinforcing the concepts of Data Structures
 Complexity analysis of algorithms: big O,
omega, and theta notation
 Analysis of Sorting and Searching
 Recursive and non-recursive algorithms.
Unit II: General Problem Solving
(GPS) techniques
 Divide and conquer
 Greedy method
 Dynamic Programming
AAD CSE SRM-AP 2
Syllabus
Unit III: General Problem Solving
(GPS) techniques
 Backtracking
 Branch-and-bound
 Randomized algorithms
Unit IV:
 Pattern matching algorithms
 Algorithm analysis
Unit V:
 Non-polynomial complexity
 ApproximationAAD
Algorithms
CSE SRM-AP 3
Text Book
Cormen, Leiserson, Rivest, Stein,
"Introduction to Algorithms", 3rd
Edition, MIT press, 2009

AAD CSE SRM-AP 4

Motivation
Design
 90% cases simple algorithms found from
Bachelor level courses
 Focus on the 10% tough ones

Analysis
 Why not just measure processing time?
 Time vs. space complexity analyses
 Upper and lower limits

5
Origin of the word 'Algorithm'
Algos means pain in Greek.
Algor in Latin means to be cold.
Therefore, neither is the root for the word \algorithm"!.
Algorithm stems from Abu Jaafar Mohammad Ibn Mousa Al-
Khwarizmi, the name of the 9th-century Arab scholar.
The Hindu-Arabic or Indo-Persian numerals originated from
India. It was invented between the 1st and 4th centuries by
Indian mathematicians.
It was re-introduced in the book On the Calculation with Hindu
Numerals written by Al-Khwarizmi.
Al-Khwarizmi's name was formerly Latinized as Algoritmi, led to
the term algorithm.
AAD CSE SRM-AP 6
What is an Algorithm?
Algorithm stems from al-Khwarizmi, the name of the
9th-century Arab scholar who stressed the
importance of methodical procedures for solving
problems.
A step-by-step procedure for solving well-posed
problem.
A sequence of computational steps that transform
the input into output.
An effective method expressed as a finite list of well-
defined instructions for calculating a function.
A finite set of instructions that, if followed,
accomplishes a particular task.
AAD CSE SRM-AP 7
What is algorithm?
• We need a well-specified problem that tells
what needs to be achieved
• Algorithm solves the problem
• It consists of a sequence of commands that
takes an input and gives output

Input Algorithm Output

8
Properties of Algorithms
An algorithm is a step-by-step procedure for solving a problem
in a finite amount of time.

1. Input
2. Output
3. Finiteness
4. Definiteness
5. Effectiveness

AAD CSE SRM-AP 9

Properties of Algorithms
1) Input
Inputs to the algorithm must be specified
2) Output
At least one quantity is produced.
3) Definiteness
Each instruction of the algorithm should be clear and unambiguous.
4) Finiteness
The process should be terminated after a finite number of steps.
5) Effectiveness
Every instruction must be basic enough to be carried out theoretically or by
using paper and pencil.

AAD CSE SRM-AP 10

Algorithm principles
• Sequence
- One command at a time
• Condition
- IF
- CASE
• Loops
- FOR
- WHILE
- REPEAT
 Input Algorithm Output

An algorithm is a step-by-step procedure for

solving a problem in a finite amount of time.
Running Time
Most algorithms transform best case
average case
input objects into output worst case
objects. 120

The running time of an 100

algorithm typically grows

Running Time
80
with the input size.
60
Average case time is often
difficult to determine. 40

We focus on the worst case 20

running time. 0
1000 2000 3000 4000
 Easier to analyze
Input Size
 Crucial to applications such as
games, finance and robotics

AAD CSE SRM-AP 13

Experimental Studies
Write a program 9000

implementing the 8000

algorithm 7000

Run the program with 6000

Time (ms)
inputs of varying size and 5000
composition 4000
Use a method like 3000
System.currentTimeMillis() to
2000
get an accurate measure
of the actual running time 1000

Plot the results 0

0 50 100
Input Size

AAD CSE SRM-AP 14

Limitations of Experiments
It is necessary to implement the algorithm,
which is usually difficult
In order to compare two algorithms, the
same hardware and software environments
must be used

AAD CSE SRM-AP 15

Theoretical Analysis
Uses a high-level description of the
algorithm instead of an implementation
Characterizes running time as a function
of the input size, n.
Takes into account all possible inputs
Allows us to evaluate the speed of an
algorithm independent of the
hardware/software environment
AAD CSE SRM-AP 16
Algorithm principles
• Sequence
- One command at a time
- Parallel and distributed computing
• Condition
- IF
- CASE
• Loops
- FOR
- WHILE
- REPEAT
Pseudocode
Pseudo-code is a mixture of natural language and
high-level programming constructs that describe the
main ideas behind a generic implementation of a data
structure or algorithm. More structured than English
prose.
The programming language constructs in pseudocode:
a) Expressions b) Method declarations
c) Decision structures d) While-loop
e) Repeat-loops f) For-loops
g) Array indexing h) Method calls
i) Method returns

AAD CSE SRM-AP 18

Pseudocode
Method declaration
Algorithm method (arg, arg…)
Input …
Output …
Control flow if a<b then
print “b is biggest”
 if … then … [else …] print “b=“ b
else
 while … do … print “a is biggest”
 repeat … until …
 for … do …
 Indentation replaces braces
AAD CSE SRM-AP 19
 Pseudocode
Method call
 var.method (arg [, arg…])

Return value
 return expression

Expressions
 ← Assignment (like = in C)

 = Equality testing (like = = in C)

 n2 superscripts and other mathematical formatting allowed

read/get – indicates a user will be inputting something

display/print/show – indicates that an output will appear on
the screen
SET, INIT: To initialize values

AAD CSE SRM-AP 20

 Pseudocode
Algorithm printN( n) Algorithm printSum_N( n)
Input n Input n
Output 1,2,…..n
Output sum
Read n
Read n
Initialize i to 1
Initialize i to 1
while i<=n do
Sum0
Write i.
while i<=n do
Increment i.
sum<-sum+ i
Increment i.
Write sum.
AAD CSE SRM-AP 21
Pseudocode
High-level description Example: find max
of an algorithm element of an array
More structured than Algorithm arrayMax(A, n)
English prose Input array A of n integers
Less detailed than a Output maximum element of
program A
Preferred notation for
describing algorithms currentMax  A[0]
Hides program design for i  1 to n  1 do
issues if A[i]  currentMax then
currentMax  A[i]
incremetnt++
return currentMax
AAD CSE SRM-AP 22
Algorithm Analysis
Algorithm analysis is an important part of computational complexity theory,
which provides theoretical estimation for the required resources of an
algorithm to solve a specific computational problem.
Most algorithms are designed to work with inputs of arbitrary length.
Analysis of algorithms is the determination of the amount of time and space
resources required to execute it.
o Time complexity analysis
o Space complexity analysis

23
Complexity
• The computational complexity, or simply complexity of
an algorithm is the amount of resources required for
running it.
• What are the resources an algorithm requires?
• CPU Time
• Memory, etc.
• Time complexity:
• How much time it takes to compute
• Measured by a function T(N)
• Space complexity:
• How much memory it takes to compute
• Measured by a function S(N) 24
Theoretical Analysis
Uses a high-level description of the algorithm
instead of an implementation
Characterizes running time as a function of the
input size, n.
Takes into account all possible inputs
Allows us to evaluate the speed of an algorithm
independent of the hardware/software
environment

AAD CSE SRM-AP 25

Primitive Operations
Basic computations
Examples:
performed by an algorithm  Assigning a value to a
Identifiable in pseudocode variable
 Calling a method
Largely independent from the  Performing an arithmetic
operation (for example,
programming language adding two numbers).
Exact definition not important  Comparing two numbers
 Indexing, into an array
(we will see why later)  Following an object
reference
Assumed to take a constant  Returning, from a
amount of time in the RAM method.
model

AAD CSE SRM-AP 26

Primitive Operations
Examples of primitive operations:
 Evaluating an expression x2+ey
 Assigning a value to a variable cnt ← cnt+1
 Indexing into an array A[5]
 Calling a method mySort(A,n)
 Returning from a method return(cnt)

AAD CSE SRM-AP 27

 The Random Access Machine
(RAM) Model
A CPU

An potentially unbounded
bank of memory cells, 2
1
each of which can hold an 0
arbitrary number or
character
Memory cells are numbered and accessing
any cell in memory takes unit time.
• Count Primitive Operations = time needed by RAM model
AAD CSE SRM-AP 28
 Simple Example - Steps

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int n){

int s=0; 1
for (int i=0; i< n; i++)
2 3 4
s = s + A[i];
5 6 7 1,2,8: Once
return s;
3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3

29
 Counting Primitive
Operations
By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by
an algorithm, as a function of the input size
Algorithm arrayMax(A, n)
# operations
currentMax  A[0] 2
for i  1 to n  1 do 1+ n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
{ increment counter i } 2(n  1)
return currentMax 1
Total 7n  2
AAD CSE SRM-AP 30
A[]={5,8,9,10,11,15,16,18,19,20
Estimating Running Time
Algorithm arrayMax executes 7n  2
primitive operations in the worst case.
Define:
a = Time taken by the fastest primitive
operation
b = Time taken by the slowest primitive
operation
Let T(n) be worst-case time of arrayMax.
Then
a (7n  2) AADT(n)  b(7n  2)
CSE SRM-AP 31
 Counting Primitive
Operations
By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by
an algorithm, as a function of the input size
Algorithm arrayMax(A, n)
# operations
currentMax  A[0] 2
for i  1 to n  1 do 1+ n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 0
{ increment counter i } 2(n  1)
return currentMax 1
Total 5n
AAD CSE SRM-AP 32
A[]={20,5,8,9,10,11,15,16,18,19
Best, Worst, and Average-
Case Complexity
int search(int arr[], int n, int x) Worst Case Analysis: we
{ calculate upper bound on
running time of an algorithm.
int i;
Average Case Analysis:
for (i=0; i<n; i++) we take all possible inputs
{ and calculate computing time
if (arr[i] == x) for all of the inputs.
return i; Best Case Analysis: we
} calculate lower bound on
running time of an algorithm.
return -1;
}

AAD CSE SRM-AP 33

Best case Vs Avg case Vs Worst case

AAD CSE SRM-AP 34

AAD CSE SRM-AP 35
Prefix Averages (Linear)
The following algorithm computes prefix averages in linear time by
keeping a running sum

X=8,4, 3, 5, 9, 3……. A= 8, 6, 5, 5………..

Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
s0 1
for i  0 to n  1 do n
s  s + X[i] n
A[i]  s / (i + 1) n
return A 1

=4n+2
Algorithm prefixAverages2 runs in O(n) time

Analysis of Algorithms 36
General Guidelines
The worst-case instructions determine
worst-case behaviour, overall. (eg.
A single n2 statement means the whole
algorithm is n2.)
Instant recognition:
 Assignments/arithmetic is O(1),
 Loops are O(n),
 Two nested loops are O(n2).
 How about three nested loops?
Analysis of Algorithms 37
Matrix-Multiplication

Algorithm: Matrix-Multiplication (X, Y,Z)

Input: X[n:n], Y[n:n]
Output:Z[n:n}
for i = 1 to n do
for j = 1 to n do
Z[i,j] := 0
for k = 1 to n do
Z[i,j] := Z[i,j]AAD
+ X[i,k] × Y[k,j]
CSE SRM-AP 38
Space Complexity
Space complexity is the amount of
memory used by the algorithm (including
the input values to the algorithm) to
execute and produce the result.
Space Complexity = Auxiliary Space
+ Input space

AAD CSE SRM-AP 39

public int sumArray(int[] array) {
int size = 0;
int sum = 0;
for (int iterator = 0; iterator < size; terator++) {
sum += array[iterator];
}
return sum;
}
array – the function’s only argument – the space taken by the array is equal
2n bytes where n is the length of the array
size – a 2-byte integer
sum – a 2-byte integer
iterator – a 2-byte integer
The total space needed for this algorithm to complete is 2n + 2 + 2 + 2
(bytes)= 2n+6= O(n) AAD CSE SRM-AP 40
Space Complexity
Algorithm arrayMax(A, n)
currentMax  A[0]
for i  1 to n  1 do
if A[i]  currentMax then
currentMax  A[i]
{ increment counter i }
return currentMax

AAD CSE SRM-AP 41