Algorithm Classification

1. Classification by implementation Classification by

implementation
1. Recursion or iteration.
2. Logical.
3. Serial or parallel or distributed.
4. Deterministic or non-deterministic
5. Exact or approximate:
2. Classification by Design Paradigm
1. Divide and conquer.
2. Dynamic programming.
3. The greedy method.
4. Linear programming:
5. Reduction
6. The probabilistic and heuristic paradigm

1
 Iterative algorithm
• An iterative algorithm executes steps in iterations.
• It aims to find successive approximation in sequence to
reach a solution.
• They are most commonly used in linear programs
where large numbers of variables are involved.

2
 Recursive vs Iterative
Algorithms:
• Approach: In recursive approach, the function calls itself until the
condition is met, whereas, in iterative approach, a function repeats until
the condition fails.

• Programming Construct Usage: Recursive algorithm uses a branching

structure, while iterative algorithm uses a looping construct.

• Time & Space Effectiveness: Recursive solutions are often less efficient
in terms of time and space, when compared to iterative solutions.

• Termination Test: Iteration terminates when the loop-continuation

condition fails; recursion terminates when a base case is recognized.

• Infinite Call: An infinite loop occurs with iteration if the loop-continuation

test never becomes false; infinite recursion occurs if the recursion step
does not reduce the problem in a manner that converges on the base
case.
3
 A practical problem
The birthday guy cuts the birthday cake and has to
ensure that everyone in the room gets a slice.
• Solution 1 – Iterative: The birthday guy uses a tray
and goes around giving everyone slice.
• Solution 2 – Recursive: Take a slice of cake from the
tray and pass the tray to the next person who takes a
slice from the tray and passes the tray to the next
person, who takes a slice from the tray and passes the
tray to the next person…

4
 Logical
• An algorithm may be viewed as controlled logical
deduction. This notion may be expressed as Algorithm
= logic + control.

5
 Serial or parallel or distributed

• Parallel algorithms take advantage of computer

architectures where several processors can work on a
problem at the same time, whereas distributed
algorithms utilize multiple machines connected with a
network.

6
 Deterministic or non-
deterministic:
• non-deterministic algorithm solves problems via
guessing although typical guesses are made more
accurate through the use of heuristics.

7
Complexity
 Intuition
• Are the following operations “fast” or “slow”?
arra linked
y list
behavior fast/slow behavior fast/slow
add at front slow add at front fast
add at back fast add at back slow
get at index fast get at index slow
resizing slow resizing fast
binary (pretty) fast binary (really) slow
search search

stac queu
k e
behavior fast/slow behavior fast/slow
push fast enqueue fast
pop fast dequeue fast
9
 Complexity
• “Complexity” is a word that has a special meaning in
computer science

• complexity: the amount of computational resources a

block of code requires in order to run

• main computational resources:

– time: how long the code takes to execute
– space: how much computer memory the code consumes

• Often, one of these resources can be traded for the other:

– e.g.: we can make some code use less memory if we don’t
mind that it will need more time to finish (and vice-versa)
10
 Time Complexity
• We usually care more about time complexity
– we want to make our code run fast!

• But we don’t merely measure how long a piece of code

takes to determine it’s time complexity

• That approach would have results strongly skewed by:

– size/kind of input
– speed of the computer’s hardware
– other programs running at the same time
– operating system Ethan: “Sweet! My code
ran
– etc
in 3.97
seconds!” 11
 Time Complexity
• Instead, we care about the growth rate as the input
size increase

• First, we have to be able to measure the input size

– the number of names to sort
– the number of nodes in a linked list
– the number of students in the IPL queue

• We usually call the input size “n”

• What happens if we double the input size (n  2n)?

Ethan: “Aww man. I doubled my input
size, and now my code takes over a
minute”
12
 Time Complexity
• We can learn about this growth rate in two ways:
– by examining code
– by running the same code over different input sizes

• Measuring the growth rate by is one of the few places

where computer science is like the other sciences
– here, we actually collect data

• But this data can be misleading

– modern computers are very complex
– some features, like cache memory, interfere with our data

13
 Time Complexity: Rule of
Thumb
• rule of thumb: this often works (but sometimes
doesn’t)

• rule of thumb for determining time complexity:

– find the statement executed most often in the code
– count how often it’s executed

• But be careful how you count!

– counting is hard 

• We’ll count most “simple” statements as 1

– this includes i = i + 1, int x = elementData[i], etc
– but not loops! (or methods that contain loops!)
14
 Time Complexity: Rule of
Thumb
• Examples:

1 int x = 4 * 10 / 3 + 2 - 10 * 42;

for (int i = 0; i < 100; i++) {

10 x += i;
0 } n2 + 100 +
1
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
n x += i + j;
2
}
}

15
 Optimizing Code
• Many programmers care a lot about efficiency

• But many inexperienced programmers obsess about it

– and the wrong kind of efficiency, at that

• Which one is faster:

System.out.println(“print”);
System.out.println(“me”); Who cares?
They’re both
or:
about the same
System.out.println(“print\nme”);

• If you’re going to optimize some code, improve it so that

you get a real benefit!
16
Don Knuth says...

Premature
optimizatio
n is the
root of all
evil!

Don Knuth
• Professor Emeritus at
Stanford
• “Father” of algorithm
analysis 17
 Growth Rates
• We care about n as it gets bigger
– it’s a lot like calculus, with n approaching infinity
• you all know calculus, right?

• So, when we see something complicated like this:

n 3  18n 2 385n  708

4 2
0.005n  13n 73842
• We can remove all the annoying terms:

n3
 n4
• And as n gets really big, this approaches 0
18
 Big O Notation
• We need a way to write a growth rate of a block of code

• Computer scientists use big O (“big oh”) notation

– O(n)
– O(n2)

• In big O notation, we ignore coefficients that are

constants
– 5n is written as O(n)
– 100n is also written as O(n)
– 0.05n2 is written as O(n2) and will eventually outgrow O(n)

• Each O([something]) specifies a different complexity class

19
 Complexity Classes
Complexi Name Example
ty Class
O(1) constant time popping a stack
O(log n) logarithmic binary search on an array
time
O(n) linear time scanning all elements of an
array
O(n log n) log-linear time binary search on a linked list
and good sorting algorithms
O(n2) quadratic time poor sorting algorithms (like
inserting n items into
SortedIntList)
O(n3) cubic time (example later today)
O(2n) exponential Really hard problems. These
time grow so fast that they’re
impractical 20
 Examples of Each Complexity
Class’s Growth Rate
• Assume that all complexity classes can process an
input of size 100 in 100ms

Input O(1) O(log O(n) O(n log O(n2) O(n3) O(2n)

Size n) n)
(n)
100 100m 100ms 100m 100ms 100ms 100ms 100ms
s s
200 100m 115ms 200m 240ms 400ms 800ms 32.7 sec
s s
400 100m 130ms 400m 550ms 1.6 sec 6.4 sec 12.4 days
s s
800 100m 145ms 800m 1.2 sec 6.4 sec 51.2 36.5
s s sec million
years
1600 100m 160ms 1.6 2.7 sec 25.6 sec 6 min 4.21 * 21
24
 Case Study: maxSum
• Given an array of ints, find the subsequence with the
maximum sum

• Additional information:
– values in the array can be negative, positive, or zero
– the subsequence must be contiguous (can’t skip
elements)
– you must compute:
• the value of the sum of this subsequence
• the starting index (inclusive) of this subsequence
• the stopping index (inclusive) of this subsequence

• This has been used as a Microsoft interview question!

22
 Case Study: maxSum
• For example: suppose you were given the following array:
0 1 2 3 4 5 6 7 8 9

14 8 -23 4 6 10 -18 5 5 11

max
subsequence
max sum: 4 + 6 + 10 + -18 + 5 + 5 + 11 = 23
starting index: 3
stopping index: 9

• Notice that we included a negative number (-18)!

– but this also let us include the 4, 6, and 10
23
 Case Study: maxSum
• First, a simple way to solve this: try every
subsequence!

• Psuedo-code:
// try every start index, from 0 to size - 1
// try every stop index, from start index to size - 1
// compute the sum from start index to stop index

• Converted to be part code, part pseudo-code:

for (int start = 0; start < list.length; start++) {
for (int stop = start; stop < list.length; stop++) {
// compute the sum from start index to stop index
}
}
24
 Case Study: maxSum
• Now, we just need to convert this pseudo-code:
// compute the sum from start index to stop index

• ...into code. Here’s one way:

int sum = 0;
for (int i = start; i <= stop; i++) {
sum += list[i];
}

• And we need to store this sum if it becomes our max sum:

if (sum > maxSum) {
maxSum = sum;
}
25
 Case Study: maxSum
• Here’s our whole algorithm, with some initialization:
int maxSum = list[0];
int maxStart = 0;
int maxStop = 0;
for (int start = 0; start < list.length; start++) {
for (int stop = start; stop < list.length; stop++)
{
int sum = 0;
for (int i = start; i <= stop; i++) {
sum += list[i]; this is the most
} frequently
if (sum > maxSum) {
executed line of
maxSum = sum;
code
maxStart = start;
maxStop = stop;
}
}
}
26
 Case Study: maxSum
• What complexity class is the previous algorithm?
– O(n3) (cubic time)

• This is pretty slow

– we recalculate the entire sum every time:
• calculate the entire sum from index 0 to index 0
• calculate the entire sum from index 0 to index 1
• ...
• calculate the entire sum from index 0 to index 998
• calculate the entire sum from index 0 to index 999

• How can we improve it?

– remember the old sum (values list[start] to list[stop-1])
– add the single new value (list[stop]) to the old sum
27
 Case Study: maxSum
• Improved code, now with a running sum:
int maxSum = list[0];
int maxStart = 0;
int maxStop = 0;
for (int start = 0; start < list.length; start++) {
int sum = 0;
for (int stop = start; stop < list.length;
stop++) {
sum += list[stop];
if (sum > maxSum) { these are the
maxSum = sum; most frequently
executed lines of
maxStart = start;
maxStop = stop; code
}
}
}
28
 Case Study: maxSum
• What complexity class is the previous algorithm?
– O(n2) (quadratic time)

• This is a big improvement over the old code

– it now runs much faster for large input sizes

• And it wasn’t that hard to convert our first version to

this improved version

• But we can still do better

– if only we can figure out how...

29
 Case Study: maxSum
• There is a better algorithm, but it’s harder to
understand
– and I’m not going to formally prove that it always works

• The main idea is that we will find the max subsequence

without computing all the sums
– this will eliminate our inner for loop
– ...which means we can find the subsequence with just a
single loop over the array

• We need to know when to reset our running sum

– this will “throw out” all previous values
– but we have to know for sure that we don’t want them!
30
 Case Study: maxSum
• Suppose we’re about to look at an index greater than 0
– for example index 10

• If we’re going to include previous values, we must include

the value at the index 9
– index 9 is immediately before index 10

• We want to use only the best subsequence that ends at 9

• And only if it helps us. When does it help?

– it helps when the sum of this old subsequence is positive
– and hurts when the sum of this old subsequence is negative

31
 Case Study: maxSum
• Best code:
int maxSum = list[0];
int maxStart = 0;
int maxStop = 0;
int sum = 0;
int start = 0;
for (int i = 0; i < list.length; i++) {
if (sum < 0) {
sum = 0;
start = i;
} these are the
sum += list[i]; most frequently
if (sum > maxSum) {
executed lines of
maxSum = sum;
maxStart = start;
code
maxStop = i;
}
}
32
 Case Study: maxSum
• What complexity class is our best algorithm?
– O(n) (linear time)

• This is again a big improvement over both other versions

• But let’s not just take my word for it

• Let’s conduct an experiment (in MaxSum.java -- available

on the website)
– we’ll give an array of ints of some size to each algorithm
– ...and then give the algorithm an array of twice that size
– ...and then give the algorithm an array of triple that size
– ...and see how long it takes
33
 MaxSum.java
• Output for an array of 1500 ints in the O(n3) algorithm:
How many numbers do you want to use? 1500
Which algorithm do you want to use? 1
Max = 172769
Max start = 677
Max stop = 971
for n = 1500, time = 0.96

Max = 198959
Max start = 1727
Max stop = 1972
for n = 3000, time = 7.543
these numbers are close
Max = 614711 to 8 (23) and 27 (33)
Max start = 251 respectively, so this
Max stop = 3870
for n = 4500, time = 25.427 algorithm exhibited O(n3)
growth
Double/single ratio = 7.857291666666667
Triple/single ratio = 26.486458333333335

34
 MaxSum.java
• Output for an array of 30,000 ints in the O(n2)
algorithm:
How many numbers do you want to use? 30000
Which algorithm do you want to use? 2
Max = 809852
Max start = 10146
Max stop = 19139
for n = 30000, time = 0.988

Max = 2170008
Max start = 9832
Max stop = 25833
for n = 60000, time = 3.935 these numbers are close
to 4 (22) and 9 (32)
Max = 4112483 respectively, so this
Max start = 74
Max stop = 88871
algorithm exhibited O(n2)
for n = 90000, time = 8.853 growth

Double/single ratio = 3.9827935222672064

Triple/single ratio = 8.960526315789474
35
 MaxSum.java
• Output for an array of 5,000,000 ints in the O(n)
algorithm:
How many numbers do you want to use? 5000000
Which algorithm do you want to use? 3
Max = 22760638
Max start = 456
Max stop = 4998134
for n = 5000000, time = 0.016

Max = 27670910
look at how
Max start = 1045808 fast it
Max stop = 9643590 processed
for n = 10000000, time = 0.031
5,000,000,
Max = 28178549 10,000,000,
Max start = 239081 and
Max stop = 8574748 15,000,000
these numbers are close
for n = 15000000, time = 0.044
ints!
to 2 and 3 respectively,
Double/single ratio = 1.9375 so this algorithm
Triple/single ratio = 2.75 exhibited O(n) growth 36