Algorithms

and
Analysis of
Algorithms

1. Introduction and Overview

Books and Grading

Reference Books
• Algorithm Design by Jon Kleinberg and Éva Tardos. Addison-Wesley, 2005.
• Some of the lecture slides are based on material from the following books:
• Introduction to Algorithms by Thomas Cormen, Charles Leiserson, Ronald Rivest, and Clifford Stein. MIT Press, 2009.
• Algorithms by Sanjoy Dasgupta, Christos Papadimitriou, and Umesh Vazirani. McGraw Hill, 2006.
• The Design and Analysis of Algorithms by Dexter Kozen. Springer, 1992.
• Data Structures and Network Algorithms by Robert Tarjan. Society for Industrial and Applied Mathematics, 1987.
• Linear Programming by Vašek Chvátal. W. H. Freeman, 1983.

Grading
• Programming Assignments: 20%
• Midterm: 30%
• Final Exam: 50%
Tentative Course Outline

1. Introduction and Overview

2. Introduction to Algorithms and Analysis, Asymptotic Notations
3. Recursion
4. Sorting Algorithms
5. Searching Algorithms
6. Divide and Conquer Algorithms
7. Greedy Algorithms
8. Dynamic Programming
9. Graph Algorithms-1
10.Graph Algorithms-2
11.Tree Algorithms-1
12.Tree Algorithms-2
13.Backtracking
14.Approximation Approaches
Outline

1. Introduction
2. Algorithms
3. Course Curriculum Overview
1. Introduction

The objective of the course is to learn practices for efficient problem solving and computing.

ü Problem Solving
- How to write a step by step procedure to solve a given problem
- What are the various paradigms of problem solving in computing à Divide and Conquer, Dynamic Programming, Greedy.
- When to choose which alternative strategy of problem solving

ü Efificiency
- How to evaluate the worst case behavior of an algorithm
- What are the various mathematical tools of algorithm analysis àAsymptotic notations, recurrance relations
- How to decide which algorithm is better

ü Dealing with hard problems

- How to figure out if a given problem is easy or hard
- What to do if we are given a hard problem
1. Introduction

• This course provides a comprehensive Pre-requisite:

introduction to algorithms, their design, analysis,
and implementation. Good to have
• We will learn various algorithmic techniques and
ü Idea of programming (Java, C, Python, C#, etc.)
strategies, as well as how to analyze their ü Basic mathematics (matrices, linear algebra, recurrence
efficiency and correctness. relations, induction)
• Topics covered include sorting, searching, graph ü Concept of data structures (list, array, stack, tree, quene)
algorithms, dynamic programming, and ü Familiarity with graph theory
algorithmic analysis.
 • The logical or mathematical model of a particular organization of data is called a data structure.
• The choice of a particular data model depends on two considerations:
ü it must be rich enough in structure to mirror the actual relationships of data in the real world.
ü structure should be simple enough that one can effectively process the data when necessary.
1. Introduction • Array, Linked List, Stack, Quene, Tree, Graph, Hashtable, …

There are many, but we named a few. We’ll learn these data structures in great detail!

Array

Linked List

Queue Stack
Graph Tree
1. Introduction

Types
of
Data Structures
1. Introduction

Array

Example: A linear array STUDENT consists of the names of six students

• Linear arrays are called one-dimensional arrays because each element in such an
array is referenced by one subscript.
• A two dimensional array is a collection of similar data elements where each
element is referenced by two subscripts.
1. Introduction

Lists (Array List, Linked List, …)

Linked List is a type of data structure in which elements are connected to each other through links or
pointers. Each element (called a node) has two parts - the data and a link pointing to the next node.

Example: Implementing graphs

1. Introduction

Stack

Stack (Last in First Out – LIFO - System)

A linear list in which insertions and deletions can take place only at one end, called the top.

Stack Applications:
• Back and forward buttons in a web browser
• UNDO/REDO functionality in text editors and image editing software
• Implementing recursion in programming
• Delimiter checking
1. Introduction

Example: Let's analyze the

recursion using an
example:

Factorial of the number 5.

The relevant recurrence

relation is:

fact(n) = n * fact(n-1)

n=5
5! = 120
1. Introduction

Quene
Queue (First in first out – FIFO – system) is a linear list in which
• Deletions can take place only at one end of the list, the ‘front’ of the list,
• Insertions can take place only at the other end of the list, the ‘rear’ of the list.

Quene applications
• Cashier line in a store
• A car wash line
• One way exits
1. Introduction

Tree

The data structure which reflects a hierarchical relationship between various elements, a rooted tree graph.

Employee

SSN Name Address Age Salary Dependents

Last First Middle Street Area

City State ZIP

 Unlike linear data structures (Array, Linked List, Queues,
Stacks, etc) which have only one logical way to traverse
them, trees can be traversed in different ways.
1. Introduction

Tree Tree can be traversed in following ways:

1. Inorder Traversal
Example: Algebraic expression
2. Preorder Traversal
(2𝑥 + 𝑦)(𝑎 − 7𝑏)!
3. Postorder Traversal
*

+ ↑

* y - 3

2 X a *

7 b
1. Introduction

Graph
Pairs of data have relationship other than hierarchy.
Example: Airline flights
1. Introduction

Both graphs and trees are structures used to represent relationships between elements,
• Trees are a specific type of acyclic graph with a hierarchical structure, a designated root, and no cycles,
• Graphs are more general and can have cycles and various types of connections.

Basis For Comparison Tree Graph

Path Only one between two vertices. More than one path is allowed.
Root node It has exactly one root node. Graph doesn't have a root node.
Loops No loops are permitted. Graph can have loops.
Complexity Less complex More complex comparatively
Traversal techniques Pre-order, In-order and Post-order. Breadth-first search and depth-first search.
Number of edges n-1 (where n is the number of nodes) Not defined
Model type Hierarchical Network
1. Introduction

The particular data structure that one chooses for a given situation depends largely on the frequency with which
specific operations are performed. Most frequently used operations:

• Traversing: Accessing each record exactly once so that certain items in the record may be processed.
(This accessing an processing is sometimes referred as ‘visiting’ the record.)
• Searching: Finding the location of the record with a given key value, or finding the locations of all the
records which satisfy one or more conditions.
• Inserting: Adding a new record to the structure.
• Deleting: Removing a record from the structure.

Some other operations:

• Sorting: Arranging the records in some logical order (e.g. alphabetically according to some NAME
key, or in numerical order according to some NUMBER key, such as social security number or account
number)
• Merging: Combining the records in two different sorted files into a single sorted file.
1. Introduction

The characteristics of data structures

2. Algorithms

• Algorithms are everywhere

• Today various algorithms shape how we Connect with people

Find our way Eat what we love E-shopping

• Map algorithmsàto reach point B from A. • Nearby restaurants • To find what we need
• Alternate routes related traffic, expected • Feedback given by others who • Tell us about great offers on these
time have been in these restaurants items
• Order food online • Make the purchase

The list go on…

 It’s got to terminate, Should be well Determines the
can not write an defined, no nonsense. strength of the design
endless novel here. of our algorithm
2. Algorithms

An algorithm is a finite sequence of unambiguous instructions for solving a

problem to obtain the required output for a legitimate input in a finite amount
of time.
• An algorithm takes the input to a problem (function) and transforms it to
Algorithm
Program

the output. (A mapping of input to output)

vs

• A computer program is a instance, or concrete representation, of an

algorithm in some programming language.

Implement the search

algorithm in Java

Concrete
representation. One poblem can
have many
algorithms.
2. Algorithms

An algorithm may be written in many ways.

Problem: Finding the maximum of two numbers, num1 and num2.

1. English Description: 2. Flowchart: 3. Pseudocode:

Start with two numbers, num1 and num2.

If num1 is greater than num2, then num1 is
the maximum.
Otherwise, num2 is the maximum.
The maximum number is the result.
2. Algorithms

What is meant by Algorithm Analysis?

• Algorithm analysis is an important part of computational complexity theory, which provides theoretical
estimation for the required resources of an algorithm to solve a specific computational problem.
• Analysis of algorithms is the determination of the amount of time and space resources required to execute it.

Why Analysis of Algorithms is important?

ü To predict the behavior of an algorithm without implementing it on a specific computer.

ü It is much more convenient to have simple measures for the efficiency of an algorithm than to implement the
algorithm and test the efficiency every time a certain parameter in the underlying computer system changes.
ü It is impossible to predict the exact behavior of an algorithm. There are too many influencing factors. The
analysis is thus only an approximation; it is not perfect.
ü More importantly, by analyzing different algorithms, we can compare them to determine the best one for our
purpose.
2. Algorithms

Very similar problems can have very different complexity. Recall: Complexity
Class Characteristic feature

• P: class of problems solvable in polynomial time. O(nk) for P Easily solvable in polynomial time.
some constant k.
- Shortest paths in a graph can be found in O(V2) for example. NP Yes, answers can be checked in polynomial time.
- Merge Sort
Co-NP No, answers can be checked in polynomial time.
• NP: class of problems verifiable in polynomial time.
- Hamiltonian cycle in a directed graph G(V,E) is a simple NP-hard All NP-hard problems are not in NP and it takes a
long time to check them.
cycle that contains each vertex in V .
- Determining whether a graph has a hamiltonian cycle is NP- NP-complete A problem that is NP and NP-hard is NP-complete.
complete but verifying that a cycle is hamiltonian is easy.
- Graph coloring.
2. Algorithms

Example: Interval Scheduling

2. Algorithms

Can only do one thing at any day: what is the

maximum number of tasks that you can do?

Arrange tasks in some order and iteratively pick non-

overlapping tasks

Write up a term paper

Party!

Exam study Cinema

Project

Saturday Sunday Monday Tuesday Wednesday

 Greedy Interval Scheduling

2. Algorithms

Interval Scheduling Problem

Input: n intervals [s(i), f(i)) for 1≤ i ≤ n]

Output: A schedule S of the n intervals

Input: n intervals; ith interval: [s(i), f(i)].
No two intervals in S conflict
Output: A valid schedule with maximum number of
|S| is maximized intervals in it (over all valid schedules).
Def: A schedule S ⊆ [n] ([n] = {1, 2, …, n})
Def: A valid schedule S has no conflicts.
Def: intervals i and j conflict if they overlap.
2. Algorithms

Conflicts:

No conflicts:

j
2. Algorithms

Example 1: No intervals overlap

R: set of requests No intervals overlap

Set S to be the empty set

While R is not empty

Choose i in R

Add i to S

Remove i from R

Return S*= S
2. Algorithms

Example 2: At most one overlap/task

R: set of requests

Set S to be the empty set

While R is not empty

Choose i in R

Add i to S

Remove i from R

Remove all tasks that conflict with i from R

Return S*= S
2. Algorithms

Making it more formal Task 4 Task 5

More than one conflict

Task 3 Task 2

Associate a
Task 1 value v(i)
with task i

Set S to be the empty set

While R is not empty

Choose i in R that minimizes v(i)
Add i to S
Remove all tasks that conflict with i from R

Return S*= S
2. Algorithms

Task 4 Task 5

v(i) = f(i) – s(i) Task 3 Task 2

Smallest duration first

Task 1

Set S to be the empty set

While R is not empty

Choose i in R that minimizes f(i) – s(i)
Add i to S
Remove all tasks that conflict with i from R

Return S*= S
 2. Algorithms

Task 4 Task 5
v(i) = s(i)
Earliest time first? Task 3 Task 2

Task 1

Set S to be the empty set

While R is not empty

Choose i in R that minimizes s(i)
Add i to S
Remove all tasks that conflict with i from R

Return S*= S
 2. Algorithms

Task 4 Task 5
Not so fast….
Earliest time first? Task 3 Task 2

Task 1

Task 6

Set S to be the empty set

While R is not empty

Choose i in R that minimizes s(i)

Add i to S
Remove all tasks that conflict with i from R

Return S*= S
 2. Algorithms

Pick job with minimum conflicts Task 4 Task 5

Task 3 Task 2

Task 1

Task 6

Set S to be the empty set

While R is not empty Task 1 à3

Task 2 à 3
Choose i in R that has smallest number of conflicts
Task 3 à2
Add i to S
Task 4 à3
Remove all tasks that conflict with i from R Task 5 à2
Task 6 à5
Return S*= S
2. Algorithms

Output: [Task 2, Task 3, Task 4] is an optimal solution because these

tasks have no conflicts with each other and any set with 4 tasks will
have at least two intervals in conflict.
2. Algorithms

Task 7

Task 4 Task 5 Task 17 Task 18

Task 3 Task 2 Task 15 Task 16

Task 1 Task 12 Task 13 Task 14

Task 6 Task 8 Task 9 Task 10 Task 11

Set S to be the empty set

While R is not empty

Choose i in R that has smallest number of conflicts

Add i to S

Remove all tasks that conflict with i from R

Return S*= S
2. Algorithms
2. Algorithms

Dealing with intractability

3. Course Curriculum Overview

Introduction and Overview

Data Structures
• Linear
ü Array
ü List
ü Stack
ü Quene

• Nonlinear
ü Graph
ü Tree
ü Heap
3. Course Curriculum Overview

Asymptotic Notations
• Big O, Big Omega, and Big Theta
• Problems on Big O
• Algorithmic Complexity with Asymptotic Notations

Recursion
• Linear Search, Greatest Common Divisor
• Factorial, Tail Recursion
• Recurrence Relations, Substitution Method
• Hanoi Towers

Divide and Conquer

• Binary Search
• Master Method
• Merge Sort
• Quick Sort
3. Course Curriculum Overview

Dynamic Programming
• Fibonacci Numbers
• Rod Cutting
• Matrix Chain Multiplication
• Longest Common Subsequence

Greedy Algorithms
• Knapsack Problem
• Minimum Spanning Tree: Kruskal Algorithm
• Job Sequencing with Deadlines
• Heap Shortest Path Algorithms
• Heap Sort • Dijkstra’s Algorithm
• Priority Quene • Bellman Ford Algorithm
• Minimum Spanning Tree: Prim’s Algorithm • Topological Sort
• Huffman’s Codes • Shortest Path by Topological Sort
• Floyd Warshall Algorithm
3. Course Curriculum Overview

String Matching
• Brute Force Matcher
• String Matching with Finite Automaton
• Pattern Pre-Processing
• The Knuth Morris Pratt Algorithm

Backtracking
• Rat in Maze
• N-Queens Algorithm
• Graph Coloring
• Hamiltonian Cycles

Branch and Bound

• Introduction to Branch and Bound
• 0/1 Knapsack Problem
• The 15 Puzzle Problem
• Solvability of 15 Puzzles
3. Course Curriculum Overview

NP Completeness

Approximation Algorithms
Algorithms
and
Analysis of
Algorithms