Algorithms for Data Science

CSOR W4246

Eleni Drinea
Computer Science Department

Columbia University

Asymptotic notation, mergesort, recurrences

Outline

1 Asymptotic notation

2 The divide & conquer principle; application: mergesort

3 Solving recurrences and running time of mergesort

Review of the last lecture

I Introduced the problem of sorting.

I Analyzed insertion-sort.
3n2 7n
I Worst-case running time: T (n) = 2 + 2 −4
I Space: in-place algorithm
I Worst-case running time analysis: a reasonable measure of
algorithmic efficiency.
I Defined polynomial-time algorithms as “efficient”.
I Argued that detailed characterizations of running times are
not convenient for understanding scalability of algorithms.
Running time in terms of # primitive steps

We need a coarser classification of running times of algorithms;

exact characterizations

I are too detailed;

I do not reveal similarities between running times in an
immediate way as n grows large;
I are often meaningless: high-level language steps will
expand by a constant factor that depends on the
hardware.
Today

1 Asymptotic notation

2 The divide & conquer principle; application: mergesort

3 Solving recurrences and running time of mergesort

Aymptotic analysis

A framework that will allow us to compare the rate of growth of

different running times as the input size n grows.

I We will express the running time as a function of

the number of primitive steps; the latter is a function
of the input size n.
I To compare functions expressing running times, we will
ignore their low-order terms and focus solely on the
highest-order term.
Asymptotic upper bounds: Big-O notation

Definition 1 (O).
We say that T (n) = O(f (n)) if there exist constants c > 0 and
n0 ≥ 0 s.t. for all n ≥ n0 , we have T (n) ≤ c · f (n) .

T(n) = O(f(n))
c f(n)

T(n)

n
n0
Asymptotic upper bounds: Big-O notation

Definition 1 (O).
We say that T (n) = O(f (n)) if there exist constants c > 0 and
n0 ≥ 0 s.t. for all n ≥ n0 , we have T (n) ≤ c · f (n) .

Examples: Show that T (n) = O(f (n)) when

I T (n) = an2 + b, a, b > 0 constants and f (n) = n2 .
I T (n) = an2 + b and f (n) = n3 .
Asymptotic lower bounds: Big-Ω notation

Definition 2 (Ω).
We say that T (n) = Ω(f (n)) if there exist constants c > 0 and
n0 ≥ 0 s.t. for all n ≥ n0 , we have T (n) ≥ c · f (n).

T(n) = Ω(f(n))
T(n)

c f(n)

n
n0
Asymptotic lower bounds: Big-Ω notation

Definition 2 (Ω).
We say that T (n) = Ω(f (n)) if there exist constants c > 0 and
n0 ≥ 0 s.t. for all n ≥ n0 , we have T (n) ≥ c · f (n).

Examples: Show that T (n) = Ω(f (n)) when

I T (n) = an2 + b, a, b > 0 constants and f (n) = n2 .
I T (n) = an2 + b, a, b > 0 constants and f (n) = n.
Asymptotic tight bounds: Θ notation
Definition 3 (Θ).
We say that T (n) = Θ(f (n)) if there exist constants c1 , c2 > 0
and n0 ≥ 0 s.t. for all n ≥ n0 , we have

c1 · f (n) ≤ T (n) ≤ c2 · f (n).

c2 f(n)

T(n)
T(n) = ϴ(f(n))
c1 f(n)

n
n0
Asymptotic tight bounds: Θ notation
Definition 3 (Θ).
We say that T (n) = Θ(f (n)) if there exist constants c1 , c2 > 0
and n0 ≥ 0 s.t. for all n ≥ n0 , we have

c1 · f (n) ≤ T (n) ≤ c2 · f (n).

Equivalent definition
T (n) = Θ(f (n)) if T (n) = O(f (n)) and T (n) = Ω(f (n))
Asymptotic tight bounds: Θ notation
Definition 3 (Θ).
We say that T (n) = Θ(f (n)) if there exist constants c1 , c2 > 0
and n0 ≥ 0 s.t. for all n ≥ n0 , we have

c1 · f (n) ≤ T (n) ≤ c2 · f (n).

Equivalent definition
T (n) = Θ(f (n)) if T (n) = O(f (n)) and T (n) = Ω(f (n))

Notational convention: log n stands for log2 n

Examples: Show that T (n) = Θ(f (n)) when
I T (n) = an2 + b, a, b > 0 constants and f (n) = n2
I T (n) = n log n + n and f (n) = n log n
Asymptotic upper bounds that are not tight: little-o

Definition 4 (o).
We say that T (n) = o(f (n)) if, for any constant c > 0, there
exists a constant n0 ≥ 0 such that for all n ≥ n0 , we have
T (n) < c · f (n) .
Asymptotic upper bounds that are not tight: little-o

Definition 4 (o).
We say that T (n) = o(f (n)) if, for any constant c > 0, there
exists a constant n0 ≥ 0 such that for all n ≥ n0 , we have
T (n) < c · f (n) .

I Intuitively, T (n) becomes insignificant relative to f (n) as

n → ∞.
T (n)
I Proof by showing that lim = 0 (if the limit exists).
n→∞ f (n)
Asymptotic upper bounds that are not tight: little-o

Definition 4 (o).
We say that T (n) = o(f (n)) if, for any constant c > 0, there
exists a constant n0 ≥ 0 such that for all n ≥ n0 , we have
T (n) < c · f (n) .

I Intuitively, T (n) becomes insignificant relative to f (n) as

n → ∞.
T (n)
I Proof by showing that lim = 0 (if the limit exists).
n→∞ f (n)

Examples: Show that T (n) = o(f (n)) when

I T (n) = an2 + b, a, b > 0 constants and f (n) = n3 .
I T (n) = n log n and f (n) = n2 .
Asymptotic lower bounds that are not tight: little-ω

Definition 5 (ω).
We say that T (n) = ω(f (n)) if, for any constant c > 0, there
exists a constant n0 ≥ 0 such that for all n ≥ n0 , we have
T (n) > c · f (n).
Asymptotic lower bounds that are not tight: little-ω

Definition 5 (ω).
We say that T (n) = ω(f (n)) if, for any constant c > 0, there
exists a constant n0 ≥ 0 such that for all n ≥ n0 , we have
T (n) > c · f (n).

I Intuitively T (n) becomes arbitrarily large relative to f (n),

as n → ∞.
T (n)
I T (n) = ω(f (n)) implies that lim = ∞, if the limit
n→∞ f (n)
exists. Then f (n) = o(T (n)).
Asymptotic lower bounds that are not tight: little-ω

Definition 5 (ω).
We say that T (n) = ω(f (n)) if, for any constant c > 0, there
exists a constant n0 ≥ 0 such that for all n ≥ n0 , we have
T (n) > c · f (n).

I Intuitively T (n) becomes arbitrarily large relative to f (n),

as n → ∞.
T (n)
I T (n) = ω(f (n)) implies that lim = ∞, if the limit
n→∞ f (n)
exists. Then f (n) = o(T (n)).

Examples: Show that T (n) = ω(f (n)) when

I T (n) = n2 and f (n) = n log n.
I T (n) = 2n and f (n) = n5 .
Basic rules for omitting low order terms from functions

1. Ignore multiplicative factors: e.g., 10n3 becomes n3

2. na dominates nb if a > b: e.g., n2 dominates n
3. Exponentials dominate polynomials: e.g., 2n dominates n4
4. Polynomials dominate logarithms: e.g., n dominates log3 n
⇒ For large enough n,

log n < n < n log n < n2 < 2n < 3n < nn

Properties of asymptotic growth rates

1. Transitivity
1.1 If f = O(g) and g = O(h), then f = O(h).
1.2 If f = Ω(g) and g = Ω(h), then f = Ω(h).
1.3 If f = Θ(g) and g = Θ(h), then f = Θ(h).
2. Sums of up to a constant number of functions
2.1 If f = O(h) and g = O(h), then f + g = O(h).
2.2 Let k be a fixed constant, and let f1 , f2 , . . . , fk , h be
functions such that for all i, fi = O(h). Then
f1 + f2 + . . . + fk = O(h).
3. Transpose symmetry
I f = O(g) if and only if g = Ω(f ).
I f = o(g) if and only if g = ω(f ).
Today

1 Asymptotic notation

2 The divide & conquer principle; application: mergesort

3 Solving recurrences and running time of mergesort

The divide & conquer principle

I Divide the problem into a number of subproblems that are

smaller instances of the same problem.

I Conquer the subproblems by solving them recursively.

I Combine the solutions to the subproblems to get the

solution to the overall problem.
Divide & Conquer applied to sorting

I Divide the problem into a number of subproblems that are

smaller instances of the same problem.
Divide the input array into two lists of equal size.
I Conquer the subproblems by solving them recursively.
Sort each list recursively. (Stop when lists have size 2.)
I Combine the solutions to the subproblems into the
solution for the original problem.
Merge the two sorted lists and output the sorted array.
Mergesort: pseudocode

Mergesort (A, lef t, right)

if right == lef t then return
end if
mid = lef t + b(right − lef t)/2c
Mergesort (A, lef t, mid)
Mergesort (A, mid + 1, right)
Merge (A, lef t, right, mid)

Remarks
I Mergesort is a recursive procedure (why?)
I Initial call: Mergesort(A, 1, n)
I Subroutine Merge merges two sorted lists of sizes bn/2c, dn/2e
into one sorted list of size n. How can we accomplish this?
Merge: intuition

Intuition: To merge two sorted lists of size n/2 repeatedly

I compare the two items in the front of the two lists;
I extract the smaller item and append it to the output;
I update the front of the list from which the item was
extracted.

Example: n = 8, L = {1, 3, 5, 7}, R = {2, 6, 8, 10}

Merge: pseudocode

Merge (A, lef t, right, mid)

L = A[lef t, mid]
R = A[mid + 1, right]
Maintain two pointers pL , pR , initialized to point to the first
elements of L, R, respectively
while both lists are nonempty do
Let x, y be the elements pointed to by pL , pR
Compare x, y and append the smaller to the output
Advance the pointer in the list with the smaller of x, y
end while
Append the remainder of the non-empty list to the output.

Remark: the output is stored directly in A[lef t, right], thus the

subarray A[lef t, right] is sorted after Merge(A, lef t, right, mid).
Merge: optional exercises

Optional exercise 1: write detailed pseudocode or actual

code for Merge

Optional exercise 2: write a recursive Merge

Analysis of Merge

1. Correctness

2. Running time

3. Space
Analysis of Merge: correctness

1. Correctness: by induction on the size of the two lists

(recommended exercise)
2. Running time

3. Space
Merge: pseudocode

Merge (A, lef t, right, mid)

L = A[lef t, mid] →not a primitive computational step!
R = A[mid + 1, right] →not a primitive computational step!
Maintain two pointers pL , pR initialized to point to the first
elements of L, R, respectively
while both lists are nonempty do
Let x, y be the elements pointed to by pL , pR
Compare x, y and append the smaller to the output
Advance the pointer in the list with the smaller of x, y
end while
Append the remainder of the non-empty list to the output.

Remark: the output is stored directly in A[lef t, right], thus the

subarray A[lef t, right] is sorted after Merge(A, lef t, right, mid).
Analysis of Merge: running time

1. Correctness: by induction on the size of the two lists

(recommended exercise)
2. Running time:
I Suppose L, R have n/2 elements each
I How many iterations before all elements from both lists have
been appended to the output?
I How much work within each iteration?

3. Space
Analysis of Merge: space

1. Correctness: by induction on the size of the two lists

(recommended exercise)
2. Running time:
IL, R have n/2 elements each
IHow many iterations before all elements from both lists have
been appended to the output? At most n − 1.
I How much work within each iteration? Constant.

⇒ Merge takes O(n) time to merge L, R (why?).

3. Space: extra Θ(n) space to store L, R (the output of
Merge is stored directly in A).
Refreshing your memory on recursive algorithms

Exercise (recommended): run Mergesort on input

1, 7, 4, 3, 5, 8, 6, 2.
Analysis of Mergesort

1. Correctness

2. Running time

3. Space
Mergesort: correctness
For simplicity, assume n = 2k for integer k ≥ 0.
We will use induction on k.
I Base case: For k = 0, the input consists of 1 item; Mergesort
returns the item.
I Induction Hypothesis: For k ≥ 0, assume that Mergesort
correctly sorts any list of size 2k .
I Induction Step: We will show that Mergesort correctly sorts
any list A of size 2k+1 .
From the pseudocode of Mergesort, we have:
ILine 3: mid takes the value 2k
ILine 4: Mergesort(A, 1, 2k ) correctly sorts the leftmost half
of the input, by the induction hypothesis.
I Line 5: Mergesort(A, 2k + 1, 2k+1 ) correctly sorts the

rightmost half of the input, by the induction hypothesis.

I Line 6: Merge correctly merges its two sorted input lists

into one sorted output of size 2k + 2k .

⇒ Mergesort correctly sorts any input of size 2k+1 .
Running time of Mergesort

The running time of Mergesort satisfies:

T (n) = 2T (n/2) + cn, for n ≥ 2, constant c > 0

T (1) = c

This structure is typical of recurrence relations

I an inequality or equation bounds T (n) in terms of an
expression involving T (m) for m < n
I a base case generally says that T (n) is constant for small
constant n
Remarks
I We ignore floor and ceiling notations.
I A recurrence does not provide an asymptotic bound for
T (n): to this end, we must solve the recurrence.
Today

1 Asymptotic notation

2 The divide & conquer principle; application: mergesort

3 Solving recurrences and running time of mergesort

Solving recurrences, method 1: recursion trees

The technique consists of three steps

1. Analyze the first few levels of the tree of recursive calls
2. Identify a pattern
3. Sum the work spent over all levels of recursion

Example: give an asymptotic bound for the recurrence

describing the running time of Mergesort

T (n) = 2T (n/2) + cn, for n ≥ 2, constant c > 0

T (1) = c
A general recurrence and its solution

The running times of many recursive algorithms can be

expressed by the following recurrence

T (n) = aT (n/b) + cnk , for a, c > 0, b > 1,k ≥ 0

What is the recursion tree for this recurrence?

I a is the branching factor
I b is the factor by which the size of each subproblem shrinks
⇒ at level i, there are ai subproblems, each of size n/bi
⇒ each subproblem at level i requires c(n/bi )k work
I the height of the tree is logb n levels
log
Pb n
Plogb n a i
ai c(n/bi )k = cnk


⇒ Total work: i=0 bk
i=0
Solving recurrences, method 2: Master theorem

Theorem 6 (Master theorem).

If T (n) = aT (dn/be) + O(nk ) for some constants a > 0, b > 1,
k ≥ 0, then

 O(nlogb a ) , if a > bk


T (n) = O(nk log n) , if a = bk

O(nk ) , if a < bk


Example: running time of Mergesort

I T (n) = 2T (n/2) + cn:
a = 2, b = 2, k = 1, bk = 2 = a ⇒ T (n) = O(n log n)
Solving recurrences, method 3: the substitution method

The technique consists of two steps

1. Guess a bound
2. Use (strong) induction to prove that the guess is correct
(See your textbook for more details on this technique.)

Remark 1 (simple vs strong induction).

1. Simple induction: the induction step at n requires that the

inductive hypothesis holds at step n − 1.
2. Strong induction is just a variant of simple induction where
the induction step at n requires that the inductive hypothesis
holds at all previous steps 1, 2, . . . , n − 1.
How would you solve...

1. T (n) = 2T (n − 1) + 1, T (1) = 2

2. T (n) = 2T 2 (n − 1), T (1) = 4

3. T (n) = T (2n/3) + T (n/3) + cn