Design and Analysis of Algorithms

CSE 5311
Lecture 3 Divide-and-Conquer

Junzhou Huang, Ph.D.

Department of Computer Science and Engineering

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 1

 Reviewing: Θ-notation

Definition:
Θ(g(n)) = { f (n) : there exist positive constants c1, c2, and
n0 such that 0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n)
for all n ≥ n0 }
Basic Manipulations:
• Drop low-order terms; ignore leading constants.
• Example: 3n3 + 90n2 – 5n + 6046 = Q(n3)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 2

 Reviewing: Insertion Sort Analysis
Worst case: Input reverse sorted.
n
T(n) = ∑Θ( j) = Θ(n2) [arithmetic series]
j=2

Average case: All permutations equally likely.

n
T(n) = ∑Θ( j / 2) = Θ(n2)
j=2

Is insertion sort a fast sorting algorithm?

• Moderately so, for small n.
• Not at all, for large n.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 3

 Reviewing: Recurrence for Merge Sort

Θ(1) if n = 1;
T(n) =
2T(n/2) + Θ(n) if n > 1.

• We shall usually omit stating the base case when T(n)

= Θ(1) for sufficiently small n, but only when it has
no effect on the asymptotic solution to the
recurrence.
• Next Lecture will provide several ways to find a good
upper bound on T(n).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 4

 Reviewing: Recursion Tree

Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

cn cn

cn/2 cn/2 cn

h = lg n cn/4 cn/4 cn
cn/4 cn/4

Θ(1) #leaves = n Θ(n)

Total = Θ(n lg n)
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 5
 Solving Recurrences
• Recurrence
– The analysis of integer multiplication from last lecture required us to solve a
recurrence
– Recurrences are a major tool for analysis of algorithms
– Divide and Conquer algorithms which are analyzable by recurrences.

• Three steps at each level of the recursion:

– Divide the problem into a number of subproblems that are smaller
instances of the same problem.
– Conquer the subproblems by solving them recursively. If the subproblem
sizes are small enough, however, just solve the subproblems in a
straightforward manner.
– Combine the solutions to the subproblems into the solution for the original
problem.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 6

 Recall: Integer Multiplication

• Let X = A B and Y = C D where A,B,C and D are n/2

bit integers
• Simple Method: XY = (2n/2A+B)(2n/2C+D)
• Running Time Recurrence
T(n) < 4T(n/2) + Θ(n)

How do we solve it?

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 7

 Substitution Method

The most general method:

1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.

Example: T(n) = 4T(n/2) + Θ(n)

• [Assume that T(1) = Θ(1).]
• Guess O(n3) . (Prove O and Ω separately.)
• Assume that T(k) ≤ ck3 for k < n .
• Prove T(n) ≤ cn3 by induction.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 8

 Example of substitution

T ( n ) = 4 T ( n / 2 ) + Θ(n)
≤ 4 c ( n / 2 )3 + Θ(n)
= ( c / 2 ) n 3 + Θ(n)
= cn 3 − (( c / 2 ) n 3 − Θ(n) ) desired – residual
≤ cn 3 desired
We can imagine Θ(n)=100n. Then, whenever (c/2)n3 – 100n ≥
0, for example, if c ≥ 200 and n ≥ 1.

residual

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 9

 Example
• We must also handle the initial conditions, that is, ground the
induction with base cases.
• Base: T(n) = Θ(1) for all n < n0, where n0 is a suitable constant.
• For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.

This bound is not tight!

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 10

 A Tighter Upper Bound?

We shall prove that T(n) = O(n2).

Assume that T(k) ≤ ck2 for k < n:

T ( n ) = 4 T ( n / 2 ) + 100n
≤ cn 2 + 100n
≤ cn 2

for no choice of c > 0. Lose!

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 11

 A Tighter Upper Bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.

Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.

T ( n ) = 4 T ( n / 2 ) + 100n
≤ 4 ( c ( n / 2 ) 2 − c ( n / 2 ))+ 100n
1 2
= c n 2 − 2 c n + 100n
1 2
= c n 2 − c n − ( c2 n −100n )
1 2
≤ c n 2 − c n if c > 100.
1 2 2

Pick c1 big enough to handle the initial conditions.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 12

 Recursion-tree Method
• A recursion tree models the costs (time) of a recursive execution
of an algorithm.
• The recursion tree method is good for generating guesses for
the substitution method.
• The recursion-tree method can be unreliable, just like any
method that uses ellipses (…).
• However, the recursion-tree method promotes intuition

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 13

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 14

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

T(n)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 15

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2

T(n/4) T(n/2)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 16

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2

(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 17

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2

(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

Θ(1)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 18

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2 n2
(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

Θ(1)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 19

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2 n2
(n/4)2 (n/2)2 5 n2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2

Θ(1)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 20

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2 n2
(n/4)2 (n/2)2 5 n2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2 25 n2
256

Θ(1)

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 21

 Example of Recursion Tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2 n2
(n/4)2 (n/2)2 5 n2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2 25 n2
256

Θ(1)
Total = 25 ( 5 2
n 1+16 + 16 + 16 +L5 3
( ) ( ) )
= Θ(n2)
geometric series
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 22
 Appendix: Geometric Series

n+1
1− x
1+ x + x2 +L+ xn = for x ≠ 1
1− x

2 1 for |x| < 1

1+ x + x +L=
1− x

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 23

 The Master Method
The master method applies to recurrences of the form
T(n) = a T(n/b) + f (n) ,
where a ≥ 1, b > 1, and f is asymptotically positive.

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 24

 Idea of Master Theorem

Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) / f (n/b) a f (n/b)
a
h = logbn
f (n/b2) f (n/b2) / f (n/b2) a2 f (n/b2)

#leaves = ah
nlogbaΤ (1)
Τ (1) = alogbn
= nlogba

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 25

 Case (I)

Compare f (n) with nlogba:

1. f (n) = O(nlogba – ε) for some constant ε > 0.
• f (n) grows polynomially slower than nlogba (by an nε factor).
Solution: T(n) = Θ(nlogba) .

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 26

 Idea of Master Theorem

Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) / f (n/b) a f (n/b)
a
h = logbn
f (n/b2) f (n/b2) / f (n/b2) a2 f (n/b2)

CASE 1: 1 The weight increases

geometrically from the root to the leaves.
The leaves hold a constant fraction of nlogbaΤ (1)
Τ (1) the total weight.
Θ(nlogba)
f(n)= nlogba – ε and a f(n/b)=a (n/b)logba – ε = b ε nlogba – ε
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 27
 Case (II)

Compare f (n) with nlogba:

2. f (n) = Θ(nlogba ) for some constant k ≥ 0.

• f (n) and nlogba grow at similar rates.
Solution: T(n) = Θ(nlogba lgn) .

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 28

 Idea of Master Theorem

Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) / f (n/b) a f (n/b)
a
h = logbn
f (n/b2) f (n/b2) / f (n/b2) a2 f (n/b2)

CASE 2: 2 (k = 0) The weight is nlogbaΤ (1)
Τ (1) approximately the same on each of
the logbn levels.
Θ(nlogbalg n)
f(n)=nlogba and af(n/b)=a (n/b)logba= nlogba
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 29
 Case (III)

Compare f (n) with nlogba:

3. f (n) = Ω(nlogba + ε) for some constant ε > 0.

• f (n) grows polynomially faster than nlogba (by an nε factor),
and f (n) satisfies the regularity condition that a f (n/b) ≤ c f
(n) for some constant c < 1.
Solution: T(n) = Θ( f (n) ) .

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 30

 Idea of master theorem

Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) / f (n/b) a f (n/b)
a
h = logbn
f (n/b2) f (n/b2) / f (n/b2) a2 f (n/b2)
CASE 3:3 The weight decreases

geometrically from the root to the leaves.
The root holds a constant fraction of the nlogbaΤ (1)
Τ (1) total weight.
Θ( f (n))
f(n)= nlogba + ε and a f(n/b)=a (n/b)logba + ε = b -ε nlogba + ε
Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 31
 Examples

Ex. T(n) = 4T(n/2) + n

a = 4, b = 2 ⇒ nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 – ε) for ε = 1.
∴ T(n) = Θ(n2).

Ex. T(n) = 4T(n/2) + n2

a = 4, b = 2 ⇒ nlogba = n2; f (n) = n2.
CASE 2: f (n) = Θ(n2lg0n), that is, k = 0.
∴ T(n) = Θ(n2lg n).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 32

 Examples

Ex. T(n) = 4T(n/2) + n3

a = 4, b = 2 ⇒ nlogba = n2; f (n) = n3.
CASE 3: f (n) = Ω(n2 + ε) for ε = 1
and 4(n/2)3 ≤ cn3 (reg. cond.) for c = ½<1
∴ T(n) = Θ(n3).

Ex. T(n) = 4T(n/2) + n2/lg n

a = 4, b = 2 ⇒ nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant ε > 0, we have nε = ω(lg n).

Dept. CSE, UT Arlington CSE5311 Design and Analysis of Algorithms 33