Asymptotics and Recurrence Equations: Analysis of Algorithms

Asymptotics and Recurrence
Equations
Analysis of Algorithms
Prepared by
John Reif, Ph.D.

Equations
a) Computational Complexity of a
Program
b) Worst Case and Expected Bounds
c) Definition of Asymptotic Equations
d) Solution of Recurrence Notation
Readings
Main Reading Selection:
CLR, Chapter 3, 4 and Appendix A
Goal
To estimate and compare growth rates
of functions
Ignore constant factors of growth
f(n) is asymptotically equal to g

(n)
f(n) ~ g(n)
if
f(n)
lim
=1
n g(n)
f(n) is little o g(n)

f(n)
=0
n g(n)
f(n) is o(g(n)) if lim
f(n) is big O g(n)

f(n)
f(n) is O(g(n)) if lim sup
c
g(n)
n
Example of f(n) is O(g(n))

c,n 0 > 0
s.t. f(n) c g(n)
for all n n 0
f(n) is order at least g(n)

f(n) is (g(n)) if
f(n)
lim inf
c
g(n)
n
n 0 , c > 0 s.t. f(n) c g(n)

for all n n 0
Example of f(n) is (g(n))
f(n) is order tight with g(n)

f(n) is (g(n)) if
n 0 , c, c' s.t. c g(n) f(n) c' g(n)
Suppose my algorithm runs in time

O(n)
Don t say:
his runs in time O(n2) so is worse
But prove:
his runs in time (n2) so is worse
Must find a worst case input of length
n for which his algorithm takes time >
cn2 for all n > n0
Use of O Notation
N is O(n2)
sometimes written
n = O(n2)
But n2 is not O(n) so can t use
identities!
The two sides of the equality do not
play a symmetric role
Use of Asymptotic Notation

Write
As
" f(n) g(n) is o(h(n))"
f(n) = g(n) + o(h(n))
Example
n
1
1
= 1 + + O 2
n -1
n
n
1 1
= 1 + + o
n
n
= 1 + o(1) as n
Convergent Power Sum

1
x =
O(1), for 0 < x < 1
1 x
i =0
A polynomial is asymptotically equal to

its leading term as x
d
( )
i
d
a
x
=
x
i
i =0
d
( )
i
d +1
a
x
=
o
x
i
i =0
d
i
d
a
x
~
a
x
i
d
i =0
Sums of Powers
for n
n
i =1
1
i ~
n d +1
d +1
d
Or equivalently
n
i =1
1
i =
n d +1 + o n d +1
d +1
d
( )
Examples
n
n2
i ~
2
i =1
n 2
3
n
i ~ 3
i =1
2nd order asymptotic expansion
n
i =1
1
1 d
d +1
i =
n + n + o n d 1
d +1
2
d
( )
Asymptotic Expansion of f(n) as

n n0
f(n) ~ ci g i (n)
i =1
If
(1)g i +1 (n) = o(g i (n))

for all i 1
and
k
(2) f(n) = c i g i (n) + o(g k (n))

i =1
for all k 1
Bounding Sums by Integrals
n +1
k =1
f(k) f(x) dx f (k + 1)
k =1

(cont d)
n
n +1
k =1
f(k) f(x) dx f (k + 1)
k =1
n +1
So
n +1
k =1
f(x) dx - f(n + 1) + f(1) f(k) f(x) dx

1
Example if f(x) = ln(x) then
ln (x)dx = xln(x) x

(cont d)
So
ln k = (n + 1) ln (n + 1) - n + (ln(n))
k =1
ln n
Since log(n) =
ln 2
n
n
+ (log n)
So log k = (n + 1) log (n + 1) ln 2
k =1
Other Approximations Derived from

Integrals
(n + 1) 2
(n + 1) 2
k log k =
log (n + 1) + (n log n)
2
4 ln 2
k =1
n
Harmonic Numbers
n
Hn =
k =1
1
k
1
H n = ln(n) + + O
n
Euler' s constant = .577...
Stirling s Approximation for Factorial

Factorial
n! = 1 2 3 (n-1) n
n! ~ 2 n n n e-n as n
So
1
log(n!) = n log n - n log e + log (2 n) + (1)
2
= n log n - (n)
Recurrence Equations
(over integers)
Homogenous of degree d
n>d
x n = a1 x n 1 + a 2 x n 2 + ... + a d x n d
Given
constant coefficients
initial values
a1, , ad
x1, x2 , , xd
Example: Fibonacci Sequence

n > 2
Fn = Fn-1 + Fn-2
F0 = 0,
F1 =1
Solution of Fibonacci Sequence

1
r1 = (1 + 5 ) = 1.618... golden ratio
2
1
r2 = (1 5 )
2
n
n
Fn = c1 r1 + c 2 r2
where
F0 = c1 + c 2 = 0
F1 = c1 r1 + c 2 r2 = 1
Solution of Fibonacci Sequence

(cont d)
Hence
n
n
1 1 + 5 1 5

Fn =
5 2 2
1
Fn ~
5
1 + 5
A Useful Theorem
c > 0, d > 0
n=1
c0
If T (n) =
n
d
aT
+
cn
n>1
b

( )
(
)
( )
n logb a
then T (n) = n d log b n
a > bd
a = bd
a < bd
Proof
d
T (n) = cn g(n) + a
log b n
Is solution
2
a
a
g(n) = 1 + d + d + ... +
b
b
a
d
b
log b n-1
Cases
a
(1) a > b g(n) ~ d
b
is last term so
log b n-1
) (
T(n) = a logb n d = n logb a
(2) a = bd g(n) = log b n
so T(n) = n log b n
d
(3) a < b g(n) upper bound by 0(1)
( )
so T(n) = n d
Example: Mergesort
input list L of length N
if N=1 then return L
else do
let L1 be the first
elements of L
of L
let L2 be the last

elements
2
M1 Mergesort (L1)
M2 Mergesort (L2)
return
Merge (M1 , M2)
Time Bounds of Mergesort

Initial Value T(1) = c1

+ T
+ c2 N
for N > 1T(N) T
2
2

for some constants c1, c2 > 1
Time Bound (cont d)

+ c 2 N
2
N > 1 T(N) 2T
guess T(N) a N log N + b
N
2 a
log + b + c 2 N
2
2
Holds if a = c1 + c2 ,
Solution
b = c1
T(N) ( c1 + c2 ) N log N + c1
Time Bound (cont d)

N > 1 T(N) 2T + c 2 N, T(1) = c1
2
Transform Variables
n = log N, N = 2n
N
n 1 = log N log 2 = log
2
Recurrence equation:
( )
= T ( 2 ) = T(1) = c
X n = T 2n = 2 X n-1 + c 2 2n
X0
Solve by usual methods for recurrence

equations
X n = O n 2n
so T ( N ) = O (N log N)
Advanced Material
Exact Solution of Recurrence
Relations
Homogenous Recurrence Relations

(no constant additive term)
Solve:
try
x n = a1 x n 1 + a 2 x n 2 + ... + a d x n d
xn = rn
Multiply by
r
n
r
Get characteristic equation:
r d a1 r d -1 a 2 r d -2 ... a d = 0
Case of Distinct Roots

Distinct Roots
r1, r2, , rd
d
x n = ci rin
i =1
x n ~ ci rin
Where ri is dominant root
ri > rj j i
Other Case
Roots are not distinct
r1= r2 = r3
Then solutions not independent,
so additional terms:
n
1 1
n
1
n
1
n
i i
x n = c r + c 2 n r + c3 n r + c r
i=4
Inhomogenous Recurrence
Equations
x n = a1 x n 1 + a 2 x n 2 + ... + a d x n d + a 0
Nonzero constant term
a0 0
Solution Method
(1) Solve homogenous equation
Yn = a1 Yn 1 + a 2 Yn 2 + ... + a n Yn d
Solution Method
1) Solve homogenous equation
Yn = a1 Yn 1 + a 2 Yn 2 + ... + a n Yn d
2) Case
, add particular solution
a
1
i
Solution Method (cont d)

Case
= 1 , add particular solution
a 0
x n = cn=
n
ia
i
3) Add particular and homogeneous

solutions, and solve for constants
This is all we usually need!!

Equations
Analysis of Algorithms
Prepared by
John Reif, Ph.D.

Asymptotics and Recurrence Equations: Analysis of Algorithms

Uploaded by

Copyright:

Available Formats

Asymptotics and Recurrence Equations: Analysis of Algorithms

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Asymptotics and Recurrence Equations: Analysis of Algorithms

Uploaded by

Copyright:

Available Formats

Asymptotics and Recurrence

John Reif, Ph.D.