Asymptotics and Recurrence Equations

Prepared by

John Reif, Ph.D.

Analysis of Algorithms

Asymptotics and Recurrence Equations

a) Computational Complexity of a Program

b) Worst Case and Expected Boundsc) Definition of Asymptotic Equationsd) Solution of Recurrence Notation

Readings

• Main Reading Selection:• CLR, Chapter 3, 4 and Appendix A

Goal

• To estimate and compare growth rates of functions

• Ignore constant factors of growth

“f(n) is asymptotically equal to g(n)”

1g(n)

f(n)lim

if

g(n)~f(n)

n

”f(n) is little o g(n) ”

• f(n) is o(g(n)) if 0g(n)

f(n)lim

n

”f(n) is big O g(n) ”

• f(n) is O(g(n)) if cg(n)

f(n)suplim

n

Example of f(n) is O(g(n))

0

0

c,n 0

s.t. f(n) c g(n)

for all n n

“f(n) is order at least g(n)”

• f(n) is (g(n)) if

cg(n)

f(n)inflim n

0

0

nn allfor

g(n)cf(n)s.t.0c,n

Example of f(n) is (g(n))

“f(n) is order tight with g(n)”

• f(n) is (g(n)) if

g(n)c'f(n)g(n)cs.t.c'c,,n0

Suppose my algorithm runs in time O(n)

• Don’t say:• “his runs in time O(n2) so is worse”

• But prove:• “his runs in time (n2) so is worse”

• Must find a worst case input of length n for which his algorithm takes time > cn2 for all n > n0

Use of O Notation

• N is O (n2) sometimes written

n = O (n2)

• But n2 is not O(n) so can’t use identities!

The two sides of the equality do not play a symmetric role

Use of Asymptotic Notation

• Write

• As

• Example

o(h(n))" is g(n)f(n)"

o(h(n)) g(n)f(n)

n as )1(o1

n

1o

n

11

n

1O

n

11

1-n

n2

Convergent Power Sum

• A polynomial is asymptotically equal to

its leading term as x

10for ,)1(1

1

0

xOx

xi

i

d

d

d

0i

ii

1dd

0i

ii

dd

0i

ii

xa~ xa

x o xa

x θ xa

Sums of Powers

• for n

• Or equivalently

1dn

1i

d n 1d

1~i

1d1dn

1i

d n on 1d

1i

Examples

• 2nd order asymptotic expansion

3

n~i

2

n~i

32n

1i

2n

1i

1dd1dn

1i

d n on 2

1n

1d

1i

Asymptotic Expansion of f(n) as n n0

• If

• and

(n)g c ~f(n) i1i

i

1i allfor

(n))o(g(n)(1)g i1i

1k allfor

(n))o(g(n)g c f(n) (2) ki

k

1ii

Bounding Sums by Integrals

n

1k

n

1k

1n

1

1)(k fdx f(x)f(k)

Bounding Sums by Integrals (cont’d)

• So

• Example then

n

1k

n

1k

1n

1

1)(k fdx f(x)f(k)

1n

1

n

1k

1n

1

dx f(x)f(k) f(1)1)f(n-dx f(x)

xxln(x)(x)dxln ln(x)f(x) if

Bounding Sums by Integrals (cont’d)

• So

• Since

• So

n

1k

θ(ln(n))n-1)(nln 1)(nkln

2ln

nln log(n)

n

1k

n) (log θ2ln

n-1)(n log 1)(nk log

Other Approximations Derived from Integrals

• Harmonic Numbers

n

1k

22

n) log(n θ2ln 4

1)(n-1)(n log

2

1)(nk logk

n

1kn k

1H

.577...constant γ sEuler'

n

1 Oγln(n)Hn

Stirling’s Approximation for Factorial

• Factorial n! = 1· 2 · 3 · · · (n-1) · n

• So

n as e n n2 ~ n! -nn

(n) θ -n logn

(1) θ n)(2 log 2

1 e logn -n logn )log(n!

Recurrence Equations (over integers)

• Homogenous of degree d

n > d

• Given constant coefficients a1, …, ad

initial values x1, x2 , …, xd

dnd2n21n1n xa ... xa xax

Example: Fibonacci Sequence

• n > 2

Fn = Fn-1 + Fn-2

F0 = 0, F1 =1

Solution of Fibonacci Sequence

“golden ratio”

where

n22

n11n

2

1

r c r cF

)51(2

1r

1.618... )51(2

1r

1r c r cF

0 c cF

22111

210

Solution of Fibonacci Sequence (cont’d)

• Hence

n

n

nn

n

2

51

5

1~F

2

51

2

51

5

1F

A Useful Theorem

• c > 0, d > 0

• If

• then

0

d

c n=1

T (n) = naT + cn n>1

b

blog a d

d db

d d

θ n a > b

T (n) = θ n log n a = b

θ n a < b

Proof

• Is solution

blog ndT (n) = cn g(n) + a d

b2 log n-1

d d d

a a ag(n) = 1 + + + ... +

b b b

Cases

b

b b

log n-1d

d

log n log a

a(1) a > b g(n) ~

b

is last term so

T(n) = θ a d =θ n

d

b

db

(2) a = b g(n) = log n

so T(n) = θ n log n

d

d

(3) a < b g(n) upper bound by 0(1)

so T(n) = θ n

Example: Mergesort

input list L of length Nif N=1 then return L

else do let L1 be the first elements of L

let L2 be the last elements of L

M1 Mergesort (L1)

M2 Mergesort (L2)

return Merge (M1 , M2)

Time Bounds of Mergesort

• Initial Value T(1) = c1

for N > 1

for some constants c1, c2 > 1

2T(N) T T c N

Time Bound (cont’d)

• N > 1

guess

Holds if

Solution

2T(N) 2T c N

2

T(N) a N log N + b

N N 2 a log + b c N

2 2

1 2 1a = c + c , b = c

1 2 1T(N) c + c N log N + c

Time Bound (cont’d)

• N > 1

• Transform Variables

• Recurrence equation:

2 1T(N) 2T c N, T(1) = c

nn = log N, N = 2

Nn 1 log N log 2 log

2

n nn n-1 2

00 1

X T 2 X + c

X T = T(1) = c

• Solve by usual methods for recurrence equations

nnX O n

so T N = O (N log N)

Advanced Material

Exact Solution of Recurrence Relations

Homogenous Recurrence Relations (no constant additive term)

• Solve:

try

Multiply by

• Get characteristic equation:

dnd2n21n1n xa ... xa xax

nn rx

n

d

r

r

0 a ...r ar a r d2-d

21-d

1d

Case of Distinct Roots

• Distinct Rootsr1, r2, …, rd

• Where ri is dominant root

niin

ni

d

1iin

r c ~ x

r c x

i j rr ji

Other Case

• Roots are not distinctr1= r2 = r3

• Then solutions not independent, so additional terms:

ni

d

4ii

n1

23

n12

n11n r c r n crn cr cx

Inhomogenous Recurrence Equations

• Nonzero constant term

• Solution Method

(1) Solve homogenous equation

0a 0

n 1 n 1 2 n 2 n n dY a Y a Y ... a Y

n 1 n 1 2 n 2 d n d 0x a x a x ... a x a

Solution Method

1) Solve homogenous equation

2) Case

, add particular solution

n 1 n 1 2 n 2 n n dY a Y a Y ... a Y

ia 1

Solution Method (cont’d)

Case

, add particular solution

3) Add particular and homogeneous solutions, and solve for constants

This is all we usually need!!

ia 1

0n

i

ax cn= n

ia

Asymptotics and Recurrence Equations

Prepared by

John Reif, Ph.D.

Analysis of Algorithms