04/19/23 1

CS 3343: Analysis of Algorithms

Lecture 3: Asymptotic Notations, Analyzing non-recursive algorithms

04/19/23 2

Outline

• Review of last lecture

• Continue on asymptotic notations

• Analyzing non-recursive algorithms

04/19/23 3

Mathematical definitions

• O(g(n)) = {f(n): positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) n≥n0}

• Ω(g(n)) = {f(n): positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) n≥n0}

• Θ(g(n)) = {f(n): positive constants c1, c2, and n0 such that 0 c1 g(n) f(n) c2 g(n) n n0}

04/19/23 4

Big-Oh

• Claim: f(n) = 3n2 + 10n + 5 O(n2)• Proof by definition:

(Hint: Need to find c and n0 such that f(n) <= cn2 for all n ≥ n0. You can be sloppy about the constant factors. Pick a comfortably large c when proving big-O or small one when proving big-Omega.)

(Note: you just need to find one concrete example of c and n0, but the condition needs to be met for all n ≥ n0. So do not try to plug in a concrete value of n and show the inequality holds.)

Proof:3n2 + 10n + 5 3n2 + 10n2 + 5, n ≥ 1

3n2 + 10n2 + 5n2, n ≥ 1 18 n2, n ≥ 1

If we let c = 18 and n0 = 1, we have f(n) c n2, n ≥ n0. Therefore by definition 3n2 + 10n + 5 O(n2).

04/19/23 5

How to prove logn < n

Let f(n) = 1 + log n, g(n) = n.Then f(n)’ = 1/n g(n)’ = 1.

f(1) = g(1) = 1.Because f(n)’ ≤ g(n)’ n ≥ 1, by the racetrack principle, we

have f(n) ≤ g(n) n ≥ 1, i.e., 1 + log n ≤ n.Therefore, log n < 1 + log n ≤ n for all n

From now on, we will use that fact that log n < n n ≥ 1 without proof.

04/19/23 6

True or false?

1. 2n2 + 1 = O(n2) T (also )

2. Sqrt(n) = O(log n) F ()

3. log n = O(sqrt(n)) T (also o)

4. n2(1 + sqrt(n)) = O(n2 log n) F ()

5. 3n2 + sqrt(n) = O(n2) T (also )

6. sqrt(n) log n = O(n) T (also o)

04/19/23 7

True or false?

1. 2n2 + 1 = O(n2) T (also )

2. Sqrt(n) = O(log n) F ()

3. log n = O(sqrt(n)) T (also o)

4. n2(1 + sqrt(n)) = O(n2 log n) F ()

5. 3n2 + sqrt(n) = O(n2) T (also )

6. sqrt(n) log n = O(n) T (also o)

04/19/23 8

Questions

• If f(n) O(g(n)) 1. compare f(n) and f(n) + g(n)

2. compare g(n) and f(n) + g(n)

3. compare h(n) f(n) and h(n) g(n) where h(n) > 0.

• How about f(n) Ω (g(n)) and f(n) Θ (g(n))?

04/19/23 9

Asymptotic notations

• O: <=

• o: <

• Ω: >=

• ω: >

• Θ: =

(in terms of growth rate)

04/19/23 10

O, Ω, and Θ

The definitions imply a constant n0 beyond which they aresatisfied. We do not care about small values of n.

Use definition to prove big-O (Ω,Θ): find constant c (c1 and c2) and n0, such that the definition of big-O (Ω,Θ) is satisfied

04/19/23 11

Use limits to compare orders of growth

0• lim f(n) / g(n) = c > 0

∞n→∞

f(n) o(g(n))

f(n) Θ (g(n))

f(n) ω (g(n))

f(n) O(g(n))

f(n) Ω(g(n))

04/19/23 12

Examples

• Compare 2n and 3n

• lim 2n / 3n = lim(2/3)n = 0

• Therefore, 2n o(3n), and 3n ω(2n)

• How about 2n and 2n+1?

2n / 2n+1 = ½, therefore 2n = Θ (2n+1)

n→∞ n→∞

04/19/23 13

L’ Hopital’s rule

lim f(n) / g(n) = lim f(n)’ / g(n)’n→∞ n→∞

Condition:

If both lim f(n) and lim g(n) = or 0

04/19/23 14

Example

• Compare n0.5 and log n

• lim n0.5 / log n = ?

• (n0.5)’ = 0.5 n-0.5

• (log n)’ = 1 / n• lim (n-0.5 / 1/n) = lim(n0.5) = ∞• Therefore, log n o(n0.5)• In fact, log n o(nε), for any ε > 0

n→∞

04/19/23 15

Stirling’s formula

nnn

ene

nnn

2/122!

!n nn en 2/1(constant)

or

04/19/23 16

Examples

• Compare 2n and n!

• Therefore, 2n = o(n!)

• Compare nn and n!

• Therefore, nn = ω(n!)

• How about log (n!)?

n

nnn

n

nnn e

nnc

e

nncn

2lim

2lim

2

!lim

0limlim!

lim nnnn

n

nnn e

nc

en

nnc

n

n

04/19/23 17

Example

)log(loglog)!log( 2/1 nnn

n

enCe

nncn

)log(

log2

1log

nn

nnnnC

04/19/23 18

Properties of asymptotic notations

• Textbook page 51 • Transitivity

f(n) = (g(n)) and g(n) = (h(n)) => f(n) = (h(n))(holds true for o, O, , and as well).

• Symmetryf(n) = (g(n)) if and only if g(n) = (f(n))

• Transpose symmetryf(n) = O(g(n)) if and only if g(n) = (f(n))f(n) = o(g(n)) if and only if g(n) = (f(n))

04/19/23 19

About exponential and logarithm functions

• Textbook page 55-56• It is important to understand what logarithms are and

where they come from.• A logarithm is simply an inverse exponential function.• Saying bx = y is equivalent to saying that

x = logb y. • Logarithms reflect how many times we can double

something until we get to n, or halve something until we get to 1.

• log21 = ?• log22 = ?

04/19/23 20

Binary Search

• In binary search we throw away half the possible number of keys after each comparison.

• How many times can we halve n before getting to 1?

• Answer: ceiling (lg n)

04/19/23 21

Logarithms and Trees

• How tall a binary tree do we need until we have n leaves?

• The number of potential leaves doubles with each level.

• How many times can we double 1 until we get to n?

• Answer: ceiling (lg n)

04/19/23 22

Logarithms and Bits

• How many numbers can you represent with k bits?

• Each bit you add doubles the possible number of bit patterns

• You can represent from 0 to 2k – 1 with k bits. A total of 2k numbers.

• How many bits do you need to represent the numbers from 0 to n?

• ceiling (lg (n+1))

04/19/23 23

logarithms

• lg n = log2 n• ln n = loge n, e ≈ 2.718• lgkn = (lg n)k

• lg lg n = lg (lg n) = lg(2)n• lg(k) n = lg lg lg … lg n• lg24 = ? • lg(2)4 = ?• Compare lgkn vs lg(k)n?

04/19/23 24

Useful rules for logarithms

For all a > 0, b > 0, c > 0, the following rules hold• logba = logca / logcb = lg a / lg b• logban = n logba•

blog

ba = a

• log (ab) = log a + log b– lg (2n) = ?

• log (a/b) = log (a) – log(b)– lg (n/2) = ?– lg (1/n) = ?

• logba = 1 / logab

04/19/23 25

Useful rules for exponentials

• For all a > 0, b > 0, c > 0, the following rules hold

• a0 = 1 (00 = ?)• a1 = a• a-1 = 1/a• (am)n = amn

• (am)n = (an)m

• aman = am+n

04/19/23 26

More advanced dominance ranking

1)(logloglogloglog/log

loglogloglog/

!log~log23!

)3(

23

123

nnnnn

nnnnnnnn

nnnnnnnn nnn

04/19/23 27

Analyzing the complexity of an algorithm

04/19/23 28

Kinds of analyses

• Worst case– Provides an upper bound on running time

• Best case – not very useful, can always cheat

• Average case– Provides the expected running time– Very useful, but treat with care: what is

“average”?

04/19/23 29

General plan for analyzing time efficiency of a non-recursive algorithm

• Decide parameter (input size)

• Identify most executed line (basic operation)

• worst-case = average-case?

• T(n) = i ti

• T(n) = Θ (f(n))

04/19/23 30

Example

repeatedElement (A, n)// determines whether all elements in a given // array are distinctfor i = 1 to n-1 {

for j = i+1 to n {if (A[i] == A[j])

return true;}

}return false;

04/19/23 31

Example

repeatedElement (A, n)// determines whether all elements in a given // array are distinctfor i = 1 to n-1 {

for j = i+1 to n {if (A[i] == A[j])

return true;}

}return false;

04/19/23 32

• Best case?

• Worst-case?

• Average case?

04/19/23 33

• Best case– A[1] = A[2]– T(n) = Θ (1)

• Worst-case– No repeated elements– T(n) = (n-1) + (n-2) + … + 1 = n (n-1) / 2 = Θ (n2)

• Average case?– What do you mean by “average”?– Need more assumptions about data distribution.

• How many possible repeats are in the data?– Average-case analysis often involves probability.

04/19/23 34

Find the order of growth for sums

• T(n) = i=1..n i = Θ (n2)• T(n) = i=1..n log (i) = ?• T(n) = i=1..n n / 2i = ?• T(n) = i=1..n 2i = ?• …

• How to find out the actual order of growth?– Math…– Textbook Appendix A.1 (page 1058-60)

04/19/23 35

Arithmetic series

• An arithmetic series is a sequence of numbers such that the difference of any two successive members of the sequence is a constant.

e.g.: 1, 2, 3, 4, 5

or 10, 12, 14, 16, 18, 20• In general:

Recursive definition

Closed form, or explicit formuladiaa

daa

i

ii

)1(1

1

Or:

04/19/23 36

Sum of arithmetic series

If a1, a2, …, an is an arithmetic series, then

2

)( 1

1

nn

ii

aana

e.g. 1 + 3 + 5 + 7 + … + 99 = ?(series definition: ai = 2i-1)

This is ∑ i = 1 to 50 (ai) = 50 * (1 + 99) / 2 = 2500

04/19/23 37

Geometric series

• A geometric series is a sequence of numbers such that the ratio between any two successive members of the sequence is a constant.

e.g.: 1, 2, 4, 8, 16, 32

or 10, 20, 40, 80, 160

or 1, ½, ¼, 1/8, 1/16• In general:

Recursive definition

Closed form, or explicit formula0

1

ara

raai

i

ii

Or:

04/19/23 38

Sum of geometric series

if r < 1

1

)1/()1(

)1/()1(1

1

0 n

rr

rr

r n

n

n

i

i if r > 1

if r = 1

?2

1lim

?2

1lim

?2

1

0

0

n

iin

n

iin

n

i

i

04/19/23 39

Sum of geometric series

if r < 1

1

)1/()1(

)1/()1(1

1

0 n

rr

rr

r n

n

n

i

i if r > 1

if r = 1

112lim2

1lim

21

1)(lim

2

1lim

)2(21212

122

021

021

1

21

021

0

111

0

n

i

in

n

iin

n

i

in

n

iin

nnnnn

i

i

04/19/23 40

Important formulas

)1()(

)1()1(

1

1

)(2

)1(

)(1

1

0

2

1

1

rr

r

r

rr

nnn

i

nn

n

nn

i

i

n

i

n

i

)lg(lg

)(lg1

)2(22)1(2

)(1

)(3

1

1

1

1

11

1

33

1

2

nni

ni

nni

nk

ni

nn

i

n

i

n

i

nnn

i

i

kkn

i

k

n

i

04/19/23 41

Sum manipulation rules

n

xii

x

mii

n

mii

i ii i

i ii ii ii

aaa

acca

baba

1

)(

Example:

?2

?)24(

1

1

n

ii

n

i

i

n

i

04/19/23 42

Sum manipulation rules

n

xii

x

mii

n

mii

i ii i

i ii ii ii

aaa

acca

baba

1

)(

Example:

n

ii

n

ii

n

i

n

i

nin

i

i

nnn

nnii

11

1 1

1

1

2

1

2

22)1(224)24(

04/19/23 43

i=1..n n / 2i = n * i=1..n (½)i = ?

• using the formula for geometric series:

i=0..n (½)i = 1 + ½ + ¼ + … (½)n = 2

• Application: algorithm for allocating dynamic memories

04/19/23 44

i=1..n log (i) = log 1 + log 2 + … + log n = log 1 x 2 x 3 x … x n = log n!

= (n log n)

• Application: algorithm for selection sort using priority queue

04/19/23 45

Recursive definition of sum of series

• T (n) = i=0..n i is equivalent to:

T(n) = T(n-1) + n

T(0) = 0

• T(n) = i=0..n ai is equivalent to:

T(n) = T(n-1) + an

T(0) = 1

Boundary condition

Recurrence

Recursive definition is often intuitive and easy to obtain. It is very useful in analyzing recursive algorithms, and some non-recursive algorithms too.

04/19/23 46

Recursive definition of sum of series

• How to solve such recurrence or more generally, recurrence in the form of:

• T(n) = aT(n-b) + f(n) or

• T(n) = aT(n/b) + f(n)