1

Fundamental Algorithms Summer 2004

Dan Barrish-Flood

2

• A problem is a precise specification of input and desired output

• An algorithm is a precise specification of a method for solving a problem.

What is an Algorithm?

3

What We Want in an Algorithm

• Correctness– Always halts– Always gives the right answer

• Efficiency– Time– Space

• Simplicity

4

A Typical Problem

Sorting

  Input:  A sequence b1, b2, ... bn

  Output:  A permutation of that sequence b1, b2, ... bn such

that

nbbbb 321

5

A Typical Algorithm

Insertion -Sort(

j 2

2

Insert sorted

sequence

and

6

8

A

length A

key A j

A j

A j

i j

i A i key

A i A i

i i

A i key

)

[ ]

[ ]

[ ] into the

[ .. ]

[ ]

[ ] [ ]

[ ]

1

3

1 1

4 1

5 0

1

7 1

1

for to

do

while

do

6

Running Time

Insertion - Sort(

j 2

2

Insert into the sorted

sequence

and

6

8

A

length A

key A j

A j

A j

i j

i A i key

A i A i

i i

A i key

)

[ ]

[ ]

[ ]

[ .. ]

[ ]

[ ] [ ]

[ ]

1

3

1 1

4 1

5 0

1

7 1

1

for to

do

while

do

1-n

)1(t c

)1(t c

t c

1-n c

1-n 0

1-n c

1 -n

8

n

2j=j7

n

2j=j6

n

2j=j5

4

2

1

c

c

timesCost

tJ is the number of times the “while” loop test in line 5 is executed for that value of j. Total running time is

c1 (n) + c2(n-1) +…+c8(n-1)

7

• Trying to determine the exact running time is tedious, complicated and platform-specific.

• Instead, we’ll bound the worst-case running time to within a constant factor for large inputs.

• Why worst-case?– It’s easier.– It’s a guarantee.– It’s well-defined.– Average case is often no better.

A Better Way to Estimate Running Time

8

Insertion Sort, Redux

• Outer loop happens about n times.

• Inner loop is done j times for each j, worst case.

• j is always < n

• So running time is upper-bounded by about

n n n 2

“Insertion sort runs in (n2) time, worst-case.”

9

• We’ll make all the “about”s and “rough”s precise.

• These so-called asymptotic running times dominate constant factors for large enough inputs.

10

Functions

• constants

• polynomials

• exponentials

• logarithms and polylogarithms

• factorial 1...)1(!

log ,log

3 ,2

52 ,

1 ,0

22

n

3

2

nnn

nn

nnn

11

,

E.g. number. real some is c where

termsof Sum generally, (More

200552 E.g

32/1

32103

0

nnn

an

nnnnn

na

c

d

i

ii

Polynomials

12

f(n) is monotonically increasing iff

I.e. It never “goes down” — always increases or stays the same.

For strictly increasing, replace in the above with <.

I.e. The function is always going up.

impliesm n

f m f n

( ) ( ).

13

Are constant functions monotonically increasing? Yes! (But not strictly increasing.)

If , then is strictly increasing.

So even increases forever to

c nc 0

n n.001 1000 ( ) .

14

Exponentialsf n a

a

a a

a a a a

a a a

a a a

a a

n

n

m n m n

n n n

m n mn n m

n n

n

n

n

( )

/

( ) ( )

,

( )

Facts:

e.g.

e.g.

If is strictly increasing to

0

1

1

1 1 12

1

1

2 2 2 2

2 2 2

1

1

21

2

15

Logarithms

.0log ,10

01log

1log

:Recall

) (log log log log

)(loglog

logln

loglg

:notation Some

.such that number that log

2

xxFor

b

nn

nn

nen

nn

nxbxnb

b

b

b

kk

16

Log Facts1. n = b log

b n (definition)

2. logb(xy)=logbx + logby

3. logban = nlogba

4. logbx = logcx / logcb

5. logb(1/a) = -logba

6. logba = 1/(logab)

7. alogbc= clog

ba

• if b > 1, then logbn is strictly increasing.

17

More About Logs

A polylogarithm is just

For and , is strictly

increasing to

log

log

.

k

k

n

k n n

0 0

18

Factorial

nn

nn

en

n

i

nn

nn

n

i

nnnn

!2 3,>nFor

:seecan weStirling,ut Even witho

))(1()( 2!

:ionapproximat sStirling'

toincreasingstrictly is !

)definition(by 1!0

1)2()1(!

1

1

19

More About Monotonically Increasing Functions

If and are monotonically

increasing, then so are

and if and are also then

is monotonically increasing, too.

Examples:

f n g n

f n g n

f g n

f n g n

f n g n

n n

n

nn

( ) ( )

( ) ( )

( ( ))

( ) ( ) ,

( ) ( )

log

log log

0

2 3

20

Bounding Functions• We are more interested in finding upper and lower

bounds on functions that describe the running times of algorithms, rather than finding the exact function.

• Also, we are only interested in considering large inputs: everything is fast for small inputs.

• So we want asymptotic bounds.• Finally, we are willing to neglect constant factors

when bounding. They are tedious to compute, and are often hardware-dependent anyway.

21

Example

05

1015

20

0 1 2 3

3n+8

n

0

10

20

30

0 1 2 3

3n+8

10n

but when n 2, 3n + 8 is upper -

bounded by

10n.

n

n

by

bounded-uppernot certainly is 83

22

Notation for Function Bounds

choices.]other many are there

;2 and 10 choose :[Proof

)(83

E.g.

allfor )()(

such that and constants positive

exist two thereiff ))(()(say We

)oh"-big(" O :BoundUpper

day.)every themuse llWe'

course. in the sdefinitionimportant most (The

0

0

0

nc

nOn

nnncgnf

nc

ngOnf

23

Big-Oh Facts

• It denotes an upper bound, not necessarily a tight bound. Thus,

O g n f n( ( )) ( )• The “=” is misleading. E.g

is meaningless.

etc. ),2(

)(

)(2

nOn

nOn

nOn

• Transitivity:

))(()(then

))(()( and ))(()( If

nhOnf

nhOngngOnf

24

Examples

We’ve seen that 3n + 8 = O(n).

Is 10n = O(3n + 8)?

Yes: choose c = 10, n0 = 1:

10n n+ 80

25

More Examples

2n2 + 5n - 8 = O(n2)?

Yes:

2n2 + 5n - 8 cn2

n2: 2 + 5/n - 8/n2 c

Choose c = 3; subtracting 2:

5/n - 8/n2 Choose no = 5

26

More Examples

constant.any by bounded-upper becannot it - to

increasingstrictly is )2/3( ,1 3/2 sinceBut

)2/3(

2/3 :2

23

?)2(3 Is

)1 ,1( 32 fact,in :Yes

?)3(2 Is

0

n

n

nnn

nn

nn

nn

nn

c

c

c

O

nc

O

27

Lower Bound Notation

f n g n

c n n n

n n

f n O g n g n f n

n n

( ) ( ( ))

.

( )

( )

( ) ( ( )), ( ) ( ( )),

iff there are positive constants

and such that f(n) cg(n) for all

(Same as definition of O, with replaced by .)

E.g.

In fact:

If then

and vice versa.

0 0

2

3 2

28

“Tight” Bound Notation (Both upper and lower)

iff there are positive constants

and such that

for all

f n g n

c c n

c g n f n c g n n n

( ) ( ( ))

, ,

( ) ( ) ( ) .

1 2 0

1 2 0

0

2

4

6

8c g n2 ( )

f n( )

n0

29

More Facts

is transitive, just like and .

iff

and

iff

and

iff

O

f n g n f n O g n

f n g n

f n g n f n O g n

g n O f n

f n g n g n f n

( ) ( ( )) ( ) ( ( ))

( ) ( ( ))

( ) ( ( )) ( ) ( ( ))

( ) ( ( ))

( ) ( ( )) ( ) ( ( ))

30

Examples

growth. clogarithmi allfor )(lg can write weTherefore,

factor.constant aby differ functions log Since

)(loglog

versa.not vicebut ),(

:because is This ).(

].2 ,1[ )(852 Also,

)(852at earlier th saw We

:because is This ).(852

).(83

so )83( and ),(83 seen that veWe'

21

32

32

022

22

22

n

nn

nOn

nn

ncnnn

nOnn

nnn

nn

nOnnOn

bb

31

Another Upper-Bound Notation

)."( respect to

withnegligible cally)(asymptoti is )("

as ))(()( ofThink

).(not is 2

)(2 E.g.

.0)(

)(

iff ]oh" little[" ))(()(say We

22

2

nlim

ng

nf

ngonf

non

non

ng

nf

ngonf

32

Functions and Bounds

.)0)(

)( so ,)( :(Proof

)).(()( then , toincreasing

strictly is )( andfunction constant a is )( If

.)conventionby 1 use but we etc.,

),3(),2( (also ).1( are functionsconstant All

:functionsConstant

lim

ng

nfng

ngonf

ngnf

n

33

Logarithms

on. so and

)(lg)lg(lg see we(*),in for lg ngSubstituti

1 while)(huge! 2lg

,4for fact that theof spitein

)(lg

-lysurprising perhaps -So

.0 where

, constantsany for )(log (*)

fact,In

)(lg

01.100100

01.100

b

nonnn

nn

n

non

a

banon

non

ab

a

34

Polynomials

)001.1( Thus,

als)(exponenti spolynomial i.e. 1 ),(

),1 (where , to)0 (where ,2 Renaming

2) base log means (recall ))2((

))2(())2(lg(

:have we

0,a ),(lgin for 2 ngSubstituti

class.next for Homework

:Proof

).( is positive ist coefficien

leading whose degree of polynomialAny

100 n

nb

a

nab

anbn

abn

d

on

oaaon

aaa

n lgon

o

nonn

n

d

35

Comparing Functions

vs. n vs.

vs. vs.

vs. vs.

vs. lgn vs.

vs.

n n n

n n n

n n n n n

n n n

n n n

n n

n n

lg

lg

lg lg( )

lg ln lg

/ lg lg

/

lg

2

2

2

2

2 2