A l i f Al ith IAnalysis of Algorithms I

Outline• running timeg

• pseudocode

• primitive operationsp p

• asymptotic notation

M t di li hMore case studies: linear search, insertion-sort,matrix multiplication

© 2010 Goodrich, Tamassia 1

p

Instructor: Yuzhen Xie

MotivationMotivation

Very similar problem can have very different solutions. How to compare between different algorithms in order to select the best one for a particular application?

CorrectnesslSimplicity

SecurityMaintainability…Response timeSpace usage

In this course, we focus on the running time, with space usagebeing also of interest in some occasions.

2

Characters of Running TimeMost algorithms transform input objects into output objects. best case

average caseRunning time typically grows with input size, n

Fi d i ti till 100

120

average caseworst case

Fixed n, running time may still varies on different inputs: best case, average case, worst caseE li h

60

80

100

ing

Tim

e

Ex. linear searchinsertion-sort

We focus on the worst case20

40

Run

n

We focus on the worst case running time.

A guarantee to the userCrucial to applications such as

01000 2000 3000 4000

Input Size

3

Crucial to applications such as games, finance and robotics

Linear Search

Algorithm LinearSearch(v, S[0, …, n-1]) Ex. Input: array S of n elements, value vOutput: position i if S[i] = v; -1 if v ∉S

[3, 6, 1, 5, 8]

Running timei ← 0while (i < n) and (S[i] != v) do

i = i +1

Running time(fixed n)

Best case?if i < n

return ielse

Average case?Worst case?

elsereturn -1

4

Experimental Evaluation of Running TimeExperimental Evaluation of Running Time

Write a program 9000Write a program implementing the algorithm

h h7000

8000

9000

Run the program with inputs of varying size and composition

4000

5000

6000

me

(ms)

Use a method like System.currentTimeMillis() to get an accurate measure 2000

3000

4000

Tim

gof the actual running time

Plot the results 0

1000

0 50 100

5

Input Size

Limitations of ExperimentsLimitations of Experiments

It is necessary to implement the algorithm whichIt is necessary to implement the algorithm, which may be difficult

Results may not be indicative of the running time onResults may not be indicative of the running time on other inputs not included in the experiment.

In order to compare two algorithms, the sameIn order to compare two algorithms, the same hardware and software environments must be used

6

Theoretical Analysis of Running TimeTheoretical Analysis of Running Time

Uses a high level description of the algorithmUses a high-level description of the algorithm instead of an implementation

Characterizes running time as a function of theCharacterizes running time as a function of the input size, n.

Takes into account all possible inputsTakes into account all possible inputs

Allows us to evaluate the speed of an algorithm independent of the ha d a e/soft a e en i onmentindependent of the hardware/software environment

7

PseudocodePseudocode

High-level description of Example: find max an algorithm

More structured than English prose Algorithm arrayMax(A, n)

pelement of an array

English prose

Less detailed than a program

Input: array A of n integersOutput: maximum element of A

Preferred notation for describing algorithms

currentMax ← A[0]for i ← 1 to n − 1 do

if A[i] > currentMax thenHides program design issues

[ ]currentMax ← A[i]

return currentMax

8

Pseudocode DetailsPseudocode Details

Control flow Method callif … then … [else …]while … do …repeat until

var.method (arg [, arg…])

Return valuerepeat … until …for … do …Indentation replaces braces

return expression

Expressions← Assignment

Method declarationAlgorithm method (arg [, arg…])

← Assignment(like = in Java)

= Equality testing(like == in Java)g ( g [, g ])

Input: …Output: …

( )n2 Superscripts and other

mathematical formatting allowed

9

The Random Access Machine (RAM) ModelThe Random Access Machine (RAM) Model

RAM model is independent

A CPU

of hardware architectures.

+ - * / < > =?

A potentially unbounded bank 2A potentially unbounded bank of memory cells, each of which can hold an arbitrary number or character

012

number or character

Memory cells are numbered and i ll i t k

10

accessing any cell in memory takes unit time.

Primitive OperationspBasic computations performed by an algorithm

Examples:by an algorithm

Identifiable in pseudocodeEvaluating an expression, e.g, + - * /, <, >, == Assigning a value to a

Largely independent from the programming language

E t d fi iti t i t t

g gvariableIndexing into an arrayCalling a methodExact definition not important

(we will see why later)

Assumed to take a constant

Calling a methodReturning from a method

Assumed to take a constant amount of time in the RAM model

11

Counting Primitive OperationsCounting Primitive OperationsBy inspecting the pseudocode, we can determine the y p g p ,maximum number of primitive operations executed by an algorithm, as a function of the input size.

Algorithm arrayMax(A, n) # operationscurrentMax ← A[0] 2for i ← 1 to n − 1 do n - 1

if A[i] > currentMax then 2(n − 1)currentMax ← A[i] 2(n − 1)currentMax ← A[i] 2(n 1)

return currentMax 1Total 5n − 2

12

Estimating Running TimeEstimating Running Time

l hAlgorithm arrayMax executes 5n − 2 primitive operations in the worst case. Define:

Time taken by the fastest primitive operationa = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation

Let T(n) be worst-case time of arrayMax. Thena (5n − 2) ≤ T(n) ≤ b(5n − 2)

Hence, the running time T(n) is bounded by two linear functions

13

Growth Rate of Running TimeGrowth Rate of Running Time

Changing the hardware/ softwareChanging the hardware/ software environment

Affects T(n) by a constant factor, butAffects T(n) by a constant factor, butDoes not alter the growth rate of T(n)

The linea g o th ate of the nning timeThe linear growth rate of the running time T(n) of arrayMax is an intrinsic property of the algorithmthe algorithm

14

Seven Important FunctionsSeven Important Functions

Seven functions that ft i l ith 1E+30often appear in algorithm

analysis:Constant ≈ 1L ith i l 1E+22

1E+241E+261E+281E+30

Cubic

Quadratic

LiLogarithmic ≈ log nLinear ≈ nN-Log-N ≈ n log nQ d ti 2

1E+141E+161E+181E+201E 22

T(n

)

Linear

Quadratic ≈ n2

Cubic ≈ n3

Exponential ≈ 2n

1E+41E+61E+8

1E+101E+12

In a log-log chart, the slope of the line corresponds to the

1E+01E+21E+4

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

15

corresponds to the growth rate

n

Growth Rates of some Important Functionsp

n lg n n n lg n n2 n3 2n

8 3 8 24 64 512 256

16 4 16 64 256 4 096 65 53616 4 16 64 256 4,096 65,536

32 5 32 160 1,024 32,768 4,294,967,296

64 6 64 384 4,096 262,144 1.84 x 1019

128 7 128 896 16,384 2,097,152 3.40 x 1038, , ,

256 8 256 2,048 65,536 16,777,216 1.15 x 1077

16

512 9 512 4,608 262,144 134,217,728 1.34 x 10154

Growth Rates of some Important Functionsp

17

Why Growth Rate Mattersy

if runtime time for n+1 time for 2n time for 4nis... time for n+1 time for 2n time for 4n

c lg n c lg (n+ 1) c (lg n + 1) c(lg n + 2)

c n c (n + 1) 2c n 4c n

~ c n lg n 2c n lg n + 4c n lg n +runtime

d lc n lg n c n lg n+ c n

2c n lg n + 2cn

4c n lg n + 4cn

c n2 ~ c n2 + 2c n 4c n2 16c n2

quadrupleswhen problemsize

c n3 ~ c n3 + 3c n2 8c n3 64c n3

size doubles

18

c 2n c 2 n+1 c 2 2n c 2 4n

Growth Rates of some Important Functionsp

n lg n n n lg n n2 n3 2n

8 3 8 24 64 512 256

16 4 16 64 256 4 096 65 53616 4 16 64 256 4,096 65,536

32 5 32 160 1,024 32,768 4,294,967,296

64 6 64 384 4,096 262,144 1.84 x 1019

128 7 128 896 16,384 2,097,152 3.40 x 1038, , ,

256 8 256 2,048 65,536 16,777,216 1.15 x 1077

19

512 9 512 4,608 262,144 134,217,728 1.34 x 10154

Constant FactorsConstant Factors

Th h i1E+26

Q d tiThe growth rate is not affected by

constant factors or 1E+181E+201E+221E+24 Quadratic

Quadratic

LinearLinear

105n2 + 108n

2lower-order terms

Examples 1E+101E+121E+141E+16

T(n

)102n + 105

n2

Examples102n + 105 is a linearfunction105 2 108 i 1E+2

1E+41E+61E+8

1E+10

n105n2 + 108n is a quadratic function 1E+0

1E+2

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

20

n

The “Big Idea”: Big-Oh NotationGiven functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n0 such that f(n) ≤ cg(n) for n ≥ n0

21

Big-Oh Notationg

Example: 10,0003n

2n+10 is O(n)2n + 10 ≤ cn(c − 2) n ≥ 10

1,000

3n

2n+10

(c 2) n ≥ 10n ≥ 10/(c − 2)Pick c = 3 and

10

100

n

n0 = 10

How about 10

(c = 4, n0 = 5) ?(c = 12, n0 = 1)?

11 10 100 1,000

n

22

n

Big-Oh Exampleg p

Example: 1,000,000^2Example:

n2 is not O(n)n2 ≤ cn ?

100,000

n^2

100n

10n

n ≤ c ?The above inequality cannot

1,000

10,000 n

inequality cannot be satisfied since c must be a constant

100

constant

A contradiction! 1

10

23

A contradiction!1 10 100 1,000

n

More Big-Oh ExamplesMore Big Oh Examples

5n-25n-2 is O(n)need c > 0 and n0 ≥ 1 such that 5n-2 ≤ c•n for n ≥ n0

this is true for c = 5 and n0 = 1this is true for c 5 and n0 1

3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)( )need c > 0 and n0 ≥ 1 such that 3n3 + 20n2 + 5 ≤ c•n3 for n ≥ n0

this is true for c = 4 and n0 = 21. (how about c = 28, n0 = 1?)

3 l 53 log n + 53 log n + 5 is O(log n)need c > 0 and n0 ≥ 1 such that 3 log n + 5 ≤ c•log n for n ≥ n0

24

need c > 0 and n0 ≥ 1 such that 3 log n + 5 ≤ c•log n for n ≥ n0

this is true for c = 8 and n0 = 2

Big-Oh and Growth Rateg

The big-Oh notation gives an upper bound on the th t f f tigrowth rate of a function

The statement “f(n) is O(g(n))” means that the growth ate of f( ) is no mo e than the g o th ate of ( )rate of f(n) is no more than the growth rate of g(n)

We can use the big-Oh notation to rank functions according to their growth rateaccording to their growth rate

f(n) is O(g(n)) g(n) is O(f(n))f( ) (g( )) g( ) (f( ))

g(n) grows more Yes Nof(n) grows more No Yes

25

f( ) gSame growth Yes Yes

Big-Oh RulesBig Oh Rules

If f(n) is a polynomial of degree dIf f(n) is a polynomial of degree d, then f(n) is O(nd), i.e.,

1. Drop lower-order termsp2. Drop constant factors

Use the smallest possible class of functionsUse the smallest possible class of functionsSay “2n is O(n)” instead of “2n is O(n2)”

Use the simplest expression of the classUse the simplest expression of the classSay “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

“b ” i Oh h i i

26

“best” Big-Oh characterization

More Big-Oh RulesMore Big Oh Rules

If d(n) is O(f(n)) and e(n) is O(g(n)) thenIf d(n) is O(f(n)) and e(n) is O(g(n)) thend(n) + e(n) is O(f(n) + g(n)) d(n) e(n) is O(f(n) g(n))( ) ( ) (f( ) g( ))

If d(n) is O(f(n)) and f(n) is O(g(n)) then d(n) is O(g(n))

If p(n) is a polynomial in n then log p(n) is O(log(n))

27

Asymptotic Algorithm Analysisy p g y

The asymptotic analysis of an algorithm determines the running time in big Oh notationdetermines the running time in big-Oh notation

To perform the asymptotic analysisWe find the worst case number of primitive operationsWe find the worst-case number of primitive operations executed as a function of the input sizeWe express this function with big-Oh notation

Example:Example:We determine that algorithm arrayMax executes at most 5n − 2 primitive operationsWe say that algorithm arrayMax “runs in O(n) time”We say that algorithm arrayMax runs in O(n) time

Since constant factors and lower-order terms are eventually dropped anyhow we can disregard them

28

eventually dropped anyhow, we can disregard them when counting primitive operations

Computing Prefix AveragesComputing Prefix Averages

We further illustrate asymptotic analysis with two algorithms for prefix averages 25

30

35XA

g

The i-th prefix average of an array X is the average of th fi t (i 1) l t f X 15

20

25

the first (i + 1) elements of X:A[i] = (X[0] + X[1] + … + X[i])/(i+1)

5

10

15

Computing the array A of prefix averages of another array X has applications to

01 2 3 4 5 6 7

29

y ppfinancial analysis

Prefix Averages (Quadratic)Prefix Averages (Quadratic)

The following algorithm computes prefix averages in g g p p gquadratic time by applying the definition

Algorithm prefixAverages1(X, n)g p f g ( , )Input array X of n integersOutput array A of prefix averages of X #operationsA ← new array of n integers nA ← new array of n integers nfor i ← 0 to n − 1 do n

s ← X[0] nfor j ← 1 to i do 1 + 2 + …+ (n − 1)

s ← s + X[j] 1 + 2 + …+ (n − 1)A[i] ← s / (i + 1) n

30

[ ] ( )return A 1

Arithmetic ProgressionArithmetic Progression

The running time of 7The running time of prefixAverages1 isO(1 + 2 + …+ n)

5

6

7

The sum of the first nintegers is n(n + 1) / 2 3

4

5

There is a simple visual proof of this fact

Th l i h 1

2

3

Thus, algorithm prefixAverages1 runs in O(n2) time

0

1

1 2 3 4 5 6

31

( )

Prefix Averages (Linear)g ( )The following algorithm computes prefix averages in linear time by keeping a running sumy p g g

Algorithm prefixAverages2(X, n)Input: array X of n integersInput: array X of n integersOutput: array A of prefix averages of X #operationsA ← new array of n integers n

0 1s ← 0 1for i ← 0 to n − 1 do n

s ← s + X[i] nA[i] ← s / (i + 1) n

return A 1

32

Algorithm prefixAverages2 runs in O(n) time

Relatives of Big-OhRelatives of Big Oh

big Omegabig-Omegaf(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1 such that g 0

f(n) ≥ c•g(n) for n ≥ n0

big-Thetabig-Thetaf(n) is Θ(g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer constant n0 ≥ 1 such that ’ ( ) ≤ f( ) ≤ ’’ ( ) f ≥c’•g(n) ≤ f(n) ≤ c’’•g(n) for n ≥ n0

33

Intuition for Asymptotic NotationIntuition for Asymptotic Notation

Big-OhBig Ohf(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)

big-Omegaf( ) i Ω( ( )) if f( ) i t ti llf(n) is Ω(g(n)) if f(n) is asymptotically greater than or equal to g(n)

big-Thetaf(n) is Θ(g(n)) if f(n) is asymptotically

l t ( )

34

equal to g(n)

Example Uses of the Relatives of Big-OhExample Uses of the Relatives of Big Oh

5n2 is Ω(n2)f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1

such that f(n) ≥ c•g(n) for n ≥ n0.Let c = 5 and n0 = 1

f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1 such that f(n) ≥ c•g(n) for n ≥ n0

5n2 is Ω(n)

5n2 is Θ(n2)

such that f(n) ≥ c g(n) for n ≥ n0.Let c = 1 and n0 = 1

f(n) is Θ(g(n)) if it is Ω(n2) and O(n2). We have already seen the former, for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1 such that f(n) < c•g(n) for n ≥ n0 .

35

Let c = 5 and n0 = 1

Cautions about Big-Oh

The constant factors do

g

The constant factors do matter when they are large.Ex.T ( ) 30000

Running time

TA(n) = 30000nTB(n) = 3n2

A t ti ll T i li

A

Asymptotically, TA is linear and TB is quadratic.

In practice if n is no more

B

In practice, if n is no more than 10000, algorithm B should be used.

10000n

36

Comparison of two Sorting Algorithms insertion sort is n2 / 4merge sort is 2 n lg n

sort a million items?insertion sort takes

hl 70 hroughly 70 hourswhile

merge sort takesroughly 40 secondsroughly 40 seconds

This is a slow machine, but if100x faster then it’s 40 minutesversus less than 0 5 seconds

by Matt Stallmann

versus less than 0.5 seconds.

sort 50 items?

37

by Matt Stallmann

Math You need to Review

Summations

Math You need to Review

properties of logarithms:Logarithmslogb(xy) = logbx + logbylogb (x/y) = logbx - logbyLogbxa = a logbxgb gb

logba = logxa/logxb

Exponents properties of exponentials:pa(b+c) = aba c

abc = (ab)c

ab /ac = a(b-c)

b l b

38

b = a logab

bc = a c*logab

E iExercises: Analysis of linear search, insertion-sort and classic matrix multiplication

39

Insertion-sort

Algorithm InsertionSort(A[1, …, n]) Ex. Input: array A of n elementsOutput: A, s.t. A[1] ≤ A[2] … ≤ A[n]

[8, 2, 4, 9, 3, 6]

Running timefor j ← 2 to n do

key ← A[j]i ← j - 1

(fixed n)Best case?Average case?j

while (i > 0) and (A[i] > key) doA[i+1] ← A[i]i ← i - 1

Average case?Worst case?

i ← i - 1A[i+1] ← key

40