analysis of algorithms

Analysis of Algorithms

Algorithm

Input Output

Analysis of Algorithms 2

Outline

Running timePseudo-codeCounting primitive operationsAsymptotic notationAsymptotic analysis


How good is Insertion-Sort?

How can you answer such questions?

What is “goodness”?

1. Measure2. Count3. Estimate


How can we quantify it?

1. Correctness2. Minimum use of “time” + “space”


1) Measure it

Write a program implementing the algorithmRun the program with inputs of varying size and compositionUse a method like System.currentTimeMillis() to get an accurate measure of the actual running timePlot the results

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 50 100

Input Size

Tim

e (

ms)


Limitations of Experiments

It is necessary to implement the algorithm, which may be difficultResults may not be indicative of the running time on other inputs not included in the experiment. In order to compare two algorithms, the same hardware and software environments must be used

2) Count Primitive Operations

The Idea Write down the pseudocode Count the number of “primitive

operations”



PseudocodeHigh-level description of an algorithmMore structured than English proseLess detailed than a programPreferred notation for describing algorithmsHides program design issues

Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A

currentMax A[0]for i 1 to n 1 do

if A[i] currentMax thencurrentMax A[i]

return currentMax

Example: find max element of an array


Pseudocode Details

Control flow if … then … [else …] while … do … repeat … until … for … do … Indentation replaces

braces

Method declarationAlgorithm method (arg [, arg…])

Input …

Output …

Method callvar.method (arg [, arg…])

Return valuereturn expression

Expressions Assignment

(like in Java) Equality testing

(like in Java)n2 Superscripts and

other mathematical formatting allowed


The Random Access Machine (RAM) Model

A CPU

An potentially unbounded bank of memory cells, each of which can hold an arbitrary number or character

01

2

Memory cells are numbered and accessing any cell in memory takes unit time.

Random Access Model (RAM)

Time complexity (running time) = number of instructions executedSpace complexity = the number of memory cells accessed



Primitive Operations

Basic computations performed by an algorithmIdentifiable in pseudocodeLargely independent from the programming languageExact definition not important (we will see why later)

Examples: Evaluating an

expression Assigning a

value to a variable

Indexing into an array

Calling a method Returning from a

method

Analyzing pseudocode (by counting)

1. For each line of pseudocode, count the number of primitive operations in it. Pay attention to the word "primitive" here; sorting an array is not a primitive operation.

2. Multiply this count with the number of times this line is executed.

3. Sum up over all lines.



Counting Primitive Operations

By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size

Algorithm arrayMax(A, n)

Cost Times

currentMax A[0] 2 1for i 1 to n 1 do 2 n

if A[i] currentMax then 2 (n 1)currentMax A[i] 2 (n 1)

{ increment counter i } 2 (n 1)return currentMax 1 1

Total 8n 3


Estimating Running Time

Algorithm arrayMax executes 8n 2 primitive operations in the worst case Definea Time taken by the fastest primitive operationb Time taken by the slowest primitive operation

Let T(n) be the actual worst-case running time of arrayMax. We have

a (8n 2) T(n) b(8n 2)

Hence, the running time T(n) is bounded by two linear functions

Insertion Sort


Hint: 1.observe each line i can be implemented using a constant number of RAM instructions, Ci

2.for each value of j in the outer loop, we let tj be the number of times that the while loop test in line 4 is executed.

Insertion Sort


Question: So what is the running time?

Time Complexity may depend on the input!

Best Case:?

Worst Case:?

Average Case:?


Complexity may depend on the input!

Best Case: Already sorted. tj=1. Running time = C * N (a linear function of N)

Worst Case: Inverse order. tj=j.



Arithmetic Progression

Neglecting the constants, the worst-base running time of insertion sort is proportional to1 2 …n

The sum of the first n integers is n(n 1) 2

Thus, algorithm insertion sort required almost n2 operations.

0

1

2

3

4

5

6

7

1 2 3 4 5 6

What about the Average Case?

Idea: Assume that each of the n!

permutations of A is equally likely. Compute the average over all

possible different inputs of length N.

Difficult to compute!In this course we focus on Worst Case Analysis!


See TimeComplexity Spreadsheet


What is “goodness”?

1. Measure wall clock time2. Count operations3. Estimate Computational Complexity


How can we quantify it?

1. Correctness2. Minimum use of “time” + “space”


Asymptotic Analysis

Uses a high-level description of the algorithm instead of an implementationAllows us to evaluate the speed of an algorithm independent of the hardware/software environmentEstimate the growth rate of T(n)A back of the envelope calculation!!!


The idea

Write down an algorithm Using Pseudocode In terms of a set of primitive operations

Count the # of steps In terms of primitive operations Considering worst case input

Bound or “estimate” the running time Ignore constant factors Bound fundamental running time


Growth Rate of Running Time

Changing the hardware/ software environment Affects T(n) by a constant factor, but Does not alter the growth rate of T(n)

The linear growth rate of the running time T(n) is an intrinsic property of algorithm arrayMax


Which growth rate is best?

T(n) = 1000n + n2 or T(n) = 2n + n3


Growth Rates

Growth rates of functions:

Linear n Quadratic n2

Cubic n3

In a log-log chart, the slope of the line corresponds to the growth rate of the function

1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+261E+281E+30

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Cubic

Quadratic

Linear

Functions Graphed Using “Normal” Scale

30Analysis of Algorithms

g(n) = 2ng(n) = 1

g(n) = lg n

g(n) = n lg n

g(n) = n

g(n) = n2

g(n) = n3


Constant Factors

The growth rate is not affected by

constant factors or

lower-order terms

Examples 102n 105 is a

linear function 105n2 108n is a

quadratic function1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+26

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Quadratic

Quadratic

Linear

Linear


Big-Oh NotationGiven functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constantsc and n0 such that

f(n) cg(n) for n n0

Example: 2n 10 is O(n) 2n 10 cn (c 2) n 10 n 10(c 2) Pick c 3 and n0 10

1

10

100

1,000

10,000

1 10 100 1,000n

3n

2n+10

n


f(n) is O(g(n)) iff f(n) cg(n) for n n0


Big-Oh Notation (cont.)

Example: the function n2 is not O(n)

n2 cn n c The above

inequality cannot be satisfied since c must be a constant

1

10

100

1,000

10,000

100,000

1,000,000

1 10 100 1,000n

n^2

100n

10n

n


More Big-Oh Examples

7n-2

7n-2 is O(n)need c > 0 and n0 1 such that 7n-2 c•n for n n0

this is true for c = 7 and n0 = 1

3n3 + 20n2 + 5

3n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n n0


3 log n + 5

3 log n + 5 is O(log n)

need c > 0 and n0 1 such that 3 log n + 5 c•log n for n n0



Big-Oh and Growth Rate

The big-Oh notation gives an upper bound on the growth rate of a functionThe statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the growth rate of g(n)

We can use the big-Oh notation to rank functions according to their growth rate


Questions

Is T(n) = 9n4 + 876n = O(n4)? Is T(n) = 9n4 + 876n = O(n3)? Is T(n) = 9n4 + 876n = O(n27)?

T(n) = n2 + 100n = O(?)T(n) = 3n + 32n3 + 767249999n2 = O(?)


Big-Oh Rules (shortcuts)

If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,

1. Drop lower-order terms

2. Drop constant factors

Use the smallest possible class of functions Say “2n is O(n)” instead of “2n is O(n2)”

Use the simplest expression of the class Say “3n 5 is O(n)” instead of “3n 5 is O(3n)”


Rank From Fast to Slow…

T(n) = O(n4) T(n) = O(n log n)

T(n) = O(n2)T(n) = O(n2 log n)

T(n) = O(n)T(n) = O(2n)T(n) = O(log n)

T(n) = O(n + 2n)

Note: Assume base of log is 2 unless

otherwise instructedi.e. log n = log2 n


Computing Prefix Averages

We further illustrate asymptotic analysis with two algorithms for prefix averagesThe i-th prefix average of an array X is average of the first (i 1) elements of X

A[i] X[0] X[1] … X[i]

Computing the array A of prefix averages of another array X has applications to financial analysis

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

X

A


Prefix Averages (Quadratic)The following algorithm computes prefix averages in quadratic time by applying the definition

Algorithm prefixAverages1(X, n)Input array X of n integersOutput array A of prefix averages of X #operations A new array of n integers nfor i 0 to n 1 do n

s X[0] nfor j 1 to i do 1 2 … (n 1)

s s X[j] 1 2 … (n 1)A[i] s (i 1) n

return A 1


Arithmetic Progression

The running time of prefixAverages1 isO(1 2 …n)

The sum of the first n integers is n(n 1) 2

There is a simple visual proof of this fact

Thus, algorithm prefixAverages1 runs in O(n2) time

0

1

2

3

4

5

6

7

1 2 3 4 5 6


Prefix Averages (Linear)The following algorithm computes prefix averages in linear time by keeping a running sum

Algorithm prefixAverages2(X, n)Input array X of n integersOutput array A of prefix averages of X #operationsA new array of n integers ns 0 1for i 0 to n 1 do n

s s X[i] nA[i] s (i 1) n

return A 1Algorithm prefixAverages2 runs in O(n) time


properties of logarithms:logb(xy) = logbx + logby

logb (x/y) = logbx - logby

logbxa = alogbx

logba = logxa/logxbproperties of exponentials:a(b+c) = aba c

abc = (ab)c

ab /ac = a(b-c)

b = a logab

bc = a c*logab

SummationsLogarithms and Exponents

Proof techniquesBasic probability

Math you may need to Review


Big-Omega Definition: f(n) is (g(n)) if there is a constant c

> 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0

An asymptotic lower Bound


Big-Theta Definition: f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer constant n0 1 such that c’•g(n) f(n) c’’•g(n) for n n0

Here we say that g(n) is an asymptotically tight bound for f(n).

Big-Theta (cont)

Based on the definitions, we have the following theorem:

f(n) = Θ(g(n)) if and only if f(n) = O(g(n)) and f(n) = Ω(g(n)).

For example, the statement n2/2 + lg n = Θ(n2) is equivalent to n2/2 + lg n = O(n2) and n2/2 + lg n = Ω(n2).

Note that asymptotic notation applies to asymptotically positive functions only, which are functions whose values are positive for all sufficiently large n.


Prove n2/2 + lg n = Θ(n2).

Proof. To prove this claim, we must determine positive constants c1, c2 and n0, s.t. c1 n2<= n2/2 + lg n <= c2 n2

c1 <= ½ + (lg n) / n2 <= c2 (divide thru by n2)

Pick c1 = ¼ , c2 = ¾ and n0 = 2 For n0 = 2, 1/4 <= ½ + (lg 2) / 4 <= ¾, TRUE When n0 > 2, the (½ + (lg 2) / 4) term grows

smaller but never less than ½, therefore n2/2 + lg n = Θ(n2).



Intuition for Asymptotic Notation

Big-Oh f(n) is O(g(n)) if f(n) is

asymptotically less than or equal to g(n)

big-Omega f(n) is (g(n)) if f(n) is

asymptotically greater than or equal to g(n)

big-Theta f(n) is (g(n)) if f(n) is

asymptotically equal to g(n)

analysis of algorithms

Documents

primitive operationsby

word primitive

analysis of algorithmswhat

arbitrary number

number of times

line of pseudocode

unit time

arginput output method