algorithmics - lecture 51 lecture 5: analysis of algorithms efficiency (ii)

Algorithmics - Lecture 5 1

LECTURE 5:

Analysis of Algorithms Efficiency (II)

In the previous lecture we saw …

Which are the basic steps in analyzing the time efficiency of an algorithm:

• Identify the input size

• Identify the dominant operation

• Count for how many times the dominant operation is executed (estimate the running time)

• If this number depends on the properties of input data we analyze:– Best case => lower bound of the running time – Worst case => upper bound of the running time – Average case => averaged running time

Today we will see that …… the main aim of efficiency analysis is to find out how increases

the running time when the problem size increases

… we don’t need very detailed expressions of the running time, but we need to identify:

– The order of growth of the running time– The efficiency class to which an algorithm belongs

Outline• What is the order of growth ?

• What is asymptotic analysis ?

• Some asymptotic notations

• Efficiency analysis of basic processing structures

• Efficiency classes

• Empirical analysis of the algorithms

What is the order of growth?In the expression of the running time one of the terms will become

significantly larger than the other ones when n becomes large : this is the so-called dominant term

T1(n)=an+b

T2(n)=a log n+b

T3(n)=a n2+bn+c

T4(n)=an+b n +c (a>1)

Dominant term: a n

Dominant term: a log n

Dominant term: a n2

Dominant term: an

What is the order of growth?Let us analyze what happens with the dominant term when the input

size is multiplied by k:

T1(n)=an

T2(n)=a log n

T3(n)=a n2

T4(n)=an

T’1(kn)= a kn=k T1(n)

T’2(kn)=a log(kn)=T2(n)+alog k

T’3(kn)=a (kn)2=k2 T3(n)

T’4(kn)=akn=(an)k =T4(n)k

What is the order of growth?The order of growth expresses how increases the dominant term of

the running time with the input size

Order of growthLinear

Logarithmic

Quadratic

Exponential

T’1(kn)= a kn=k T1(n)

T’2(kn)=a log(kn)=T2(n)+alog k

T’3(kn)=a (kn)2=k2 T3(n)

T’4(kn)=akn=(an)k

How can be interpreted the order of growth?

Between two algorithms it is considered that the one having a smaller order of growth is more efficient

However, this is true only for large enough input sizes

Example. Let us consider T1(n)=10n+10 (linear order of growth)T2(n)=n2 (quadratic order of growth)

If n<=10 then T1(n)>T2(n)

In this case the order of growth is relevant only for n>10

A comparison of orders of growth

n log2n nlog2n n2 2n

10 3.3 33 100 1024

100 6.6 664 10000 1030

1000 10 9965 1000000 10301

10000 13 132877 100000000 103010

The multiplicative constants in the dominant term can be ignored

Comparing orders of growth

The order of growth of two running times T1(n) and T2(n) can be compared by computing the limit of T1(n)/T2(n) when n goes to infinity.

If the limit is 0 then T1(n) has a smaller order of growth than T2(n)

If the limit is a finite constant c (c>0) then T1(n) and T2(n) have the same order of growth

If the limit is infinity then T1(n) has a larger order of growth than T2(n)

What is asymptotic analysis ?

• The differences between orders of growths are more significant for larger input size

• Analyzing the running times on small inputs does not allow us to distinguish between efficient and inefficient algorithms

• Asymptotic analysis deals with analyzing the properties of the running time when the input size goes to infinity

(this means a large input size)

What is asymptotic analysis ?• Depending on the behavior of the running time when the input

size becomes large, the algorithm can belong to different classes of efficiency

• There are several standard notations used in algorithms efficiency analysis :

(big – Theta) O (big – O) Ω (big – Omega)

• What is the order of growth ?

• What is asympotic analysis ?

Outline

Let f,g: N-> R+

Definition.f(n) (g(n)) iff there exist c1, c2 > 0 and n0 N such that c1g(n) ≤f(n) ≤ c2g(n) for all n≥n0

Notation. f(n)= (g(n)) (same order of growth as g(n))

Examples.1. T(n) = 3n+3 T(n) (n) c1=2, c2=4, n0=3, g(n)=n2. T(n)= n2+10 nlgn +5 T(n) (n2) c1=1, c2=2, n0=40, g(n)=n2

- notation

0 20 40 60 80 1000

Graphical illustration. f(n) is bounded, for large values of n, both above and below by g(n) multiplied by some positive constants

f(n)= n2+10 nlgn +5

c2g(n)=2n2

c1g(n)=n2

Doesn’t matter

c1g(n) ≤ f(n) ≤ c2g(n)

1. If T(n)=aknk+ak-1nk-1+…+a1n+a0 then T(n) (nk)

Proof. Since T(n)>0 for all n it follows that ak>0.

Then T(n)/nk ->ak (as n->).

Thus for all ε>0 there exists N(ε) such that |T(n)/nk- ak|< ε => ak- ε<T(n)/nk<ak+ ε for all n>N(ε)

Let us suppose that ak- ε>0.

Then by taking c1=(ak- ε), c2=ak+ ε and n0=N(ε) one obtains

c1nk < T(n) <c2nk for all n>n0, i.e. T(n) (nk)

- notation. Properties

2. (c g(n))= (g(n)) for all constant c

Proof. Let f(n) (cg(n)).

Then c1cg(n) ≤ f(n) ≤ c2cg(n) for all n≥n0. By taking c’1= cc1 and c’2= c c2 we obtain that f(n) (g(n)). Thus (cg(n)) (g(n)). Similarly we can prove that (g(n)) (cg(n)), so (cg(n))= (g(n)).

Particular cases:a) (c)= (1) b) (logah(n))= (logbh(n)) for all a,b >1

The logarithm’s base is not significant in specifying the efficiency class. We will use logarithms in base 2

3. f(n) (f(n)) (reflexivity)

4. f(n) (g(n)) => g(n) (f(n)) (symmetry)

5. f(n) (g(n)) , g(n) (h(n)) => f(n) (h(n)) (transitivity)

6. (f(n)+g(n)) = (max{f(n),g(n)})

3. 3n<=T(n) <=4n-1 T(n) (n) c1=3, c2=4, n0=1

4. Multiplying two matrices: T(m,n,p)=4mnp+5mp+4m+2Extension of the definition:f(m,n,p) (g(m,n,p)) iff there exist c1, c2 >0 and m0,n0 ,p0 N such that c1g(m,n,p) <=f(m,n,p) <=c2g(m,n,p) for all m>=m0, n>=n0, p>=p0

Thus T(m,n,p) (mnp)

5. Sequential search: 6<= T(n) <= 3(n+1)If T(n)=6 then we cannot find c1 such that 6 >= c1n for large values of n.Thus T(n) does not belong to (n). There exist running times which do not belong to a big-theta class.

- notation. More examples

Definition.f(n) O(g(n)) iff there exist c >0 and n0 N such that f(n) <=c g(n) for all n>=n0

Notation. f(n)= O(g(n)) (an order of growth at most as that of g(n))

Examples.

1. T(n) = 3n+3 T(n) O(n) c=4, n0=3, g(n)=n2. 6<= T(n) <= 3(n+1) T(n) O(n) c=4, n0=3, g(n)=n

O - notation

Graphical illustration. f(n) is bounded above, for large values of n, by g(n) multiplied by a positive constant

O - notation

0 20 40 60 80 1000

f(n)= 10nlgn +5

cg(n)=n2

f(n)<=cg(n)

doesn’t matter

O – notation. Properties1. If T(n)=aknk+ak-1nk-1+…+a1n+a0 then T(n) O(nd) for all d>=k

Proof. Since T(n)>0 for all n it follows that ak>0. Then T(n)/nk -> ak (as n->). Thus for all ε>0 there exists N(ε) such that T(n)/nk <= ak+ ε for all n>N(ε) Hence T(n) <= (ak+ ε)nk <= (ak+ε)nd

Then by taking c=ak+ ε and n0=N(ε) one obtains T(n) <cnd for all n>n0, i.e. T(n) O(nd)

Example.n O (n2) (it is correct but is more useful to write n O (n))

O – notation. Properties2. f(n) O(f(n)) (reflexivity)

3. f(n) O(g(n)) , g(n) O(h(n)) => f(n) O(h(n)) (transitivity)

4. (g(n)) is a subset of O(g(n)

Remark. The inclusion is a strict one: there exist elements of O(g(n)) which do not belong to (g(n))

Example: f(n)=10nlgn+5, g(n)=n2

f(n)<=g(n) for all n>=36 f(n)O(g(n))But it don’t exist constants c and n0 such that: cn2 <= 10nlgn+5 for all n >= n0

O – notation. PropertiesWhen by a worst case analysis we obtain that

T(n) <= g(n) we can say that the running time of the algorithm belongs to O(g(n))

Example. Sequential search: 6<= T(n) <= 3(n+1)

Thus the running time of sequential search belongs to O(n)

Ω – notationDefinition.f(n) Ω(g(n)) iff there exist c > 0 and n0 N such that

cg(n) <= f(n) for all n>=n0

Notation. f(n)= Ω(g(n)) (an order of growth at least as that of g(n))

Examples.1. T(n) = 3n+3 T(n) Ω(n) c=3, n0=1, g(n)=n

2. 6<= T(n) <= 3(n+1) T(n) Ω(1) c=6, n0=1, g(n)=1

Ω – notationGraphical illustration. f(n) is bounded below by g(n) multiplied by a

positive constant

0 20 40 60 80 1000

sn’t

cg(n)<=f(n)

f(n)=10nlgn+5

cg(n)=20n

Ω – notation. Properties1. If T(n)=aknk+ak-1nk-1+…+a1n+a0 then T(n) Ω(nd) for all d<=k

Proof. Since T(n)>0 for all n it follows that ak>0.

Then T(n)/nk -> ak (as n->). Thus for all ε>0 there exists N(ε) such that ak - ε <= T(n)/nk for all n>N(ε) Hence (ak -ε)nd <=(ak-ε)nk <= T(n) Then by taking c=ak- ε and n0=N(ε) one obtains cnd <= T(n) for all n>n0, i.e. T(n) Ω(nd)

Example.n2 Ω (n)

Ω – notation. Properties2. (g(n)) Ω(g(n)

Proof. It suffices to consider only the lower bound from big-theta definition

Remark. The inclusion is a strict one: there exist elements of Ω(g(n)) which do not belong to (g(n))

Example: f(n)=10nlgn+5, g(n)=n f(n) >= 10g(n) for all n>=1 f(n) Ω(g(n))But it don’t exist constants c and n0 such that: 10nlgn+5<=cn for all n >= n0

3. (g(n))=O(g(n))Ω(g(n)

Efficiency analysis of basic processing structures

• Sequential structureP: P1 (g1(n)) O(g1(n)) Ω(g1(n))

P2 (g2(n)) O(g2(n)) Ω(g2(n))

… … … Pk (gk(n)) O(gk(n)) Ω(gk(n))

---------------------------------------------------- (max{g1(n),g2(n), …, gk(n)})

O(max{g1(n),g2(n), …, gk(n)})

Ω(max{g1(n),g2(n), …, gk(n)})

• Conditional statementP:

IF <condition> THEN P1 (g1(n)) O(g1(n)) Ω(g1(n))

ELSE P2 (g2(n)) O(g2(n)) Ω(g2(n))

--------------------------------------------------------------------------- O(max{g1(n),g2(n)})

Ω(min{g1(n),g2(n)})

• Loop statementP:

FOR i:=1, n DO P1 (1) (n) FOR i:=1,n DO FOR j:=1,n DO P1 (1) (n2)

Remark: If the counting variables vary between 1 and n the complexity is nk (k is the number superposed loops)

Remark. If the limits of the counters are modified inside the loop body

then the analysis need to be modifiedExample:m:=1FOR i:=1,n DO m:=3*m {m=3i} FOR j:=1,m DO processing step from (1)

The complexity of the sequence is: 3+32+…+3n = (3n+1-1)/2-1The complexity (3n)

Efficiency classesSome frequently encountered efficiency classes:

Name of the class Asymptotic notation

Example

logarithmic O(lgn) Binary search

linear O(n) Sequential search

quadratic O(n2) Insertion sort

cubic O(n3) Multiplication of two n*n matrices

exponential O(2n) Processing all subsets of a set with n elements

factorial O(n!) Processing all permutations of order n

Algoritmica - Curs 5 37

ExampleLet, x[1..n] be an array with values from the set {1,…,n}. This array

either has distinct elements or there is a unique pair of indices (i,j) such that i<>j and x[i]=x[j]

Particular case: n=8, x=[2,1,4,5,3,8,7,6] all elements are distinct x=[2,1,4,5,3,8,5,6] there is a pair of identical elements

Find an efficient algorithm (both with respect the running time and the memory space) to check if all elements are distinct or not

ExampleVariant 1:check1(x[1..n])i:=1d=Truewhile (d=True) and (i<n) do d:=NOT (search(x[i+1..n],x[i])) i:=i+1 endwhilereturn dProblem size: n1<= T(n)<=T’(n-1)+T’(n-2)+…+T’(1)1<=T(n)<=n(n-1)/2T(n) Ω (1), T(n) O (n2)

search(x[left..right],v)i:=leftwhile x[i]<>v AND i<right do i:=i+1 endwhileif x[i]=v then return True else return FalseendifSubproblem size: k=f-s+11<= T’(k)<=k

Best case: x[1]=x[2]Worst case: distinct elements

ExampleVariant 2:check2(x[1..n])Integer f[1..n] // frequenciesf[1..n]:=0 for i:=1 to n do f[x[i]]:=f[x[i]]+1i:=1while f[i]<2 AND i<n do i:=i+1if f[i]>=2 then return False else return TrueendifProblem size: nn+1<= T(n)<=2nT(n) (n)

Variant 3:check3(x[1..n])Integer f[1..n] // frequenciesf[1..n]:=0i:=1 while i<=n do f[x[i]]:=f[x[i]]+1 if f[x[i]]>=2 then return False i:=i+1 endifendwhilereturn True

Problem size: n4<= T(n)<=2n

T(n) O (n)

ExampleVariant 4:Variants 2 and 3 need and

additional memory space of size O(n)

Can we solve the problem in linear running time by using an additional memory space of size O(1) ?

Idea: the elements are distinct if and only if the array contains all elements of the set {1,2,…,n}.In only one value can be duplicate then this is equivalent to the fact that the sum is n(n+1)/2

check4(x[1..n])s:=0for i:=1 to n do s:=s+x[i] endforif s=n(n+1)/2 then return True else return FalseEndifProblem size: nT(n) = nT(n) (n)

Remark. Variant 4 is better than variant 3 with respect to the size of the memory space but the average running time is smaller in variant 3 than in variant 4

Empirical analysis of the algorithmsSometimes the mathematical analysis of efficiency is too difficult to

apply … in these cases the empirical analysis could be useful

It can be used to:• Develop a hypothesis about the algorithm’s efficiency• Compare the efficiency of several algorithms designed to solve

the same problem• Establish the efficiency of an algorithm’s implementation• Check the accuracy of a theoretical assertion about algorithm’s

efficiency

General plan for empirical analysis1. Establish the aim of the analysis

2. Choose an efficiency measure (e.g. number of executions of some operations or time needed to execute a sequence of processing steps)

3. Decide on the characteristics of the input sample (size, range …)

4. Implement the algorithm in a programming language

5. Generate a set of data inputs

6. Execute the program for each data sample and record the results

7. Analyze the obtained results

General plan for empirical analysis

Efficiency measure. It is chosen depending on the aim of the empirical analysis:

• If the aim is to estimate the efficiency class an adequate efficiency measure is the number of operations

• If the aim is to analyze/compare the implementation of an algorithm on a given machine an adequate efficiency measure is the physical time

Set of input data. Different input data must be generated in order to conduct a useful empirical analysis

Some rules in generating input data:

• The input data in the set should be of different sizes and values (the entire range of values should be represented)

• All characteristics of input data should be represented in the sample set (different configurations)

• The data should be typical (not only exceptions)

Algorithm’s implementation. Some monitoring processing step should be included:

• Counting variables (when the efficiency measure is the number of executions)

• Calls of some functions which return the current time (in order to estimate the time needed to execute a processing sequence)

Next lecture will be on …… basic sorting algorithms

… on their correctness and

… on their efficiency

algorithmics - lecture 51 lecture 5: analysis of algorithms efficiency (ii)

Documents

cs 410 applied algorithms applied algorithms lecture #...

alg0183 algorithms & data structures lecture 4 experimental...

efficient algorithms lecture 1

cse373:’datastructures’&’algorithms’...

web algorithmics

experimental algorithmics applied to on...

algorithms - cs.princeton.edu · scientiﬁc method applied...

algorithmics - lecture 21 lecture 2: algorithms description...

algorithms pseudocode; examples - gabriel...

data streams: algorithms...

lecture 3.4: recursive algorithms

algorithmics 08 heaps.ppt

algorithmics - lecture 71 lecture 7: algorithms design...

coemalkapur.ac.incoemalkapur.ac.in/dist/pdf/cse.pdf ·...

algorithms lecture 5: sorting algorithms ii

lecture 14: graph algorithms

parameterized algorithmics for computational social choice...

algorithmics - lecture 41 lecture 4: analysis of algorithms...

course 9: algorithms design techniques - decrease and...

algorithmics - lecture 8-91 lecture 8: divide and conquer