analysis of algorithmsds172/wiki.files/01-analysis.pdf · data structures and algorithms in java....

Analysis of algorithms


Textbooks

Main textbook.

Introduction to Algorithms. Cormen, Leiserson, Rivestand Stein.

Additional textbooks.

Algorithm design: foundations, analysis, and Internetexamples. Michael T. Goodrich, Roberto Tamassia.

Data structures and algorithms in Java. Michael T.Goodrich, Roberto Tamassia.

Data structures and algorithms. Alfred V. Aho, John E.Hopcroft, Jeffrey D. Ullman.

Data structures & their algorithms. Harry R. Lewis, LarryDenenberg.

Algorithms in C++. Robert Sedgewick.


Algorithms

An algorithm is a well-defined computational procedurethat takes some value, or set of values, as input andproduces some value, or set of values, as output.

Algorithm is a tool for solving a well-specifiedcomputational problem. A computational problem isdefined by an input/output relationship.

Example

In the sorting problem,

Input A sequence of n number a1, . . . , an.

Output A permutation a′1, . . . , a′n of the input numbers

which is ordered, namely a′1 ≤ a′2 ≤ · · · ≤ a′n.


Insertion sort

Insertion-Sort(A)

(1) for j = 2 to A.length(2) key← A[j ]

% Insert A[j ] into the sorted sequence A[1..j − 1](3) i ← j − 1(4) while i > 0 and A[i ] > key(5) A[i + 1]← A[i ](6) i ← i − 1(7) A[i + 1]← key

45 726 1


Insertion sort

Insertion-Sort(A)



45 726 1 j = 2


Insertion sort

Insertion-Sort(A)



45 72 6 1 j = 3


Insertion sort

Insertion-Sort(A)



45 72 6 1 j = 4


Insertion sort

Insertion-Sort(A)



45 72 6 1 j = 5


Insertion sort

Insertion-Sort(A)



45 72 61 j = 6


Insertion sort

Insertion-Sort(A)



4 5 72 61


Insertion sort — Correctness

At the start of each iteration of the for loop, A[1..j − 1]contains the elements that are initially in A[1..j − 1] insorted order.

At the end of the algorithm (j = n + 1), the array issorted.

45 72 6 1


Analysis of algortihms

We want to measure the resources used by an algorithm:

Running time.Memory usage.

The complexity of an algorithm is a function of the inputsize.

In the sorting problem — the number of values.In graph problems — the number of nodes and edges.

For a fixed input size, the algorithm complexity dependson the specific input. We can consider several options.

Best case.Worst case.Average case.


Analysis of insertion sort

Insertion-Sort(A)

(1) for j = 2 to A.length(2) key← A[j ](3) i ← j − 1(4) while i > 0 and A[i ] > key(5) A[i + 1]← A[i ](6) i ← i − 1(7) A[i + 1]← key

cost timesc1 nc2 n − 1c3 n − 1c4

∑nj=2 tj

c5∑n

j=2(tj − 1)

c6∑n

j=2(tj − 1)

c7 n − 1

T (n) = c1n + c2(n − 1) + c3(n − 1) + c4

n∑j=2

tj+

c5

n∑j=2

(tj − 1) + c6

n∑j=2

(tj − 1) + c7(n − 1)


Insertion sort — Best case

The best case is when the array is sorted. In this case,case tj = 1 for all j .

The running time is

T (n) = c1n + c2(n − 1) + c3(n − 1) + c4(n − 1) + c7(n − 1)

= (c1 + c2 + c3 + c4 + c7)n − (c2 + c3 + c4 + c7)

= a′n − b′


Insertion sort — Worst case

The worst case is when the array is sorted in reverseorder. In this case tj = j for all j .∑n

j=2 j = n(n+1)2− 1 and

∑nj=2(j − 1) = n(n−1)

2.

The running time is

T (n) = c1n + c2(n − 1) + c3(n − 1) + c4

(n(n + 1)

2− 1

)+ c5

(n(n − 1)

2

)+ c6

(n(n − 1)

2

)+ c7(n − 1)

=(c4

2+

c52

+c62

)n2

+(c1 + c2 + c3 +

c42− c5

2− c6

2+ c7

)n

− (c2 + c3 + c4 + c7)

= an2 + bn − c



The worst-case running time of insertion sort isan2 + bn + c for some constants a, b, c .

The exact values of a, b, c depend on the computer onwhich we run the algorithm.

In theoretical computer science, we are interested in theasymptotic complexity, namely the behavior of theworst-case complexity function as n approaches infinity.

In the expression an2 + bn + c , the terms bn and cbecome negligible compared to an2 when n approachesinfinity.

Thus, the time complexity of insertion sort is ≈ an2.

We are not interested in the constant a. Thus, we saythat the running time of insertion sort grows like n2 whenn approaches infinity.


Order of growth

n n log n n2 n3 2n

10 33.2 100 1,000 1,02420 86.4 400 8,000 1,048,57630 147.2 900 27,000 1,073,741,824

100 664.4 10,000 1,000,000200 1528.8 40,000 8,000,000300 2468.6 90,000 27,000,000

1000 9965.8 1,000,000 1,000,000,0002000 21931.6 4,000,000 8,000,000,000


Θ

Θ(g) is the set of all function that grow like g when napproaches infinity.

Formally,

Θ(g) = f : there are positive constants c1, c2, and n0

such that 0 ≤ c1g(n) ≤ f (n) ≤ c2g(n) for all n ≥ n0


Example

Claim12n2 − 3n ∈ Θ(n2).

Proof.

We need to show that there are c1, c2, n0 such that for n ≥ n0,

c1n2 ≤ 1

2n2 − 3n ≤ c2n

2.

Clearly, 12n2 − 3n ≤ 1

2n2, so we can take c2 = 1/2.

Furthermore,

c1n2 ≤ 1

2n2 − 3n⇐⇒ 3n ≤

(1

2− c1

)n2 ⇐⇒ 3

12− c1

≤ n

We can take c1 = 1/4 and n0 = 12.


Example — Polynomials

Claim

Let f (n) = bknk + bk−1n

k−1 + · · ·+ b1n + b0, where bk > 0,be a polynomial of degree k . Then f (n) ∈ Θ(nk).

Proof.

Let ai = |bi |. Then,

f (n) ≤ aknk + ak−1n

k−1 + · · ·+ a1n + a0

≤ aknk + ak−1n

k + · · ·+ a1nk + a0n

k

= (ak + ak−1 + · · ·+ a0)nk

f (n) ≥ aknk − ak−1n

k−1 − · · · − a1n − a0

≥ aknk − ak−1n

k−1 − · · · − a1nk−1 − a0n

k−1

= aknk − (ak−1 + · · ·+ a1 + a0)nk−1

so f (n) ≥ 12akn

k for large enough n.Analysis of algorithms

Example

Some functions in Θ(n2) are

n2 + 10n

3n2 + 100n + 5

3n2 +√n + log n


O

O(g) is the set of all function that grow like g or slowerwhen n approaches infinity.

Formally,

O(g) = f : there are positive constants c and n0

such that 0 ≤ f (n) ≤ cg(n) for all n ≥ n0


Example

Some functions in Θ(n2) are

n2 + 10n

3n2 + 100n + 5

3n2 +√n + log n

Some functions in O(n2) are

n2 + 10n

3n2 + 100n + 5

3n2 +√n + log n

n

n1.999

log n

log log n


Ω

Ω(g) is the set of all function that grow at least as fastas g when n approaches infinity.

Formally,

Ω(g) = f : there are positive constants c and n0

such that 0 ≤ cg(n) ≤ f (n) for all n ≥ n0

Θ(g) = O(g) ∩ Ω(g).


Example

Some functions in Ω(n2) are

n2 + 10n

3n2 + 100n + 5

3n2 +√n + log n

n2 log log n

4n3

2n


Asymptotic notation in equations and inequalities

Stands alone on right-hand side means set membership.Example: n = O(n2) means n ∈ O(n2).

In general, asymptotic notation on the right side standsfor some anonymous function.Example: 2n2 + 3n + 1 = 2n2 + Θ(n) means that2n2 + 3n+ 1 = 2n2 + f (n) for some function f (n) in Θ(n).

On the left-hand side stands for any anonymous function.Example: 2n2 + Θ(n) = Θ(n2) means that for anyfunction g(n) in Θ(n) there is a function f (n) in Θ(n2)such that 2n2 + g(n) = f (n).


Asymptotic notations intuition

Intuitively,

O is like ≤.Ω is like ≥.Θ is like =.

Unlike real numbers, some functions cannot be compared.

For example, f (n) = n and g(n) =

n2 if n is even

1 if n is odd.


Properties

Transitivity

f ∈ Θ(g) and g ∈ Θ(h) implies f ∈ Θ(h).

Same for O,Ω.

Reflexivity

f ∈ Θ(f ).

Same for O,Ω.

Symmetry

f ∈ Θ(g) if and only if g ∈ Θ(f ).

Transpose symmetry

f ∈ O(g) if and only if g ∈ Ω(f ).


The Divide and Conquer Paradigm

Divide

Divide the problem into several (smaller) sub-problems.

Solve the sub-problems recursively.

Stop the recursion when the size of a sub-problem is smallenough. Then, either the sub-problem does not requiresolving, or apply some naive algorithm on thesub-problem.

Conquer

Combine the solutions of the sub-problems into thesolution for the original problem.


Merge sort

Devide

Divide the array into two sub-array of equal sizes.

Sort the two sub-arrays recursively.

Stop the recursion when the size of the sub-array is 1.An array of size 1 is always sorted, so no work is needed.

Conquer

Merge the two sorted sub-arrays to produce the sortedarray.

34 825 1 2 6


Merge sort

Devide




Conquer


34 825 1 2 6

merge


Merge sort

Devide




Conquer


34 82 5 1 2 6


Merge sort

Devide




Conquer


34 82 5 1 2 6

merge


Merge sort

Devide




Conquer


34 82 5 1 2 6


Merge sort

Devide




Conquer


34 82 5 1 2 6

merge


Merge sort

Devide




Conquer


34 82 5 1 2 6


Merge sort

Devide




Conquer


34 82 5 1 2 6

merge


Merge sort

Devide




Conquer


34 82 5 1 2 6


Merge sort

Devide




Conquer


34 82 5 1 2 6

merge


Merge sort

Devide




Conquer


3 4 82 51 2 6


Merge sort

Merge-Sort(A, left, right)

(1) if left < right(2) mid← b(left + right)/2c(3) Merge-Sort(A, left,mid)(4) Merge-Sort(A,mid + 1, right)(5) Merge(A, left,mid, right)

The array A is sorted by calling Merge-Sort(A, 1,A.length).


Merge

Merge(A, left,mid, right)(1) n1 ← mid− left + 1(2) n2 ← right−mid(3) Create array L[1..n1 + 1](4) Create array R[1..n2 + 1](5) for i = 1 to n1(6) L[i ]← A[left + i − 1](7) for j = 1 to n2(8) R[j ]← A[mid + j ](9) L[n1 + 1]←∞(10) R[n2 + 1]←∞

(11) i ← 1(12) j ← 1(13) for k = left to right(14) if L[i ] ≤ R[j ](15) A[k]← L[i ](16) i ← i + 1(17) else(18) A[k]← R[j ](19) j ← j + 1

9 10 11 12 13 14 15 16

A 25 842 1 3 6

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

25 842 1 3 6

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

25 842 1 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

25 841 1 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

25 821 1 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

22 821 1 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

22 321 1 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

22 321 4 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

52 321 4 3 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

52 321 4 6 6

i j

k

Merge(A, 9, 12, 16)


Merge



9 10 11 12 13 14 15 16

A

L 25 842 1 3 6∞ R ∞

52 321 4 6 8

i j

k

Merge(A, 9, 12, 16)


Analysis of merge sort

Merge-Sort(A, left, right)

(1) if left < right(2) mid← b(left + right)/2c(3) Merge-Sort(A, left,mid)(4) Merge-Sort(A,mid + 1, left)(5) Merge(A, left,mid, right)

costΘ(1)Θ(1)T (n/2)T (n/2)Θ(n)

T (n) =

Θ(1) if n = 1

2T (n/2) + Θ(n) if n > 1


analysis of algorithmsds172/wiki.files/01-analysis.pdf · data structures and algorithms in java....

Documents