cosc 3101a - design and analysis of algorithms 2 asymptotic notations continued proof of...

COSC 3101A - Design and Analysis of Algorithms

2

Asymptotic Notations Continued

Proof of Correctness: Loop Invariant

Designing Algorithms: Divide and Conquer

5/11/2004 Lecture 2 COSC3101A 2

Typical Running Time Functions

• 1 (constant running time):

– Instructions are executed once or a few times

• logN (logarithmic)

– A big problem is solved by cutting the original problem in smaller

sizes, by a constant fraction at each step

• N (linear)

– A small amount of processing is done on each input element

• N logN

– A problem is solved by dividing it into smaller problems, solving

them independently and combining the solution

5/11/2004 Lecture 2 COSC3101A 3

Typical Running Time Functions

• N2 (quadratic)

– Typical for algorithms that process all pairs of data items (double

nested loops)

• N3 (cubic)

– Processing of triples of data (triple nested loops)

• NK (polynomial)

• 2N (exponential)

– Few exponential algorithms are appropriate for practical use

5/11/2004 Lecture 2 COSC3101A 4

Logarithms

• In algorithm analysis we often use the notation “log n”

without specifying the base

nn

nn

elogln

loglg 2

yxlogBinary logarithm

Natural logarithm

)lg(lglglg

)(lglg

nn

nn kk

xy log

xylog yx loglog

y

xlog yx loglog

xalog xb ba loglog

abx logxba log

5/11/2004 Lecture 2 COSC3101A 5

Review: Asymptotic Notations(1)

))(( ng))(( ngO))(( ng

)()(0:,0,0|)())(( 00 ncgnfnnncnfngo

)()(0:,0,0|)())(( 00 nfncgnnncnfng

)()(0:,0,0|)())(( 00 nfncgnnncnfng

)()()(:,0,0,0|)())(( 120021 ngcnfngcnnnccnfng

)()(0:,0,0|)())(( 00 ncgnfnnncnfngO

))(( ngo ))(( ng

5/11/2004 Lecture 2 COSC3101A 6


))(( ng))(( ngO ))(( ng

))(()( ngonf

))(( ngo ))(( ng

if and only if 0)(

)(lim

ng

nfn

))(()( ngnf if and only if )(

)(lim

ng

nfn

L0L L0

5/11/2004 Lecture 2 COSC3101A 7


• A way to describe behavior of functions in the limit

– How we indicate running times of algorithms

– Describe the running time of an algorithm as n grows to

• O notation: asymptotic “less than”: f(n) “≤”

g(n)

notation: asymptotic “greater than”: f(n) “≥” g(n)

notation: asymptotic “equality”: f(n) “=” g(n)

5/11/2004 Lecture 2 COSC3101A 8

Big-O Examples(1)

– 2n2 = O(n3):

– n2 = O(n2):

– 1000n2+1000n = O(n2):

– n = O(n2):

2n2 ≤ cn3 2 ≤ cn c = 1 and n0= 2

n2 ≤ cn2 c ≥ 1 c = 1 and n0= 1

1000n2+1000n ≤ cn2 1000n+1000 ≤ cn c=2000 and n0 = 1

n ≤ cn2 cn ≥ 1 c = 1 and n0= 1

5/11/2004 Lecture 2 COSC3101A 9

Big-O Examples(2)

• E.g.: prove that n2 ≠ O(n)

– Assume c & n0 such that: n≥ n0: n2 ≤ cn

– Choose n = max (n0, c)

– n2 = n n ≥ n c n2 ≥ cn

contradiction!!!

5/11/2004 Lecture 2 COSC3101A 10

More on Asymptotic Notations

• There is no unique set of values for n0 and c in proving the

asymptotic bounds

• Prove that 100n + 5 = O(n2)

– 100n + 5 ≤ 100n + n = 101n ≤ 101n2

for all n ≥ 5

n0 = 5 and c = 101 is a solution

– 100n + 5 ≤ 100n + 5n = 105n ≤ 105n2

for all n ≥ 1

n0 = 1 and c = 105 is also a solution

Must find SOME constants c and n0 that satisfy the asymptotic notation relation

5/11/2004 Lecture 2 COSC3101A 11

Big- Examples

– 5n2 = (n)

– 100n + 5 ≠ (n2)

– n = (2n), n3 = (n2), n = (logn)

c, n0 such that: 0 cn 5n2 cn 5n2 c = 1 and n0 = 1

c, n0 such that: 0 cn2 100n + 5

100n + 5 100n + 5n ( n 1) = 105n

cn2 105n n(cn – 105) 0

Since n is positive cn – 105 0 n 105/c

contradiction: n cannot be smaller than a constant

5/11/2004 Lecture 2 COSC3101A 12

Examples

– n2/2 –n/2 = (n2)

• ½ n2 - ½ n ≤ ½ n2 n ≥ 0 c2= ½

• ½ n2 - ½ n ≥ ½ n2 - ½ n * ½ n ( n ≥ 2 ) = ¼ n2 c1= ¼

– n ≠ (n2): c1 n2 ≤ n ≤ c2 n2 only holds for: n ≤ 1/c1

– 6n3 ≠ (n2): c1 n2 ≤ 6n3 ≤ c2 n2 only holds for: n ≤

c2 /6

– n ≠ (logn): c1 logn ≤ n ≤ c2 logn

c2 ≥ n/logn, n≥ n0 – impossible

5/11/2004 Lecture 2 COSC3101A 13

Comparisons of Functions

• Theorem:f(n) = (g(n)) f = O(g(n)) and f = (g(n))

• Transitivity:– f(n) = (g(n)) and g(n) = (h(n)) f(n) = (h(n))– Same for O and

• Reflexivity:– f(n) = (f(n))– Same for O and

• Symmetry:– f(n) = (g(n)) if and only if g(n) = (f(n))

• Transpose symmetry:– f(n) = O(g(n)) if and only if g(n) = (f(n))

5/11/2004 Lecture 2 COSC3101A 14

More Examples(1)

• For each of the following pairs of functions, either f(n) is O(g(n)), f(n) is Ω(g(n)), or f(n) = Θ(g(n)). Determine which relationship is correct.

– f(n) = log n2; g(n) = log n + 5

– f(n) = n; g(n) = log n2

– f(n) = log log n; g(n) = log n

– f(n) = n; g(n) = log2 n

– f(n) = n log n + n; g(n) = log n

– f(n) = 10; g(n) = log 10

– f(n) = 2n; g(n) = 10n2

– f(n) = 2n; g(n) = 3n

f(n) = (g(n))

f(n) = (g(n))

f(n) = O(g(n))

f(n) = (g(n))

f(n) = (g(n))

f(n) = (g(n))

f(n) = (g(n))

f(n) = O(g(n))

5/11/2004 Lecture 2 COSC3101A 15

More Examples(2)

notation

– n2/2 – n/2

– (6n3 + 1)lgn/(n + 1)

– n vs. n2

notation

– n vs. 2n

– n3 vs. n2

– n vs. logn

– n vs. n2

= (n2)

n ≠ (n2)

= (n2lgn)

n = (2n)

• O notation

– 2n2 vs. n3

– n2 vs. n2

– n3 vs. nlogn

n3 = (n2)

n = (logn)

n (n2)

2n2 = O(n3)

n2 = O(n2)

n3 O(nlgn)

5/11/2004 Lecture 2 COSC3101A 16

Asymptotic Notations in Equations

• On the right-hand side (n2) stands for some anonymous function in (n2)2n2 + 3n + 1 = 2n2 + (n) means:

There exists a function f(n) (n) such that

2n2 + 3n + 1 = 2n2 + f(n)

• On the left-hand side2n2 + (n) = (n2)No matter how the anonymous function is chosen on

the left-hand side, there is a way to choose the anonymous function on the right-hand side to make the equation valid.

5/11/2004 Lecture 2 COSC3101A 17

Limits and Comparisons of Functions

Using limits for comparing orders of growth:

• compare ½ n (n-1) and n2

))(()(:)(thangrowthoforderlargerahas)(,

))(()(:)(asgrowthofordersamethehas)(,

))(()(:)(thangrowthofordersmallerahas)(,0

)(

)(lim

ngntngnt

ngntngntc

ngontngnt

ng

ntn

2

111lim

2

1lim

2

1)1(21

lim2

2

2

nn

nn

n

nn

nnn

5/11/2004 Lecture 2 COSC3101A 18

Limits and Comparisons of Functions

L’Hopital rule:

)('

)('lim

)(

)(lim

ng

nt

ng

ntnn

n• compare and

n

nn

lglim

n

nn

lglim

n

en

n

2

1

lg1

limn

ne

n limlg2 0

nlg

5/11/2004 Lecture 2 COSC3101A 19

Loop Invariant

• A loop invariant is a relation among program variables that – is true when control enters a loop, – remains true each time the program executes the

body of the loop, – and is still true when control exits the loop.

• Understanding loop invariants can help us – analyze algorithms, – check for errors, – and derive algorithms from specifications.

5/11/2004 Lecture 2 COSC3101A 20

Proving Loop Invariants

• Initialization (base case): – It is true prior to the first iteration of the loop

• Maintenance (inductive step): – If it is true before an iteration of the loop, it remains true before

the next iteration

• Termination: – When the loop terminates, the invariant - usually along with the

reason that the loop terminated - gives us a useful property that

helps show that the algorithm is correct

– Stop the induction when the loop terminates

• Proving loop invariants works like induction

5/11/2004 Lecture 2 COSC3101A 21

Loop Invariant for Insertion Sort(1)

Alg.: INSERTION-SORT(A)

for j ← 2 to n

do key ← A[ j ]

Insert A[ j ] into the sorted sequence A[1 . . j -1]

i ← j - 1

while i > 0 and A[i] > keydo A[i + 1] ← A[i] i ← i – 1

A[i + 1] ← key

a8a7a6a5a4a3a2a1

1 2 3 4 5 6 7 8

key

Invariant: at the start of the for loop the elements in A[1 . . j-1] are in sorted order

5/11/2004 Lecture 2 COSC3101A 22


• Initialization:

– Just before the first iteration, j = 2:

the subarray A[1 . . j-1] = A[1],

(the element originally in A[1]) – is

sorted

5/11/2004 Lecture 2 COSC3101A 23


• Maintenance: – the while inner loop moves A[j -1], A[j -2], A[j -3],

and so on, by one position to the right until the proper position for key (which has the value that started out in A[j]) is found

– At that point, the value of key is placed into this position.

5/11/2004 Lecture 2 COSC3101A 24


• Termination: – The outer for loop ends when j > n (i.e, j = n + 1)

j-1 = n– Replace n with j-1 in the loop invariant:

• the subarray A[1 . . n] consists of the elements originally in A[1 . . n], but in sorted order

• The entire array is sorted!

jj - 1

5/11/2004 Lecture 2 COSC3101A 25

Steps in Designing Algorithms(1)

1. Understand the problem

• Specify the range of inputs the algorithm should handle

2. Learn about the model of the implementation technology

• RAM (Random-access machine), sequential execution

3. Choosing between an exact and an approximate solution

• Some problems cannot be solved exactly: nonlinear equations,

evaluating definite integrals

• Exact solutions may be unacceptably slow

4. Choose the appropriate data structures

5/11/2004 Lecture 2 COSC3101A 26


5. Choose an algorithm design technique

• General approach to solving problems algorithmically that is

applicable to a variety of computational problems

• Provide guidance for developing solutions to new problems

6. Specify the algorithm

• Pseudocode: mixture of natural and programming language

7. Prove the algorithm’s correctness

• Algorithm yields the correct result for any legitimate input, in a

finite amount of time

• Mathematical induction, loop-invariants

5/11/2004 Lecture 2 COSC3101A 27


8. Analyze the Algorithm

• Predicting the amount of resources required:

• memory: how much space is needed?

• computational time: how fast the algorithm runs?

• FACT: running time grows with the size of the input

• Input size (number of elements in the input)

– Size of an array, polynomial degree, # of elements in a matrix, # of bits in the binary

representation of the input, vertices and edges in a graph

Def: Running time = the number of primitive operations (steps)

executed before termination

– Arithmetic operations (+, -, *), data movement, control, decision making ( if, while),

comparison

5/11/2004 Lecture 2 COSC3101A 28


9. Coding the algorithm

• Verify the ranges of the input

• Efficient/inefficient implementation

• It is hard to prove the correctness of a program (typically done

by testing)

5/11/2004 Lecture 2 COSC3101A 29

Classification of Algorithms

• By problem types

– Sorting

– Searching

– String processing

– Graph problems

– Combinatorial problems

– Geometric problems

– Numerical problems

• By design paradigms

– Divide-and-conquer

– Incremental

– Dynamic programming

– Greedy algorithms

– Randomized/probabilistic

5/11/2004 Lecture 2 COSC3101A 30

Divide-and-Conquer

• Divide the problem into a number of subproblems

– Similar sub-problems of smaller size

• Conquer the sub-problems

– Solve the sub-problems recursively

– Sub-problem size small enough solve the problems in

straightforward manner

• Combine the solutions to the sub-problems

– Obtain the solution for the original problem

5/11/2004 Lecture 2 COSC3101A 31

Merge Sort Approach

• To sort an array A[p . . r]:

• Divide– Divide the n-element sequence to be sorted into two

subsequences of n/2 elements each

• Conquer

– Sort the subsequences recursively using merge sort

– When the size of the sequences is 1 there is nothing

more to do

• Combine

– Merge the two sorted subsequences

5/11/2004 Lecture 2 COSC3101A 32

Merge Sort

Alg.: MERGE-SORT(A, p, r)

if p < r Check for base case

then q ← (p + r)/2 Divide

MERGE-SORT(A, p, q) Conquer

MERGE-SORT(A, q + 1, r) Conquer

MERGE(A, p, q, r) Combine

• Initial call: MERGE-SORT(A, 1, n)

1 2 3 4 5 6 7 8

62317425

p rq

5/11/2004 Lecture 2 COSC3101A 33

Example – n Power of 2

1 2 3 4 5 6 7 8

q = 462317425

1 2 3 4

7425

5 6 7 8

6231

1 2

25

3 4

74

5 6

31

7 8

62

1

5

2

2

3

4

4

7 1

6

3

7

2

8

6

5

Example

5/11/2004 Lecture 2 COSC3101A 34

Example – n Power of 2

1

5

2

2

3

4

4

7 1

6

3

7

2

8

6

5

1 2 3 4 5 6 7 8

76543221

1 2 3 4

7542

5 6 7 8

6321

1 2

52

3 4

74

5 6

31

7 8

62

5/11/2004 Lecture 2 COSC3101A 35

Example – n Not a Power of 2

62537416274

1 2 3 4 5 6 7 8 9 10 11

q = 6

416274

1 2 3 4 5 6

62537

7 8 9 10 11

q = 9q = 3

274

1 2 3

416

4 5 6

537

7 8 9

62

10 11

74

1 2

2

3

16

4 5

4

6

37

7 8

5

9

2

10

6

11

4

1

7

2

6

4

1

5

7

7

3

8

5/11/2004 Lecture 2 COSC3101A 36

Example – n Not a Power of 2

77665443221

1 2 3 4 5 6 7 8 9 10 11

764421

1 2 3 4 5 6

76532

7 8 9 10 11

742

1 2 3

641

4 5 6

753

7 8 9

62

10 11

2

3

4

6

5

9

2

10

6

11

4

1

7

2

6

4

1

5

7

7

3

8

74

1 2

61

4 5

73

7 8

5/11/2004 Lecture 2 COSC3101A 37

Merging

• Input: Array A and indices p, q, r such that p ≤ q < r– Subarrays A[p . . q] and A[q + 1 . . r] are sorted

• Output: One single sorted subarray A[p . . r]

1 2 3 4 5 6 7 8

63217542

p rq

5/11/2004 Lecture 2 COSC3101A 38

Merging

• Idea for merging:

– Two piles of sorted cards

• Choose the smaller of the two top cards

• Remove it and place it in the output pile

– Repeat the process until one pile is empty

– Take the remaining input pile and place it face-down

onto the output pile

5/11/2004 Lecture 2 COSC3101A 39

Merge - Pseudocode

Alg.: MERGE(A, p, q, r)1. Compute n1 and n2

2. Copy the first n1 elements into L[1 . . n1 + 1] and the next n2 elements into R[1 . . n2 + 1]

3. L[n1 + 1] ← ; R[n2 + 1] ←

4. i ← 1; j ← 15. for k ← p to r6. do if L[ i ] ≤ R[ j ]7. then A[k] ← L[ i ]8. i ←i + 19. else A[k] ← R[ j ]10. j ← j + 1

p q

7542

6321rq + 1

L

R

1 2 3 4 5 6 7 8

63217542

p rq

n1 n2

5/11/2004 Lecture 2 COSC3101A 40

Example: MERGE(A, 9, 12, 16)p rq

5/11/2004 Lecture 2 COSC3101A 41

Example: MERGE(A, 9, 12, 16)

5/11/2004 Lecture 2 COSC3101A 42

Example (cont.)

5/11/2004 Lecture 2 COSC3101A 43

Example (cont.)

5/11/2004 Lecture 2 COSC3101A 44

Example (cont.)

Done!

5/11/2004 Lecture 2 COSC3101A 45

Running Time of Merge

• Initialization (copying into temporary arrays):

(n1 + n2) = (n)

• Adding the elements to the final array (the last

for loop):

– n iterations, each taking constant time (n)

• Total time for Merge: (n)

5/11/2004 Lecture 2 COSC3101A 46

Analyzing Divide-and Conquer Algorithms

• The recurrence is based on the three steps of the paradigm:– T(n) – running time on a problem of size n– Divide the problem into a subproblems, each of size

n/b: takes D(n)– Conquer (solve) the subproblems aT(n/b) – Combine the solutions C(n)

(1) if n ≤ c

T(n) = aT(n/b) + D(n) + C(n)otherwise

5/11/2004 Lecture 2 COSC3101A 47

MERGE-SORT Running Time

• Divide: – compute q as the average of p and r: D(n) = (1)

• Conquer: – recursively solve 2 subproblems, each of size n/2

2T (n/2)

• Combine: – MERGE on an n-element subarray takes (n) time

C(n) = (n)

(1) if n =1

T(n) = 2T(n/2) + (n) if n > 1

5/11/2004 Lecture 2 COSC3101A 48

Correctness of Merge Sort

• Loop invariant

(at the start of the for loop)

– A[p…k-1] contains the k-p smallest elements

of L[1 . . n1 + 1] and R[1 . . n2 + 1] in

sorted order

– L[i] and R[j] are the smallest elements not yet

copied back to A

p r

5/11/2004 Lecture 2 COSC3101A 49

Proof of the Loop Invariant

• Initialization

– Prior to first iteration: k = p

subarray A[p..k-1] is empty

– A[p..k-1] contains the k – p = 0 smallest elements

of L and R

– L and R are sorted arrays (i = j = 1)

L[1] and R[1] are the smallest elements in L and R

5/11/2004 Lecture 2 COSC3101A 50


• Maintenance

– Assume L[i] ≤ R[j] L[i] is the smallest element not

yet copied to A

– After copying L[i] into A[k], A[p..k] contains the k – p + 1 smallest elements of L and R

– Incrementing k (for loop) and i reestablishes the loop

invariant

5/11/2004 Lecture 2 COSC3101A 51


• Termination

– At termination k = r + 1

– By the loop invariant: A[p..k-1] = A[p…r] contains

the k – p = r – p + 1 smallest elements of L and R in

sorted order

– Exactly the number of elements to be sorted

MERGE(A, p, q, r) is correctk = r + 1

5/11/2004 Lecture 2 COSC3101A 52

Readings

• Chapters 2.2, 3• Appendix A

cosc 3101a - design and analysis of algorithms 2 asymptotic notations continued proof of...

Documents