11

Computer AlgorithmsLecture 6

Recurrence

Ch. 4 (till Master Theorem)

Some of these slides are courtesy of D. Plaisted et al, UNC and M. Nicolescu, UNR

Merge SortSorting Problem: Sort a sequence of n elements into non-decreasing order.

• Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each

• Conquer: Sort the two subsequences recursively using merge sort.

• Combine: Merge the two sorted subsequences to produce the sorted answer.

2

Analysis of Merge Sort

• Running time T(n) of Merge Sort:• Divide: computing the middle takes (1) • Conquer: solving 2 subproblems takes 2T(n/2) • Combine: merging n elements takes (n) • Total:

– T(n) = (1) if n = 1– T(n) = 2T(n/2) + (n) if n > 1

T(n) = (n lg n)

3

4

Recurrences

• Running time of algorithms with recursive calls can be described using recurrences

• A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs.

• For divide-and-conquer algorithms:

• Example: Merge Sort

(1) if 1( )

2 ( / 2) ( ) if 1

nT n

T n n n

solving_trivial_problem if 1( )

num_pieces ( / subproblem_size_factor) dividing combining if 1

nT n

T n n

4

5

Recurrence Relations• Equation or an inequality that characterizes a function by its

values on smaller inputs.

• Solution Methods (Chapter 4)– Iteration Method– Substitution Method.– Recursion-tree Method.– Master Method.

• Recurrence relations arise when we analyze the running time of iterative or recursive algorithms.– Ex: Divide and Conquer.

T(n) = (1) if n cT(n) = a T(n/b) + D(n) + C(n) otherwise

5

6

Iteration Method

T(n) = c + T(n/2)

T(n) = c + T(n/2)

= c + c + T(n/4)

= c + c + c + T(n/8)

Assume n = 2k

T(n) = c + c + … + c + T(1)

= clgn + T(1)

= Θ(lgn)

k times

T(n/2) = c + T(n/4)

T(n/4) = c + T(n/8)

77

Iteration Method – Example

T(n) = n + 2T(n/2)

T(n) = n + 2T(n/2)

= n + 2(n/2 + 2T(n/4))

= n + n + 4T(n/4)

= n + n + 4(n/4 + 2T(n/8))

= n + n + n + 8T(n/8)

… = in + 2iT(n/2i)

= kn + 2kT(1)

= nlgn + nT(1) = Θ(nlgn)

Assume: n = 2k

T(n/2) = n/2 + 2T(n/4)

88

Iteration Method – Example

T(n) = n + T(n-1)

T(n) = n + T(n-1)

= n + (n-1) + T(n-2)

= n + (n-1) + (n-2) + T(n-3)

… = n + (n-1) + (n-2) + … + 2 + T(1)

= n(n-1)/2 - 1 + T(1)

= n2/2 -n/2 -1 + T(1) = Θ(n2)

9

Substitution method

• Two steps– Guess the form of the solution– Use mathematical induction to find the constants and show the

solution works• Useful when it is easy to guess the form of the answer• Can be used to establish either upper or lower bound on a recurrence• Example:

– determine upper bound for – guess: T(n) = O(n lg n) – prove: T(n) cn lg n for some c

nnTnT )2/(2)(

9

10

Substitution method

• Guess a solution– T(n) = O(g(n))– Induction goal: apply the definition of the asymptotic notation

• T(n) ≤ d g(n), for some d > 0 and n ≥ n0

– Induction hypothesis: T(k) ≤ d g(k) for all k < n

• Prove the induction goal– Use the induction hypothesis to find some values of the constants d

and n0 for which the induction goal holds

11

Example – Exact Function

Recurrence: T(n) = 1 if n = 1

T(n) = 2T(n/2) + n if n > 1

Guess: T(n) = n lg n + n. Induction:

• Basis: n = 1 n lgn + n = 1 = T(n).• Hypothesis: n = k/2: T(k/2) = (k/2) lg (k/2) + (k/2)• Inductive Step: n = k: T(k) = 2 T(k /2) + k

= 2 ((k /2)lg(k /2) + (k /2)) + k

= k (lg(k /2)) + 2 k

= k lg k – k + 2 k

= k lg k + k

12

ExampleT(n) = T(n-1) + n

• Guess: T(n) = O(n2)

– Induction goal: T(n) ≤ c n2, for some c and n ≥ n0

– Induction hypothesis: T(n-1) ≤ c(n-1)2

• Proof of induction goal:

T(n) = T(n-1) + n ≤ c (n-1)2 + n

= cn2 – (2cn – c - n) ≤ cn2

if: 2cn – c – n ≥ 0 c ≥ n/(2n-1) c ≥ 1/(2 – 1/n)

– For n ≥ 1 2 – 1/n ≥ 1 any c ≥ 1 will work

13

Substitution Method. Summary

• Guess the form of the solution, then use mathematical induction to show it correct.

– Substitute guessed answer for the function when the inductive hypothesis is applied to smaller values – hence, the name.

• Works well when the solution is easy to guess.

• No general way to guess the correct solution.

14

Making a good guess

• Need experience and creativity• Use recursion trees to generate good guesses• If a recurrence is similar to one you are familiar, then guessing a

similar solution is reasonable– T(n) = O(n lg n) Why?– The additional term (17) cannot substantially affect the solution to

the recurrence (when n is large)

nnTnT )172/(2)(

14

15

Recursion-tree Method

• Making a good guess is sometimes difficult with the substitution method.

• Use recursion trees to devise good guesses.• Convert the recurrence into a tree:

– Each node represents the cost incurred at that level of recursion– Sum up the costs of all levels

• Recursion Trees– Show successive expansions of recurrences using trees.– Keep track of the time spent on the subproblems of a divide and conquer

algorithm.– Help organize the algebraic bookkeeping necessary to solve a recurrence.

16

Example

• Running time of Merge Sort:T(n) = (1) if n = 1

T(n) = 2T(n/2) + (n) if n > 1

• Rewrite the recurrence asT(n) = c if n = 1

T(n) = 2T(n/2) + cn if n > 1

c > 0: Running time for the base case and

time per array element for the divide and

combine steps.

17

Recursion Tree for Merge Sort

For the original problem, we have a cost of cn, plus two subproblems each of size (n/2) and running time T(n/2).

cn

T(n/2) T(n/2)

Each of the size n/2 problems has a cost of cn/2 plus two subproblems, each costing T(n/4).

cn

cn/2 cn/2

T(n/4) T(n/4) T(n/4) T(n/4)

Cost of divide and merge.

Cost of sorting subproblems.

18

Recursion Tree for Merge SortContinue expanding until the problem size reduces to 1.

cn

cn/2 cn/2

cn/4 cn/4 cn/4 cn/4

c c c cc c

lg n

cn

cn

cn

cn

Total : cnlgn+cn

19

Recursion Tree for Merge SortContinue expanding until the problem size reduces to 1.

cn

cn/2 cn/2

cn/4 cn/4 cn/4 cn/4

c c c cc c

• Each level has total cost cn.

• Each time we go down one level, the number of subproblems doubles, but the cost per subproblem halves cost per level remains the same.

• There are lg n + 1 levels, height is lg n. (Assuming n is a power of 2.)

• Can be proved by induction.

• Total cost = sum of costs at each level = (lg n + 1)cn = cnlgn + cn = (n lgn).

20

T(n) = 2T(n/2) + n2

• Subproblem size at level i is: n/2i

• Subproblem size hits 1 when 1 = n/2i i = lgn• Cost of the problem at level i = (n/2i)2 No. of nodes at level i = 2i • Total cost:

T(n) = O(n2)

22

0

21lg

0

2lg1lg

0

2

2)(

211

1)(

2

1

2

1)1(2

2)( nnOnnOnnnT

nnT

i

in

i

in

n

ii

T(1) T(1) T(1)T(1) T(1)

T(n/4) T(n/4) T(n/4) T(n/4)

T(n/2)T(n/2)

T(n/2)=2(T/4)+(n/2)2

T(n/4)=2(T/8)+(n/4)2

Recursion trees?

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

21

Example of recursion tree

T(n)

Solve T(n) = T(n/4) + T(n/2) + n2:

22

Example of recursion tree

T(n/4) T(n/2)

n2

Solve T(n) = T(n/4) + T(n/2) + n2:

23

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

n2

(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4)

24

Example of recursion tree

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2 (n/2)2

Q(1)

…

Solve T(n) = T(n/4) + T(n/2) + n2:

n2

25

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2 (n/2)2

Q(1)

…

2nn2

26

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2 (n/2)2

Q(1)

…

2165 n

2nn2

27

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2

Q(1)

…

2165 n

2n

225625 n

n2

(n/2)2

…

28

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2

Q(1)

…

2165 n

2n

225625 n

13

1652

165

1652 n

…

Total =

= Q(n2)

n2

(n/2)2

geometric series

29

Appendix: geometric series

1

11 2

xxx

for |x| < 1

1

11

12

xx

xxxn

n

for x ¹ 1

Return to last slide viewed.

30

31

T(n) = 3T(n/4) + cn2

Subproblem size at level i is: n/4i

Subproblem size hits 1 when 1 = n/4i i = log4n

Cost of a node at level i = c(n/4i)2

Number of nodes at level i = 3i last level has 3log4

n = nlog4

3 nodes

Total cost:

T(n) = O(n2)

)(

163

1

1

16

3

16

3)( 23log23log2

0

3log21log

0

444

4

nOnOcnnOcnnOcnnTi

iin

i

32

T(n) = T(n/3) + T(2n/3) + n

The longest path from the root to

a leaf is:

n (2/3)n (2/3)2 n … 1

Subproblem size hits 1 when

1 = (2/3)in i=log3/2n

Cost of the problem at level i = n

Total cost:

T(n) = O(nlgn)

nnn

nnnnnnnnTn

i

n

i

lg2/3lg

1

2/3lg

lglog1...)(

2/32/3 log

02/3

log

0

for all levels

T(1)T(1)

T(1)T(1)

33

Other Examples

• Use the recursion-tree method to determine a guess for the recurrences

– T(n) = 3T(n/4) + (n2).– T(n) = T(n/3) + T(2n/3) + O(n).

34

Recursion Trees – Caution Note

• Recursion trees only generate guesses.– Verify guesses using substitution method.

• A small amount of “sloppiness” can be tolerated. Why?• If careful when drawing out a recursion tree and summing the costs,

can be used as direct proof.

35

Recursion-Tree Method. Summary• Recursion tree

– Each node represents the cost of a single subproblem somewhere in the set of recursive function invocations

– Each level denotes iteration– Sum the costs within each level of the tree to obtain a set of per-level costs– Sum all the per-level costs to determine the total cost of all levels of the

recursion• Useful when the recurrence describes the running time of a divide-and-

conquer algorithm• Useful for generating a good guess, which is then verified by the

substitution method

Use the recursion-tree method to “guess” the solution (rather than for proving)

35

3636

Example Recurrences

• T(n) = T(n-1) + n Θ(n2)– Recursive algorithm that loops through the input to eliminate one item

• T(n) = T(n/2) + c Θ(lgn)– Recursive algorithm that halves the input in one step

• T(n) = T(n/2) + n Θ(n)– Recursive algorithm that halves the input but must examine every item

in the input

• T(n) = 2T(n/2) + 1 Θ(n)– Recursive algorithm that splits the input into 2 halves and does a

constant amount of other work

37

• Three methods for solving recurrences (obtaining asymptotic Θ or O bounds on the solution)– Substitution method– Recursion-tree method– Master method

• Notes – some technical details are neglected– Assumption of integer arguments to functions: T(n)– Boundary condition for small n T(n) is constant for small n– Floors, ceilings

• Mathematical induction is used to prove our solution for recurrence– Showing base case holds– Assume the nth step is true– Prove the (n+1)th step by the result before the nth step

37