amortized analysis and master method - deepak garg · when amortized cost > actual cost, store...

Amortized Analysis and Master Method

Deliverables

Copyright @ gdeepak.com 2 6/5/2012 7:15 PM

Amortized Analysis

Aggregate Analysis

Accounting Method

Potential Analysis

Solving Recurrence Relations

Master Method

Amortized Analysis

Analyze a sequence of operations on a data structure.

Goal is to show that although some individual operations may be expensive, on average the cost per operation is small.

Such analysis is not immediately obvious and needs deeper knowledge of problem parameters and complexity bounds.

No probability is involved. It is average cost in the worst case. Aggregate analysis, Accounting method, and Potential method


Aggregate Analysis

It requires knowledge of which series of operations are possible. It is common case with data structures, which have state that persists between operations.

Idea is that a worst case operation can alter the state in a way that worst case cannot occur for a long time, thus amortizing its cost. E.g. doubling an array will be an operation for a heavy cost.

Aggregate analysis determines the upper bound T(n) on the total cost of a sequence of n operations, then calculates average cost to be T(n)/n


Aggregate Analysis-Example

k-bit binary counter A[0 . . k − 1] of bits, where A[0] is the least significant bit and A[k − 1] is the most significant bit. Counts upward from 0.

Value of counter is

Initially, counter value is 0, so A[0 . . k − 1] = 0.

Copyright @ gdeepak.com 5

1

[ ].2k

i

i o

A i

6/5/2012 7:15 PM

Binary counter algorithm

Increment(A, k)

i ← 0

while (i < k and A[i ] = 1)

do A[i ] ← 0

i ←i + 1

if i < k

then A[i ] ← 1


Binary Counter example k = 3


0

1

2

3

4

5

6

7

…

0 0 0

0 0 1

0 1 0

0 1 1

1 0 0

1 0 1

1 1 0

1 1 1

…

Counter Value

0

1

3

4

7

8

10

11

…

cost

6/5/2012 7:15 PM

Binary counter- analysis

Cost of Increment = # of bits flipped

Each call could flip k bits, so n Increments takes O(nk) time.

Observation is that not every bit flips every time.

bit flips how often times in n Increments

0 every time n

1 1/2 the time n/2

2 1/4 the time n/4

...

i 1/2i the time n/2i

i ≥ k never 0


Binary counter-Aggregate analysis

Therefore, total # of flips =

< n =

= 2n .

Average cost per operation = O(1)


1

2ii o

1

2

n

ii o

n

1

12

n

6/5/2012 7:15 PM

Stack Operations example

• Stack operations

• Push(S, x): O(1) ⇒ O(n) for any sequence of n operations.

• Pop(S): O(1) ⇒ O(n) for any sequence of n operations

• Multipop(S, k)

while S is not empty and k > 0

do Pop(S)

k ← k − 1


Stack Complexity for Multipop

Running time of Multipop is linear in #of Pop operations

# of iterations of while loop is min(s, k), where s = # of objects on stack therefore, total cost = min(s, k).

Worst-case cost of Multipop is O(n)

For Sequence of n Push, Pop, Multipop operations worst-case cost of sequence is O(n2).


Aggregate Analysis for stacks

Each object can be popped only once per time that it is pushed.

Have ≤ n Pushes ⇒≤n Pops, including those in Multipop.

Therefore, total cost = O(n) and average over n operations ⇒ O(1) per operation

No probability is involved and shows worst-case O(n) cost for sequence


Accounting method

A set of elementary operations are chosen to be used in the algorithm and their cost is set to 1.

Actually, cost differs because a prepayment may be necessary to complete an operation with some payment left over to be used later from savings.

Assign different charges to different operations. Some are charged more than actual cost and some are charged less.

Amortized cost = amount we charge


Accounting Method


When amortized cost > actual cost, store the difference on specific objects in the data structure as credit.

Use credit later to pay for operations whose actual cost > amortized cost.

Differs from aggregate analysis because different operations can have different costs while in aggregate analysis, all operations have same cost.

Need credit to never go negative

6/5/2012 7:15 PM

Stack

• Intuition: When pushing an object, pay `2.

• ` 1 pays for the PUSH.

• ` 1 is prepayment for it being popped by either POP or Multipop.

• Each object has ` 1, which is credit, credit can never go negative

• Total amortized cost, = O(n), is an upper bound on total actual cost.


operation Actual cost Amortized Cost

PUSH 1 2

Pop 1 0

Multipop min(k, s) 0

6/5/2012 7:15 PM

Accounting Method

Let ci = actual cost of ith operation, = amortized cost of ith operation. Then require ≥ for all sequences of n operations. Total credit stored = - ≥ 0


ic

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

6/5/2012 7:15 PM

Binary Counter-Example

• Charge `2 to set a bit to 1. `1 pays for setting a bit to 1. • `1 is prepayment for flipping it back to 0. • Have ` 1 of credit for every 1 in the counter. • Therefore, credit ≥ 0. • Amortized cost of increment: Cost of resetting bits to 0 is

paid by credit. • At most 1 bit is set to 1. Therefore, amortized cost ≤ ` 2.

For n operations, amortized cost = O(n).


Potential Method

• Like the accounting method, but think of credit as potential stored with entire data structure.

• Accounting method stores credit with specific objects. • Potential method stores potential in the data structure as

a whole. • Can release potential to pay for future operations. • Most flexible of the amortized analysis methods. • Let Di = data structure after ith operation ,


Potential Method

• D0 = initial data structure

• ci = actual cost of ith operation ,

• = amortized cost of ith operation .

• Potential function Φ : Di → R

• Φ (Di) is the potential associated with data structure Di.

• = ci + Φ (Di ) − Φ (Di−1)

• Increase in potential due to ith operation


ic

ic

6/5/2012 7:15 PM

Potential Method

Total amortized cost = + ΦDi –ΦDi-1 telescoping: every term except D0 and Dn is added and subtracted = + ΦDn –ΦD0

≥

If we require that Φ (Di ) ≥Φ (D0) for all i , then amortized cost is an upper bound on actual cost Φ (D0) = 0, Φ (Di ) ≥ 0


1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

6/5/2012 7:15 PM

Stack Example

• Φ = # of objects in stack

• D0 = empty stack ⇒ Φ(D0) = 0.

• Since # of objects in stack is always ≥ 0, Φ(Di ) ≥ 0 = Φ(D0) for all i.

• Therefore, amortized cost of a sequence of n operations = O(n).


Operation Actual cost Φ s=#of objects initially

Amortized Cost

Push 1 (s + 1) − s =1 1+ 1 = 2

Pop 1 (s − 1) − s = −1 1− 1 = 0

Multipop k’=min(k, s) (s−k’) − s = −k’ k’ − k’ = 0

6/5/2012 7:15 PM

Binary Counter example k = 3


0

1

2

3

4

5

6

7

…

0 0 0

0 0 1

0 1 0

0 1 1

1 0 0

1 0 1

1 1 0

1 1 1

…

Counter Value

0

1

3

4

7

8

10

11

…

cost

6/5/2012 7:15 PM

Potential Method


Potential Method


ic

6/5/2012 7:15 PM

Solving Recurrence Equations for Complexity – 3 methods

Guess the solution and prove

Using Recursion Tree Method

Master Method


Guess the solution and prove

1. Guess the form of the solution

2. Verify by induction and show that the solution works for a set of constants

T(n)=2T(n/2) + n Which means that now the problem has been divided into two sub problems and the size of the sub problem is n/2


Guess and Substitute by Example

We guess that it works for T(n) = O(n lg n)

To prove that T(n) ≤ cn lgn for an appropriate c.

Assuming that the guess works we will have

T(n) = 2T(n/2)+ n

= n lg(n/2) +n + n

= n lg n – n lg 2 + n +n

= n lg n + n ≤ cnlg n

this is true for a specific value of c


2( lg )2 2 2

n n nn

6/5/2012 7:15 PM

Guess by reasoning and experience

Looks similar to binary search, try O(logn)

Assume T(k) = O(log2 k) for all k < n

Show that T(n) = O(log2 n)

We need to prove T(n/2) ≤ c log2(n/2)


dnTnT )2/()(

6/5/2012 7:15 PM

Guess by reasoning and experience


dnTnT )2/()(

dnc )2/(log2

dcnc 2loglog 22

nc 2log

if c ≥ d

Residual dcnc 2log

6/5/2012 7:15 PM

Recursion Tree

Guessing may be difficult for recurrences which does not resemble any common recurrence

Need to check the cost of tree at each level of recursion and sum costs of all the levels. For this information of depth of the tree, no of leaves of the tree is required

In the end answer can be verified by substitution using the above as a guess.


2)4/(3)( nnTnT

cnnTnTnT )3/2(2)3/()(

6/5/2012 7:15 PM

Recursion Tree (Level 1)


2)4/(3)( nnTnT

cn2

T(n/4 ) T(n/4 ) T(n/4 )

6/5/2012 7:15 PM

Recursion Tree (Level 2)


2)4/(3)( nnTnT

cn2 cn2

cost

2

4

nc

2

4

nc

2

4

nc

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

3/16cn2

6/5/2012 7:15 PM


2)4/(3)( nnTnT

cn2 cn2

cost

2

4

nc

2

4

nc

2

4

nc

2

16

nc

3/16cn2

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc (3/16)2cn2

What is the cost at each level? 2

16

3cn

d

6/5/2012 7:15 PM

Important to grasp

• Data is divided into 4 parts at each level


14

d

n

04

log

d

n

04loglog dn

nd log4log

nd 4log

6/5/2012 7:15 PM

Important to grasp

No of leaves in the tree with three children of every node at each level


nd 4log33

6/5/2012 7:15 PM


)3(16

3...

16

3

16

3)( 4log2

1

2

2

22 n

d

cncncncnnT

)3(16

34

4log

1log

0

2 nn

i

i

cn

xx

k

k

1

10

let x = 3/16

)3(16

34log

0

2 n

i

i

cn

)3()16/3(1

14log2 n

cn

)3(13

164log2 n

cn )(13

16)(

3log2 4ncnnT

)()( 2nOnT

6/5/2012 7:15 PM

Proving after Guess through recursion tree

Now we have reasonable guess with the help of recursion tree , let us prove it by guessing

Assume T(k) = O(k2) for all k < n Show that T(n) = O(n2)

Given that T(n/4) = O((n/4)2), then T(n/4) ≤ c(n/4)2


2)4/(3)( nnTnT 22)4/(3 nnc

22 16/3 ncn 2cn

13

16c

6/5/2012 7:15 PM

Recurrence equations-Master Method

• Where a ≥ 1 and b>1 are constants and f(n) is an asymptotically positive function

• We are dividing a problem of size n into a sub problems, each of size n/b, where a and b are positive constants. a sub problems are solved recursively each in time T(n/b) where a and b are positive constants. Cost of dividing the problem and combining the results of the sub problems is described by the function f(n) which can be written as D(n)+C(n).


dnnfbnaT

dncnT

if)()/(

if )(

6/5/2012 7:15 PM

Three cases of Master Method

Compare f (n) with nlogb

a:

1. f (n) = O(nlogba – ε) for some constant ε > 0.

f (n) grows polynomially slower than nlogba (by an nε factor).

Solution: T(n) = Θ(nlogba) .

2. f (n) = Θ(nlogba lgkn) for some constant k ≥ 0.

f (n) and nlogba grow at similar rates.

Solution: T(n) = Θ(nlogba lgk+1n)


Three Cases of Master Method

Compare f (n) with nlogba:

3. f (n) = Ω(nlogba + ε) for some constant ε > 0.

f (n) grows polynomially faster than nlogba (by an nε factor),

and f (n) satisfies the regularity condition that

a f (n/b) ≤ c f (n) for some constant c < 1.

Solution: T(n) = Θ( f (n))


Master Theorem

There is a gap between cases 1 and 2 when f (n) is smaller than but not polynomially smaller. Similarly, there is a gap between cases 2 and 3 when f (n) is larger but not polynomially larger. If the function f (n) falls into these gaps, or if regularity condition in case 3 fails to hold, master method cannot be used.


.1 somefor )()/( provided

)),((is)(then),(is)(if 3.

)log(is)(then),log(is)(if 2.

)(is)(then),(is)(if 1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

dnnfbnaT

dncnT

if)()/(

if )(

6/5/2012 7:15 PM

Easy Way


( ) ( ) knT n aT n

b

blog ak

k k

k k

if a > b then T(n) = O(n ) (Bottom Heavy)

if a b then T(n) = O(n log(n)) (Steady State)

if a < b then T(n) = O(n ) (Top Heavy)

6/5/2012 7:15 PM

T(n)=4T(n/2)+n

a=4 = n2

b=2

f(n)=n

We are in case 1 so T(n) is (n2)


abnlog





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

2

2

2

is f( ) ( )?

is f( ) ( log )?

is f( ) ( )?

k

n O n

n n n

n n

6/5/2012 7:15 PM

T(n)=2T(n/2)+nlogn

a=2 = n

b=2

f(n)=nlogn

We are in case 2 and k=1 so T(n) is (n log2 n)


abnlog





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

1

1

1

is f( ) ( )?

is f( ) ( log )?

is f( ) ( )?

k

n O n

n n n

n n

6/5/2012 7:15 PM

T(n)=4T(n/2)+n3

a=4 = 2

b=2

f(n)=nlogn

We are in case 3 says T(n) is (n3).

For n=1/2 4*(1/2)3/2 ≤ δ (1/2)3 ¼ ≤ δ (1/8) for δ = 1/2






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

abnlog 2

2

2

is f( ) ( )?

is f( ) ( log )?

is f( ) ( )?

k

n O n

n n n

n n

6/5/2012 7:15 PM

T(n)=8T(n/2)+n2

Solution: logba=3, so case 1 says T(n) is (n3).






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

dnnfbnaT

dncnT

if)()/(

if )(

6/5/2012 7:15 PM

T(n)=9T(n/3)+n3

Solution: logba=2, so case 3 says T(n) is (n3)






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

dnnfbnaT

dncnT

if)()/(

if )(

6/5/2012 7:15 PM

Master Theorem

T(n)=T(n/2)+1 (binary search)

Solution: logba=0, so case 2 says T(n) is (log n).






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

6/5/2012 7:15 PM

T(n) = 4T(n/2) + n2/lgn

a = 4, b = 2 ⇒ nlogba = n2; f (n) = n2/lgn.

Master method does not apply






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

6/5/2012 7:15 PM


)()/()( nfbnaTnT

)(nf

a

)/( bnf)/( bnf )/( bnf

)/( 2bnf

a

)/( 2bnf )/( 2bnf )/( 2bnf

a

)/( 2bnf )/( 2bnf )/( 2bnf

a

)/( 2bnf )/( 2bnf

)(nf

)/( bnaf

)/( 22 bnfa

6/5/2012 7:15 PM

Depth of the tree

Data is divided by b at each level of the tree


1db

n

0log

db

n

log log 0dn b

nbd loglog

nd blog

6/5/2012 7:15 PM

Proof-Master Method Case 1: The Weight decreases Geometrically from the root to the leaves. The root hold a constant fraction of the total weight. First term is dominating. Case 2: Weight is approximately same on all levels. It is polynomial or slowly decreasing function. Case 3: The Weight increases Geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight. Last term is dominating.


1)(log

0

log

1)(log

0

log

2233

22

2

)/()1(

)/()1(

. . .

)()/()/()/(

)()/()/(

))/())/((

)()/()(

n

i

iia

n

i

iin

b

b

b

b

bnfaTn

bnfaTa

nfbnafbnfabnTa

nfbnafbnTa

bnbnfbnaTa

nfbnaTnT

6/5/2012 7:15 PM


cn3

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

cn

2 c

n

2 c

n

2 c

n

2

cn

( ) 4 ( / 2)T n T n n

Total cn

Total 4cn

2 = 2cn

Total 16cn

4 = 4cn

6/5/2012 7:15 PM


cn3

cn

2

3

cn

8

3 c

n

8

3 c

n

8

3 c

n

8

3

cn

2

3

cn

8

3 c

n

8

3 c

n

8

3 c

n

8

3

cn

2

3

cn

8

3 c

n

8

3 c

n

8

3 c

n

8

3

cn

2

3

cn

8

3 c

n

8

3 c

n

8

3 c

n

8

3

cn3

cn

2

3 c

n

2

3 c

n

2

3 c

n

2

3

cn3

3( ) 4 ( / 2)T n T n n

Total cn3

4cn

2

3 =

1

2 cn3

16cn

8

3 =

1

32 cn3

6/5/2012 7:15 PM


cn

cn

cn

2 c

n

2

cn

cn

2

cn

4 c

n

4

cn

2

cn

4 c

n

4

Total cn

Total cn

Total cn

T(n)=2T(n/2) + n

6/5/2012 7:15 PM


T(n)= T(n/3)+T(2n/3)+n

6/5/2012 7:15 PM

Questions, Comments and Suggestions


Question 1

For recurrence T(n) = T(n-1) + n what will be the big oh complexity.

A) O(n)

B) O(nlogn)

C) O(n2)

D) O(2n)


Question 2

The recurrence relation that arises in relation to the complexity of binary search is

(A) T(n) = T(n/2) + k, k is a constant

(B) T(n) = 2T(n/2) + k, k is a constant

(C) T(n) = T(n/2) + log n

(D) T(n) = T(n/2) + n


Question 3

Which of the following sorting algorithms has the lowest worst-case complexity?

(A) Merge sort

(B) Bubble sort

(C) Quick sort

(D) Selection sort


amortized analysis and master method - deepak garg · when amortized cost > actual cost, store...

Documents