correctness of huffman codes introduction to randomized algorithms quickselect & quicksort...

1

Correctness of Huffman Codes

Introduction To Randomized Algorithms

QuickSelect & QuickSort

Monday, July 14th

2

Outline For Today

1. Correctness of Huffman Codes

2. QuickSelect

3. Probability Review

4. QuickSelect Runtime Analysis

5. QuickSort

3

Outline For Today


2. QuickSelect



5. QuickSort

Goal: Make the binary blob as small as

possible, satisfying the protocol.

Recap: Encoding-Decoding

010010100010010100011110110010010101010110100001110100010011000010010101011010100010

encoder

decoder

Ex: Variable Length Prefix-free Encoding

Ex: A = a, b, c, d

abcd

010110111

110010

decode


Ex: A = a, b, c, d

abcd

010110111

110010

decodec


Ex: A = a, b, c, d

abcd

010110111

110010

decodeca


Ex: A = a, b, c, d

abcd

010110111

110010

decodecab

Recap: Prefix-free Encodings Binary TreesWe can represent each prefix-free code Ɣ as a binary

tree T and vice-versa.

abcd

010110111

Code 1

b

c d

0 1

a0 1

0 1Encoding of letter x = path

from the root to the leaf with

x

# bits for x = depthT(x)

Recap: Formal Problem Statement

Input: An alphabet A, and frequencies 𝓕 of letters in A

Output: A binary tree T, where letters of A are the

leaves of T, that has the minimum average bit length

(ABL):

Recap: Observations About Optimal T

Observation 1: The optimal binary tree T is full, i.e.,

each non-leaf vertex u has exactly 2 children

a

0 1

c

0 1

b

0 1

e0

a

0 1

c

0 1

b

0 1

e

Why?T T`

Recap: Observation 2 About Optimal TClaim: In any optimal tree T if leaf x has depth i, and leaf

y has depth j, s.t i < j => f(x) ≥ f(y)

Exchange Argument: Replace x and y and get a better

tree T`.

Corollary

In any optimal tree T the two lowest

frequency letters are both in the lowest

level of the tree!

Recap: Huffman’s Key Insight

Observation 1 => optimal Ts are full => each leaf has

a sibling

Corollary => 2 lowest freq. letters x, y are at the same

level

Changing letters across the same level does not

change the cost of T

b

c d

0 1

a0 1

0 1

There is an optimal tree T,

in which the two lowest

frequency letters are

siblings (in the lowest level

of the tree).

Possible Greedy Algorithm

Possible greedy algorithm:

1. If x, y are siblings, treat them as a single meta-letter

xy

2. Find an optimal tree T* with A-x, y + xy

3. Expand xy back into x and y in T*

Possible Greedy Algorithm (Example)

xy t

0 1

z0 1

Ex: A = x, y, z, t, and let x, y be the two lowest freq.

letters

Let A` = xy, z, t

t

0 1

z0 1

x y

0 1

T* T

Huffman’s Algorithm (1951)

procedure Huffman(A, 𝓕): if (|A|=2): return T where branch 0, 1 point to A[0] and A[1], respectively

let x, y be lowest two frequency letters let A` = A-x,y+xy let ` = - x, y + xy: f(x) + f(y)𝓕 𝓕 T* = Huffman(A`, `)𝓕 expand x, y in T* to get Treturn T`

Huffman’s Algorithm Correctness (1)

By induction on the |A|

Base case: |A| = 2 => return simple full tree with 2

leaves

IH: Assume true for all alphabets of size k-1

Huffman will get a Tk-1opt with meta-letter xy and

expand xy


xy t

0 1z

0 1t

0 1z

0 1

x y0 1

Tk-1opt T

f(xy)*depth(xy)=(f(x) +

f(y))*depth(xy)

(f(x) + f(y))*(depth(xy) + 1)

Total diff = f(x) + f(y)


Take any optimal Z, we’ll argue ABL(T) ≤ ABL(Z)

By corollary, we can assume in Z, x,y are also siblings

at the lowest level.

Consider Z` by merging xy in Z => Z` is valid prefix-

code for A` of size k-1

21

Correctness

t

0 1

z

0 1

x y0 1

t

0 1

z

0 1

xy

Z Z`

ABL(Z) = ABL(Z`) + f(x) + f(y)

ABL(T) = ABL(T`) + f(x) + f(y)

By IH: ABL(T`) ≤ ABL(T`) => ABL(T) ≤

ABL(Z)

Q.E.D

Total diff is again f(x) + f(y)!

Huffman’s Algorithm Runtime

Exercise: Make Huffman run in O(|A|log(|A|))?

23

Outline For Today


2. QuickSelect



5. QuickSort

24

Given a fixed input, they may give:

1. Different outputs

2. Different runtimes

depending on the outcomes of the coins

Compared to their deterministic counterparts:

often simpler, more practical, elegant

Randomized Algorithms

Randomized AlgorithmInput Outpu

t

flip coins

25

Input: An array A of n integers, and an integer 1 ≤ k ≤

n

Output: Find the rank-k element in A: kth smallest

element

If k = 1 find min

If k = n find max

If k = n/2 find median

Naïve Solution: Sort A, return kth element O(nlog(n))

Problem of Selection

Can we do better? Maybe O(n)?

26

QuickSelect

Given A, k

Pick a pivot p from A uniformly at random,

Partition A into AL: those < p and AR: those > p

If p is the rank-k element return p, i.e. |Al| = k-1

Otherwise recurse on either AL or AR depending on the

sizes of AL and AR .

27

QuickSelect Simulation

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

28

QuickSelect Simulationpivot

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

29


105 7 13 8 14 1 19 11 4 10 98 16

k = 6

30


105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

31


105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

32


7 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

33


7 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

34


7 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

35


7 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

36


7 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

37


7 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

38


7 1 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

39


7 1 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

40


7 1 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

41


7 1 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

42


7 1 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

43


7 1 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

44


7 1 4 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

k = 6

45


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 11 19 14 13 105

k = 6

46


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 10 11 19 14 13 105

k = 6

47


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 10 11 19 14 13 105

k = 6

48


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 98 10 11 19 14 13 105

k = 6

49


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 98 10 11 19 14 13 105

k = 6

50


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 16 98 10 11 19 14 13 105

k = 6

51


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 8 16 98 10 11 19 14 13 105

k = 6

6th smallest element is to the right of 8

4th smallest

52


16 98 10 11 19 14 13 105

k = 2

53


16 98 10 11 19 14 13 105

k = 2

54


16 98 10 11 19 14 13 105

k = 2

16

55


16 98 10 11 19 14 13 105

k = 2

16

56


16 98 10 11 19 14 13 105

k = 2

98 16

57


16 98 10 11 19 14 13 105

k = 2

98 16

58


16 98 10 11 19 14 13 105

k = 2

10 98 16

59


16 98 10 11 19 14 13 105

k = 2

10 98 16

60


16 98 10 11 19 14 13 105

k = 2

10 19 98 16

61


16 98 10 11 19 14 13 105

k = 2

10 19 98 16

62


16 98 10 11 19 14 13 105

k = 2

10 14 19 98 16

63


16 98 10 11 19 14 13 105

k = 2

10 14 19 98 16

64


16 98 10 11 19 14 13 105

k = 2

10 13 14 19 98 16

65


16 98 10 11 19 14 13 105

k = 2

10 13 14 19 98 16

66


16 98 10 11 19 14 13 105

k = 2

10 105 13 14 19 98 16

67


16 98 10 11 19 14 13 105

k = 2

10 11 105 13 14 19 98 16

2nd smallest

return 11

68

QuickSelect Pseudocode

procedure QuickSelect(A, k):pick a pivot element p uniformly at

random;for i from 1 to n:

if A[i] < p put A[i] into AL

else put A[i] into AR

if |AL| = k-1; return p; // p is the kth element

else if |AL| ≥ k-1 return QuickSelect(AL, k)

else: return QuickSelect(AR, k-1-|AL|)

69

Correctness of QuickSelect

Informally:

If p is the kth smallest element we correctly return

it

O.w. the kth smallest element is either on AL or AR.

We pick the correct subarray according to # elements

< p

and update the rank k correctly

Can be made formal with an inductive proof.

70

Runtime of QuickSelect

Assume we are selecting the median (rank = n/2)?

Question 1: What’s the best scenario?

First pivot is the median: O(n) runtime

Question 2: What’s the worst scenario?

We iteratively pick the max or min element as pivot

End up having n-1 iterations O(n2)

Runtime of QuickSelect is fundamentally a probability

question!

How long does QuickSelect take on average?

Or what’s the expected runtime of QuickSelect?

71

Terminology Clarification Worst-Case vs Average-Case

Worst-case/Average-case refer to assumptions about

the input

Worst-case: Under any input (or the worst input).

Average-case: Under an “average” input according to

some distribution.

Our randomized algorithms analyses will be worst-case

No assumptions about the input distribution.

Given any input (or worst input) what’s the average or

expected run-time of algorithms?

72

Outline For Today


2. QuickSelect



5. QuickSort

73

Definition (Sample Space Ω): Set of all possible

outcomes

Ex 1: Rolling two dice

Ω = (1,1), (1, 2), …, (6, 6)

Ex 2: QuickSelect Ω, all possible sequences of pivot

picks

Ω: (kth), (nth, kth),…, (nth, (n-1)st, 1st, kth)

Each outcome i ∈Ω has a probability p(i) ≥ 0

Sample Space Ω

74

Definition (Event S ⊆ Ω): a set of outcomes from Ω

Ex: Rolling a 10 or more with 2 dice

S = (4, 6), (6, 4), (5, 5), (5, 6), (6, 5), (6, 6)

Ex: Picking the kth element as pivot in at most 2 picks

S = (kth), (nth, kth), (n-1st, kth), …, (1st, kth)

The probability of each event S:

Event S⊆ Ω

Pr(rolling a 10 or more) = 6/36

75

Definition (Random Variable): a fn: Ω -> (real 𝕽numbers)

Ex: X: the sum of the dices

Random Variable

Ω = (1,1), (1, 2), …, (6,

6)

2 3 12Ex: Y: run-time of QuickSelect

Ω: (kth) (nth, kth), …, (nth, n-1st, …,

kth) n ~2n ~n

2

76

Definition (Indicator Random Variable): A RV X from Ω-

> 0, 1

With probability p, X=1

With probability 1-p, X=0

Indicator Random Variable

Examples Later

77

Definition (Expectation E[X]): average value of X

Expectation

Value of X under

outcome i

probability of i

Equivalently:

Assuming X takes non-negative integer

values

78

Expectation Examples

1. Let X be sum of 2 dices: E[X]

=2. Let Y be an indicator random variable.

Y=1 with prob. p, Y=0 with 1-p

3. Assume we have a coin that comes heads with prob.

p.

Let Z be # times we have to flip the coin to get a head

Due to independence of consequent coin flips.

79

Facts About Expectation:

Facts About Expectation

80

Let Z = Σ Xj

**Linearity of Expectation**

Even if Xj depend on each other (i.e, not

independent)Extremely useful when trying to understand the

average value of a complicated random variable Z!.

We express Z as a sum of simpler random variables.

81

Ex: Birthday Paradox

If there are k people in a room, on average, how

many pairs of people have the same birthday?

Let Z be # pairs of people with the same birthday.

Z = 0, if no one shares birthdays

Z = 1, if exactly one pair of people have the same

birthday

… E[Z] is difficult to compute from the definition of

expectation.

82

Ex: Birthday Paradox

Let X(i,j) be an indicator random variable

X(i,j) = 1 if i, j have the same birthday,

X(i,j) = 0 otherwise

Then: when k = 28, E[Z] >

1!

83

Outline For Today


2. QuickSelect



5. QuickSort

84

Back to Expected Runtime of QuickSelect Let Z be the runtime of QuickSelect

Run time question is equivalent to: What’s E[Z]?

Trick: Try to break Z into simpler random

variables.

85

QuickSelect Execution

Termination

…

Recursion 1: work done = n

Recursion 2: work done = r2

Recursion 3: work done = r3

work done = 0

Recursion k: work done = rk

Recursion k+1: work done = rk+1

…

Total Work: Sum of work done across all

recursions

86

Phases of QuickSelect

Termination

…

…

Phase 1: calls when the array size [n, 0.75n]

Phase 2: calls when the array size [0.75n, 0.752n)

Phase j-1: calls when the array size [0.75j-1n, 0.75jn)

Phase log4/3n

87

Expectd Runtime In Terms of Phases

, where Xj is the work done in phase j.

Can we bound E[Xj]?

88

Bounding the Work Of Each Phase Consider phase j

At each recursion during phase j, the work done ≤

(¾)j-1n

Let Yj be the # recursive calls made during phase j

Let’s try to bound the expected number of recursive calls during

phase j.

89

Bounding the # Calls Of Each Phase

e1 e2 … … ek/4 … … … … … e3k/4 … … ek-1 ek

Let’s say phase j starts with (¾)j-1n ≤ k < (¾)jn

Guaranteed to exit the phase when we cut k

by ¾!

Let e1,e2, …, ek be our elements in increasing

order

Observation: A phase is guaranteed to end when p,

is between [ek/4, e3k/4], irrespective of the rank of the

item we’re searching for!if pivot is from here, phase is guaranteed to end

90

Bounding the Work Of Each Phase

e1 e2 … … ek/4 … … … … … e3k/4 … … ek-1 ek

Q: What’s the probability of picking p from [ek/4,

e3k/4]?

A: 50%

Expected # picks to pick a central pivot is ≤ 2.

Therefore, expected # recursion to end phase j:

91

Final Calculations

Q.E.D.

92

Summary: QuickSelect’s Runtime Analysis1. Defined r.v. Z as the run-time of QuickSelect

2. Broke the executions into log4/3n phases,

according to the sizes of the arrays in the

recursions

3. Defined Xj as the runtime during phase j

4. Expressed Z as

5. Bounded Xj: Yj:*(3/4)jn, Yj is # recursions in

phase j

6. Bounded E[Yj] by 2

7. Using (5) and (6) bounded E[Z] by O(n).

93

Outline For Today


2. QuickSelect



5. QuickSort

94

Back To Sorting: QuickSort

Input: Given an array A of size n

Output: Elements of A in increasing order

Pick a pivot p from A uniformly at random,

Partition A into AL: those < p and AR: those > p

Sort AL and AR recursively

Output: [AL, p, AR]

95

QuickSort Pseudocode

procedure QuickSort(A):if |A| = 1 return A[0]pick a pivot element p uniformly at

random;for i from 1 to n:

if A[i] < p put A[i] into AL

else put A[i] into AR

return [QuickSort(AL), p, QuickSort(AR)]

96

QuickSort Simulation

105 7 13 8 14 1 19 11 4 10 98 16

97

QuickSort Simulationpivot

105 7 13 8 14 1 19 11 4 10 98 16

98


105 7 13 8 14 1 19 11 4 10 98 16

99


105

105 7 13 8 14 1 19 11 4 10 98 16

100


105

105 7 13 8 14 1 19 11 4 10 98 16

101


7 105

105 7 13 8 14 1 19 11 4 10 98 16

102


7 105

105 7 13 8 14 1 19 11 4 10 98 16

103


7 13 105

105 7 13 8 14 1 19 11 4 10 98 16

104


7 13 105

105 7 13 8 14 1 19 11 4 10 98 16

105


7 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

106


7 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

107


7 1 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

108


7 1 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

109


7 1 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

110


7 1 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

111


7 1 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

112


7 1 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

113


7 1 4 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

114


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 11 19 14 13 105

115


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 10 11 19 14 13 105

116


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 10 11 19 14 13 105

117


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 98 10 11 19 14 13 105

118


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 98 10 11 19 14 13 105

119


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 16 98 10 11 19 14 13 105

120


105 7 13 8 14 1 19 11 4 10 98 16

pivot

7 1 4 8 16 98 10 11 19 14 13 105

Recurse

Recurse

Total work done at a recursive call with m elements:,

m-1 comparisons with the pivot + m copies = O(m).

** Run-time of QuickSort = O(# comparisons made)**

121

Analysis Roadmap

1. Let Z be the runtime of QuickSort, i.e. #

comparisons made by Quicksort

2. Let X(i, j) be the # times (i, j) gets compared.

3. Express Z as:

4. Solve E[X(i, j)] and sum them up to solve E[Z].

122

Counting # Comparisons Made By QuickSort Let X(i, j) be the # times (i, j) gets compared.

Fix a particular recursive call:105 7 13 8 14 1 19 11 4 10 98 16

7 1 4 8 16 98 10 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

Observation 1: All comparisons are made against the

pivot!Observation 2: The pivot will never be compared to

anything else!

123

Counting # (i, j) Comparisons

105 7 13 8 14 1 19 11 4 10 98 16

7 1 4 8 16 98 10 11 19 14 13 105

105 7 13 8 14 1 19 11 4 10 98 16

Condition for (i, j) comparison: i, j are compared only

when

1. They are in the some recursive call together

2. One of them is a pivot

And they can never be compared again.

Therefore X(i, j) can only be 0 or 1 (i.e., is an indicator

r.v.)

124

E[X(i,j)]

Recall: Expected value of indicator R.V.: E[X] =

Pr(X=1).

Question: What’s Pr(X(i,j) = 1)?

125

(i, j) Comparison Simulation (1)

e1 e2

… … … ei … … ej … … em-

1

em

Recursive Call 1

… ei …… ej … … em-

1

em

No Comparison

Recursive Call 2

pivot

pivot

… ei …… ej …

pivot

… ei … ej …

No Comparison

Recursive Call 3

Cannot be compared again! Total 0 comparisions!

i and j are at different recursions

126

(i, j) Comparison Simulation (2)

e1 e2

… … … ei … … ej … … em-

1

em

Recursive Call 1

No Comparison

Recursive Call 2

pivot

pivot

… … ei … … ej …

pivot

No Comparison

Recursive Call 3

1 (ei, ej) comparison! (and only 1)

e1 e2

… … … ei … … ej …

Observation: (ei, ej) is compared only if ei or ej is the first pivot to be picked among [ei, ej] block.

127

Pr(X(i,j) = 1)

Pr(X(i,j) = 1) = probability that ei or ej is the

first element to be picked amongst [ei, ei+1,

…, ej]:

128

Final Calculations

For fixed i: 1/2 + 1/3 + ... + 1/(n-i+1)

≤ ln(n)

Q.E.D

129

On Monday: Min Cut and Max Cut

correctness of huffman codes introduction to randomized algorithms quickselect & quicksort...

Documents