Probability Theory Overview and Analysis of Randomized

Algorithms

Prepared by

John Reif, Ph.D.

Analysis of Algorithms

Probability Theory Topics

a) Random Variables: Binomial and Geometric

b) Useful Probabilistic Bounds and Inequalities

Readings

• Main Reading Selections:– CLR, Chapter 5 and Appendix C

Probability Measures

• A probability measure (Prob) is a mapping from a set of events to the reals such that:1) For any event A

0 Prob(A) 12) Prob (all possible events) = 13) If A, B are mutually exclusive

events, thenProb(A B) = Prob (A) + Prob (B)

Conditional Probability

• Define

Prob(A B)Prob(A | B)

Prob(B)

for Prob(B) > 0

Bayes’ Theorem

• If A1, …, An are mutually exclusive and contain all events then

ii n

jj=1

j j j

PProb(A | B)

P

where P = Prob(B | A ) Prob(A )

Random Variable A (Over Real Numbers)

• Density Function

Af (x) = Prob(A=x)

Random Variable A (cont’d)

• Prob Distribution Function

x

A A

-

F (x) = Prob(A x) = f (x) dx

Random Variable A (cont’d)

• If for Random Variables A,B

• Then “A upper bounds B” and “B lower bounds A”

A Bx F (x) F (x)

A

B

F (x) Prob (A x)

F (x) Prob (B x)

Expectation of Random Variable A

• Ā is also called “average of A” and “mean of A” = A

A

-

E(A) = A = x f (x) dx

Variance of Random Variable A

2 2 2 2A

2 2A

-

σ (A A) A (A)

where 2nd momoment A x f (x) dx

Variance of Random Variable A (cont’d)

Discrete Random Variable A

Discrete Random Variable A (cont’d)

Discrete Random Variable A Over Nonnegative Numbers

• Expectation

Ax=0

E(A) = A = x f (x)

Pair-Wise Independent Random Variables

• A,B independent if

Prob(A B) = Prob(A) X Prob(B)

• Equivalent definition of independence

A B A B

A B A B

A B A B

f (x) = f (x) f (x)

M (s) = M (s) M (s)

G (z) = G (z) G (z)

Bounding Numbers of Permutations

• n! = n x (n-1) x 2 x 1= number of permutations of n objects

• Stirling’s formulan! = f(n) x (1+o(1)), where

n -nf(n) = n e 2πn

Bounding Numbers of Permutations (cont’d)

• Note– Tighter bound

= number of permutations of n objects taken k at a time

1 1(12n+1) 12nf(n) e < n! < f(n) e

n!

(n-k)!

Bounding Numbers of Combinations

= number of (unordered)

combinations of n objects

taken k at a time

• Bounds (due to Erdos & Spencer, p. 18)

n n!

k k! (n-k)!

2 3

2

k kk 2n 6nn n e

~ (1 o(1))k k!

3

4for k = o n

Bernoulli Variable

• Ai is 1 with prob P and 0 with prob 1-P

• Binomial Variable• B is sum of n independent Bernoulli

variables Ai each with some probability p

BINOMIAL with parameters n,p

B 0

i=1 n

probability P B B+1

procedure

begin

for to do

with do

output

end

B is Binomial Variable with Parameters n,p

n pmean

2 np (1-p)variance

B is Binomial Variable with Parameters n,p (cont’d)

x n-xn Prob(B=x) = p (1-p)

xdensity fn

xk n-k

k=0

n Prob(B x) = p (1-p)

kdistribution fn

Poisson Trial

• Ai is 1 with prob Pi and 0 with prob 1-Pi

• Suppose B’ is the

sum of n independent Poisson trials

Ai with probability Pi for i > 1, …, n

Hoeffding’s Theorem

• B’ is upper bounded by a Binomial Variable B

• Parameters n,p where

n

ii=1

Pp=

n

Geometric Variable V• parameter p

xx 0 Prob(V=x) = p(1-p)

GEOMETRIC parameter p

V 0

loop with probability 1-p

exit

procedure

begin

goto

Probabilistic Inequalities

• For Random Variable A

2 2 2

A

A (A)

mean

variance

Markov and Chebychev Probabilistic Inequalities

• Markov Inequality (uses only mean)

• Chebychev Inequality (uses mean and variance)

Prob (A x)x

2

2Prob ( A )

Example of use of Markov and Chebychev Probabilistic Inequalities

• If B is a Binomial with parameters n,p

2

npThen Prob (B x)

xnp (1-p)

Prob ( B np )

Gaussian Density Function

2x-

21

(x) = e2π

Normal Distribution

• Bounds x > 0 (Feller, p. 175)

x

-

(X) = (Y) dY

3

1 1 Ψ(x)(x) 1 (x)

x x x

x [0,1]

x 1 xx (1) (x) - (0) =

22πe 2π

Sums of Independently Distributed Variables

• Let Sn be the sum of n independently distributed variables A1, …, An

• Each with mean and variance

• So Sn has mean and variance 2

n

2

n

Strong Law of Large Numbers: Limiting to Normal Distribution• The probability density function of

to normal distribution (x)

• Hence

Prob

nn

(S - )T = as nlimits

n( S - x) (x) as n

Strong Law of Large Numbers (cont’d)

• So

Prob

(since 1- (x) (x)/x)

n( S - x) 2(1 (x))

2 (x)/x

Moment Generating Functions and Chernoff Bounds

Advanced Material

Moments of Random Variable A (cont’d)

• n’th Moments of Random Variable A

• Moment generating function

n nA

-

A x f (x) dx

sxA A

-

sA

M (s) e f (x) dx

= E(e )

Moments of Random Variable A (cont’d)

• NoteS is a formal parameter

n

0

nAAn

d M (s)

dss

Moments of Discrete Random Variable A

• n’th moment

n nA

x=0

A x f (x)

Probability Generating Function of Discrete Random Variable A

2 " ' ' 2Avariance σ (1) (1) ( (1))A A AG G G

x AA A

x=0

G (z) = z f (x) E(z )

" 2A2nd derivative G (1) = A A

A1st derivative G ' (1) = A

• If A1, …, An independent with same distribution

1

1 1

1 2 n

n

B A

n n

B A B A

Then if B = A A ... A

f (x) = f (x)

M (s) = M (s) , G (z) = G (z)

1 iA Af (x) = f (x) for i = 1, ... n

Moments of AND of Independent Random Variables

Generating Function of Binomial Variable B with Parameters n,p

• Interesting fact

xn k k n-k

k=0

nG(z) = (1-p+pz) = z p (1-p)

k

1Prob(B= ) =

n

Generating Function of Geometric Variable with parameter p

k k

k=0

pG(Z) = Z (p(1-p) ) =

1-(1-p)Z

Chernoff Bound of Random Variable A

• Uses all moments• Uses moment generating function

By setting x = ’ (s) 1st derivative minimizes bounds

-sxA

γ(s) - sxA

γ(s) - s γ'(s)

Prob (A x) e M (s) for s 0

= e where γ(s) = ln (M (s))

e

Chernoff Bound of Discrete Random Variable A

• Choose z = z0 to minimize above bound

• Need Probability Generating function

-xAProb (A x) z G (z) for z 1

x AA A

x 0

G (z) = z f (x) = E(z )

Chernoff Bounds for Binomial B with parameters n,p

• Above mean x

n-x x

x xx-μ 1

-x - μ 2

Prob (B x)

n-

n-x x

1 e since 1 e

x x

e for x μe

Chernoff Bounds for Binomial B with parameters n,p (cont’d)

• Below mean x

n-x x

Prob (B x)

n-

n-x x

Anguin-Valiant’s Bounds for Binomial B with Parameters n,p

• Just above mean

• Just below mean

2 μ-ε

2

= np for 0 < < 1

Prob (B (1+ ) ) e

2 μ-ε

3

for 0 < < 1

Prob (B (1- ) ) e

Anguin-Valiant’s Bounds for Binomial B with Parameters n,p (cont’d)

Tails are bounded by Normal distributions

Binomial Variable B with Parameters p,N and Expectation = pN• By Chernoff

Bound for p < ½

• Raghavan-Spencer bound for any > 0

nN 2

NProb (B ) 2 p

2

μ

(1+ )Prob (B (1+ ) )

(1+ )

e

Probability Theory

Prepared by

John Reif, Ph.D.

Analysis of Algorithms