Independent random variables

Will MonroeJuly 24, 2017

with materials byMehran Sahamiand Chris Piech

Announcements: Midterm

Tomorrow!

Tuesday, July 25, 7:00-9:00pm

Building 320-105(main quad, Geology Corner)

One page of notes (front & back)

No books/computers/calculators

Review: Joint distributions

A joint distribution combines multiple random variables. Its PDF or PMF gives the probability or relative likelihood of both random variables taking on specific values.

pX ,Y (a ,b)=P(X=a ,Y=b)

Review: Joint PMF

A joint probability mass function gives the probability of more than one discrete random variable each taking on a specific value (an AND of the 2+ values).

pX ,Y (a ,b)=P(X=a ,Y=b)

01

2

0.050.100.05

0.200.100.10

0.100.100.20

0 1 2Y

X

Review: Joint PDF

A joint probability density function gives the relative likelihood of more than one continuous random variable each taking on a specific value.

P(a1<X≤a2,b1<Y≤b2)=

∫a1

a2

dx∫b1

b2

dy f X ,Y (x , y)

Review: Joint CDF

F X ,Y (x , y)=P(X≤x ,Y≤ y)

x

y

to 0 asx → -∞,y → -∞,

to 1 asx → +∞,y → +∞,

plot by Academo

Probabilities from joint CDFsP(a1<X≤a2,b1<Y≤b2)=F X ,Y (a2,b2)

−FX ,Y (a1,b2)

−FX ,Y (a2,b1)

+F X ,Y (a1,b1)

a1

a2

b1

b2

Review: Marginalization

Marginal probabilities give the distribution of a subset of the variables (often, just one) of a joint distribution.

Sum/integrate over the variables you don’t care about.

pX (a)=∑y

pX ,Y (a , y)

f X (a)=∫−∞

∞

dy f X ,Y (a , y)

Review: Non-negative RVexpectation lemma

You can integrate y times the PMF, or you can integrate 1 minus the CDF!

E [Y ]=∫0

∞

dy P(Y > y)

=∫0

∞

dy (1−FY ( y))

FY ( y)

y

Non-negative RV expectation lemma:Rearranging terms

E [X ]=0 P (X=0)+1 P(X=1)+2 P(X=2)+3 P(X=3)+⋯

=0 P(X=0)+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯

[Addendum]

1

2

3...

=0 P(X=0)+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯

=0 P(X=0)+1 P(X≥1)+1 P(X=1)+2 P(X≥2)+1 P(X=1)+2 P(X=2)+3 P (X≥3)+⋯

=∑i=1

∞

P(X≥i)

Non-negative RV expectation lemma:Graphically

E [X ]=∑

[Addendum]

0 P(X=0) 1 P(X=1) 2 P(X=2) 3 P(X=3) 4 P(X=4)0

0.05

0.1

0.15

0.2

0.25

0.3

P(X ≥ 1)P(X ≥ 2)

P(X ≥ 3)P(X ≥ 4)

x

p_

X(x

)

Review: Multinomial random variable

An multinomial random variable records the number of times each outcome occurs, when an experiment with multiple outcomes (e.g. die roll) is run multiple times.

X1 ,…, Xm∼MN (n , p1, p2,…, pm)

P(X1=c1 , X2=c2 ,…, Xm=cm)

=( nc1 , c2 ,…, cm

) p1c1 p2

c2… pmcmvector!

A question from last class

“Are X and Y independent?”

pX ,Y (a ,b)=P(X=a ,Y=b)

01

2

0.050.100.05

0.200.100.10

0.100.100.20

0 1 2Y

X

P(X=0,Y=0)=P(X=0)P(Y=0)

Independence ofdiscrete random variables

Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!).

X⊥Y iff ∀ x , y :

P(X=x ,Y= y)=P(X=x)P(Y= y)- or -

pX ,Y (x , y)=pX (x) pY ( y)

Coin flips

P(X=x ,Y= y)=(nx) px(1−p)n−x

(my ) py(1−p)m− y

=P(X=x)P(Y= y)

n flips m flipsX: number of headsin first n flips

Y: number of headsin next m flips

∴X⊥Y

Coin flips

n flips m flipsX: number of headsin first n flips

X⟂Z

Z: total number of heads in n + m flips

Z=0→X=0

Web server hits

Your web server gets N requests in a day. N ~ Poi(λ).

Each request comes independently from human (prob. p) or bot (1 – p).

X: # requests from humans in dayY: # requests from bots in day

Knowing N:

P(X=i ,Y= j)=P (X=i ,Y= j|N=i+ j)P (N=i+ j)

X∼Bin (N , p)Y∼Bin (N ,1−p)

+P (X=i ,Y= j|N≠i+ j)P(N≠i+ j)

=(i+ ji ) p

i(1−p) j =e−λ λ

i+ j

(i+ j)!

Web server hits

P(X=i ,Y= j)=(i+ ji ) p

i(1−p) je−λ λ

i+ j

(i+ j)!

=(i+ j)!i ! j !

pi(1−p) j e−λ λ

iλ

j

(i+ j)!

=e−λpiλ

i

i !(1−p) jλ j

j !

=e−λ p (λ p)i

i !e−λ(1− p) [λ(1−p)] j

j !

=P (X=i)P(Y= j)

X∼Poi (λ p)Y∼Poi (λ(1−p))

∴X⊥Y

Independence ofcontinuous random variables

Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!).

X⊥Y iff ∀ x , y :

f X ,Y (x , y)=f X (x) f Y ( y)- or -

F X ,Y (x , y)=FX (x)FY ( y)- or -

f X ,Y (x , y)=g(x)h( y)

Density functions and independence

f X ,Y (x , y)=6 e−3 x e−2 y for 0<{x , y}<∞X⊥Y : yes!

X⊥Y : yes!

X⊥Y : no!

f X ,Y (x , y)=4 x y for 0<{x , y}<1

f X ,Y (x , y)=4 x y for 0<x<1− y<1

g(x) h(y)

g(x) h(y)

https://bit.ly/1a2ki4G → https://b.socrative.com/login/student/Room: CS109SUMMER17

The Joy of Meetings

P(X+10<Y )+P(Y +10<X )=2 P(X+10<Y )(symmetry)

2 people set up a meeting for 12pm.

Each arrives independently, uniformly between 12:00 and 12:30.

X ~ Uni(0, 30): mins after 12 for person 1Y ~ Uni(0, 30): mins after 12 for person 2

P(first to arrive waits > 10 minutes for the other) = ?

=2 ∬x , y : x+10< y

dx dy f X ,Y (x , y)

(symmetry)

The Joy of Meetings

P(X+10<Y )+P(Y +10<X )=2 P(X+10<Y )

=2 ∬x , y : x+10< y

dx dy f X ,Y (x , y)

=2 ∫y=10

30

dy ∫x=0

y−10

dx ( 130 )

2

=2 ∬x , y : x+10< y

dx dy f X (x) f Y ( y)

=2

302 ∫y=10

30

dy ( y−10)

(symmetry)

(independence)

=2

302 [ y2

2−10 y ]

y=10

30

=2

302 [( 302

2−300)−( 102

2−100)]=4

9

Independence ofcontinuous random variables

Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!).

X⊥Y iff ∀ x , y :

f X ,Y (x , y)=f X (x) f Y ( y)- or -

F X ,Y (x , y)=FX (x)FY ( y)- or -

f X ,Y (x , y)=g(x)h( y)

Setting records

Let X₁, X₂, … be a sequence ofindependent and identically distributed (I.I.D.) continuous random variables.

“record value”: an Xₙ that beats all previous Xᵢ ⇒ Xₙ = max(X₁, …, Xₙ)

Aᵢ: event that Xᵢ is a “record value”

An+1⊥An ?

https://bit.ly/1a2ki4G → https://b.socrative.com/login/student/Room: CS109SUMMER17

Independence is symmetricX⊥Y ⇔ Y ⊥X

Let X₁, X₂, … be a sequence ofindependent and identically distributed (I.I.D.) continuous random variables.

“record value”: an Xₙ that beats all previous Xᵢ ⇒ Xₙ = max(X₁, …, Xₙ)

Aᵢ: event that Xᵢ is a “record value”

E⊥F ⇔ F⊥E

An+1⊥An ?An⊥ An+1 ?https://bit.ly/1a2ki4G → https://b.socrative.com/login/student/

Room: CS109SUMMER17

yes!

P(An An+1)

=1n⋅

1n+1

=P (An)P(An+1)

Break time!

Sum of independent binomials

X∼Bin (n , p) Y∼Bin (m, p)

n flips m flipsX: number of headsin first n flips

Y: number of headsin next m flips

X+Y∼Bin (n+m, p)

More generally:

X i∼Bin (ni , p) ⇒ ∑i=1

N

X i∼Bin (∑i=1

N

ni , p)all X i independent

Sum of independent Poissons

X∼Poi (λ1) Y∼Poi (λ2)

λ₁ chips/cookieX: number of chipsin first cookie

Y: number of chipsin second cookie

X+Y∼Poi(λ1+λ2)

More generally:

X i∼Poi(λi) ⇒ ∑i=1

N

X i∼Poi(∑i=1

N

λ i)

λ₂ chips/cookie

all X i independent

Convolution

A convolution is the distribution of the sum of two independent random variables.

f X+Y (a)=∫−∞

∞

dy f X (a− y) f Y ( y)

Dance Dance Convolution

X, Y: independent discrete random variables

P(X+Y=a)=∑y

P(X+Y=a ,Y= y)(law of total probability)

=∑y

P(X=a− y)P(Y= y)

X, Y: independent continuous random variables

f X+Y (a)=∫−∞

∞

dy f X (a− y) f Y ( y)

Blurring a photo

images: Stig Nygaard (left), Daniel Paxton (right)

=∫0

1

dy f X (a− y) f Y ( y)

Sum of independent uniforms

f X+Y (a)=∫−∞

∞

dy f X (a− y) f Y ( y)

X∼Uni (0 ,1) Y∼Uni (0,1)

0 10

1

0 10

1

1

Case 1: if 0 ≤ a ≤ 1, then we need 0 ≤ y ≤ a (for a – y to be in [0, 1])Case 2: if 1 ≤ a ≤ 2, then we need a – 1 ≤ y ≤ 1

={ ∫0

ady⋅1=a 0≤a≤1

∫a−1

1dy⋅1=2−a 1≤a≤2

0 otherwise 0 1 20

1

Sum of independent normals

More generally:

X∼N (μ1 ,σ12) Y∼N (μ2 ,σ2

2)

X+Y∼N (μ1+μ2 ,σ12+σ2

2)

X i∼N (μi ,σi2) ⇒ ∑

i=1

N

X i∼N (∑i=1

N

μi ,∑i=1

N

σ i2)

all X i independent

Virus infections

M∼Bin (50 ,0.1)≈X∼N (5,4.5)

150 computers in a dorm:

50 Macs (each independently infected with probability 0.1)

100 PCs (each independently infected with probability 0.4)

What is P(≥ 40 machines infected)?

M: # infected Macs

P: # infected PCs P∼Bin (100 ,0.4)≈Y∼N (40,24)

P(M+P≥40)≈P(X+Y≥39.5)W=X+Y∼N (5+40, 4.5+24)=N (45,28.5)

P(W≥39.5)=P (W−45√28.5

≥39.5−45√28.5 )≈1−Φ(−1.03)≈0.8485