encs 6161 - ch5 and 6

8/9/2019 ENCS 6161 - Ch5 and 6

1/48

Chapter 5, 6

Multiple Random VariablesENCS6161 - Probability and Stochastic

ProcessesConcordia University

8/9/2019 ENCS 6161 - Ch5 and 6

2/48

Vector Random VariablesA vector r.v. X is a function X : S Rn, where S isthe sample space of a random experiment.

Example: randomly pick up a student name from alist. S = {all student names on the list}. Let be agiven outcome, e.g. Tom

H() : height of student W() : weight of student

A() : age of student

H,W,A are r.v.s.

Let X = (H,W,A), then X is a vector r.v.

ENCS6161 p.1/47

8/9/2019 ENCS 6161 - Ch5 and 6

3/48

EventsEach event involving X = (X1, X2, , Xn) has acorresponding region in Rn.

Example: X = (X1, X2) is a two-dimensional r.v.

A = {X1 + X2 10}

B = {min(X1, X2) 5}

C = {X21 + X

22 100}

ENCS6161 p.2/47

8/9/2019 ENCS 6161 - Ch5 and 6

4/48

Pairs of Random VariablesPairs of discrete random variables

Joint probability mass function

PX,Y(xj , yk) = P{X = xj

Y = yk} = P{X = xj , Y = yk}

Obviously jk PX,Y(xj , yk) = 1.Marginal Probability Mass FunctionPX(xj) = P{X = xj} = P{X = xj , Y = anything} =

k=1PX,Y(xj , yk)

Similarly PY(yk) =

j=1 PX,Y(xj, yk).

ENCS6161 p.3/47

8/9/2019 ENCS 6161 - Ch5 and 6

5/48

Pairs of Random VariablesThe joint CDF of X and Y (for both discrete andcontinuous r.v.s)

FX,Y(x, y) = P{X x, Y y}

-

6

x

(x,y)

y

ENCS6161 p.4/47

8/9/2019 ENCS 6161 - Ch5 and 6

6/48

Pairs of Random VariablesProperties of the joint CDF:

1. FX,Y(x1, y1) FX,Y(x2, y2), if x1 x2, y1 y2.2. FX,Y(, y) = FX,Y(x, ) = 03. FX,Y(, ) = 14. FX(x) = P

{X

x

}= P

{X

x, Y = anything

}= P{X x, Y } = FX,Y(X, )FY(y) = FX,Y(, y)FX(x), FY(y) : Marginal cdf

ENCS6161 p.5/47

8/9/2019 ENCS 6161 - Ch5 and 6

7/48

Pairs of Random VariablesThe joint pdf of two jointly continuous r.v.s.

fX,Y(x, y) =

2

xy FX,Y(x, y)

Obviously,

fX,Y(x, y)dxdy = 1

and

FX,Y(x, y) =

x

y

fX,Y(x, y)dydx

ENCS6161 p.6/47

8/9/2019 ENCS 6161 - Ch5 and 6

8/48

Pairs of Random VariablesThe probability

P

{a

X

b, c

Y

d

}=

b

a d

c

fX,Y(x, y)dydx

In general,

P{(X, Y) A} = AfX,Y(x, y)dxdy

Example:

-

6

x

y

1

1

0

A

10

x0

fX,Y(x, y)dydx

ENCS6161 p.7/47

8/9/2019 ENCS 6161 - Ch5 and 6

9/48

Pairs of Random VariablesMarginal pdf:

fX(x) =

d

dx FX(x) =

d

dxFX,Y(x, )=

d

dx

x

fX,Y(x, y)dydx

=

fX,Y(x, y)dy

fY(y) =

fX,Y(x, y)dx

ENCS6161 p.8/47

8/9/2019 ENCS 6161 - Ch5 and 6

10/48

Pairs of Random VariablesExample:

fX,Y(x, y) =

1 if 0 x 1, 0 y 1

0 otherwise.Find FX,Y(x, y)

1) x 0 or y 0, FX,Y(x, y) = 02) 0

x

1, and 0

y

1

FX,Y(x, y) =x0

y0

1 dydx = xy

3) 0 x 1, and y > 1FX,Y(x, y) =

x

0 1

0

1 dydx = x

4) x > 1 and 0 y < 1FX,Y(x, y) = y

5) x > 1 and y > 1

FX,Y(x, y) = 1

ENCS6161 p.9/47

8/9/2019 ENCS 6161 - Ch5 and 6

11/48

IndependencePX,Y(xj , yk) = PX(xj)PY(yk), for all xj and yk(discrete r.v.s)or FX,Y(x, y) = FX(x)FY(y) for all x and y

or fX,Y(x, y) = fX(x)fY(y) for all x and y

Example:a)

fX,Y =

1 0 x 1, 0 y 10 otherwise -

6

b)

fX,Y = 1 0 x 2, 0 y x0 otherwise -

6

ENCS6161 p.10/47

8/9/2019 ENCS 6161 - Ch5 and 6

12/48

Conditional ProbabilityIf X is discrete,

FY(y | x) = P{Y y, X = x}P{X = x} for P{X = x} > 0

fY(y | x) = ddy

FY(y | x)

If X is continuous, P{X = x} = 0

FY(y | x) = limh0 FY(y | x < X x + h) = limh0P

{Y

y, x < X

x + h

}P{x < X x + h}

= limh0

y

x+hx

fX,Y(x, y)dxdy

x+h

xfX(x)dx

= limh0

y fX,Y(x, y

)dy hfX(x) h =

y fX,Y(x, y

)dy

fX(x)

fY(y

|x) =

d

dy

FY(y

|x) =

fX,Y(x, y)

fX(x)

ENCS6161 p.11/47

8/9/2019 ENCS 6161 - Ch5 and 6

14/48

Conditional ProbabilityExample: A r.v. X is uniformly selected in [0, 1], andthen Y is selected uniformly in [0, x]. Find fY(y)

Solution:fX,Y(x, y) = fX(x)fY(y|x) = 1 1

x=

1

xfor 0 x 1, 0 y x and is 0 elsewhere.

-

6

x1

y

0

1 fY(y) =

fX,Y(x, y)dx

= 1

y

1

xdx = ln yfor 0 y 1 and fY(y) = 0 elsewhere.

ENCS6161 p.13/47

8/9/2019 ENCS 6161 - Ch5 and 6

15/48

Conditional ProbabilityExample:

- - -x {0, 1}

Detector

Channel

fY(y | x) Y R X

X = 0 A (volts), we assume P{X = 0} = P0X = 1 +A (volts), P{X = 1} = P1 = 1 P0Decide X = 0, if P{X = 0 | y} P{X = 1 | y}Decide X = 1, if P{X = 0 | y} < P{X = 1 | y}This is called the Maximum a posterior probability

(MAP) detection.

ENCS6161 p.14/47

8/9/2019 ENCS 6161 - Ch5 and 6

16/48

Conditional ProbabilityBinary communication over Additive White GaussianNoise (AWGN) channel

fY(y|0) = 12

e(y+A)222

fY(y|1) =1

2 e(yA)2

22

Apply the MAP detection, we need to findP

{X = 0

|y}

and P{

X = 1|y}

. Note here X is discrete,Y is continuous.

ENCS6161 p.15/47

8/9/2019 ENCS 6161 - Ch5 and 6

17/48

Conditional ProbabilityUse the similar approach (considering x < X x + h, and leth 0), we have

P{

X = 0|y}

=P{X = 0}fY(y|0)

fY(y)

P{X = 1|y} = P{X = 1}fY(y|0)fY(y)

Decide X = 0, if

P{X = 0|y} P{X = 1|y} y 22A

lnp0

p1Decide X = 1, if

P{X = 0|y} < P{X = 1|y} y >2

2A ln

p0

p1

When p0 = p1 =12 : Decide X = 0, if y 0

Decide X = 1, if y > 0

ENCS6161 p.16/47

8/9/2019 ENCS 6161 - Ch5 and 6

18/48

Conditional ProbabilityProb of error:considering the special case p0 = p1 =

12

P = P0P{

X = 1|X = 0} + P1P{

X = 0|X = 1}= P0P{Y > 0|X = 0} + P1P{Y 0|X = 1}

P{Y > 0|X = 0} = 12

0

e(y+A)2

22 dy = QA

Similarly,

P{Y 0|X = 1} = Q

A

P = Q

A

A , P , P

ENCS6161 p.17/47

8/9/2019 ENCS 6161 - Ch5 and 6

20/48

Multiple Random VariablesJoint cdf

FX1,Xn(x1, xn) = P[X1 x1, , X1 xn]

Joint pdffX1,Xn(x1, xn) =

n

x1, xnFX1,Xn(x1, xn)If discrete, joint pmf

PX1,Xn(x1, xn) = P[X1 = x1, Xn = xn]Marginal pdf

fXi(xi) =

all x1xn except xi

f(x1, xn)dx1 dxn

ENCS6161 p.19/47

8/9/2019 ENCS 6161 - Ch5 and 6

21/48

IndependenceX1, Xn are independent iff

FX1,Xn(x1,

xn) = FX1(x1)

FXn(xn)

for all x1, , xnIf we use pdf,

fX1,Xn(x1, xn) = fX1(x1) fXn(xn)

for all x1, , xn

ENCS6161 p.20/47

8/9/2019 ENCS 6161 - Ch5 and 6

22/48

Functions of Several r.v.sOne function of several r.v.s

Z = g(X1,

Xn)

Let Rz = {x = (x1, , xn) s.t. g(x) z} thenFz(z) = P

{X

Rz

}=

xRz

fX1, ,Xn(x1, , xn)dx1 dxn

ENCS6161 p.21/47

8/9/2019 ENCS 6161 - Ch5 and 6

23/48

Functions of Several r.v.sExample: Z = X+ Y, find Fz(z) and fz(z) in terms offX,Y(x, y)

x

y

y=z-x

Z = X+ Y z Y z XFz(z) =

zx

fX,Y(x, y)dydx

fz(z) =d

dzFz(z) =

fX,Y(x, z x)dx

If X and Y are independent

fz(z) =

fX(x)fY(z x)dx

ENCS6161 p.22/47

8/9/2019 ENCS 6161 - Ch5 and 6

25/48

Functions of Several r.v.sSince X, Y are indep. exponentially distributed

fZ(z) =

0 y fX(yz)fY(y)dy =

0 yeyz

ey

dy

=1

(1 + z)2for z > 0

ENCS6161 p.24/47

8/9/2019 ENCS 6161 - Ch5 and 6

26/48

Transformation of Random VectorsTransformation of Random Vectors

Z1 = g1(X1

Xn) Z2 = g2(X1

Xn)

Zn = gn(X1

Xn)

The joint CDF of Z is

FZ1Zn(z1

zn) = P

{Z1

z1,

, Zn

zn

}= x:gk(x)zk

fX1Xn(x1 xn)dx1 dxn

ENCS6161 p.25/47

8/9/2019 ENCS 6161 - Ch5 and 6

27/48

pdf of Linear TransformationsIf Z = AX, where A is a n n invertible matrix.

fZ(z) = fZ1Zn(z1,

, zn)

=fX1Xn(x1, , xn)

|A|x=A1z

=fX(A1z)

|A|

|A| is the absolute value of the determinant of A.e.g if A =

a b

c d

, then |A| = |ad bc|

ENCS6161 p.26/47

8/9/2019 ENCS 6161 - Ch5 and 6

28/48

pdf of General TransformationsZ1 = g1(X), Z2 = g2(X), , Zn = gn(X) whereX = (X1, , Xn)

We assume that the set of equations:z1 = g1(x), , zn = gn(x)has a unique solution given by

x1 = h1(z),

, xn = hn(z)

The joint pdf of Z is given by

fZ1Zn(z1 zn) =fX1Xn(h1(z), , hn(z))

|J(x1,

, xn)

|= fX1Xn(h1(z), , hn(z)) |J(z1, , zn)| ()where J(x1, , xn) is called the Jacobian of thetransformation.

ENCS6161 p.27/47

8/9/2019 ENCS 6161 - Ch5 and 6

29/48

pdf of General TransformationsThe Jacobian of the transformation

J(x1,

, xn) = detg1x1

g1xn gnx1

gnxn

and

J(z1, , zn) = deth1z1

h1zn

hnz1

hnzn

= 1J(x1 xn)

x=h(z)

Linear transformation is a special case of ()

ENCS6161 p.28/47

8/9/2019 ENCS 6161 - Ch5 and 6

30/48

pdf of General TransformationsExample: let X and Y be zero-mean unit-varianceindependent Gaussian r.vs. Find the joint pdf of Vand W defined by:

V = (X2 + Y2) 12

W = (X, Y) = arctan(Y /X) W [0, 2)

This is a transformation from Cartesian to Polarcoordinates. The inverse transformation is:

x

y

v

wx

y x = v cos(w)y = v sin(w)

ENCS6161 p.29/47

8/9/2019 ENCS 6161 - Ch5 and 6

31/48

pdf of General TransformationsThe Jacobian

J(v, w) = cos w v sin wsin w v cos w = v cos

2 w + v sin2 w = v

Since X and Y are zero-mean unit-varianceindependent Gaussian r.v.s,

fX,Y(x, y) =1

2 ex2

2 1

2 ey2

2 =1

2 ex2+y2

2

The joint pdf of V, W is then

fV,W(v, w) = v 1

2 e

(v2 cos2 w+v2 sin2 w)

2 =

v

2 ev2

2

for v 0 and 0 w < 2

ENCS6161 p.30/47

8/9/2019 ENCS 6161 - Ch5 and 6

32/48

pdf of General TransformationsThe marginal pdf of V and W

fV(v) =

fV,W(v, w)dw =

2

0

v

2e

v2

2 dw = vev2

2

for v 0. This is called the Rayleigh Distribution.fW(w) =

fV,W(v, w)dv =1

2

0ve

v2

2 dv =1

2

for 0 w < 2Since

fV,W(v, w) = fV(v)fW(w)

V, W are independent.

ENCS6161 p.31/47

8/9/2019 ENCS 6161 - Ch5 and 6

33/48

Expected Value of Functions of r.v.sLet Z = g(X1, X2, , Xn) thenE[Z] =

g(x1, , xn)fX1Xn(x1, , xn)dx1 dxnFor discrete case,

E[Z] =

all possible x

g(x1, , xn)PX1Xn(x1, , xn)

Example: Z = X1 + X2 + + XnE[Z] = E[X1 + X2 + + Xn]

=

(x1 + + xn)fX1Xn(x1 xn)dx1 dxn= E[X1] + + E[Xn]

ENCS6161 p.32/47

8/9/2019 ENCS 6161 - Ch5 and 6

34/48

Expected Value of Functions of r.v.sExample: Z = X1X2 XnE[Z] =

x1 xnfX1Xn(x1 xn)dx1 dxnIf X1, X2, , Xn are indep.

E[Z] = E[X1X2 Xn] = E[X1]E[X2] E[Xn]The (j,k)-th moment of two r.v.s X&Y is

E[XjYk] =

xjykfX,Y(x, y)dxdy

If j = k = 1 , it is called the correlation.

E[XY] =

xyfX,Y(x, y)dxdy

If E[XY] = 0, we call X&Y are orthogonal.

ENCS6161 p.33/47

8/9/2019 ENCS 6161 - Ch5 and 6

35/48

Expected Value of Functions of r.v.sThe (j,k)-th central moment of X, Y is

E

(X E(X))j (Y E(Y))k

when j = 2, k = 0, V ar(X)j = 0, k = 2, V ar(Y)

When j = k = 1, it is called the covariance of X, Y

Cov(X, Y) = E[(X E(X))(Y E(Y))]= E[XY] E[X]E[Y] = Cov(Y, X)

The correlation coefficient of X and Y is defined as

X,Y = Cov(X, Y)XYwhere X =

V ar(X) and Y =

V ar(Y).

ENCS6161 p.34/47

8/9/2019 ENCS 6161 - Ch5 and 6

36/48

Expected Value of Functions of r.v.sThe correlation coefficient 1 X,Y 1Proof:

0 EX E[X]X Y E[Y]Y 2

= 1 2X,Y + 1 = 2(1 X,Y)

If X,Y = 0, X, Y are said to be uncorrelated.If X, Y are independent,E[XY] = E[X]E[Y] Cov(X, Y) = 0 X,Y = 0.Hence, X, Y are uncorrelated.The converse is not always true. It is true in the caseof Gaussian r.v.s ( will be discussed later)

ENCS6161 p.35/47

8/9/2019 ENCS 6161 - Ch5 and 6

37/48

Expected Value of Functions of r.v.sExample: is uniform in [0, 2).Let X = cos and Y = sin X and Y are not independent, since X2 + Y2 = 1.

However

E[XY] = E[sin cos ] = E[1

2sin(2)]

=20

1

2

1

2sin(2)d = 0

We can also show E[X] = E[Y] = 0. So

Cov(X, Y) = E[XY] E[X]E[Y] = 0 X,Y = 0

X, Y are uncorrelated but not independent.

ENCS6161 p.36/47

8/9/2019 ENCS 6161 - Ch5 and 6

38/48

Joint Characteristic FunctionJoint Characteristic Function

X1Xn(w1, , wn) = E

ej(w1X1++wnXn)

For two variablesX,Y(w1, w2) = E

ej(w1X+w2Y)

Marginal characteristic function

X(w) = X,Y(w, 0) Y(w) = X,Y(0, w)

If X, Y are independent

X,Y(w1, w2) = E[ejw1X+jw2Y]

= E[ejw1X ]E[ejw2Y] = X(w1)Y(w2)

ENCS6161 p.37/47

8/9/2019 ENCS 6161 - Ch5 and 6

39/48

Joint Characteristic FunctionIf Z = aX+ bY

Z(w) = E[ejw(aX+bY)] = X,Y(aw,bw)

If Z = X+ Y, X and Y are independent

Z(w) = X,Y(w, w) = X(w)Y(w)

ENCS6161 p.38/47

8/9/2019 ENCS 6161 - Ch5 and 6

40/48

Jointly Gaussian Random VariablesConsider a vector of random variablesX = (X1, X2, Xn). Each with mean mi = E[Xi], fori = 1,

, n and the covariance matrix

K =

V ar(X1) Cov(X1, X2) Cov(X1, Xn)

...... ...

Cov(Xn, X1)

V ar(Xn)

Let m = [m1, , mn]T be the mean vector andx = [x1, , xn]T where ()T denotes transpose. ThenX1, X2,

Xn are said to be the jointly Gaussian if

their joint pdf is:

fX(x) =1

(2)n2

|K

|

12

exp

1

2(x m)TK1(x m)

where |K| is the determinant of K.

ENCS6161 p.39/47

8/9/2019 ENCS 6161 - Ch5 and 6

41/48

Jointly Gaussian Random Variables

For n = 1, let V ar(X1) = 2, then fX(x) =

12

e(xm)2

22

For n = 2, denote the r.v.s by X and Y. Let

m =

mXmY

K = 2X XYXYXYXY

2Y

Then

|K

|= 2X

2Y(1

2XY)

K1 =1

2X2Y(1 2XY)

2Y XYXYXYXY 2X

and

fX,Y(x, y) = 12XY

1 2XY

exp 12(1 2XY)

( x mxx

)2

2XY( x mxx

)(y my

y) + (

y myy

)2ENCS6161 p.40/47

8/9/2019 ENCS 6161 - Ch5 and 6

42/48

Jointly Gaussian Random VariablesThe marginal pdf of X:

fX(x) =1

2xe

(xmx)2

22x

The marginal pdf of Y:fY(y) =

12y

e

(ymy)2

22y

If XY = 0

X, Y are independent.

The conditional pdf.

fX|Y(x

|y) =

fX,Y(x, y)

fY(y)

=

exp

1

2(12XY

)2X

x XY XY (y my) mx

222X(1 2XY)

N(XY XY

(y my) + mx

E[X|Y], 2X(1 2XY)

V ar(X|Y)

)

ENCS6161 p.41/47

8/9/2019 ENCS 6161 - Ch5 and 6

43/48

8/9/2019 ENCS 6161 - Ch5 and 6

44/48

Linear Transformation of Gaussian r.v.s

Since K is symmetric, it is always possible to find a matrix A s.t.

= AKAT is diagonal. =

1 0

2

. . .

0 n

so

fY(y) =1

(2)n2 || 12 e

12 (ym)T1(ym)

=1

21e

(y1m1)2

21

1

2ne

(ynmn)2

2n

That is, we can transform X into n independent Gaussian r.v.s

Y1 Yn with means mi and variance i.

ENCS6161 p.43/47

8/9/2019 ENCS 6161 - Ch5 and 6

45/48

Mean Square Estimation

We use g(X) to estimate Y, write as Y = g(X). The

cost associated with the estimation error is C(Y Y).

Y X

e.g.

C(Y Y) = (Y Y)2= (Y g(X))2

The mean square error

E[C] = E[(Y g(X))2]Case 1: if Y = a

E[(Y

a)2] = E[Y2]

2aE[Y] + a2 = f(a)df

da= 0 a = E[Y] = Y

The mean square error: E[(Y

Y)

2] = V ar[Y]

ENCS6161 p.44/47

S i i

8/9/2019 ENCS 6161 - Ch5 and 6

46/48


Case 2: if Y = aX+ b, then E[(Y aX b)2] = f(a, b)

fa = 0f

b = 0 a = XY

Y

X

, b = Y

aX

Y = XY YX

(X X) + YThis is called the MMSE linear estimation.

The mean square error:

E[(Y Y)2] = 2Y(1 2XY)If XY = 0, Y = Y, error=

2

Y

, reduces to case 1.

If XY = 1, Y = YX (X X) + Y, and error= 0

ENCS6161 p.45/47

M S E i i

8/9/2019 ENCS 6161 - Ch5 and 6

47/48


Case 3: Y is a general function of X.

E[(Y Y)2] = E[(Y g(X))2] = E[E[(Y g(X))2|X]]

=

E[(Y g(x))2|x]fX(x)dxFor any x, choose g(x) to minimize E[(Y g(x))2|X = x]

g(x) = E[Y

|X = x]

This is called the MMSE estimation.

Example: X, Y joint Gaussian.

E[Y|X] = X,YY

X (X X) + YThe MMSE estimation is linear for Gaussian.

ENCS6161 p.46/47

M S E ti ti

8/9/2019 ENCS 6161 - Ch5 and 6

48/48


Example: X uniform(1, 1), Y = X2. We haveE[X] = 0

XY = E[XY]

E[X]E[Y] = E[XY] = E[X3] = 0So, the MMSE linear estimation:

Y = Yand the error is 2Y.

The MMSE estimation:g(x) = E[Y|X = x] = E[X2|X = x] = x2

So Y = X2 and the error is 0.

ENCS6161 p.47/47

encs 6161 - ch5 and 6

Documents