mathematical expectation

Dr Sonnet Nguyen, USTH 2011

1

Topic 3: Mathematical Expectation

Course: Probability and StatisticsLect. Dr Quang Hng/ Sonnet Nguyen

Topic 3: Mathematical Expectation 2/50

Contentso Expectationo Variance, standard deviationo Momentso Conditional expectation, conditional variance,

conditional momentso Chebyshevs inequalityo Weak and Strong Laws of Large Numbers o Other Measures of Central Tendencyo Other Measures of Dispersion


2


Expectation

1

or , or briefly the is a very important concept in P&S.

For a discrete random variable having the p

Mathematical Expectation E

ossible values ,....,

xpect

,the of

ed ValueExpectation

expectation inX x x

X

1 11

1

s defined as

( ) ( ) ..... ( ) ( )

or equivalently, if ( ) ( ), then ( ) ( ),

where the last summation is taken over all appropriate values of .

n

n n j jj

n

j j j jj

E X x P X x x P X x x P X x

P X x f x E X x f x

x

=

=

= = + + = = =

= = =


Expectation

1 2

1

As a special case when the probabilities are all equal, we have

which is called the arithmetic mean, or simply the mean, of , ,..., .

For a continuous random variab

1( ) n

jj

n

E X xn

x x x

=

=

le having density function ( ), the expectation of X is defined as

provided that the integral

( )

converges absolute

( )

ly.

X

E X x f x dx

f x

+

-

=


3


Expectation

X

The expectation of is very often called the of X and is denoted by , or simply , when the particular random variable is understood.The mean, or expectation, of gives a single val

mea

ue

n

that ac

X

X

m m

ts as a representative or of , and for this reason it is often called

average of the valuesmeasure of central tende a .ncy

X


ExampleSuppose that a game is to be played with a single die assumed fair. In this game a player wins $20 if a 2 turns up, $40 if a 4 turns up; loses $30 if a 6 turns up; while the player neither wins nor loses if any other face turns up. Find the expected sum of money to be won.

1 1 1 1 1 1 E(X)=0 + 20 0 + 40 + 0 +

Solutio

(-30) = 5.6 6 6 6 6 6

n.

+


4


Functions of Random Variables

{ | ( ) } { |

Let be a discrete random variable with probability function ( ). Then ( ) is also a discrete random variable, and

the probability function of is ( ) ( ) ( ) ( )

x g x y x g

Xf x Y g X

Yh y P Y y P X x f x

=

=

= = = = =( ) }

1 2

1 21 1

1

.

If takes on the values , ,...., , and the values

, ,...., ( ), then ( ) ( ) ( ).

Therefore, [ ( )] ( ) ( ) ( ) ( ).

x y

nm n

m i i j ji j

n

j jj

X x x x Y

y y y m n y h y g x f x

E g X g x f x g x f x

=

= =

=

=

= =


Functions of Random VariablesSimilarly, if is a continuous random variable having probability

density ( ), then it can be shown that [ ( )] ( ) ( ) .

Note that [ ( )] do not involve the probability function and the pr

X

f x E g X g x f x dx

E g X

+

-

=

obability density function of ( ). Generalizations are easily made to functions of two or more random variables. For example, if and are two continuous random variables having joint density

Y g X

X Y

=

function ( , ), then the of ( , ) is given by

[ ( , )]

expectation

( , ) ( , )

f x yg X Y

E g X Y g x y f x y dx dy+ +

- -

=


5


Theorem on Expectation

a1) If is any constant, then ( ) ( ).a2) If and are any random variables, then ( ) ( ) ( ).a3) If and are independent rand

Theorem 1:

om variables, then ( )

cE c X c E XX Y

E X Y E X E YX Y

E XY

=

+ = +

( ) ( ). E X E Y=


The Variance and Standard Deviation

2 2

variance ( ) [( ) ]

Another quantity of great importance is called the and is defined by:

. The variance is a nonnegative number. The positive square root of the variance is c

[( ( )) ]XVar X E X E X E Xm= - = -

2

alled the and is given by

standard deviation

( ) [( ) ] .X XVar X E Xs m= = -


6


The Variance and Standard Deviation

2 2 2

1

1 2If is a discrete random variable taking the values , , . . . , and having probability functi

[(

on ( ), then the variance is given by

If takes on an in

) ] ( ) ( )

f

. n

X X j X jj

n

E X x

X x x xf

X

f x

x

s m m=

= - = -

2 2

1

1 2inite number of values , , . . . then

provided that the series converges.

If is a continuous random variable having density function ( ),

then the variance is g

( ) ( ),

i

ve

X j X jj

x

f

f

x

x

x

X x

s m

=

= -

2 2 2n by:

provided that the integral conver

[( ) ] ( ) ( ) ,

es.

g

X X XE X x f x dxs m m+

-

= - = -


What does the variance measure?

The variance (or the standard deviation) is a measure of the , or , of the values of If th

dispersionsca e values tend

the random va to be concent

tterrate

riable ad near t

bout the mean .he mean, the

var

m; while

. The situa

iance

tion i

is small if the va

s indicated graphlues tend to be distributed far fr

ically in the following figure foro

m the mean, the va

the case of two co

riance is

ntinuous

large

distributions having the same mean .m


7


Units of Variance and Standard Deviation

2

Note that if has certain dimensions or units, such as cm, then the variance of has units cm while the standard deviation has the same unit as , i.e., cm. For this reason, the standard deviati

XX

X

X2

on is often used. When no confusion can result, the standard deviation is often denoted by instead of , and the variance in

such case is .

s s

s


Theorems on Variance

2 2 2 2 2 2

2

2

a1) [( ) ] ( ) ( ) [ ( )] .a2) If is any constant, then ( ) ( ). a3) The quantity [( ) ] is a minimum when ( ).a4) If and are in

Theorem 2:

dependent rand

X X XE X E X E X E Xc Var c X c Var X

E X a a E XX Y

s m m= - = - = -

=

- =

2 2 2

om variables, ( ) ( ) ( ) or .In words, the variance of a sum of independent variables equals the sum of their variances.

X Y X YVar X Y Var X Var Y s s s = + = +


8


Standardized Random VariablesLet be a random variable with mean and standard deviation ( >0). Then we can define an associated

given by .

An important property of * is that it h

standardized random -variable *

mea

as

X

X

X

X

s

ms

ms

=

an of zerovariance of 1 standardized

( *) 0, ( *) 1

and a , which accounts for the name , i.e.,

. The values of a standardized variable are sometimes ca standard scorlled , and is then se i ts a d E X Var X

X= =

o be expressed in standard units (i.e., is taken as the unit in measuring Standardized variables are useful for comparing different distribu

)t

. i s.

on

X m-


Moments

20 1 2

The of a random variable X about the mean , also called the , is defined as where =0,1,2,.....

It follows that

rth

=1,

momentrth central mom

=0, and = , i.e., the second

ent[( ) ]rr rE X

m

m m m s

m m= -

central moment or second moment about the mean is the variance.


9


Moments

1

1

If is a discrete random variable taking the values ,......, and having probability function ( ), then the is

[( ) ] ( ) ( ).

If takes on an infinite number of va

rth m t

l

omenn

nr r

r j jj

X x xf x

E X x f x

X

m m m=

= - = -

1 2

1

ues , , . . . , then

( ) ( ) provided that the series converges.

If is a continuous random variable having density function ( ),

then [( ) ] ( ) ( ) provided that

rr j j

j

r rr

x x

x f x

X f x

E X x f x dx

m m

m m m

=

+

-

= -

= - = -

the

integral converges.


Raw Moment or Moment about the origin

' '

'

1

The , also called the , is defined as: where 0, 1, 2, . . . .

The relationship betwee

rth moment of X about the origin rth raw moment (

n these moments is given by

..1

)

.

rr

r r r

E X r

rm m m m

m

-

=

= - +

=

' '

0

' '1 0

' 2 ' ' 32 2 3 3 2

' ' ' 2 44 4 3 2

( 1) ... ( 1) .

As special cases we have, using and 1,

, 3 2 ,

4 6 3 .

j j r rr j

rj

m m m m

m m m

m m m m m m m m

m m m m m m m

-

+ - + + -

= =

= - = - +

= - + -


10


Moment Generating Functions

The of is defined by , that is, assuming convergence,

moment generating function( ) ( )

( ) ( ) (discrete variable)

( ) ( ) (cont

n

i

tXX

t xX

t xX

XM t E eM t e f x

M t e f x dx+

-

=

=

=

uous variable).


Moment Generating Functions

' ' 2 '2

0

'

0

Homework. Show that the Taylor series expansion of the moment generating function is

( ) 1 .... ...,

where ( ) is the th derivative of ( )

evaluated at

r rX r r

r

r

r X Xrt

M t t t t t

d M t r M tdt

m m m m

m

=

=

= = + + + + +

=

0.Note: Since the coefficients in this expansion enable us to find the m moment geoments, neratin the reason fo g funct

r the name is apparent n io .

t =


11


Characteristic FunctionsIf putting , where is the imaginary unit, in the moment generating function we obtain an important function called the . We denote this by characteristic funct

ion

( ) ( ) ( i XX X

t i i

M i E e w

w

f w w= =

=

).

( ) ( ) (discrete variable)

( ) ( ) (contin

It follows that, assuming convergence,

Sin

uous variable

ce the series and the integral always converge

).

1,

i xX

i xX

i x

e f x

e f x dx

e

w

w

w

f w

f w+

-

=

=

absolutely.


Characteristic Functions

2' ' '

20

'

0

Homework. Show that the Taylor series expansion of the characteristic function is

( ) ( )( ) ( ) 1 ( ) .... ..... 2! !

where ( ) ( )

rr

X r rr

rr

r Xr

i ii ir

did

w

w wf w m w m w m m

m f ww

=

=

= = + + + + +

= -


12


Theorems on Moment Generating Functions and Characteristic Functions

/( )/

If ( ) is the moment generating function of the random variable and and ( 0) are constants, then the moment generating function of ( ) / is

T

heorem 3:

( ) at bX a b X

XM tX a b b

X a btM t e Mb+

+

=

/)/

X

(

Similarly, for characteristic function ( )

.

( ) . a i bX a b Xe bw

f w

wf w f+

=



If and are independent random variables having moment generating functions ( ) and ( ), respectively, then In words, the moment generating fu

Theorem 4:

( )nction

( ) ( ).

X Y X Y

X Y

X YM t M t

M t M t M t+ =

X

of a sum of independent random variables is equal to the product of their moment generating functions.Similarly, for characteristic functions ( ) and ( ) of two independent random variables and

Y

Xf w f w

X+Y X Y( ) = ( ) . ( )Y

f w f w f w


13



Suppose that and are random variables having moment generating functions ( ) and ( ), respectively. Then and have the s

Theorem 5 (Uniquen

ame probability

ess

dis

Theore

tribution if and onl

m)

X Y

X YM t M t X

Y y if ( ) ( ) identically.

Similarly, probability distribution is uniquely determined by its characteristic function. Thus, suppose that X and Y are random variables having characteristic function

X YM t M t=

X

X

s ( ) and( ) respectively, then and have the same probability

distribution if and only if ( ) = ( ) identically.Y

Y

X Yf w

f wf w f w


Relation between the density function and the characteristic function

An important reason for introducing the characteristic function is that represents the Fourier transform of the density function ( ). From the theory of Fourier transforms, we can easily

N

ote:

deterf x

mine the density function from the characteristic function. In fact,

which is often called an inversion formula, or inverse Fourier transform.Another reason f

1( ) ( )2

o

i xXf x e d

w f w wp

+-

-

=

r using the charact. function is that it always exists whereas the moment generating function may not exist.


14


Variance for Joint DistributionsThe results given above for one variable can be extended to two or more variables. For example, if and are two continuous random variables having joint density function ( , ), the means, or expe

X Yf x y

X

2 2 2X

2 2 2

ctations, of and are

,

and the variance

( ) ( , ) , ( ) ( , )

[( ) ]= ( ) ( , ) ,

[( ) ]= ( ) ( , ) .

s are

Y

X X

Y Y Y

E X x f x y dx dy E Y y f x y dx dy

E X x f x y dx dy

E Y y f x y dx d

X

y

Y

m m

s m m

s m m

- - - -

- -

- -

= = = =

= - -

= - -


CovarianceAnother quantity that arises in the case of two variables and is the

defined by In terms of the join

covariance(

t density f, ) [( )( )

unction ( , ), we have

].

( ( )

XY X Y

XY X

Cov X Y E X Y

x

X Y

f x ys m m

s m

= = - -

= -

Similar remarks can be made for two discrete random variables. ( , ), ( , ),

where the sums are taken over all

) ( , ) .

( )( ) ( , ),

the

Y

XY

X Yx y y

X Yx y

xx f x

y f x y dx dy

x y f x y

y y f x ym m

m

s m m

- -

-

= -

=

-

=

discrete values of and .X Y


15


Properties of covariance

2 2 2

a1) ( ) ( ) ( ) ( ) .a2) If and are independent random variables, then ( , ) 0.a3) ( ) ( ) ( ) 2 ( , ). OR 2 .a

Theorem 6:

4

XY X Y

XY

X Y X Y XY

E XY E X E Y E XYX Y

Cov X YVar X Y Var X Var Y Cov X Y

s m m

s

s s s s

= - = -

= = = +

= +

) .XY X Ys s s


Correlation CoefficientIf and are independent, then ( , ) 0. On the other hand, if and are completely dependent, e.g. when , then

( , ) . From this we are led to a measure of the dependence of

XY

XY X Y

X Y Cov X YX Y X Y

Cov X Y

s

s s s

= =

== =

correlation coefficient coefficient of correla

the variables and given by

.

We call the , or .From the property a4) of covariance we see that -1 1. In the case where =0

tion

XY

X Y

X Ys

rs s

rr

r

=

(i.e., zero covariance), we call the variables

and . In such cases ( =0) the variables may or may not b

uncorrelatee independ nt

de .

XY r


16


Conditional Expectation, Variance, and Moments

11

conditional density function

of given

If and have joint density function ( , ), then we have seen in Topic 2, the

( , ) is ( | ) , where ( ) is the ( )

marginal density function of

X Y

Y

f x y

f x yf y x f xf x

X =

. We can define the , or

, of given by:

where " " is

to be interpreted as in the continuous c

conditional expectation conditional mean

( | ) (

ase.

| ) , E Y

X

Y X

XX x y f y x dy x

x X x dx

-

=

+

= =

<


Conditional Expectation, Variance, and Moments

1

Similar properties (a1) and (a2) of Theorem 1 also hold for conditional expectation. We note the following properties:1. ( | ) ( ), when and are independent.

2. ( ) ( | ) ( ) .

It

E Y X x E Y X Y

E Y E Y X x f x dx

-

= =

= =is often convenient to calculate expectations by use of the

property 2, rather than directly.


17


Example

The average travel time to a distant city is hours by car or hours by bus. A woman cannot decide whether to drive or take the bus, so she tosses a coin. What is her expected travel time?

c b


Example - Solution

car bus

Here we are dealing with the joint distribution of the outcome of the toss, , and the travel time, , where if 0 and if 1. Presumably, both Y and Y ar

Solution.

e indecar bus

X YY Y X Y Y X= = = =

car car

bus bus

pendent of X, so that by Property 1 above: , and .Then Propert

E(Y|X=0)=E(Y |X=0)

y 2 (with the int

=E(Y )=cE(Y|X=l)=E(Y |X=1)=E(Y )=

egral replaced by a sumb) gives

( ) ( | 0) ( 0) ( | 1

, for a fair coin,

) ( 12

.) c bE Y E Y X P X E Y X P X += = = + = = =


18


Conditional Variance and Conditional Moments

2 22 2

2

We can define the of given by

[( ) | ] ( ) ( | )

where ( | ) . We can also define the of about any value

conditional variance

rth conditional mome given as

nt

[(

Y X

E Y X x y f y x dy

E Y X xY

a X

E

m m

m

-

- = = -

= =

) | ] ( ) ( | ) .

The usual theorems for variance and moments extend to conditional variance and moments.

r rY a X x y a f y x dy

-

- = = -


Chebyshevs InequalityAn important theorem in probability and statistics that reveals a general property of discrete or continuous random variables having finite mean and variance is known under the name of Chebyshev's ine

2

quality.

Suppose that is a random variable (discrete or continuous) having mean and variance , which are finite. Then if

Theorem 7 (Chebyshev's Inequa

is any positive number,

lity):

P(|X- | )

X

m

e

e

s

2

2 2or, with 1 k P(|X- |, ) .kk

se s m s

e =


19


Proof of the Chebyshevs Inequality

2 2 2

2

| |

2

We present the proof for continuous r.v. A proof for discrete r.v. is similar. If ( ) is the density function of , then

[( ) ] ( ) ( )

( ) ( )

(x

f xX

E X x f x dx

x f x dx

fm e

s m m

m

e

-

-

= - = -

-

2

| |

2

2

) (| | ).

Thus, (| | ) .

xx dx P x

P x

m ee m e

sm e

e

- = -

-


Example of the ChebyshevsInequality

2

Let 2 in Chebyshev's inequality, we see that

or .

In words, the probability of differing from its me

1P(|X- | 2 ) = 0.25

an by more than

P(|X

2 st

- |


20


1 2

The following theorem, called the , is an interesting consequence of Chebyshev's

law of large numbers

Theorem 8 (Law of Large Numberinequality.

: Let , , . . . , be mutually independent r

s)andom

nX X X

2

1

1

variables (discrete or continuous), each having finite mean and variance .

If for 1, 2,...., then .

Since is the arithmetic mean of , . . . , ,

lim

this

0

th

P

eo

nn

n nii

nn

Sn

S X n

S X Xn

m e

m s

=

=

- =

=

rem

states that the probability of the arithmetic mean differing

from its expected value by more than approaches zero as n .

nSn

e

Weak Law of Large Numbers


A stronger result, which we might expect to be true, is that

, but this is actually false.lim

lim with probabili

However, we

ty one, i.e.

can prove

that .

This r

P lim =1

n

x

n n

x x

Sn

S Sn n

m

m m

=

= =

esult is often called the , and, by contrast, that of Theorem 8 is called the

. When the "

strong law of large numbersweak law of large

numbers law " is referred to wit

of larghout qua

el

nif

umbersicati weak laon, the is impw lied.

Weak and Strong Law of Large Numbers


21


Other Measures of Central Tendency

measure of cent

As we have alread

ral tende

y seen, the mean, or expectation, of a random variable provides a

for the values of a distribution. Although the mean is used most, two

ncyother

X\

\ measures of central tendency are also employed. These are the and tmode medhe ian.


Other Measures of Central Tendency: MODE

1. . The of a discrete random variable is that value which occurs most often or, in other words, has the greatest probability of occurring. Sometimes we have two, three

MODE m

, or

ode

mor\ e values that have relatively large probabilities of occurrence. In such cases, we say that t bimodal trimodal multimodahe distribution is , , or , respectively. The mode of a con o

l

tinu\ us random variable is the value (or values) of where the probability density function has a relative maximum.

XX


22


Other Measures of Central Tendency: MEDIAN

2. . The is that value for which

.

In the case of a continuous distribution we ha

MEDIAN median1 1( ) and ( )2 2

( )ve

, or equivalently:0.5 ( ) ( ) 0 .5 .

P X m P X m

P X m P X m F m

m

< >

< = = > = and the median separates the density curve into two parts having equal areas of 1/2 each. In the case of a discrete distribution a unique median may not exist.


Percentiles

It is often convenient to subdivide the area under a density curve by use of ordinates so that the area to the left of the ordinate is some percentage of the total unit area. The values corresponding

0.10

to such areas are called , or briefly . Thus, for example, the area to the left of the ordinate at in the above Fig is .For instance, the area

percentile val

to the left o

ues percentil

f w l

es

ou

x

x

a

a

0.10

d be 0.10, or 10%, and would be called the 10 percentile (also called

median would be the 50 pethe first

decile). The (or firc ftenti h de ile c le).thx th


23


Percentiles

For any (0,1), there exists a real number such that , where is the area under the curve from - to . Formally,

( ) ( ).x

x SS x

S f x dx F xa

a a

a a

a a

a a

a-

=

= = =


PercentilesLet be a number between 0 and 1. The of the distribution of a continuous random variable , denotedby ( ), is defined by ( ( )).Thus (0.75), t

(100 ) percentile

he 75th percentile, is such th

thX

F

a

h a a h ah

a

=at the area under

the graph of ( ) to the left of (0.75) is equal 0.75. When we say that an individual's test score was at

the 90 percentile of the population, we mean that 90% of allExam e.

pl

f x

th

h

population scores were below that score and 10% were above. Similarly, 30 percentile is the score that exceeds 30% of all scores and is exceeded by 70% of all scores.

th


24


Other Measures of DispersionJust as there are various measures of central tendency besides the mean, there are various measures of dispersion or scatter of a random variable besides the variance or standard deviation. Some of th

0.25 0.75

0.75 0.25

0.75 0.25

e most common are:1. . If and represent the 25 and 75 percentile values, the difference is called the

( )

SEMI-INTERQUARTILE RANGE

interqu and 2

artile range

x x thth x x

x x-

- is the .

2. . The (M.D.) of a random variable is defined as the expectation of , i.

semi-interqua

e., assuming

rtile range

MEAN DEVIATION meanconvergence,

deviation

. .( ) ( )

X X

M D X E X x

m

m m

-

= - = - ( ) (discrete variable) . .( ) ( ) ( ) (continuous variable)

f x

M D X E X x f x dxm m

-= - = -


SkewnessOften a distribution is not symmetric about any value but instead has one of its tails longer than the other. If the longer tail occurs to the right, as in Fig. A, the distribution is said to be skewed to the right

skewed to

, while if the longer tail occurs to the left, as in Fig. B, it is said to be

. Measures describing this asymmetry are c the left coefficients o

alled , or brief skewness sfly kewn

33

3 3 3

3

. One such measure is given by: [( ) ] .

The measure will be positive or negative according to whether the distribution is skewed to the right or l

ess

eft, respecti

E X mma

s sa

-= =

3

vely. For a symmetric distribution, 0.a =


25


Skewness and Kurtosis

Fig. A Fig. B Fig. C


Kurtosis

In some cases a distribution may have its values concentrated near the mean so that the distribution has a large peak as indicated by the solid curve of Fig. C. In other cases the distribution may be relatively flat as in the dashed curve of Fig. C. Measures of the degree of peakedness of a distribution are called

or briefly . A meacoefficients

of kur sure often used is tosis, kurtosi given bs y

4

42 2 2 4

normal

[( ) ] .( [( ) ])

This is usually compared with the (discussed in the next topic), which has a coefficient of kur

curvtosis equal to .

e3

E XE X

mmb

m s-

= =-

mathematical expectation

Documents