elements of probabilty … a. n. kolmogorov (1933) + … carlos mana benasque-2014 astrofísica y...
TRANSCRIPT
ELEMENTS OF PROBABILTY
… A. N. Kolmogorov (1933) + …
),,( PBCarlos Mana Carlos Mana Benasque-2014Benasque-2014Astrofísica y Física de PartículasAstrofísica y Física de PartículasCIEMATCIEMAT
Event: Object of questions that we make about the result of the experiment suchthat the possible answers are: “it occurs” or “it does not occur”
elementary = those that can not be decomposed in others of lesser entity
Sample Space: {Set of all the possible elementary results of a random experiment}
exclusive: if one happens, no other occursexhaustive: any possible elemental result has to be included in
The elementary events have to be:
sure: get any result contained in
impossible: to get a result that is not contained in
random event: any event that is neither impossible nor sure
k
ke
jkjkee jk ,;0is a partition of
)(1) Sample Space
}{ ke
)dim(
Finitedrawing a die
denumerable throw a coin and stop when we get head
654321 ,,,,, eeeeee
,,,, xxxcxxcxcc
non-denumerable decay time of a particle t
),( B2) Measurable Space
B
Sample Space
-algebra (Boole) Class closed under complementation and denumerable union
We are interested in a class of events that:
1) Contains all possible results of the experiment on which we are interested
2) Is closed under union and complementation
BAABABAA 21121 ;,
BWhy algebra ?
;;;;0; 212121 BAABAABAABB
1) has all the elementary events (to which we shall assign probabilities)2) has all the events we are interested in (deduce their probabilities)B
So now:
654321 ,,,,, eeeeee
Several possible algebras:
A={get an even number}
Ā={get an odd number} AAEB ,,0,
0,min EB
of subsetspossibleallEB ,0,max
n)(dim
k
nSubsets with k elements n
n
k k
n2
0
elements
Minimal:
Maximal:
10
0
n 1:
n
n:0
EXAMPLE:
Interest in evenness:
Generalize the Boole algebra such that and can be performed infinite number of times resulting on events of the same class (closed)
BABA
iiii
11
)dim( denumerable
has structure of σ-algebraB
Remember that: 1) All -algebras are Boole algebras 2) Not all Boole algebras are -algebras
has the structure of Boole algebra )dim( finite B
,...1
BAi
i BABA c
In general, in non-denumerable topological spaces there are subsets that can not be considered as events
We are mainly interested innR
linear set of points
Among its possible subsets are the intervals
points
(degenerated interval) aaa ,
,Rany interval,
denumerable or not, is a subset of R
finite or denumerable of intervals is an interval
finite or denumerable ofintervals is not, in general, an interval
Which are the “elementary” events?
0,,, aaaaaa
)dim( non-denumerable
:R
…but…
Generate a σ-algebra, for instance, from half open intervals on the right
ba,
aaanbabanbababnabannn
,1,,1,,,1,111
Contains all half-open intervals on the right
2) Form the set A by adding their denumerable unions and complements
It has all intervals and points (degenerated intervals)
Its elements are Borel sets (borelians)
3.1) There is at least one σ-algebra containing A (add sets to close under denumerable union and complementation)
1) Initial Set (Ω):
RBQZN ,,Ω: intervals assigns P to intervals
3.2) The intersection of any number of σ-algebras is a σ-algebra There exists a smallest σ-algebra
containing ABorel σ-algebra : Minimum σ-algebra of subsets of generated by R)( RB ba,
],[),,(],,( bababa(May do with as well)
RBA :3.1) Measure Set function
i) -additive
),,( B3) Measure Space
11
)(i
ii
i AA 0;,,1,
21
jiji
ji AAAA
0)(: ABA ii) Non-negative
All Borel Sets of Rn are Lebesgue MeasurableThere are non-denumerable subsets of R with zero Lebesgue measure (Cantor Ternary Set)If axiom of choice (Solovay 1970) not all subsets of R are measurable are not Borel (Ex.: Vitali’s set)
1)( 3.2) Probability Measure
),,( PB►Probability Space
Remember that:
P(notation )
►Measure Space ),,( B
Any bounded measure can be converted in a probability measure
RBA 1,0:(certainty)
(univoque)
For any succession of disjoint of sets of B
),...(1)( AAc
Random variables
Associate to each elemental event of the sample space Ω one, andonly one, real number through a function
(misfortunately called “random variable”)
RwXwwX )(:)(
Results of the experiment notnecessarily numeric,…
Induced Space
(f(w):ΩΔ Borel measurable measurable wrt the σ-algebra associated to Ω)
),(),( II BB X
What is random is the outcome of the experiment before it is done; our knowledge on the result before observation,…
Is neither variable nor random),(),( RII BRBE
)( kXP )),(( baXP
Usually:
),,(),,( III PBPB
or
To keep the structure of the σ-algebra it isnecessary that X(w) be Lebesgue (…Borel)measurable
IBABAX ;)(1
RBABAX ;)(1
654321 ,,,,, eeeeee
A={get an even number}
Ā={get an odd number} AAEB ,,0,
EXAMPLE:
Interest in evenness:
RwXwwX )(:)(
1)()()( 321 eXeXeX
1)()()( 654 eXeXeX
],(1 aX
Ba 01Beeea },,{11 321
Ba 1
1)
1)()()( 531 eXeXeX
1)()()( 642 eXeXeX
Ba 01BAeeea },,{11 442
Ba 1
2)
Types of Random Quantities
Discrete Continuous
finite or denumerable set
),,( QBRwwX :)(
),,( PBR RAccording to the Range of X(w) …
non-denumerable set
R
A dPAXP )()()( 1
n
kAk k
xX1
)()( 1simple function:
ofpartitionfinitenkAk ,,1; ► simple random quantity:
RnkRxkX ,,1;
1
)()(k
Ak kxX 1elementary function:
► elementary random quantity:
ofpartitioncountablekAk ,1;
RkRxkX ,1;
setedenumerablnonRX
)()()()( dQAPxXPkAkk 1
)(X
R
A dPAXP )()()( 1
AA
dwfdP )()(
► absolutely continuous:
(Radon-Nikodym Theorem)
► singular
Radon-Nikodym Theorem (1913;1930)
),( Btwo σ-finite measures over the measurable space ,
AAA
wdwfwdd
dvwdvA )()()()(
If
BA
measurable function over with range in (non-negative)unique
)(wf B ,0
(if same properties as ))(wf)(wg 0)()(| xgxfx
Probability density function:
)(w Lebesgue measure
),,( QBRwwX :)(
AXBR
dxxpAQ)(1
)(
),,( PBR R
BAAAcontinuousabsolutely 00)(:
such that
thenwfif )(
0)( xPRxW
Each set Wk has, at most, k points for otherwise
If , then some
W if the denumerable union of finite sets
i
ik xP 1)(
Wx x
Wx
Set of points of R with finite probabilities
so if it is ∞-denumerable, it is not possible that all the points have the same probability
If X is AC 0)( aXP0])([ a but {X=a} is not an impossible result
Remember that:
1K
KWW partition of kW 1)(21
1 xPxW
21)(31
2 xPxW
kxPkxWK1)(1
1 …
…kW
kWxIf , then
The set of points of R with finite probabilities is denumerable
i
ixP 1)(
W is a denumerable set
DISTRIBUTION FUNCTION
Gen.Def: One-dimensional DF RRxF X :
RxxFxF ;)()(lim
0
1)(lim;0)(lim xFxF xx
)()0( xFxF 1)(;0)( FF
Distribution Function
1) Continuous on the right:
2) Monotonous non-decreasing:
)()( 21 xFxF
3) Limits:
2121, xxandRxxif
The Distribution Function of a Random Quantity has all the information needed to describe the properties of the random process.
wXDistribution Function of a Random Quantity
Def.- DF associated to the Random Quantity is the function
RxxXPxXPxF ;,)()(
X
)(11)( xFxXPxXP )()( xFxXP
Properties
)()(,)( 122121 xFxFxxXPxXxP
RxxXPxXPxF ;,)()(
Properties of the DF (some)DF defined
bx
axxF
1
0)( Rba ,
RxXIf takes values in
Set of points of discontinuity of the DF is finite or denumerable
)()( xFxF )()(/ xFxFRxD)()( xFxF
monótona crecienteQxrDx )( such that )()()( xFxrxF
2121 /, xxDxx )()()()( 2211 xrxFxFxr
associated is different for each xis a one-to-one relation
)(xr
monotonous non-decreasing
2)
1)
3)
QxrDx )(
)()()(lim0
xXPxFxF
has a jump of amplitude
)()(,)( xFxFxxXPxXxP
At each point of discontinuity …
)( xXP )(xF
Distribution Function Probability Measure
For each DF there exists a unique probability measure defined over Borel Sets that assigns the probability to each half-open interval
)()( 12 xFxF Rxx 21,
Reciprocally, to each probability measure defined on the measurable space , corresponds a DF
Random Quantitydiscrete
continuous Distribution Function
),( B
singular or absolutely continuous
Discrete Random Quantity XRwX :)(
RX Range of
),,( QB ),,( PBR R
:)(wX finite or
denumerable set
1k
kp
,..., 21 xxX takes values
with probabilities ,..., 21 pp
kp real, non-negative and
ii pxXP )(
1)(;0)( FF
k
kxk xpxXPxF )()()( ],(1DF:
1) Step-wise and monotonous non-decreasing
2) Constant everywhere but on points of discontinuity where it has a jump
kkkk pxXPxFxF )()()(
EXAMPLE:
1i
ip
kk kXPp2
1)(
,...2,1X
k
kxk xpxXPxF )()()( ],(1
kkkk pxXPxFxF )()()(
1)(;0)( FF
Continuous Random Quantity
)(xF continuous everywhere in R )()( xFxF )()()()( xFxXPxFxF
RwX :)(
RX Range of
),,( QB ),,( PBR R
:)(wX non-denumerable set
AC: Radon-Nikodym A AA
dwwpdd
dPdPAP )()(
Lebesgue measure
)(xp Probability Density Function
2) bounded in every bounded interval of and Riemann integrable on it
3)
Rxxp ;0)(1)
R
1)(
dxxp
x
duupxFxXP )()()(
dx
xdFxp
)()( uniquely a.e.
0
21
11)(
xxp
x
duupxF )()(
)arctan(1
2
1x
EXAMPLE:
EXAMPLE: )1,0(~ CsX
1 3n
nnX
X 2,0]supp[ nX
2
1)2()0( nn XPXP
Example
)()( xXPxF
x
duupxF )()(
General Distribution Function (Lebesgue Decomposition)
1)(|)(
dxxpxp
x
duupxF )()(
C SD N
j
N
k
Dkk
ACjj
N
i
Dii xFaxFbxFaxF
1 11
)()()()(
discretediscrete Abs. continuousAbs. continuous SingularSingular
almost everywhere
)(xF0)( xF
continuous
almost everywhere)()( xFxp
pdf:
(Dirac Delta, Cantor,…)
(Normal, Gamma,…)
(Poisson, Binomial,…)
)( nxXP 1)(
nnxXP
Step Function (simple or elementary) with denumerable number
of jumps
CONDITIONAL PROBABILITY and BAYES THEOREM
Given a probability space
● The information assigned to an event
depends on the information we have
Two consecutive extractions without replacement:
What is the probability to get a red ball in the second extraction?
1) I do not know the outcome of the first : P(r)=1/2
2) It was black: P(r)=2/3
All probabilities are conditional
),,( PBBA
Consider
and two not disjoint sets
BBE
)()()()( BAPBAPEAPAP
BBA, 0BA
A and B A and not B
What is the probability for A to happen
if we know that B has already occured?)( BAP
)()( BAPCBAP )()(1)( BPCBBPCBBP
Probability to happen
)(
),()(
BP
BAPBAP
)(1 BPC
Notation: ),,,()( CBAPCBAP
Conditional Probability
Statistical Independence
),(),( BAPBAP
),,( PB
0)( BP
Statistical Independence
)()( APBAP
)()(: APBAP Correlation
Generalization: ),,,( 21 nAAAP
!n possible arrangements
)(),,(),,(),,,( 311221 nnnn APAAAPAAAPAAAP
A Bdoes not depend on
)()(: APBAP )()( APBAP
That B has already happened does not change the probability of ocurrence of A
)(),,(),,( 3221 nnn APAAAPAAAP ),,(),,( 221 nn AAPAAAP
…should say ”inconditionally” independent
Caution !
BAAAA n ,,, 21 For a finite collection of n events
are independent iff:
)()(),( jiji APAPAAP
)()()(),,( kjikji APAPAPAAAP
)()(),,( mpmp APAPAAP
for each subset AAA mp ,,
kji nkji ,,1,, ji nji ,,1,
)()( APBAP A Bindependent of …
)()(),( CBPCAPCBAP It may happen that depends on through A B C
Conditional dependence
)()(),( BPAPBAP but
Theorem of
Total Probability
)()()()(
11k
n
kk
n
kBAPBAPAPAP
nkBk ,1,
0
jji
i BB j
n
jB
1
Partition of the Sample Space
n
kkk
n
kk BPBAPBAPAP
11
)()|(),()(
n
kkk
n
kk BPBAPBAP
11
)()()(
C
BCPBCAPBAP )(),()(
Theorem of Total Probability with Conditional Probabilities)()|(),|(),(),|(),,( BPBCPCBAPCBPCBAPCBAP
C
CBAPBAP ),,(),(
… to find out a method by which we might judge concerning the probability that an event has to happen, in given circumstances, upon supposition that we know nothing concerning it but that, under the same circumstances, it has happened a certain number of times, and failed a certain other number of times.
Common sense is indeed sufficient to shew us that, form the observation of what has in former instances been the consequence of a certain cause or action, one may make a judgement what is likely to be the consequence of it another time.
…some rule could be found, according to which we ought to estimate the chance that the probability for the happening of an event perfectly unknown, should lie between any two named degrees of probability, antecedently to any experiments made about it; …
Bayes Theorem
LII. An Essay towards solving a Problem in the Doctrine of Chances. By the late Rev. Mr. Bayes, communicated by Mr. Price, in a letter to John Canton, M. A. and F. R. S.
Dear Sir,Read Dec. 23, 1763. I now send you an essay which I have found among the papers of our deceased friend Mr. Bayes, and which, in my opinion, has great merit, and well deserves to be preserved. Experimental philosophy, you will find, is nearly interested in the subject of it; and on this account there seems to be particular reason for thinking that a communication of it to the Royal Society cannot be improper.
The Reverend Thomas Bayes, F.R.S. (1701?-1761)
)()()()(),( APABPBPBAPBAP
)(
)()()(
AP
HPHAPAHP ii
i
nkH k ,1, Partition of the Sample Space
ni ,1
Probability of occurrence of the event (cause, hypothesis) Hi “a priori”, before we know if event A has happened
Probability of occurrence of event A having occurred (cause, hypothesis) Hi
Probability (“a posteriori”) fo event Hi to happen having observed the occurrence of event (efect) A
Probability thatProbability that HHii be the cause (hypothesis) of the observed effect A be the cause (hypothesis) of the observed effect A
normalisation
)(
)()()(
AP
BPBAPABP
n
kkk
iii
HPHAP
HPHAPAHP
1
)()(
)()()(
n
kkk
ii
i
HPHHPHAP
HPHHPHHAPHAHP
100
000
0
)()()(
)()(),(),(
Other forms of
Bayes Theorem:
BT+ Total Probability Theorem
+ general hypothesis (H0) (all probabilities are condicional)
… and many more to come…
n
kkk HPHAPAP
1
)()()(
nkH k ,1, Partition of the Sample Space
),( 0HAP
),(),( 221121 xXxXPxxF
),( 21 xxp
2211 ),()( dxxxpxp
)()|()()|(),( 22111221 xpxxpxpxxpxxp
1212 ),()( dxxxpxp
)(
),()|(
1
2112 xp
xxpxxp
Def
)(
),()|(
2
2121 xp
xxpxxp
Def
)0)(( xp
)()|( 212 xpxxp
)()|( 121 xpxxp
Marginal and Conditional DensitiesMarginal and Conditional Densities
21
221121 ),(),(xx
dwwwpdwxxF
(Stieltjes-Lebesgue integral )
EXAMPLE: CAUSE-EFFECT
is sic:1H
:12 HH is healthy
Hypothesis o causes to analyze
exclusive y exhaustive
Dat
a
incidence in the population
“a priori” probabilities 1000
1)( 1 HP
1000
999)(1)( 12 HPHP
if T denotes the event
T={give positive in the test } 100
99)( 1 HTP
100
2)( 1 HTP
Certain disease occurs in 1 out of 1000 individuals
There is a diagnostic test such that:
If a person is sic, gives positive in the 99% of the cases
If a person is healthy, gives positive in 2% of the cases
A person has given positive in the test. What are the chances that he is sic?
A person has given positive.
What are the chances that he is sic? ???)( 1 THP
)(
)()()( 11
1 TP
HPHTPTHP
2
1
)()()(k
kk HPHTPTP
Bayes Theorem:Bayes Theorem:
Total Probability Theorem:Total Probability Theorem:
047.0
1000999
1002
10001
10099
10001
10099
)()()(
)()()(
1
111
n
kkk HPHTPTP
HPHTPTHP
The test is costly, agresive,… if a person gives positive… The test is costly, agresive,… if a person gives positive…
What are the chances that he is healty?What are the chances that he is healty?953.0)(1)( 11 THPTHP
The disease is serious… The disease is serious…
What are the chances to be sic giving negative in the test?What are the chances to be sic giving negative in the test?
511111 10
)(1
)())(1(
)(1
)()()(
TP
HPHTP
TP
HPHTPTHP
be sicbe sic:1H T={give positive in the test } T={give positive in the test }
99.0)( 1 HTP !!!!
xHP )( 1Incidence of the disease in the population
Probability to be sic giving positive )1()()(
)()(
11
11
xHTPxHTP
xHTPTHP
)1()()(
)1()()(
11
1
1xHTPxHTP
xHTPTHP
Probability to be healthy giving positive
Probability to be sic giving negative
)(1
))(1(
)(1
)()( 11
1 TP
xHTP
TP
xHTPTHP
Probabilities of interest as function of known data and incidence of the disease in the population
)1()()(1
))(1(
11
1
xHTPxHTP
xHTP
x
xTHP
101102)( 1
x
xTHP
972
)1(2)( 1
x
xTHP
972
99)( 1
From the given data:Interest to minimise )( 1 THP )( 1 THP
maximize )( 1 THP
)( 1HPx
optimum
eficiency
healthy giving positive
sic undetected
undergo costly and agresive treatment
correctly detected sic
incidence in the population
),( cHP
),( cHP ),( cHP
),( cHP
HH
Receiver Operating Characteristic
(ROC curve)
)max())(( 22 dxxfd
c
duufcFcHP )()(),( 11
c
duufcFcHP )()(),( 22
))(()()( 1212 xFFxfcFx
1
0
121
1
0
))(()( dxxFFdxxfA
For each cut value c
Tru
e po
siti
ve P
(+|H
)
)1,0( )1,1(
5.0A),( xx
)),(( xxf
optimum
random
u
dxxfduufduufuFA )()()()( 1221
)( HP False positives
)1,0(T
rue
posi
tives
Tru
e po
sitiv
es
)( HP
False PositivesFalse Positives
)( HP
Reverse selection criteria
)0,0(
STOCHASTIC CHARACTERISTICS
n
kAk k
xX1
)()( 1
),,( QBRX :)(
),,( PBR
1
)()(k
Ak kxX 1
R
Akk dPAPxXPk
)()()()( 1
AAR
A dwfdPdPAXP )()()()()( 1
k
kk xXPxg )()(
RR
dxxfxgxdFxg )()()()(
R
dPXgXgEYE )()]([)]([][
Def.: Expectation:Def.: Expectation:
)(X Discrete
)(X Absolutely continuous
)]([ XgY
As usual:
(Stieltjes-Lebesgue integral )
Mathematical ExpectationMathematical Expectation
R
dxxxpXE )(][
R
nnn dxxpxXE )(][
i
ii XccX 0 i
ii XEccXE ][][ 0
i
iXX i
iXEXE ][][
)()( 1 RLxpxn
nmn nmn 02 nif 10
niiX 1}{ independent
MomentsMoments (wrt origin)
Mean:Mean:
Linear operator
Rci ►
►
R
nn dxxpcxcXE )()(])[(
Rc
])[(min 2cXERc c
Moments Moments wrt point
…Moments wrt Mean
R
dxxpxXEXV )0()()(])[(][ 222 Variance:Variance:
R
nnn dxxpxXE )()(])[(
33
1 3
44
2
221
2][ XY cYV NOT Linear► XccY 10
Skewness: Kurtosis:
Moments wrt MeanMoments wrt Mean
01
11
limlim1
1
n
kk
k
k kka
a
Poisson!
)(k
ekXPk
!)(
00 kkekXPX
k
k
n
k
nn
n
)1(
1)(
2xxp
Cauchy
dxxpxn |)(| 1n
)1( nforCPV
No moments (no mean, no variance,…)
)()( 1 RLxpxn Watch!!Watch!!
Abs. Conv.
22
6)(
nnXP
,2,1n
1];[ kXE kk
,2,1,0k
Right
Dispersion: Variance
Position: Mean: , …
Mode: )(sup0 xpx x Median: 2/1)()( mm xXPxF qxXPxF )()(quantile:
2
0x 0
0
0
344
2
Asymmetry: Skewness 33
1 Symmetric
Left
0
0
0
Peaked: Kurtosis Normal
01 Mode < Median < Mean 01 Mode > Median > Mean
Global PictureGlobal Picture
][XE
mx
2121221121 ][)])([(],[ XXEXXEXXV
},{ 21 XX independent 0],[ 21 XXV
1],[
121
2112
XXV
Covariance Covariance (and “Linear Correlation” )(and “Linear Correlation” )
Linear relation: baXX 12112
212 cXaX Quadratic:
Comments
0,2
1211 thenisXforif
1|| 12 Holder inequality:
)()()(),(),( 222
),(211
),(12121
2121
ijDOxx
gx
x
ggXXgY
)(),()],([][ 22121 ijDOgXXgEYE
)()(][ 22
),(211
),(12121
xx
gx
x
gYEY
][2][][ 21
),(212
2
21
2
121)2,1()2,1(
XXVx
g
x
gXV
x
gXV
x
g
1221
),(21
22
2
2
21
2
121)2,1()2,1(
2
x
g
x
g
x
g
x
g
22][][ YYEYEYV
),,( 21 XXgY Taylor Expansion for the Variance of
(mind for the re-maind-er…)
Useful expression:
iiXE )(
INTEGRAL TRANSFORMS
Fourier (… Laplace) TransformMellin Transform
CRf :
dxxfet ixt )()()(1 RLf Rt
Properties:
Fourier Transform (Characteristic Function)
Probability Density…
][)( ixteEt
1)( t
)()( tt
1)0( ►
► bounded
►Schwarz symmetry
►Uniformly continuous in R
Exists for all
Inversion Theorem (Lévy,1925)
dttexp itx )(2
1)(
dttexXp kitxk )(
2
1)(
b
b
itxk dtte
bxXp k
/
/
)(2
)(
Discrete:
Reticular: bnaxk ZnbRba ,0,,
► One-to-one correspondence between DF and CF► Two DF with same CF are the same a.e.
CRt :
)()(|,0 tt
(all necessary but not sufficient)
)(X
niiii xpX 1)}(~{
nXXX 1
If are n independent random quantities
)()(][)( 1)( 1 tteEt n
XXitX
n
21 XXX )()()()()( 2121 tttttX
][][)()( )( tXigiYtY eEeEtXgY
bXaY Rba ,
)()( btet Xiat
Y
Useful Relations:
►
►
If distribution of is symmetric, then is a real function X )(tX
)()()()( tttt XXXX
►
)|(~ 111 nPoX
)|(~ 222 nPoX21 XXX
)1()(it
i ei et
C
zzw
n
n
dzezi
enXP
S )/1(2)1(
2/
2
1
2)(
Pole of order n+1 at z=0
0
2/21
)1()1(
)(]0),({Re
p
pn
pnpzzfs
)2()( 21)(
2/
2
1 21
n
n
IenXP
)()(21
2121)()()(itit ee
X eettt
X: Discrete reticular: a=0, b=1
],(;:
2/1
2
1
zC
c
Example
)|(~ kkk xPoX
),|(~ kkkk xNX
),|(~ kkkk baxCaX
),|(~ kkk baxGaX
)|(~ SxPoX
),|(~ SSxNX
nXXX 1
nS 1
nS 1
221
2nS
nS aaa 1
nS bbb 1
),|(~ SS baxCaX
),|(~ SbaxGaXnS bbb 1
Some Useful Cases:
)1,0(~ CsX
][)( iXteEt0
)()(][
t
k
kkk t
tiXE
itiXt eeEt 212
1][)(
1 3n
nnX
X
2,0]supp[ nX
2
1][
2)0()1 XE
i
8
1
8
3][
8
3)0( 22)2 XE
2
1)2()0( nn XPXP
Moments of a Distribution (F-LT usually called “moment generating functions)
][),,( )(1
111 ntXtXin eEtt
0,,
1
1
),,()(][
n
j
j
i
ijiji
tt
n
jk
k
ik
kkkk
jki tt
ttiXXE
Example
CRf :
0
1)();( dxxxfsfM s
)(1 RLf
Cs Obviously, if exists …
)()(lim0
xOxfx
)()(lim xOxfx
)Re(s
Strip of holomorphyProbability Densities…
i
i
sdsxsfMi
xf
),(
2
1)(
,
,)Re(
][);( 1 sXEsfM
Mellin Transform
Re
Im
,);( sfM
xexf )(1 1)(2 xexf
)();( 2,1 ssfM ,0 0,1
0 1 0
baXY 0,, aRba
)1()( 1 bbsMasM Xs
Y
1XY1,1 ba )2()( sMsM XY
niiii xpX 1)}(~{
),0[ ix
nXXXX 21independent
Non- negative
121 XXX
)()()( 1 sMsMsM nX
)2()()( 21 sMsMsM X
Useful Relations:
►
►
c
)|(~ 222 axExX
)|(~ 111 axExX1
)()(
s
i
ia
ssM21XXX
121
2
)(
)()(
sX aa
ssM
ic
ic
z dzzxaai
aaxp 2
2121 )()(
2)(
,0
0c real
)2(2)( 21021 xaaKaaxp
Newman series…
)ln()1(2)1(
)(]),({Re 212
21 xaann
xaanzzfs
n
n
Example
)|(~ 222 axExX
)|(~ 111 axExX
21XXX
)2(2~ 21021 xaaKaa
121 XXX
21221 )(~ xaaaa
),|(~ 1111 baxGaX
),|(~ 2222 baxGaX)2(
)()(
2~ 21
12/)(2
1
2
21
21 21
21
xaaKxa
a
bb
aav
bbbb
012 bb21
121
)()()(
)(~
21
121
21
21bb
bbb
xaa
xaa
bb
bb
121 a
|)|(~ 021 xaKa
XXX
),0|(~ 111 xNX
),0|(~ 222 xNX
…Densities with support in R…
Some Useful Cases:
0
21 )()()( dwwpxwpxp
0
211)/()()( dwwwxpwpxpobviously…
DISTRIBUTIONS AND GENERALIZED FUNCTIONS
CTDxT ,)(:
DT
cCD `
2121 ,,, TTT
,, TT nn
nn}{
RRf : Locally Lebesgue integrable (LLI) defines a distribution
dxxxfT f )()(,
“regular” distributions (“singular” the rest)
is a Distribution
Linear functional:
Continuous functional:
Distribution Functional:
C ,
,,)(
GTGTae
DGT DGT , C ,
0,0suppsupp TT ,,; TTDD
,, TT ppp DTTD ,)1(,
,,}{ TTiffTT nn
nn
)(,),(, axTaxTTSa
)(,||
1),(, a
xTa
axTTPa
Some Basic Properties (see text for more)
~,,
~TT Fourier Transform
)()(,,, aaxS aa
)0(||
1)(,
||
1,,
aax
aP axa
)(2 ]/1,/1[ xn
f nnn 1
0|)0()0(|lim|,,|lim nnnn
212121 ,,)0()0(, )0(,
)0(,,
,, nnT
)()( ),0[ xxH 1
0
),0[ )()()(, dxxdxxxTH 1
,)0(,, HH
Two examples
LLI: defines a distribution
Tempered Distributions cCD `
“rapidly decreasing” (Schwartz Space)
xexf )( RxRx )()( ),0( xexf x
1
SS
)(1 RLf R
mC
xa
xf
)(
)(2 0Nmfor some
defines a Tempered Distribution fT
)(2 ]/1,/1[ xn
f nnn 1 0|,,| nnT
0C 1C admissible admissible
….
EXAMPLE:
andCCRS |:{ },0||lim 0||
NmnDx nm
x
EXAMPLE:
EXAMPLE:
Sx nn )}({
0,0|)(|max NmkxDx nn
mk
Rx
0)(, nn xT
Convergence to zero
T
),,( QB RX :)(
]),(()()( 1 xXQxXPxF LLI: defines a distribution
,FT
Probability Density “Distribution” ,,, FF TTT
T does not have to be generated by a LLI function
In general In general T is a Probability Distribution if:T is a Probability Distribution if: 00, T
11, T
…but any LLI function defines a probability distribution if
0)(0)()(, xdxxxfT f
1)(1, dxxfT f
,,1)( SDx
Probability Distributions
0
)(!
)(n
nn itn
t
0
)(2
1
!)(
2
1)(
n
nitxnitx dtiten
dttexp
0
) )0,(!
)1(n
nnnp x
nT dtex itx
2
1)0,(
dxxpet itx )()(
delta distribution
0
)
0
) )0(!
),0,(!
)1(,n
nn
n
nnnp n
xn
T
pDPROB TTT )1(
)(X ,,)( 21 xxXrec )( kk xXPp
1
)(k
kkD xpT
RXrec )( )()( xxp RA1
Delta Distribution unifies discrete and AC random quantities:
Discrete:
Continuous:
CF:
)()()()()()()()()( 221 bxHxFbxHaxHaFaxHxHxFxF
)()()()()()()()0()0( ),[2),0[1121 xxfxxfaaFaFFDf ba 11
´,,, FF TDfT
b
ma
mmm
mm dxxxfdxxxfaaFaFFXDfXE )()()()()0(, 2
0
11201
b
a
dxxfdxxfaFaFF )()()()()0(1 2
0
1121
Example
)(1 aF
)(2 aF
)0(1F
1
)(1 xF
)(2 xF
ba0
LIMIT THEOREMS and
CONVERGENCE
Find the limit behaviour of a sequence of random quantities
convergence criteria
General Problem:
,...,...,, 21 nXXX
,...1
,...,2
,1
21211
n
kkn X
nZ
XXZXZ
Example:
How is distributed when ?nZ )( n
~“distance”
1) More or less strong,2) May have convergence for some criteria and not for others
Convergence in:
Distribution
Probability
Norm)(RLp
Almost Sure
Uniform
Central Limit Theorem
Weak Law of Large Numbers
Strong Law of Lage Numbers
Glivenko-Cantelli Theorem
Convergence in Quadratic Mean
Glivenko-Cantelli Theorem
Logarithmic
Chebyshev Theorem
X 2, with finite mean and variance
2
1
kkXP
k
XgEkXgP
)]([)(
)(~ xFX 0)( XgY
?kYP
kXgX )(1 kXgX )(2 21 X
21
)()()()(][ xdFxgxdFxgYE
0
0)( Xg
2
))(()( 2 kXgkPXkPxkdF
kXg )(
2)()( XXg
Bienaymé-Chebyshev Inequality
Let be r.q. with the same distribution and finite mean
,...,...,, 21 nXXX ),...(),...,(),( 2211 nn xFxFxF
nX X if, and only if
0;0)(lim XxXP nn
o, equivalently,
converges in probability to
Convergence in Probability
Let
Def.:
0;1)(lim XxXP nn
and
,...,...,, 21 nXXX Weak Law of Large Numbers (J. Bernouilli…)
0;0lim nn ZP
lim(Prob)
,...1
,...,2
,1
21211
n
kkn X
nZ
XXZXZThe sequence converges in Probability to
LLN in practice:…
WLLN: When n is very large, the probability that Zn differs from μ by a small amount is very small
Zn gets more and more concentrated around the real number μ
But “very small “ is not zero: it may happen that for some k>n, Zk differs from μ by more than ε…
,...,...,, 21 nXXX
nX X if, and only if
0;0)(lim XxXP nn
o, equivalently,
converges “almost sure” to
Convergence Almost Sure Let
Def.:
0;1)(lim XxXP nn
Strong Law of Large Numbers (E.Borel, A.N. Kolmogorov,…)
0;0)(lim xZP nn
Prob(lim)
Let be r.q. with the same distribution and finite mean ,...,...,, 21 nXXX
,...1
,...,2
,1
21211
n
kkn X
nZ
XXZXZThe sequence converges Amost Sure to
LLN in practice:…
WLLN: When n is very large, the probability that Zn differs from μ by a small amount is very small
Zn gets more and more concentrated around the real number μ
But “very small “ is not zero: it may happen that for some k>n, Zk differs from μ by more than ε…SLLN: as n grows, the probability for this to happen tends to zero
,...,...,, 21 nXXX ),...(),...,(),( 2211 nn xFxFxF
nX )(~ xFX if, and only if
)(;)()(lim)()(lim FCxxXPxXPxFxF nnnn
o, equivalently,
tends to be distributed as
Rtxxnn ;)()(lim
Convergence in DistributionLet
Def.:
and their corresponding
,...,...,, 21 nXXX
,...1
,...,2
,1
21211
n
kkn X
nZ
XXZXZ
Sequence of independent r.q.
How is distributed when ?nZ )( n
same distribution
2,finite mean and variance
Form the sequence
nzNX
nZ
n
kkn
,~1
1
1,0~~
zN
n
ZZ n
n
Central Limit Theorem (Lindberg-Levy,…)
easy dem. from CF
standarized
n
kkn X
nZ
1
1 1n 2n
5n 10n
20n 50n
Unifom DF
Parabolic DF
n
kkn X
nZ
1
1
1n 2n
5n 10n
20n 50n
Cauchy DF
n
kkn X
nZ
1
1
1n 2n
5n 10n
20n 50n
,...,...,, 21 nxxx
Glivenko-Cantelli Theorem
If observations are iid: 10)()(suplimlim xFxFP nxnn
The Empiric Distribution Function converges uniformly to the Distribution Function of the r.q. X
)(ne
)1(e one observation of X }{ 1x
n
kkxn xI
nxF
1],( )(
1)(
number of values xi lower or equal to x
n
Empiric Distribution Function
)(xF
Uniform Convergence
(example of aplication of Chebishev Th.)
)(xf
0)()(suplim
xfxfnSxxn
converges uniformly to 1)( nn xfThe sequence if, and only ifDef.:
experiment
independent, identically distributed
RSffn :,
],( 0xA )(xIY A is a Bernouilli random quantity )()()1( 0xFAXPYP
)(1)()0( 0xFAXPYP )(1)()( 00 xFxFet it
Y
n
knkxn xnFxIZ
1],( )()(
)(xnFZW nn knk FFk
nkWP
)1()(
nFWE
)1( FnFWV
2
)1()()(
n
FFxFxFP n
Demonstration of Convergence in Probability:
n
kkxn xI
nxF
1],( )(
1)(Empiric Distribution Function
with Characteristic Function
1)
2) Sum of iid Bernouilli random quantities
2
1
kkXP
3) Bienaymé-Chebyshev Inequality
is a Binomial r.q.
FFE n
n
FFFV n
)1(
n
WxFn )(
Convergence in Lp(R) Norm
1)( nn wX )(wX
if, and only if,
Converges in norm to Def.:
p=2 convergence in quadratic mean
)(RLp
nRLwX pn ;)()( andRLwX p )()( 0)()(lim p
nn wXwXE
p
n wXwXEnnn )()()(|)(0 00that is:
Logarithmic Divergence of a pdf Xp xx ;)(~
from its true pdf )(xp
Kullback-Leibler “Discrepancy” (see Lect. on Information)
xx
xx d
p
ppppD
X
KL )(~)(
log)(]|~[
1)( iip xA sequence of pdf “Converges Logarithmically” to a density iff)(xp
0]|[lim kKLk ppD
Logarithmic Convergence