![Page 1: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/1.jpg)
ERG2310A-I p. I-81
Probability and Random Variables
Concept of Probability:
When the outcome of an event is not always the same, probability is the measure of the chance of obtaining a particular possible outcome
outcomeslikely equally possible ofnumber totaloutcomesfavourablepossibleofnumber
outcomes favourable =)(P
NN
AP A
N ∞→= lim)( Where N is total number of event occurrence,
NA is the number of occurrence of outcome A
e.g. dice tossing: P2 = 1/6 ; P2 or 4 or 6 = 1/2
![Page 2: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/2.jpg)
ERG2310A-I p. I-82
Common Properties of Probability
• 0≤ P(A) ≤ 1
• If there are N possible outcomes A1 , A2 , … , AN then
• Conditional Probability:
probability of the outcome of an event is conditional on the outcome of another event
∑=
=N
iiAP
11)(
P(A)B)andP(AA)|P(B =
P(B)B)andP(AB)|P(A =;
P(B)A)P(BAPB)P(A |)(| =Bayes’ theorem
![Page 3: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/3.jpg)
ERG2310A-I p. I-83
Common Properties of Probability
• Mutually exclusiveness
P(A or B) = P(A) + P(B); P(A and B) = 0
• Statistically Independence
P(B)P(A) B)and P(A(B)| P(A (A)|P(B
⋅=⇒== );) APBP
P(B) B)andP(AB)| P(A;
P(A)B)andP(AA)|P(B ==Q
Thus, A and B are statistically independent.
Thus, A and B are mutually exclusive.
![Page 4: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/4.jpg)
ERG2310A-I p. I-84
Communication Example
In a communication channel, signal may be corrupted by noises.
m0
m1
r0
r1
P(r0|m0)
P(r0|m1)P(r1|m0)
P(r1|m1)
If r0 is received, m0 should be chosen ifP(m0|r0)P(r0) > P(m1|r0)P(r0)
By Bayes’ theorem P(r0|m0)P(m0) > P(r0|m1)P(m1)
Similarly, if r1 is received, m1 should be chosen ifP(m1|r1)P(r1) > P(m0|r1)P(r1) ⇒ P(r1|m1)P(m1) > P(r1|m0)P(m0)
Probability of correct reception: P(c) = P(ro|mo)P(mo) + P(r1|m1)P(m1)
Probability of error: P(ε) = 1-P(c)
![Page 5: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/5.jpg)
ERG2310A-I p. I-85
Random Variables
A random variable X(.) is the rule or functional relationship which assigns real numbers X(λi) to each possible outcome λi in an experiment.
For example, in coin tossing, we can assign X(head) = 1, X(tail) = -1
If X(λ) assumes a finite number of distinct values
discrete random variable
If X(λ) assumes any values within an interval
continuous random variable
![Page 6: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/6.jpg)
ERG2310A-I p. I-86
Cumulative Distribution Function
The cumulative distribution function, FX(x) , associated with a random variable X is:
)( xXPxFX ≤=Properties:
•
•
•
•
1)(0 ≤≤ xFX1)(;0)( =∞=−∞ FF
2121 )()( xxxFxF ≤≤ if
)()( 1221 xFxFxXxP XX −=≤<
FX(x)
x0
1
FX(x)
x0
1
(Non-decreasing)
![Page 7: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/7.jpg)
ERG2310A-I p. I-87
Probability Density Function
The probability density function, fX(x) , associated with a random variable X is:
dxxdF
xf XX
)()( =
∫∞−
==≤x
XX dfxFxXP ββ )()(
∫∞
∞−
= 1)( dxxf X
∫=−=≤<2
1
)()()( 1221
x
xXXX dxxfxFxFxXxP
0)( ≥xf X
Properties:
•
•
•
•
FX(x)
x0
1
fX(x)
x0
for all x
![Page 8: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/8.jpg)
ERG2310A-I p. I-88
Statistical Averages of Random Variables
The statistical average or expected value of a random variable X is defined as
Xi
ii mxPxXE == ∑ )(
EX is called the first moment of X and mX is the average or mean value of X.
Similarly, the second moment EX2 is
∑=i
ii xPxXE )( 22
Its square root is called the root-mean-square (rms) value of X.
XmdxxxfXE == ∫∞
∞−
)(
∫∞
∞−
= dxxfxXE )( 22
or
or
The variance of the random variable X is defined as
222222 )()()( XXXXX mXEdxxfmXmXE −=−=−= ∫∞
∞−
σσ or
The square root of the variance is called the standard deviation, σX, of the random variable X.
![Page 9: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/9.jpg)
ERG2310A-I p. I-89
Statistical Averages of Random Variables
∑∑==
=
N
iii
N
iii XEaXaE
11
∑∑==
=
N
iii
N
iii XVaraXaVar
1
2
1
Expected value of linear combination of N random variables is equivalent to linear combination of expected values of individual random variables
For N statistically independent random variables: X1, X2, … , XN
Covariance of a pair of random variables: X, Y
YXYXXY mmXYEmYmXE −=−−= ))((µ
If X and Y are statistically independent, µXY=0
![Page 10: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/10.jpg)
ERG2310A-I p. I-90
A random variable that is equally likely to take on any value within a given range is said to be uniformly distributed.
Uniform Distribution
![Page 11: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/11.jpg)
ERG2310A-I p. I-91
Consider an experiment having only two possible outcomes, A and B, which are mutually exclusive.
Let the probabilities be P(A) = p and P(B) = 1 − p = q.
The experiment is repeated n times and the probability of A occurring itimes is ,)( ini qp
in
iAP −
== where (binomial coefficient).
)!(!!ini
nin
−=
Binomial Distribution
The mean value of the binomial distribution is np and the variance is (npq).
![Page 12: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/12.jpg)
ERG2310A-I p. I-92
Poisson Distribution
It describes the number of events that occur in an interval of timegiven that the average rate of occurrence is known.
λλλ ==== − XXEi
eiXPi
!
)( where
where λ is the average value of the rate of occurrence of the desired event.The mean value of the Poisson distribution is λ and the variance is λ.Example: The probability of error on a single transmission in a digital communication
system is Pe=10-4. What is the probability of more than three errors in 1000 transmission?
We find the probability of three errors or less is
1.0)1000(!
)3(3
0====≤ ∑
=
−e
i
i
PXi
eXP λλλ where
( ) ( ) ( ) ( ) 999996.0!31.0
!21.0
!11.0
!01.0)3(
32101.0 ≅
+++=≤ −eXP
6104313 −≅≤−=> xKPXP
![Page 13: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/13.jpg)
ERG2310A-I p. I-93
Gaussian Distribution
Central-Limit theorem: The sum of N independent, identically distributedrandom variables approaches a Gaussian distribution when N is very large.
22 2/)(
21)( σµ
σπ−−= xexf
The Gaussian pdf is continuous and is defined by
where µ is the mean , and σ2 is the variance .
cumulative distribution function:
∫∞−
−−=
≤=x
y
X
dye
xXPxF22 2/)(
21
)(
σµ
σπ
![Page 14: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/14.jpg)
ERG2310A-I p. I-94
Gaussian Distribution
Zero-mean unit-variance Gaussian random variable: 2/2
21)( xexg −=π
⇒ Probability distribution function: ∫∫∞−
−
∞−
==Ωx
yx
dyedyygx 2/2
21)()(π
Define Q-function: ∫∞
−=Ω−=x
y dyexxQ 2/2
21)(1)(π
(monotonic decreasing)
In general, for a random variable X with pdf:22 2/)(
21)( σµ
σπ−−= xexf
−
=>
−
Ω=≤σ
µσ
µ xQxXPxxXP )()( ;
Define: error function (erf) and complementary error function (erfc) :
=
+=Ω
221
21)( xerfcxerfx
21Q(x) ;
∫∫∞
−− =−==x
yx
y dyexerfxerfcdyexerf22 2)(1)(2)(
0 ππ ;
Thus,
![Page 15: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/15.jpg)
ERG2310A-I p. I-95
Q-function
∫∞
−=x
y dyexQ 2/2
21)(π
1for x >>≅−
π2)(
2/2
xexQ
x
![Page 16: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/16.jpg)
ERG2310A-I p. I-96
Random Processes
A random process is a set of indexed random variables (sample functions) defined in the same probability space.
In communications, the index is usually in terms of time.
xi(t) is called a sample functionof the sample space.
The set of all possible sample functions xi(t) is called ensemble and defines the random process X(t).
For a specific i, xi(t) is a time function.For a specific ti, X(ti) denotes a random variable.
![Page 17: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/17.jpg)
ERG2310A-I p. I-97
Random Processes: Properties
Consider a random process X(t) , let X(tk) denote the random variable obtained by observing the process X(t) at time tk .
Mean: mX(tk) =EX(tk)
Variance:σX2 (tk) =EX2(tk)-[mX (tk)] 2
Autocorrelation: RXtk ,tj=EX(tk)X(tj) for any tk and tj
Autocovariance: CXtk ,tj=E [X(tk)- mX (tk)][X(tj)- mX (tj)]
![Page 18: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/18.jpg)
ERG2310A-I p. I-98
Ensemble Average
The statistical average that is determined from the measurements
made at some fixed time t = ti on all sample functions.
Time Average
The statistical average that is determined from the measurementsmade on one sample function for a period of time.
Ensemble Average and Time Average
![Page 19: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/19.jpg)
ERG2310A-I p. I-99
Stationarity and Ergodicity
Stationary Random Process
Ensemble average (mean, variance) does not change with time (constant).
Ergodic Random process
Ensemble average and time average are equal.
An ergodic random process is stationary but a stationary random process is not necessarily ergodic.
The statistical properties can be determined from one sample function in the ensemble
Correlation and covariance depend only on time difference.
![Page 20: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/20.jpg)
ERG2310A-I p. I-100
Time varying mean
Time varying variance
Stationary
Stationary Random Process
![Page 21: Probability and Random Variableswebpages.eng.wayne.edu/ece4700/Lecture Notes/lecture1-3.pdf · ERG2310A-I p. I-88 Statistical Averages of Random Variables The statistical average](https://reader030.vdocuments.site/reader030/viewer/2022011919/60173540b552967ac075e93a/html5/thumbnails/21.jpg)
ERG2310A-I p. I-101
Stationarity and Ergodicity: Example
Consider the random process with sample function )cos()( Θ+= tAtn oωwhere ωo is constant and is a random variable with pdf
otherwisef
πθπθ≤
=Θ 02
1)(Θ
Compute statistical ensemble averages:
22)(cos)()(cos)(
02
)cos()()cos()(
222222 AdtAdftAtn
dtAdftAtn
oo
oo
∫∫
∫∫
−
∞
∞−Θ
−
∞
∞−Θ
=+=+=
=+=+=
π
π
π
π
πθθωθθθω
πθθωθθθω i.e. mean =0
i.e. variance=A2/2
Compute time averages:
∫
∫
−∞→
−∞→
=+=
=+=
T
ToT
T
ToT
AdttAT
tn
dttAT
tn
2)(cos
21lim)(
0)cos(21lim)(
2222 θω
θω i.e. mean =0
i.e. variance=A2/2
As ensemble average=time average, the random process is ergodic.
otherwisef 4
0
2)(
πθπθ ≤
=Θ If then tAAtntAtn oo ωπ
ωπ
2cos2
)(cos22)(22
2 +== and
Time dependent ! ⇒ Not stationary !