lecture01 intro probability theory
TRANSCRIPT
Copyright © Syed Ali Khayam 2009
CSE-801: Stochastic Systems
Introduction to Probability Theory
Syed Ali Khayam
School of Electrical Engineering & Computer Science
National University of Sciences & Technology (NUST)
Pakistan
Copyright © Syed Ali Khayam 2008
Course Information
Lecture Timings: Wednesdays: 5:30pm-7:20pm
Fridays: 5:30pm-6:20pm
Office Hours Wednesdays: 4:00pm-5:30pm
Fridays: 4:00pm-5:30pm
Office is located on top floor, last room on your left in the north wing
The course will be managed through Moodle: lms.nseecs.edu.pk
Course password: given in class
2
Copyright © Syed Ali Khayam 2008
Textbook
3
Copyright © Syed Ali Khayam 2008
Course Outline
Syllabus Introduction to Probability Theory Functions of Random Variables Limits and Inequalities Stochastic Processes Prediction and Estimation Markov Chains and Processes (time permitting) Assorted Topics (time permitting)
Grading Final Exam: 40% Midterm Exam: 30% Quizzes: 20% Homework Assignments: 10%
4
Copyright © Syed Ali Khayam 2008
Course Outline
Grading Final Exam: 40% Midterm Exam: 30% Quizzes: 20% Homework Assignments: 10%
Lot’s of extra credit for extra effort We will also have a voluntary user study as well Anyone volunteering and seeing it through will get 10 extra credit points
5
Copyright © Syed Ali Khayam 2008
Policies
Quizzes are announced and will take place at the start of Friday classes
Exams will be closed book, but you will be allowed to bring an A4-sized cheat sheet to the exam
Late homeworks submissions will not be accepted
Strongest possible disciplinary action will be taken in case of plagiarism or cheating in exams, homeworks or quizzes
It is mandatory to maintain at least 75% class attendance to sit in the Final Test
6
Copyright © Syed Ali Khayam 2008
Credits and Acknowledgements
I would like to thank Dr. Garcia for providing online lecture notes on the textbook’s website
Throughout this course, I will be borrowing examples and explanations from the Stochastic Systems Course taught by Professor Hayder Radha at Michigan State University
7
Copyright © Syed Ali Khayam 2008
Why this course?
Stochastic theory is an extension of probability theory
This course on Stochastic theory will teach mathematical tools that are commonly-used in a variety of engineering, computer science and IT disciplines
We will focus solely on performance modeling of phenomena observed in communications engineering
Applications and examples will be provided as required
8
Copyright © Syed Ali Khayam 2008
What will we cover in this lecture?
This lecture is intended to be an introduction to elementary probability theory
We will cover: Random Experiments and Random Variables
Axioms of Probability
Mutual Exclusivity
Conditional Probability
Independence
Law of Total Probability
Bayes’ Theorem
9
Copyright © Syed Ali Khayam 2008
Definition of Probability
Probability: [m-w.org]1 : the quality or state of being probable
2 : something (as an event or circumstance) that is probable3 a (1) : the ratio of the number of outcomes in an exhaustive set of equally likely outcomes that produce a given event to the total number of possible outcomes (2) : the chance that a given event will occur b : a branch of mathematics concerned with the study of probabilities4 : a logical relation between statements such that evidence confirming one confirms the other to some degree
10
Copyright © Syed Ali Khayam 2008
Definition of Probability
Do you know which famous person was so opposed to probability theory that he said:
“God does not play dice with the universe.”?
11
Copyright © Syed Ali Khayam 2008
Definition of Probability
And do you know which famous person said:
“God does play dice with the universe. All the evidence points to him being an inveterate gambler, who throws the dice on every possible occasion.”?
12
Copyright © Syed Ali Khayam 2008
Definition of a Random Experiment
A random experiment comprises of: A procedure
An outcome
Procedure
(e.g., flipping a coin)
Outcome
(e.g., the value
observed [head, tail] after
flipping the coin)
Sample Space
(Set of All Possible
Outcomes)
13
Copyright © Syed Ali Khayam 2008
Definition of a Random Experiment: Outcomes, Events and the Sample Space
An outcome cannot be further decomposed into other outcomes
{s1 = the value 1}, …, {s6 = the value 6}
An event is a set of outcomes that are of interest to usA = {s: such that s is an even number}
The set of all possible outcomes, S, is called the sample space
S = {s1, s2, s3, s4, s5, s6}
outcome event sample space
14
Copyright © Syed Ali Khayam 2008
Definition of a Random Experiment: Outcomes, Events and the Sample Space
s1
s2
s3
s4
s5
s6
S
15
Copyright © Syed Ali Khayam 2008
Definition of a Random Experiment: Outcomes, Events and the Sample Space
Example of a Random Experiment: Experiment: Roll a fair die once and record the number of dots on the
top face
S = {1, 2, 3, 4, 5, 6}
A = “the outcome is even” = {2, 4, 6}
B = “the outcome is greater than 4” = {5, 6}
16
Copyright © Syed Ali Khayam 2008
Axioms of Probability Probability of any event A is non-negative:
Pr{A} ≥ 0
The probability that an outcome belongs to the sample space is 1:
Pr{S} = 1
The probability of the union of mutually exclusive events is equal to the sum of their probabilities:
If A1 ∩ A2=Ø, => Pr{A1 U A2} = Pr{A1} + Pr{A2}
17
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity
For mutually exclusive events A1, A2,…, AN, we have:
1 1
Pr PrNN
i ii i
A A
s1
s2
s3
s4
s5
s6
S
A1
A2
Find Pr{A1 U A2}
and Pr{A1}+Pr{A2}
in the fair die
example
18
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity
In general, we have:
Pr{A1 U A2} = Pr{A1} + Pr{A2} – Pr{A1 ∩ A2}
s1
s2
s3
s4
s5
s6
S
19
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity: Example
Experiment: Roll a fair dice twice and record the number of dots on the top face:
S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),
(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),
(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }
20
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity: Example
Find the probability of the following events:A1 = “first roll gives an odd number”
A2 = “second roll gives an odd number”
C = “the sum of the two rolls is odd”
21
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity: Example
Define three events:A1 = “first roll gives an odd number”
A2 = “second roll gives an odd number”
C = “the sum of the two rolls is odd”
Find the probability of C using probability of A1 and A2
22
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity: Example
A1
A2
23
S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),
(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),
(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),
(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity: Example
Pr{A1} = “first roll gives an odd number” = 18/36 = 1/2
Pr{A2} = “second roll gives an odd number” = 18/36 = 1/2
C = “the sum of the two rolls is odd”
24
Copyright © Syed Ali Khayam 2008
Mutual Exclusivity: Example
C = “the sum of the two rolls is odd”
Let
C1 = “first roll is odd and second is even”=
C2 = “first roll is even and second is odd”=
Since C1 and C2 are mutually exclusive:
21A A
1 2A A
2 11 2C A A A A
1 2 1 2
2 11 2
1 2 1 2
Pr Pr Pr{ } Pr{ }
Pr Pr Pr Pr
Pr 1 Pr 1 Pr Pr
12
C C C C C
A A A A
A A A A
25
Copyright © Syed Ali Khayam 2008
Conditional Probability
Given that event B has already occurred, what is the probability that event A will occur?
Given that event B has already occurred, reduces the sample space of A
s1
s2
s3
s4
s5
s6
S
s1
s2
s3
s4
s5
s6
Event B has
already occurred
=> s2, s4, s3
cannot occur
S
26
Copyright © Syed Ali Khayam 2008
Conditional Probability Given that event B has already occurred, we define a new
conditional sample space that only contains B’s outcomes
The new event space for A is the intersection of A and B: EA|B = A ∩ B
s1
s2
s3
s4
s5
s6
S
s1
s2
s3
s4
s5
s6
Event B has
already
occurred
S
What’s missing here?S|B = {s1, s5, s6}
EA|B= A ∩ B = {s6}
27
Copyright © Syed Ali Khayam 2008
Conditional Probability Consider that the example below corresponds to an experiment
where we throw a fair dice and record the number of dots on its face
For this experiment, what is Pr {A|B} in the example below?Pr{A|B} = Pr{s6|B}= 1/3
We need to normalize all probabilities in a conditional sample space with Pr{B}
s1
s2
s3
s4
s5
s6
S
s1
s2
s3
s4
s5
s6
Event B has
already
occurred
S
S|B = {s1, s5, s6} 28
Copyright © Syed Ali Khayam 2008
Conditional Probability We need to normalize all probabilities in a conditional sample
space with Pr{B}Pr{s1|B} = Pr{s1}/Pr{B} = (1/6)/(1/2) = 1/3Pr{s5|B} = Pr{s5}/Pr{B} = (1/6)/(1/2) = 1/3Pr{s6|B} = Pr{s6}/Pr{B} = (1/6)/(1/2) = 1/3
s1
s2
s3
s4
s5
s6
S
s1
s2
s3
s4
s5
s6
Event B has
already
occurred
S
S|B = {s1, s5, s6}
29
Copyright © Syed Ali Khayam 2008
Conditional Probability The probability of an event A in the conditional sample space is:
For the dice example: Pr{A|B} = Pr{A∩B}/Pr{B} = Pr{s6}/Pr{B} = (1/6)/(1/2) = 1/3
s1
s2
s3
s4
s5
s6
S
s1
s2
s3
s4
s5
s6
Event B has
already
occurred
S
S|B = {s1, s5, s6}
PrPr
Pr
A BA B
B
30
Copyright © Syed Ali Khayam 2008
Independence
Two events are independent if they do not provide any information about each other
In other words, the fact that B has already happened does not affect the probability of A’s outcomes
Pr PrA B A
31
Copyright © Syed Ali Khayam 2008
Independence
Note that
The above condition implies that
Pr Pr only when Pr Pr PrA B A A B A B
Pr Pr Pr PrAB A B A B
32
Copyright © Syed Ali Khayam 2008
Independence
In general, for n mutually independent events, A1, …, An, we have:
1 1
Pr Prn n
i ii i
A A
1 2Pr , , , Pr ,k i i ip k ij kA A A A A A A
33
Copyright © Syed Ali Khayam 2008
Independence: Example
Are events A and C independent? Assume that all outcomes are equally likely
s4
s1
s2
s3
s6
s5
S
34
Copyright © Syed Ali Khayam 2008
Independence: Example
Are events A and C independent?
Yes: Pr{A ∩ C} = Pr{s5} = 1/6
Pr{A}Pr{C} = (3/6)x(2/6) = 1/6
s4
s1
s2
s3
s6
s5
S
35
Copyright © Syed Ali Khayam 2008
Independence: Example
Are events A and B independent? Assume that all outcomes are equally likely
s4
s1
s2
s3
s6
s5
S
36
Copyright © Syed Ali Khayam 2008
Independence: Example
Are events A and B independent?
NO: Pr{A ∩ B} = Pr{s5} = 1/6
Pr{A}Pr{B} = (3/6)x(3/6) = 1/4
s4
s1
s2
s3
s6
s5
S
37
Copyright © Syed Ali Khayam 2008
Independence: Example
Are events A and B independent? Assume that all outcomes are equally likely
s1
s2
s3
s4
s5
s6
S
A
B
38
Copyright © Syed Ali Khayam 2008
Independence: Example
Are events A and B independent?
NO: Pr{A ∩ B} = Pr{Ø} = 0
Pr{A}Pr{B} = (2/6)x(3/6) = 1/6
Recall that A and B are mutually exclusive
s1
s2
s3
s4
s5
s6
S
A
B
39
Copyright © Syed Ali Khayam 2008
What you need to remember from what we have studied so far…
1. Outcomes, events and sample space:
2. For mutually exclusive events A1, A2,…, AN, we have:
3. In general, we have:
outcome event sample space
1 2 1 2Pr Pr PrA A A A
1 2 1 2 1 2Pr Pr Pr PrA A A A A A
40
Copyright © Syed Ali Khayam 2008
What you need to remember from what we have studied so far…
4. Conditional probability reduces the sample space:
5. Two events A and B are independent only if
6. For independent events:
PrPr
Pr
A BA B
B
Pr PrA B A
Pr Pr PrA B A B
41
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…1. Whenever you see two events which have an OR relationship (i.e., event A or
event B), their joint event will be their union, {A U B}
Example: On a binary channel, find the probability of error?
An error occurs when
A: “a 0 is transmitted and a 1 is received” OR
B: “a 1 is transmitted and a 0 is received”
Thus probability of error is: Pr{A U B}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
(See Appendix of this lecture for a explanation of the binary channel)
42
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…
2. Whenever you see two events which have an AND relationship (i.e., both event A and event B), their joint event will be their intersection, {A ∩ B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is received?
An error occurs when
A: “a 0 is transmitted” AND
B: “a 1 is received”
Thus probability of above event is: Pr{A ∩ B}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
43
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…
3. Whenever you see two events which have an OR relationship (i.e., A U B), check if they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}
Example: On a binary channel, find the probability of error?
An error occurs when
A: “a 0 is transmitted and a 1 is received” OR
B: “a 1 is transmitted and a 0 is received”
Thus probability of error is: Pr{error} = Pr{A U B}
Are A and B are mutually exclusive?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
44
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…
3. Whenever you see two events which have an OR relationship (i.e., A U B), check if they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}
Example: On a binary channel, find the probability of error?
An error occurs when
A: “a 0 is transmitted and a 1 is received” OR
B: “a 1 is transmitted and a 0 is received”
Thus probability of error is: Pr{error} = Pr{A U B}
YES!
A and B are mutually exclusive; transmission of a 0 precludes the possibility of transmission of a 1, and vice versa. Therefore, we can set
Pr{error} = Pr{A U B} = Pr{A} + Pr{B}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
45
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…
4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is received?
A: “a 0 is transmitted” AND
B: “a 1 is received”
Probability of above event is: Pr{A ∩ B}
Are A and B independent?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
46
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…
4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is received?
A: “a 1 is received” AND
B: “a 0 is transmitted”
Probability of above event is: Pr{A ∩ B}
Are A and B independent?
NO! (See Appendix of this lecture for a more detailed explanation)
Pr{A|B}=Pr{R1|T0} ≠ Pr{A}=Pr{R1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
47
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if
they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}Example 2: On a binary channel, find the probability that a 0 is transmitted and a 1 is
received?A: “at time n+1, a 1 is received when a 0 is transmitted” ANDB: “at time n, a 0 is received when a 1 is transmitted”Probability of above event is: Pr{A ∩ B}Are A and B independent?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
48
Copyright © Syed Ali Khayam 2008
Four “Rules of Thumb” from what we have studied so far…
4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}
Example 2: On a binary channel, find the probability that a 0 is transmitted and a 1 is received?A: “at time n+1, a 1 is received when a 0 is transmitted” ANDB: “at time n, a 0 is received when a 1 is transmitted”Probability of above event is: Pr{A ∩ B}Are A and B independent?YES!
Pr{A|B}=Pr{R1|T0}=Pr{A}=> Pr{A ∩ B} = Pr{A}Pr{B} = Pr{R1|T0} Pr{R0|T1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
49
Copyright © Syed Ali Khayam 2008
Partition of a Sample Space
B1, B2,…, BN form a partition of a sample space we have:
S = B1 U B2 U … U BN
Bi ∩ Bj = Ø, i ≠ j
B1
B2
B3 B4
s2s4
s6
s1 s5
s3
50
Copyright © Syed Ali Khayam 2008
Total Probability
If B1, B2,…, BN form a partition then for any event A(A ∩ Bi) ∩ (A ∩ Bj) = Ø, i ≠ j
=> A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
51
Copyright © Syed Ali Khayam 2008
Total Probability
Thus event A can be expressed as the union of mutually exclusive events:
A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)
=> Pr{A} = Pr{A ∩ B1} + Pr{A ∩ B2} + … + Pr{A ∩ BN}
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
52
Copyright © Syed Ali Khayam 2008
Total Probability
If B1, B2,…, BN form a partition then for any event A:
Pr{A} = Pr{A ∩ B1} + Pr{A ∩ B2} + … + Pr{A ∩ BN}
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
53
Copyright © Syed Ali Khayam 2008
Total Probability
Using the definition of conditional probability:
Pr{A| Bi} = Pr{A ∩ Bi} / Pr{Bi}
=> Pr{A ∩ Bi} = Pr{A| Bi} Pr{Bi}
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
54
Copyright © Syed Ali Khayam 2008
The Law of Total Probability
The Law of Total Probability states:
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
If B1, B2,…, BN form a partition then for any event A
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + … + Pr{A|BN} Pr{BN}
55
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem
Based on the Law of Total Probability, Thomas Bayes decided to look at the probability of a partition given a particular event, the so-called inverse probability
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
56
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem
Based on the Law of Total Probability, Thomas Bayes decided to look at the probability of a partition given a particular event, the so-called inverse probability
Pr{Bi|A} = Pr{A ∩ Bi} / Pr{A}
Since Pr{A ∩ Bi} = Pr{A|Bi} Pr{Bi}, we obtain
Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
57
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem
Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
From the Law of Total Probability, we have:
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + … + Pr{A|BN} Pr{BN}
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
1
Pr PrPr
Pr Pr
i ii N
j jj
A B BB A
A B BBayes’ Rule
58
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem
B1, B2,…, BN are known as a priori events these events are known before the experiment
Pr{Bi} is known as a priori probability
Pr{Bi|A} is known as a posteriori probability Experiment is performed; Event A is observed; now what is the
probability that Bi has occurred
B1
B2
B3
B4
As2
s4s6
s1 s5
s3
A
59
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Bayes’ Theorem is best understood through a classical example of a memory-less binary channel shown below
What we already know about this channel is: A priori probabilities: Pr{T0}, Pr{T1}
Channel probabilities: Pr{R0|T0}, Pr{R1|T0} = 1 - Pr{R0|T0}, Pr{R1|T1}, Pr{R0|T1} = 1 - Pr{R1|T1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
60
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Given that we know Pr{T0} and Pr{T1}, we want to find: Pr{Ti|Ri}: The probability that Ti was transmitted given that Ri has been
received, i = 0, 1; or the probability of successful symbol transmission
Pr{Ti|Rj}: The probability that Ti was transmitted given that Rj has been received, i ≠ j: or the probability of symbol error
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}61
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Pr{correct transmission}: The probability that Ti was transmitted given that Ri has been received, i = 0, 1; or the probability of successful symbol transmission
Pr{correct transmission} = Pr{ (T0 ∩ R0) U (T1 ∩ R1) }
= Pr{T0|R0} Pr{R0} + Pr{T1|R1} Pr{R1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
Let’s focus on this first
62
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Let’s first focus on finding Pr{T0|R0}
From Bayes’ Rule, we know that
Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
63
Copyright © Syed Ali Khayam 2008
From Bayes’ Theorem, we know that
Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}
Bayes’ Theorem: Example
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
We know these
But we don’t know this
Let’s now focus on finding
Pr{R0} in terms of what we
already know
64
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Let’s now focus on finding Pr{R0} in terms of what we already know
From the Law of Total Probability, we have
Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
We know all of these terms
65
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Let’s now focus on finding Pr{R0} in terms of what we already know
From the Law of Total Probability, we have
Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
We know all of these terms
66
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}
And Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}
=> Pr{T0|R0} = Pr{R0|T0} Pr{T0} / (Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1})
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
All the terms in this
expression are known
We can now compute
Pr{T0|R0}
67
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Pr{T0|R0} = Pr{R0|T0} Pr{T0} / (Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1})
Using similar computations, we can show that
Pr{T1|R1} = Pr{R1|T1} Pr{T1} / (Pr{R1|T1} Pr{T1} + Pr{R1|T0} Pr{T0})
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1} 68
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Pr{T0|R0} = Pr{R0|T0} Pr{T0} / (Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1})
Pr{T1|R1} = Pr{R1|T1} Pr{T1} / (Pr{R1|T1} Pr{T1} + Pr{R1|T0} Pr{T0})
Plug the above values into the original equation to get
Pr{correct transmission} = Pr{T0|R0} Pr{R0} + Pr{T1|R1} Pr{R1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1} 69
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example A Priori Probabilities
Pr{T0} = 0.45
Pr{T1} = 1 - Pr{T0} = 0.55
Channel probabilities: Pr{R0|T0} = 0.94
Pr{R1|T0} = 1 - Pr{R0|T0} = 0.06
Pr{R1|T1} = 0.91
Pr{R0|T1} = 1 - Pr{R1|T1} = 0.09
T0
T1
R0
R1
Pr{R0|T0}=0.94
Pr{R1|T1}=0.91 70
Copyright © Syed Ali Khayam 2008
Bayes’ Theorem: Example
Then:
Pr{T0|R0} = Pr{R0|T0} Pr{T0} / (Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1})
=0.94x0.45 / (0.94x0.45 + 0.09x0.55) = 0.8952
Pr{T1|R1} = Pr{R1|T1} Pr{T1} / (Pr{R1|T1} Pr{T1} + Pr{R1|T0} Pr{T0})
=0.91x0.55 / (0.91x0.55 + 0.06x0.45) = 0.9488
T0
T1
R0
R1
Pr{R0|T0}=0.94
Pr{R1|T1}=0.9171
Copyright © Syed Ali Khayam 2008
Lecture 1: Appendix A
Additional Examples and Explanations
72
Copyright © Syed Ali Khayam 2008
Background on a Memoryless Binary Communication Channel Memoryless: Bit transmission at time n+i, i>0 has no dependence on bit
transmission at time n
Binary: Only two symbols are transmitted, represented by T0 and T1
Prior Probabilities: Probabilities of T0 and T1 are calculated ahead of time from the data; Pr{T1}= 1 - Pr{T0}
Crossover Probabilities: Pr{R0|T1} and Pr{R1|T0} are called crossover or bit-error probabilities. These probabilities are also calculated ahead of time by sending training signals on the channel; Pr{R0|T0}=1-Pr{R1|T0}, Pr{R0|T1}=1-Pr{R1|T1}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}73
Copyright © Syed Ali Khayam 2008
Example 1
On a binary channel, find the probability that a 0 is transmitted and a 1 is received?
A: “a 1 is received” AND
B: “a 0 is transmitted”
Probability of above event is: Pr{A ∩ B}
Are A and B independent?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}
74
Copyright © Syed Ali Khayam 2008
Example 1
What is the sample space of our experiment?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}Spring 2008 75
Copyright © Syed Ali Khayam 2008
Example 1
What is the sample space of our experiment?
Sample Space, S = {(T0 ∩ R0), (T0 ∩ R1), (T1 ∩ R0), (T1 ∩ R1)}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}Spring 2008 76
Copyright © Syed Ali Khayam 2008
Example 1
A: “a 1 is received” AND B: “a 0 is transmitted”
Are A and B independent?
A and B are independent when only when Pr{A ∩ B} = Pr{A}Pr{B}. So the main question is thefollowing:
Is Pr{A ∩ B} = Pr{A}Pr{B}?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}77
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Is Pr{A ∩ B} = Pr{A}Pr{B} ?
The LHS of the above equation is: Pr{A ∩ B} = Pr{A|B}Pr{B} = Pr{R1|T0}Pr{T0}
The RHS is:Pr{A}Pr{B} = Pr{R1}Pr{T0}
So the above question can be rephrased as:Is Pr{R1|T0}Pr{T0} = Pr{R1}Pr{T0} ?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}78
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Is Pr{R1|T0}Pr{T0} = Pr{R1}Pr{T0} ?
We can get rid of Pr{T0} from both sides, so we are left with the following question:
Is Pr{R1|T0} = Pr{R1} ?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}79
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Is Pr{R1|T0} = Pr{R1} ?
It can be intuitively deduced that the above equality relation does not hold in general because from the figure below we can see that Pr{R1} should be a function of both Pr{R1|T0} and Pr{R1|T1}.
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}80
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Is Pr{R1|T0} = Pr{R1} ?
It can be intuitively deduced that the above equality relation does not hold in general. Mathematically, we can show this by computing the Pr{R1}:Pr{R1} = Pr{ (T0 is tx’d AND R1 is rec’d) OR (T1 is tx’d AND R1 is rec’d)}Pr{R1} = Pr{ (T0 ∩ R1) U (T1 ∩ R1)}
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}81
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Is Pr{R1|T0} = Pr{R1} ?
Pr{R1} = Pr{ (T0 ∩ R1) U (T1 ∩ R1)}
Clearly, (T0 ∩ R1) and (T1 ∩ R1) are mutually exclusive events. => Pr{R1} = Pr{T0 ∩ R1} + Pr{T1 ∩ R1}or Pr{R1} = Pr{R1 ∩ T0} + Pr{R1 ∩ T1}which gives Pr{R1} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1}Now we rephrase the question posed on the top of this slide as:
Is Pr{R1|T0} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} ?
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}82
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Is Pr{R1|T0} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} ?
For the above relation to be satisfied, the following relationship must be satisfied:Pr{T0} = 1=> Pr{T1} = 1- Pr{T0} = 0
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}83
Copyright © Syed Ali Khayam 2008
Example 1A: “a 1 is received” AND B: “a 0 is transmitted”
Final Result: Pr{T0} = 1 => A and B are independent
So events A and B are independent when T0 is the only symbol being transmitted
T0
T1
R0
R1
Pr{R0|T0}
Pr{R1|T1}84