lecture01 intro probability theory

CSE-801: Stochastic Systems

Introduction to Probability Theory

Syed Ali Khayam

School of Electrical Engineering & Computer Science

National University of Sciences & Technology (NUST)

Pakistan

Course Information

Lecture Timings: Wednesdays: 5:30pm-7:20pm

Fridays: 5:30pm-6:20pm

Office Hours Wednesdays: 4:00pm-5:30pm

Fridays: 4:00pm-5:30pm

Office is located on top floor, last room on your left in the north wing

The course will be managed through Moodle: lms.nseecs.edu.pk

Course password: given in class

Textbook

Course Outline

Syllabus Introduction to Probability Theory Functions of Random Variables Limits and Inequalities Stochastic Processes Prediction and Estimation Markov Chains and Processes (time permitting) Assorted Topics (time permitting)

Grading Final Exam: 40% Midterm Exam: 30% Quizzes: 20% Homework Assignments: 10%

Course Outline

Grading Final Exam: 40% Midterm Exam: 30% Quizzes: 20% Homework Assignments: 10%

Lot’s of extra credit for extra effort We will also have a voluntary user study as well Anyone volunteering and seeing it through will get 10 extra credit points

Policies

Quizzes are announced and will take place at the start of Friday classes

Exams will be closed book, but you will be allowed to bring an A4-sized cheat sheet to the exam

Late homeworks submissions will not be accepted

Strongest possible disciplinary action will be taken in case of plagiarism or cheating in exams, homeworks or quizzes

It is mandatory to maintain at least 75% class attendance to sit in the Final Test

Credits and Acknowledgements

I would like to thank Dr. Garcia for providing online lecture notes on the textbook’s website

Throughout this course, I will be borrowing examples and explanations from the Stochastic Systems Course taught by Professor Hayder Radha at Michigan State University

Why this course?

Stochastic theory is an extension of probability theory

This course on Stochastic theory will teach mathematical tools that are commonly-used in a variety of engineering, computer science and IT disciplines

We will focus solely on performance modeling of phenomena observed in communications engineering

Applications and examples will be provided as required

What will we cover in this lecture?

This lecture is intended to be an introduction to elementary probability theory

We will cover: Random Experiments and Random Variables

Axioms of Probability

Mutual Exclusivity

Conditional Probability

Independence

Law of Total Probability

Bayes’ Theorem

Definition of Probability

Probability: [m-w.org]1 : the quality or state of being probable

2 : something (as an event or circumstance) that is probable3 a (1) : the ratio of the number of outcomes in an exhaustive set of equally likely outcomes that produce a given event to the total number of possible outcomes (2) : the chance that a given event will occur b : a branch of mathematics concerned with the study of probabilities4 : a logical relation between statements such that evidence confirming one confirms the other to some degree

Do you know which famous person was so opposed to probability theory that he said:

“God does not play dice with the universe.”?

And do you know which famous person said:

“God does play dice with the universe. All the evidence points to him being an inveterate gambler, who throws the dice on every possible occasion.”?

Definition of a Random Experiment

A random experiment comprises of: A procedure

An outcome

Procedure

(e.g., flipping a coin)

Outcome

(e.g., the value

observed [head, tail] after

flipping the coin)

Sample Space

(Set of All Possible

Outcomes)

Definition of a Random Experiment: Outcomes, Events and the Sample Space

An outcome cannot be further decomposed into other outcomes

{s1 = the value 1}, …, {s6 = the value 6}

An event is a set of outcomes that are of interest to usA = {s: such that s is an even number}

The set of all possible outcomes, S, is called the sample space

S = {s1, s2, s3, s4, s5, s6}

outcome event sample space

Example of a Random Experiment: Experiment: Roll a fair die once and record the number of dots on the

top face

S = {1, 2, 3, 4, 5, 6}

A = “the outcome is even” = {2, 4, 6}

B = “the outcome is greater than 4” = {5, 6}

Axioms of Probability Probability of any event A is non-negative:

Pr{A} ≥ 0

The probability that an outcome belongs to the sample space is 1:

Pr{S} = 1

The probability of the union of mutually exclusive events is equal to the sum of their probabilities:

If A1 ∩ A2=Ø, => Pr{A1 U A2} = Pr{A1} + Pr{A2}

Mutual Exclusivity

For mutually exclusive events A1, A2,…, AN, we have:

Pr PrNN

i ii i

Find Pr{A1 U A2}

and Pr{A1}+Pr{A2}

in the fair die

example

Mutual Exclusivity

In general, we have:

Pr{A1 U A2} = Pr{A1} + Pr{A2} – Pr{A1 ∩ A2}

Mutual Exclusivity: Example

Experiment: Roll a fair dice twice and record the number of dots on the top face:

S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),

(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }

Find the probability of the following events:A1 = “first roll gives an odd number”

A2 = “second roll gives an odd number”

C = “the sum of the two rolls is odd”

Define three events:A1 = “first roll gives an odd number”

A2 = “second roll gives an odd number”

Find the probability of C using probability of A1 and A2

S = { (1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),

(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),

(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),

(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }

Pr{A1} = “first roll gives an odd number” = 18/36 = 1/2

Pr{A2} = “second roll gives an odd number” = 18/36 = 1/2

C1 = “first roll is odd and second is even”=

C2 = “first roll is even and second is odd”=

Since C1 and C2 are mutually exclusive:

1 2A A

2 11 2C A A A A

1 2 1 2

2 11 2

1 2 1 2

Pr Pr Pr{ } Pr{ }

Pr Pr Pr Pr

Pr 1 Pr 1 Pr Pr

C C C C C

A A A A

Conditional Probability

Given that event B has already occurred, what is the probability that event A will occur?

Given that event B has already occurred, reduces the sample space of A

Event B has

already occurred

=> s2, s4, s3

cannot occur

Conditional Probability Given that event B has already occurred, we define a new

conditional sample space that only contains B’s outcomes

The new event space for A is the intersection of A and B: EA|B = A ∩ B

Event B has

already

occurred

What’s missing here?S|B = {s1, s5, s6}

EA|B= A ∩ B = {s6}

Conditional Probability Consider that the example below corresponds to an experiment

where we throw a fair dice and record the number of dots on its face

For this experiment, what is Pr {A|B} in the example below?Pr{A|B} = Pr{s6|B}= 1/3

We need to normalize all probabilities in a conditional sample space with Pr{B}

Event B has

already

occurred

S|B = {s1, s5, s6} 28

Conditional Probability We need to normalize all probabilities in a conditional sample

space with Pr{B}Pr{s1|B} = Pr{s1}/Pr{B} = (1/6)/(1/2) = 1/3Pr{s5|B} = Pr{s5}/Pr{B} = (1/6)/(1/2) = 1/3Pr{s6|B} = Pr{s6}/Pr{B} = (1/6)/(1/2) = 1/3

Event B has

already

occurred

S|B = {s1, s5, s6}

Conditional Probability The probability of an event A in the conditional sample space is:

For the dice example: Pr{A|B} = Pr{A∩B}/Pr{B} = Pr{s6}/Pr{B} = (1/6)/(1/2) = 1/3

Event B has

already

occurred

S|B = {s1, s5, s6}

A BA B

Independence

Two events are independent if they do not provide any information about each other

In other words, the fact that B has already happened does not affect the probability of A’s outcomes

Pr PrA B A

Independence

Note that

The above condition implies that

Pr Pr only when Pr Pr PrA B A A B A B

Pr Pr Pr PrAB A B A B

Independence

In general, for n mutually independent events, A1, …, An, we have:

Pr Prn n

i ii i

1 2Pr , , , Pr ,k i i ip k ij kA A A A A A A

Independence: Example

Are events A and C independent? Assume that all outcomes are equally likely

Are events A and C independent?

Yes: Pr{A ∩ C} = Pr{s5} = 1/6

Pr{A}Pr{C} = (3/6)x(2/6) = 1/6

Are events A and B independent? Assume that all outcomes are equally likely

Are events A and B independent?

NO: Pr{A ∩ B} = Pr{s5} = 1/6

Pr{A}Pr{B} = (3/6)x(3/6) = 1/4

Are events A and B independent? Assume that all outcomes are equally likely

Are events A and B independent?

NO: Pr{A ∩ B} = Pr{Ø} = 0

Pr{A}Pr{B} = (2/6)x(3/6) = 1/6

Recall that A and B are mutually exclusive

What you need to remember from what we have studied so far…

1. Outcomes, events and sample space:

2. For mutually exclusive events A1, A2,…, AN, we have:

3. In general, we have:

outcome event sample space

1 2 1 2Pr Pr PrA A A A

1 2 1 2 1 2Pr Pr Pr PrA A A A A A

What you need to remember from what we have studied so far…

4. Conditional probability reduces the sample space:

5. Two events A and B are independent only if

6. For independent events:

A BA B

Pr PrA B A

Pr Pr PrA B A B

Four “Rules of Thumb” from what we have studied so far…1. Whenever you see two events which have an OR relationship (i.e., event A or

event B), their joint event will be their union, {A U B}

Example: On a binary channel, find the probability of error?

An error occurs when

A: “a 0 is transmitted and a 1 is received” OR

B: “a 1 is transmitted and a 0 is received”

Thus probability of error is: Pr{A U B}

Pr{R0|T0}

Pr{R1|T1}

(See Appendix of this lecture for a explanation of the binary channel)

Four “Rules of Thumb” from what we have studied so far…

2. Whenever you see two events which have an AND relationship (i.e., both event A and event B), their joint event will be their intersection, {A ∩ B}

Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is received?

A: “a 0 is transmitted” AND

B: “a 1 is received”

Thus probability of above event is: Pr{A ∩ B}

Pr{R0|T0}

Pr{R1|T1}

3. Whenever you see two events which have an OR relationship (i.e., A U B), check if they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}

Thus probability of error is: Pr{error} = Pr{A U B}

Are A and B are mutually exclusive?

Pr{R0|T0}

Pr{R1|T1}

3. Whenever you see two events which have an OR relationship (i.e., A U B), check if they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}

Thus probability of error is: Pr{error} = Pr{A U B}

A and B are mutually exclusive; transmission of a 0 precludes the possibility of transmission of a 1, and vice versa. Therefore, we can set

Pr{error} = Pr{A U B} = Pr{A} + Pr{B}

Pr{R0|T0}

Pr{R1|T1}

4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}

A: “a 0 is transmitted” AND

B: “a 1 is received”

Probability of above event is: Pr{A ∩ B}

Are A and B independent?

Pr{R0|T0}

Pr{R1|T1}

A: “a 1 is received” AND

B: “a 0 is transmitted”

NO! (See Appendix of this lecture for a more detailed explanation)

Pr{A|B}=Pr{R1|T0} ≠ Pr{A}=Pr{R1}

Pr{R0|T0}

Pr{R1|T1}

Four “Rules of Thumb” from what we have studied so far…4. Whenever you see two events which have an AND relationship (i.e., A ∩ B), check if

they are independent. If so, set Pr{A ∩ B} = Pr{A}Pr{B}Example 2: On a binary channel, find the probability that a 0 is transmitted and a 1 is

received?A: “at time n+1, a 1 is received when a 0 is transmitted” ANDB: “at time n, a 0 is received when a 1 is transmitted”Probability of above event is: Pr{A ∩ B}Are A and B independent?

Pr{R0|T0}

Pr{R1|T1}

Example 2: On a binary channel, find the probability that a 0 is transmitted and a 1 is received?A: “at time n+1, a 1 is received when a 0 is transmitted” ANDB: “at time n, a 0 is received when a 1 is transmitted”Probability of above event is: Pr{A ∩ B}Are A and B independent?YES!

Pr{A|B}=Pr{R1|T0}=Pr{A}=> Pr{A ∩ B} = Pr{A}Pr{B} = Pr{R1|T0} Pr{R0|T1}

Pr{R0|T0}

Pr{R1|T1}

Partition of a Sample Space

B1, B2,…, BN form a partition of a sample space we have:

S = B1 U B2 U … U BN

Bi ∩ Bj = Ø, i ≠ j

Total Probability

If B1, B2,…, BN form a partition then for any event A(A ∩ Bi) ∩ (A ∩ Bj) = Ø, i ≠ j

=> A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)

Total Probability

Thus event A can be expressed as the union of mutually exclusive events:

A = (A ∩ B1) U (A ∩ B2) U … U (A ∩ BN)

=> Pr{A} = Pr{A ∩ B1} + Pr{A ∩ B2} + … + Pr{A ∩ BN}

Total Probability

If B1, B2,…, BN form a partition then for any event A:

Pr{A} = Pr{A ∩ B1} + Pr{A ∩ B2} + … + Pr{A ∩ BN}

Total Probability

Using the definition of conditional probability:

Pr{A| Bi} = Pr{A ∩ Bi} / Pr{Bi}

=> Pr{A ∩ Bi} = Pr{A| Bi} Pr{Bi}

The Law of Total Probability

The Law of Total Probability states:

If B1, B2,…, BN form a partition then for any event A

Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + … + Pr{A|BN} Pr{BN}

Bayes’ Theorem

Based on the Law of Total Probability, Thomas Bayes decided to look at the probability of a partition given a particular event, the so-called inverse probability

Bayes’ Theorem

Based on the Law of Total Probability, Thomas Bayes decided to look at the probability of a partition given a particular event, the so-called inverse probability

Pr{Bi|A} = Pr{A ∩ Bi} / Pr{A}

Since Pr{A ∩ Bi} = Pr{A|Bi} Pr{Bi}, we obtain

Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}

Bayes’ Theorem

Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}

From the Law of Total Probability, we have:

Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + … + Pr{A|BN} Pr{BN}

Pr PrPr

i ii N

A B BB A

A B BBayes’ Rule

Bayes’ Theorem

B1, B2,…, BN are known as a priori events these events are known before the experiment

Pr{Bi} is known as a priori probability

Pr{Bi|A} is known as a posteriori probability Experiment is performed; Event A is observed; now what is the

probability that Bi has occurred

Bayes’ Theorem: Example

Bayes’ Theorem is best understood through a classical example of a memory-less binary channel shown below

What we already know about this channel is: A priori probabilities: Pr{T0}, Pr{T1}

Channel probabilities: Pr{R0|T0}, Pr{R1|T0} = 1 - Pr{R0|T0}, Pr{R1|T1}, Pr{R0|T1} = 1 - Pr{R1|T1}

Pr{R0|T0}

Pr{R1|T1}

Given that we know Pr{T0} and Pr{T1}, we want to find: Pr{Ti|Ri}: The probability that Ti was transmitted given that Ri has been

received, i = 0, 1; or the probability of successful symbol transmission

Pr{Ti|Rj}: The probability that Ti was transmitted given that Rj has been received, i ≠ j: or the probability of symbol error

Pr{R0|T0}

Pr{R1|T1}61

Pr{correct transmission}: The probability that Ti was transmitted given that Ri has been received, i = 0, 1; or the probability of successful symbol transmission

Pr{correct transmission} = Pr{ (T0 ∩ R0) U (T1 ∩ R1) }

= Pr{T0|R0} Pr{R0} + Pr{T1|R1} Pr{R1}

Pr{R0|T0}

Pr{R1|T1}

Let’s focus on this first

Let’s first focus on finding Pr{T0|R0}

From Bayes’ Rule, we know that

Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}

Pr{R0|T0}

Pr{R1|T1}

From Bayes’ Theorem, we know that

Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}

Pr{R0|T0}

Pr{R1|T1}

We know these

But we don’t know this

Let’s now focus on finding

Pr{R0} in terms of what we

already know

Let’s now focus on finding Pr{R0} in terms of what we already know

From the Law of Total Probability, we have

Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}

Pr{R0|T0}

Pr{R1|T1}

We know all of these terms

Let’s now focus on finding Pr{R0} in terms of what we already know

From the Law of Total Probability, we have

Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}

Pr{R0|T0}

Pr{R1|T1}

We know all of these terms

Pr{T0|R0} = Pr{R0|T0} Pr{T0} / Pr{R0}

And Pr{R0} = Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1}

=> Pr{T0|R0} = Pr{R0|T0} Pr{T0} / (Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1})

Pr{R0|T0}

Pr{R1|T1}

All the terms in this

expression are known

We can now compute

Pr{T0|R0}

Pr{T0|R0} = Pr{R0|T0} Pr{T0} / (Pr{R0|T0} Pr{T0} + Pr{R0|T1} Pr{T1})

Using similar computations, we can show that

Pr{R0|T0}

Pr{R1|T1} 68

Plug the above values into the original equation to get

Pr{correct transmission} = Pr{T0|R0} Pr{R0} + Pr{T1|R1} Pr{R1}

Pr{R0|T0}

Pr{R1|T1} 69

Bayes’ Theorem: Example A Priori Probabilities

Pr{T0} = 0.45

Pr{T1} = 1 - Pr{T0} = 0.55

Channel probabilities: Pr{R0|T0} = 0.94

Pr{R1|T0} = 1 - Pr{R0|T0} = 0.06

Pr{R1|T1} = 0.91

Pr{R0|T1} = 1 - Pr{R1|T1} = 0.09

Pr{R0|T0}=0.94

Pr{R1|T1}=0.91 70

=0.94x0.45 / (0.94x0.45 + 0.09x0.55) = 0.8952

=0.91x0.55 / (0.91x0.55 + 0.06x0.45) = 0.9488

Pr{R0|T0}=0.94

Pr{R1|T1}=0.9171

Lecture 1: Appendix A

Additional Examples and Explanations

Background on a Memoryless Binary Communication Channel Memoryless: Bit transmission at time n+i, i>0 has no dependence on bit

transmission at time n

Binary: Only two symbols are transmitted, represented by T0 and T1

Prior Probabilities: Probabilities of T0 and T1 are calculated ahead of time from the data; Pr{T1}= 1 - Pr{T0}

Crossover Probabilities: Pr{R0|T1} and Pr{R1|T0} are called crossover or bit-error probabilities. These probabilities are also calculated ahead of time by sending training signals on the channel; Pr{R0|T0}=1-Pr{R1|T0}, Pr{R0|T1}=1-Pr{R1|T1}

Pr{R0|T0}

Pr{R1|T1}73

Example 1

On a binary channel, find the probability that a 0 is transmitted and a 1 is received?

A: “a 1 is received” AND

B: “a 0 is transmitted”

Pr{R0|T0}

Pr{R1|T1}

Example 1

What is the sample space of our experiment?

Pr{R0|T0}

Pr{R1|T1}Spring 2008 75

Example 1

What is the sample space of our experiment?

Sample Space, S = {(T0 ∩ R0), (T0 ∩ R1), (T1 ∩ R0), (T1 ∩ R1)}

Pr{R0|T0}

Pr{R1|T1}Spring 2008 76

Example 1

A: “a 1 is received” AND B: “a 0 is transmitted”

A and B are independent when only when Pr{A ∩ B} = Pr{A}Pr{B}. So the main question is thefollowing:

Is Pr{A ∩ B} = Pr{A}Pr{B}?

Pr{R0|T0}

Pr{R1|T1}77

Example 1A: “a 1 is received” AND B: “a 0 is transmitted”

Is Pr{A ∩ B} = Pr{A}Pr{B} ?

The LHS of the above equation is: Pr{A ∩ B} = Pr{A|B}Pr{B} = Pr{R1|T0}Pr{T0}

The RHS is:Pr{A}Pr{B} = Pr{R1}Pr{T0}

So the above question can be rephrased as:Is Pr{R1|T0}Pr{T0} = Pr{R1}Pr{T0} ?

Pr{R0|T0}

Pr{R1|T1}78

Is Pr{R1|T0}Pr{T0} = Pr{R1}Pr{T0} ?

We can get rid of Pr{T0} from both sides, so we are left with the following question:

Is Pr{R1|T0} = Pr{R1} ?

Pr{R0|T0}

Pr{R1|T1}79

Is Pr{R1|T0} = Pr{R1} ?

It can be intuitively deduced that the above equality relation does not hold in general because from the figure below we can see that Pr{R1} should be a function of both Pr{R1|T0} and Pr{R1|T1}.

Pr{R0|T0}

Pr{R1|T1}80

Is Pr{R1|T0} = Pr{R1} ?

It can be intuitively deduced that the above equality relation does not hold in general. Mathematically, we can show this by computing the Pr{R1}:Pr{R1} = Pr{ (T0 is tx’d AND R1 is rec’d) OR (T1 is tx’d AND R1 is rec’d)}Pr{R1} = Pr{ (T0 ∩ R1) U (T1 ∩ R1)}

Pr{R0|T0}

Pr{R1|T1}81

Is Pr{R1|T0} = Pr{R1} ?

Pr{R1} = Pr{ (T0 ∩ R1) U (T1 ∩ R1)}

Clearly, (T0 ∩ R1) and (T1 ∩ R1) are mutually exclusive events. => Pr{R1} = Pr{T0 ∩ R1} + Pr{T1 ∩ R1}or Pr{R1} = Pr{R1 ∩ T0} + Pr{R1 ∩ T1}which gives Pr{R1} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1}Now we rephrase the question posed on the top of this slide as:

Is Pr{R1|T0} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} ?

Pr{R0|T0}

Pr{R1|T1}82

Is Pr{R1|T0} = Pr{R1|T0}Pr{T0} + Pr{R1|T1}Pr{T1} ?

For the above relation to be satisfied, the following relationship must be satisfied:Pr{T0} = 1=> Pr{T1} = 1- Pr{T0} = 0

Pr{R0|T0}

Pr{R1|T1}83

Final Result: Pr{T0} = 1 => A and B are independent

So events A and B are independent when T0 is the only symbol being transmitted

Pr{R0|T0}

Pr{R1|T1}84

lecture01 intro probability theory

t0prt0 prr1prt0

t0prt0 prr1

t0 prt0

t1 prt1

prr0 prr0

b1 pr

t0 t0 r0

t0 prt0

Documents

intro probability

jan 2014 intro to bayesian probability, statistical...

intro to probability -...

1 chapters 14 and 15 (intro stats – 3 edition)...

lecture01-intro to module autumn 2013-14

intro to probability -...

13.1 intro to probability

lecture 15: probability review bayes nets intro

probability theory intro

probability & statistics: intro, summary statistics

[pols 4150] intro. to probability theory, discrete and...

1 intro to probability zhi wei 2 outline basic concepts in...

lecture 1: intro/refresher in matrix...

lecture01 intro simone

7330 lecture01 intro and review-f09.ppt

intro to artificial intelligence csc384h, winter 2009 hojjat...

probability day 1 ct - mrs. allison's blogday 1 - intro to...

intro to probability and statistics

intro to probability

02 intro to probability