Bayes Rule for probability

P A P B AP A B

P A P B A P A P B A

Let A1, A2 , … , Ak denote a set of events such that

1 1

i ii

k k

P A P B AP A B

P A P B A P A P B A

An generalization of Bayes Rule

1 2 and k i jS A A A A A

for all i and j. Then

Example:We have three urns. Urn 1 contains 14 red balls and 12 black balls. Urn 2 contains 6 red balls and 20 black balls. Urn 3 contains 3 red balls and 23 black balls.An Urn is selected at random and a ball is selected from that urn.

If the ball turns out to be red what is the probability that it came from the first urn? second urn? third Urn?

Urn 1 Urn 2 Urn 3

Solution:

Note: the desired conditional probability is in the reverse direction of the given conditional probabilities. This is the case when Bayes rule should be used

1 2 3

14 6 3, ,

26 26 26P B A P B A P B A

1 2 3S A A A

Let Ai = the event that we select urn i

1 2 3

1

3P A P A P A

Let B = the event that we select a red ball

We want for 1, 2,3.iP A B i

Bayes rule states

1 11

1 1 2 2 3 3

P A P B AP A B

P A P B A P A P B A P A P B A

1 143 26

6 31 14 1 13 26 3 26 3 26

14 14

14 6 3 23

613 26

2 6 31 14 1 13 26 3 26 3 26

6 6

14 6 3 23P A B

313 26

3 6 31 14 1 13 26 3 26 3 26

3 3

14 6 3 23P A B

Example:

Suppose that an electronic device is manufactured by a company.During a period of a week

– 15% of this product is manufactured on Monday,– 23% on Tuesday, – 26% on Wednesday , – 24% on Thursday and – 12% on Friday.

Also during a period of a week – 5% of the product is manufactured on Monday is

defective– 3 % of the product is manufactured on Tuesday is

defective, – 1 % of the product is manufactured on Wednesday

is defective , – 2 % of the product is manufactured on Thursday is

defective and – 6 % of the product is manufactured on Friday is

defective.

If the electronic device manufactured by this plant turns out to be defective, what is the probability that is as manufactured on Monday, Tuesday, Wednesday, Thursday or Friday?

Solution:

LetA1 = the event that the product is manufactured on

Monday

A2 = the event that the product is manufactured on Tuesday

A3 = the event that the product is manufactured on Wednesday

A4 = the event that the product is manufactured on Thursday

A5 = the event that the product is manufactured on Friday

Let B = the event that the product is defective

Now

P[A1] = 0.15, P[A2] = 0.23, P[A3] = 0.26, P[A4] = 0.24 and P[A5] = 0.12Also

P[B|A1] = 0.05, P[B|A2] = 0.03, P[B|A3] = 0.01,

P[B|A4] = 0.02 and P[B|A5] = 0.06We want to find

P[A1|B], P[A2|B], P[A3|B], P[A4|B] and P[A5|B] .

1 1 5 5

i i

i

P A P B AP A B

P A P B A P A P B A

We will apply Bayes Rule

i P[Ai] P[B|Ai] P[Ai]P[B|Ai] P[Ai|B]

1 0.15 0.05 0.0075 0.2586

2 0.23 0.03 0.0069 0.2379

3 0.26 0.01 0.0026 0.0897

4 0.24 0.02 0.0048 0.1655

5 0.12 0.06 0.0072 0.2483

Total 1.00 0.0290 1.0000

The sure thing principle and Simpson’s paradox

The sure thing principleSuppose andP A C P B C

thenP A C P B C

P A P B

Example – to illustrate

Let A = the event that horse A wins the race.

B = the event that horse B wins the race.

C = the event that the track is dry

= the event that the track is muddyC

Proof:

implies P A C P B C

P A C P B CP C P C

P A P A C P A C

or P A C P B C

implies P A C P B C

P A C P B CP C P C

or P A C P B C

P B C P B C P B

Simpson’s ParadoxDoes

andP D S C P D S C implyP D S C P D S C

?P D S P D S Example to illustrate

D = death due to lung cancerS = smokerC = lives in city, = lives in countryC

If we let

Then the statement would be true using the Sure Thing Principle

This logic is incorrectThe events

and A D S B D S

and A D S B D S

are not defined and do not make sense.

The conditional probabilities

are defined.

and P D S P D S

P D S C P D S CP D SP D S

P S P S

similarly

P D S C P S CP D S C P S C

P S C P S P SP S C

P D S C P C S P D S C P C S

P D S P D S C P C S P D S C P C S

Solution

is greater than

P D S P D S C P C S P D S C P C S

P D S P D S C P C S P D S C P C S

if andP D S C P D S C

P D S C P D S C

whether

, 1 ,P C S P C S P C S

depends also on the values of

and 1P C S P C S P C S

thanP D S P D S C P C S P D S C P C S

P D S P D S C P C S P D S C P C S

Suppose 0.90 =0.60 andP D S C P D S C

0.40 0.10P D S C P D S C

whether.10, 1 .90,P C S P C S P C S

and

=.80 and 1 0.20P C S P C S P C S

.90 .10 .40 .90 .45

.60 .80 .10 .20 .50

1 2 3

The Monty Hall Problem

Behind one of the three doors there is a valuable prize. Behind the other two doors is a worthless prize. You are asked to pick one of the doors. After you have selected, Monty Hall opens one of the doors and reveals a worthless prize. He then asks you do you want to switch your choice.

1. Should you change your choice?

2. Should you keep your first choice? or

3. It does not matter.

Solution

Suppose you choice is door #1, and Monty reveals that door #3 has a worthless prize behind it. We can always renumber the doors so that this is the case.

LetAi = the event that the valuable prize is behind door

number i. i = 1, 2, 3.

P [A1] = P [A2] = P [A3] =1/3

S = A1 A2 A3 and Ai Aj =

1 1 2

1 3 1 1 21 2

P A A AP A A P A A A

P A A

11 3

231 2

1

2

P A

P A A

2 3

1

2P A A

Another Solution (the correct solution)

The probability that you pick the correct door is 1/3 . If you pick the correct door Monty will pick randomly between the two worthless doors. If you did not pick the correct door Monty will choose the worthless door to open with with probability 1.

Let

Bi = the event that Monty opens door i. i = 1, 2, 3.

Again P [A1] = P [A2] = P [A3] =1/3

and S = A1 A2 A3 and Ai Aj =

1 1 2 1 3 1

1 10, ,

2 2P B A P B A P B A

Also

We want to compute P [A1|B3] andP [A2|B3].

1 2 2 2 3 20, 0, 1P B A P B A P B A

and1 3 2 3 3 30, 1, 0P B A P B A P B A

1 3

1 33

P A BP A B

P B

1 3 1

1 3 2 3 3 31 2 3

P A P B A

P A P B A P A P B A P A P B A

1 13 2

1 1 1 13 2 3 3

1 1

1 0 1 2 3

and

2 3

2 33

P A BP A B

P B

2 3 2

1 3 2 3 3 31 2 3

P A P B A

P A P B A P A P B A P A P B A

13

1 1 1 13 2 3 3

1 2 2

1 0 1 2 3

1 2 3

Another Problem

We have three chests each having 2 drawers

In chest 1 there is a gold coin in each drawer.

In chest 2 there is a silver coin in each drawer.

In chest 3 there is a gold coin in the top drawer and a silver coin in the bottom drawer..

One of the chests is selected at random. Then the drawer is selected at random. The coin in that drawer turns out to be gold.

What is the probability that the coin in the other drawer is also gold? Is it ½ ?

1 2 3

SolutionLetCi = the event that we select Chest i. i = 1, 2, 3.

P [C1] = P [C2] = P [C3] =1/3

S = C1 C2 C3 and Ci Cj =

We want to compute

LetD1 = the event that we select top drawer in the chest.

G = the event the coin in the drawer is goldLet

= (C1 D1) (C1 D2) (C3 D1)

D2 = the event that we select bottom drawer in the chest.

P[C1|G].

1

1

P C GP C G

P G

1 1 1 1 2 1 3

1 1 1 2 1 3

P C C D C D C D

P C D C D C D

1 1 1 2

1 1 1 2 1 3

P C D C D

P C D C D C D

Thus

1 1 1 1 2 1 3

1

1 1 1 2 1 3

P C C D C D C DP C G

P C D C D C D

1 1 1 2

1 1 1 2 3 1

P C D P C D

P C D P C D P C D

1 1 1 1 2 1

1 1 1 1 2 1 3 1 3

P C P D C P C P D C

P C P D C P C P D C P C P D C

1 1 1 13 2 3 2

1 1 1 1 1 13 2 3 2 3 2

2

3

Comment: There are 6 drawers and three of those drawers contain gold coins. Of those three drawers two are in a chest that has a gold coin in the other drawer.

Random Variables

an important concept in probability

A random variable , X, is a numerical quantity whose value is determined be a random experiment

Examples1. Two dice are rolled and X is the sum of the two upward

faces.

2. A coin is tossed n = 3 times and X is the number of times that a head occurs.

3. We count the number of earthquakes, X, that occur in the San Francisco region from 2000 A. D, to 2050A. D.

4. Today the TSX composite index is 11,050.00, X is the value of the index in thirty days

Examples – R.V.’s - continued5. A point is selected at random from a square whose sides are

of length 1. X is the distance of the point from the lower left hand corner.

6. A chord is selected at random from a circle. X is the length of the chord.

point

X

chord

X

Definition – The probability function, p(x), of a random variable, X.

For any random variable, X, and any real number, x, we define

p x P X x P X x

where {X = x} = the set of all outcomes (event) with X = x.

Definition – The cumulative distribution function, F(x), of a random variable, X.

For any random variable, X, and any real number, x, we define

F x P X x P X x

where {X ≤ x} = the set of all outcomes (event) with X ≤ x.

(1,1)

2

(1,2)

3

(1,3)

4

(1,4)

5

(1,5)

6

(1,6)

7

(2,1)

3

(2,2)

4

(2,3)

5

(2,4)

6

(2,5)

7

(2,6)

8

(3,1)

4

(3,2)

5

(3,3)

6

(3,4)

7

(3,5)

8

(3,6)

9

(4,1)

5

(4,2)

6

(4,3)

7

(4,4)

8

(4,5)

9

(4,6)

10

(5,1)

6

(5,2)

7

(5,3)

8

(5,4)

9

(5,5)

10

(5,6)

11

(6,1)

7

(6,2)

8

(6,3)

9

(6,4)

10

(6,5)

11

(6,6)

12

Examples1. Two dice are rolled and X is the sum of the two upward

faces. S , sample space is shown below with the value of X for each outcome

12 2 1,1

36p P X P

23 3 1,2 , 2,1

36p P X P

34 4 1,3 , 2,2 , 3,1

36p P X P

4 5 6 5 45 , 6 , 7 , 8 , 9

36 36 36 36 36p p p p p

3 2 110 , 11 , 12

36 36 36p p p

and 0 for all other p x x

: for all other X x x Note

Graph

0.00

0.06

0.12

0.18

2 3 4 5 6 7 8 9 10 11 12

x

p(x)

The cumulative distribution function, F(x)

For any random variable, X, and any real number, x, we define

F x P X x P X x

where {X ≤ x} = the set of all outcomes (event) with X ≤ x.

Note {X ≤ x} =if x < 2. Thus F(x) = 0.

{X ≤ x} ={(1,1)} if 2 ≤ x < 3. Thus F(x) = 1/36

{X ≤ x} ={(1,1) ,(1,2),(1,2)} if 3 ≤ x < 4. Thus F(x) = 3/36

0

0.2

0.4

0.6

0.8

1

1.2

0 5 10

Continuing we find

F x

136

336

636

1036

1536

2136

2636

3036

3336

3536

0 2

2 3

3 4

4 5

5 6

6 7

7 8

8 9

9 10

10 11

11 12

121

x

x

x

x

x

x

x

x

x

x

x

x

F(x) is a step function

2. A coin is tossed n = 3 times and X is the number of times that a head occurs.

The sample Space S = {HHH (3), HHT (2), HTH (2), THH (2), HTT (1), THT (1), TTH (1), TTT (0)}

for each outcome X is shown in brackets

10 0 TTT

8p P X P

31 1 HTT,THT,TTH

8p P X P

32 2 HHT,HTH,THH

8p P X P

13 3 HHH

8p P X P

0 for other .p x P X x P x

Graphprobability function

0

0.1

0.2

0.3

0.4

0 1 2 3

p(x)

x

0

0.2

0.4

0.6

0.8

1

1.2

-1 0 1 2 3 4

GraphCumulative distribution function

F(x)

x

Examples – R.V.’s - continued5. A point is selected at random from a square whose sides are

of length 1. X is the distance of the point from the lower left hand corner.

6. A chord is selected at random from a circle. X is the length of the chord.

point

X

chord

X

Examples – R.V.’s - continued5. A point is selected at random from a square whose sides are

of length 1. X is the distance of the point from the lower left hand corner.

An event, E, is any subset of the square, S.

P[E] = (area of E)/(Area of S) = area of E

point

X

S

E

Thus p(x) = 0 for all values of x. The probability function for this example is not very informative

set of all points a dist 0

from lower left corner

xp x P X x P

S

The probability function

set of all points within a

dist from lower left cornerF x P X x P

x

The Cumulative distribution function

S

0 1x 1 2x 2 x

xx

x

2

4

000 1

Area 1 2

1 2

x

x

xF x P X x

A x

x

S

0 1x 1 2x 2 x

xx

xA

Computation of Area A 1 2x

xA

x

1

1

22

2 1x

2 1x

2

2 2 21 12 1

2 2 2

xA x x x

2 2 2 1 2 21 1 tan 14 4

x x x x x

2tan 1x

1 2tan 1x

0

1

-1 0 1 2

2

2 1 2 2

0 0

0 14

1 tan 1 1 24

1 2

x

xx

F x P X xx x x x

x

F x

The probability density function, f(x), of a continuous random variableSuppose that X is a random variable.

Let f(x) denote a function define for - < x < with the following properties:

1. f(x) ≥ 0

2. 1.f x dx

3. .

b

a

P a X b f x dx Then f(x) is called the probability density function of X.

The random, X, is called continuous.

Probability density function, f(x)

1.f x dx

.b

a

P a X b f x dx

Cumulative distribution function, F(x)

F x

.x

F x P X x f t dt

Thus if X is a continuous random variable with probability density function, f(x) then the cumulative distribution function of X is given by:

.x

F x P X x f t dt

Also because of the fundamental theorem of calculus.

dF xF x f x

dx

ExampleA point is selected at random from a square whose sides are of length 1. X is the distance of the point from the lower left hand corner.

point

X

2

2 1 2 2

00

0 14 1 2

1 tan 14 2

1

xxx

F x P X xx

x x xx

Now

2 1 2 2

0 0 or 2

0 12

1 tan 1 1 24

x x

xf x F x x

dx x x x

dx

2 1 2 21 tan 14

dx x x

dx

3

21 22 1 2

2x x x

1 2 2 1 22 tan 1 tan 1d

x x x xdx

Also

3

2

1 2

22 tan 1

2 1

xx x x

x

2 1 2tan 1d

x xdx

Now

321 2 2

2

1 1tan 1 1 2

21 1

dx x x

dx x

12

1tan

1

du

du u

and

32

2 1 2

2tan 1

1

d xx x

dx x

2 1 2 21 tan 14

dx x x

dx

1 22 tan 12

x x x

Finally

1 2

0 0 or 2

0 12

2 tan 1 1 22

x x

xf x F x x

x x x x

Graph of f(x)

0

0.5

1

1.5

2

-1 0 1 2