© 2009 Winton 1

Empirical and Discrete Distributions

© 2009 Winton 2

E i i l Di t ib tiEmpirical Distributions• An empirical distribution is one for which each

possible event is assigned a probability derived f i l b ifrom experimental observation– It is assumed that the events are independent and the

sum of the probabilities is 1sum of the probabilities is 1• An empirical distribution may represent either a

continuous or a discrete distributioncontinuous or a discrete distribution– If it represents a discrete distribution, then sampling is

done “on step”p– If it represents a continuous distribution, then sampling

is done via “interpolation”

© 2009 Winton 3

Discrete vs. Continuous Sampling

• The way the empirical table is described usually d i if i i l di ib i i bdetermines if an empirical distribution is to be handled discretely or continuously

discrete description continuous descriptionvalue probability value probability10 .1 0 – 10- .120 .15 10 – 20- .1535 .4 20 – 35- .440 .3 35 – 40- .360 .05 40 – 60- .05

© 2009 Winton 4

Empirical Distribution Linear Interpolationrsample 60

Empirical Distribution Linear Interpolationdiscrete case continuous casevalue prob value prob

50

40

value prob value prob10 .1 0 – 10- .120 .15 10 – 20- .1535 4 20 35 4

30

20

35 .4 20 – 35- .440 .3 35 – 40- .360 .05 40 – 60- .05

0 5 1

10

0 x

• To use linear interpolation for continuous sampling, the discrete points on the end of each step need to be

0 .5 1

discrete points on the end of each step need to be connected by line segments– This is represented in the graph by the green line segments– The steps are represented in blue

© 2009 Winton 5

Sampling On Steprsample 60

Sampling On Stepdiscrete case continuous casevalue prob value prob

50

40

value prob value prob10 .1 0 – 10- .120 .15 10 – 20- .1535 4 20 35 4

30

20

35 .4 20 – 35- .440 .3 35 – 40- .360 .05 40 – 60- .05

0 5 1

10

0 x

• In the discrete case, sampling on step is accomplished by accumulating probabilities from the original table

0 .5 1

– For x = 0.4, accumulate probabilities until the cumulative probability exceeds 0.4

– rsample is the event value at the point this happens • the cumulative probability 0.1+0.15+0.4 is the first to exceed 0.4,

– the rsample value is 35

© 2009 Winton 6

Sampling By Linear Interpolationrsample 60

Sampling By Linear Interpolationdiscrete case continuous casevalue prob value prob

50

40

value prob value prob10 .1 0 – 10- .120 .15 10 – 20- .1535 4 20 35 4

30

20

35 .4 20 – 35- .440 .3 35 – 40- .360 .05 40 – 60- .05

0 5 1

10

0 x

• In the continuous case, the end points of the probability accumulation are needed– For x = 0.4, the values are x=0.25 and x=0.65 which represent the points

0 .5 1

(.25,20) and (.65,35) on the graph – From basic college algebra, the slope of the line segment is

(35-20)/(.65-.25) = 15/.4 = 37.5– slope = 37.5 = (35-rsample)/(.65-.4) so

rsample = 35 - (37.5.25) = 35 – 9.375 = 25.625

© 2009 Winton 7

Discrete Distributions• Historical perspective behind the names used with discrete

distributionsdistributions– James Bernoulli (1654-1705) was a Swiss mathematician whose book

Ars Conjectandi (published posthumously in 1713) was the first j (p p y )significant book on probability

• It gathered together the ideas on counting, and among other things provided a proof of the binomial theoremprovided a proof of the binomial theorem

– Siméon-Denis Poisson (1781-1840) was a professor of mathematics at the Faculté des Sciences whose 1837 text Recherchés sur la probabilité des jugements en matière criminelle et en matière civile introduced the discrete distribution now called the Poisson distribution

• Keep in mind that scholars such as these evolved their theories• Keep in mind that scholars such as these evolved their theories with the objective of providing sophisticated abstract models of real-world phenomenap– An effort which, among other things, gave birth to the calculus as a

major modeling tool

© 2009 Winton 8

I. Bernoulli DistributionBernoulli events, trials, and processes

• Bernoulli event• Bernoulli event– One for which the probability the event occurs is p and

the probability the event does not occur is 1-pthe probability the event does not occur is 1 p• The event is has two possible outcomes (usually viewed as

success or failure) occurring with probability p and 1-p, respectivelyrespectively

• Bernoulli trialAn instantiation of a Bernoulli event– An instantiation of a Bernoulli event

• Bernoulli processA sequence of Bernoulli trials where probability p of– A sequence of Bernoulli trials where probability p of success remains the same from trial to trial

– This means that for n trials, the probability of n , p yconsecutive successes is pn

© 2009 Winton 9

Bernoulli DistributionBernoulli Distribution• Bernoulli distribution

– Given by the pair of probabilities of a Bernoulli event • Too simple to be interesting in isolationToo simple to be interesting in isolation

– Implicitly used in “yes-no” decision processes where the choice occurs with the same probability from trial to trial p y

• e.g., the customer chooses to go down aisle 1 with probability p

– It can be cast in the same kind of mathematical notation used to describe more complex distributions

© 2009 Winton 10

Bernoulli Distribution pdfBernoulli Distribution pdfpz(1-p)1-z for z = 0,1 Note: this seemingly p ( p) ,

p(z) =0 otherwise

p(z)

bizarre description is chosen since it mimics the case of the

1

binomial distribution when n=1

1-pp

• The expected value of the distribution is given by)(( )

z10

• The standard deviation is given byp1p0p)(1 E(X) -

p)(1pp)p(1p)p)(0(1σ 22 • Notational overkill, but useful for understanding other distributions

p)(1pp)-p(1p)-p)(0-(1σ -

© 2009 Winton 11

SamplingSampling• Sampling from a discrete distribution requires a function thatSampling from a discrete distribution, requires a function that

corresponds to the distribution function of a continuous distribution f given by

x

f(z)dz F(x)

• This is given by the mass function F(x) of the distribution, which is the step function obtained from the cumulative (discrete) distribution

( )( )

p ( )given by the sequence of partial sums

F th B lli di t ib ti F( ) h th t ti

x

0p(z)

• For the Bernoulli distribution, F(x) has the construction 0 for - x < 0

F(x) = 1-p for 0 x < 1( ) p1 for x 1

– A non-decreasing function (an so can be sampled on step)

© 2009 Winton 12

Graph of F(x) and Sampling FunctionGraph of F(x) and Sampling FunctionF(x)

1• Distribution function F(x)

F(x)

p

z1-p

• Sampling function

z10

l– for random value x drawn from [0,1),0 if 0 x < 1-p

rsample =1 if 1-p x < 1

rsample

1p– demonstrates that sampling from a

discrete distribution, even one as simple as the Bernoulli distribution, can be ie ed in the same manner ascan be viewed in the same manner as for continuous distributions 1

x0 1-p

© 2009 Winton 13

II Bi i l Di t ib tiII. Binomial DistributionTh B lli di ib i h f il• The Bernoulli distribution represents the success or failure of a single Bernoulli trial

• The Binomial Distribution represents the number of• The Binomial Distribution represents the number of successes and failures in n independent Bernoulli trials for some given value of n– For example, if a manufactured item is defective with probability

p, then the binomial distribution represents the number of successes and failures in a lot of n items

• Unlike a barrel of apples, where one bad apple can cause others to be bad

– Sampling from this distribution gives a count of the number of p g gdefective items in a sample lot

– Another example is the number of heads obtained in tossing a coin n timesn times

© 2009 Winton 14

Binomial Theorem (1)o a eo e ( )• The binomial distribution gets its name from the

bi i l hbinomial theorem

n!nwhereba

nba k-nk

nn

• It is worth pointing out that if a = b = 1 this becomes

k)!-(nk!k

wherebak

ba0

It is worth pointing out that if a b 1, this becomes

n211

nnn

• If S is a set of size n the number of k element subsets

k0

If S is a set of size n, the number of k element subsets of S is given by

kn

k)!(k!n!

kk)!-(nk!

© 2009 Winton 15

Binomial Theorem (2)o a eo e ( )• This formula is the result of a simple counting

l i hanalysis: there are

k)!(nn!1)k-(n...1)-(nn

ordered ways to select k elements from n t h th 1st it ( 1) th 2nd d

k)!-(n

– n ways to choose the 1st item, (n-1) the 2nd , and so on– Any given selection is a permutation of its k elements, so

the underlying subset is counted k! timesthe underlying subset is counted k! times– Dividing by k! eliminates the duplicates

C 2n t th t t l b f b t f• Consequence: 2n counts the total number of subsets of an n-element set

© 2009 Winton 16

Binomial Distribution pdf (1)Binomial Distribution pdf (1)• For n independent Bernoulli trials the pdf of the p p

binomial distribution is given byn

p(z) = 0 h i

n...,1,0,zfor p)(1pz

znz

0 otherwise• By the binomial theorem

1 p))-(1 (p p(z) nn

0

verifying that p(z) is a pdf0

© 2009 Winton 17

Binomial Distribution pdf (2)Binomial Distribution pdf (2)Wh h i it f it ith• When choosing z items from among n items with probability p for an item being defective, the term

znz p)(1p zn

represents the probability that z are defective (and concomitantly that (n-z) are not defective)

concomitantly that (n z) are not defective)

© 2009 Winton 18

E t d V l d V iExpected Value and Variance• E(X) = np for a binomial distribution on n items

where probability of success is p • The calculation is accomplished by

zp)(1pn!zp)(1pn

z)(zpE(X)n

z-nzz-nzn

ii

n

term in common

np p)-1np(p p)-(1p 11-n

np npp)-(1p 11-n

1-nz-n1-zn

z-n1-zn

zp)(1pz)!-(nz!

zp)(1pz

z)(zp E(X)10

ii1

1-z1-z 11

present in every summand apply the binomial theorem to this (note that n-z = (n-1) - (z-1) )

• It can be similarly shown that the standard deviation is )p1(np

( ( ) ( ) )

)p(p

© 2009 Winton 19

Graph of pdf and Mass Function F(x)Graph of pdf and Mass Function F(x)

• The binomial distribution with n=10 and p=0.7 appears as f ll ( 7)follows: (mean = np = 7)

It di f ti• Its corresponding mass function F(z) is given by

© 2009 Winton 20

Sampling FunctionSa p g u c o109

rsample

87655432

A t i l li t ti i t l t th rsample

( )0 1

10 x

• A typical sampling tactic is to accumulate the sum

increasing rsample until the sum's value exceeds the random value between 0 and 1 drawn for x

0

p(z)

between 0 and 1 drawn for x– The final rsample summation limit is the sample value – In contrast to a continuous pdf described by some formula, the

function for a finite discrete pdf has to be given in its relational u ct o o a te d sc ete pd as to be g ve ts e at o aform by a table of pairs, which in turn mandates the kind of "search" algorithm approach used above to obtain rsample

© 2009 Winton 21

III Poisson DistributionIII. Poisson Distribution(values z = 0, 1, 2, . . .)

• With a little work it can be shown that the Poisson di ib i i h li i i f h bi i ldistribution is the limiting case of the binomial distribution using p= /n 0 as n

• The expected value E(X) = • The standard deviation is λ• The pdf is given by eλp(z)

λz

z!p( )

© 2009 Winton 22

III. Poisson Distributionhistory and use

• This distribution dates back to Poisson's 1837 text di i il d i i l i ff hi hregarding civil and criminal matters, in effect scotching the

tale that its first use was for modeling deaths from the kicks of horses in the Prussian armykicks of horses in the Prussian army

• In addition to modeling the number of arrivals over some interval of time the distribution has also been used tointerval of time the distribution has also been used to model the number of defects on a manufactured article– Recall the relationship to the exponential distribution; a Poisson

process has exponentially distributed interarrival times

• In general the Poisson distribution is used for situations h th b bilit f t i i llwhere the probability of an event occurring is very small,

but the number of trials is very large (so the event is expected to actually occur a few times)expected to actually occur a few times)– Less cumbersome than the binomial distribution

© 2009 Winton 23

Graph of pdf and Sampling Functionp p p g• With = 2 0.25

0.3p(z) mean

0.15

0.2

9

rsample

0

0.05

0.1

z

987

0 1 2 3 4 5 6 7 8 9 10 . . .65432210 x

0 1x

© 2009 Winton 24

IV Geometric DistributionIV. Geometric Distribution• The geometric distribution gets its name from the geometric series:

20

n2

0

n

0

n

r)-(11 r1)(n ,

r)-(1r rn ,

r-11 r 1, r for

i fl f h i i• The pdf for the geometric distribution is given by

21zforpp)(1 1-z

various flavors of the geometric series

p(z) = otherwise

• The geometric distribution is the discrete analog of the exponential

...2,1,zfor pp)(1

The geometric distribution is the discrete analog of the exponential distribution– Like the exponential distribution, it is "memoryless"; i.e.,

P(X > a+b | X > a) = P(X > b)– The geometric distribution is the only discrete distribution with this

property just as the exponential distribution is the only continuousproperty just as the exponential distribution is the only continuous one behaving in this manner

© 2009 Winton 25

Memoryless Propertye o y ess ope y• Being memoryless is characterized by

P(X > a+b | X > a) = P(X > b)P(X > a+b | X > a) = P(X > b)• The interpretation is that if we haven’t had an arrival in “a” seconds, the

probability of an arrival in the next “b” seconds is the same as if the “a” seconds had not elapsed– Only the geometric and exponential distributions have this property

• This is NOT a property associated with independent events since among• This is NOT a property associated with independent events, since among other things there are three events involved: X > a+b, X > a, and X > b– Recall that A and B are independent if P(A ∩ B) = P(A)P(B) – So long as P(B) is non-zero, P(A | B) = P(A ∩ B )/P(B)

• If effect, if A and B are independent events and B has already occurred then by this equation P(A) is unchanged; i e as one wouldoccurred, then by this equation P(A) is unchanged; i.e., as one would expect P(A|B) = P(A) when A and B are independent

– The events X > a + b and X > a are certainly NOT independent since independence would infer P(X > a+b | X > a) = P(X > a+b) – NOT!

© 2009 Winton 26

Expected Value and StandardExpected Value and Standard Deviation

• With expected value is given by11

p1

p)1-(11p pp)-z(1 E(X) 2

1

1-z

– by applying the 3rd form of the geometric series

• The standard deviation is given byg y

pp1

p

© 2009 Winton 27

GraphGraph• A plot of the geometric distribution with p = 0 3 is given by• A plot of the geometric distribution with p = 0.3 is given by

0 35p(z)

mean0 250.3

0.35

0 15

0.2

0.25

0 050.1

0.15

0

0.05

1 2 3 4 5 6 7 8 9 10z

1 2 3 4 5 6 7 8 9 10

© 2009 Winton 28

Utili tiUtilizationA t i l f th t i di t ib ti i f d li• A typical use of the geometric distribution is for modeling the number of failures before the first success in a sequence of independent Bernoulli trialssequence of independent Bernoulli trials

• This is the scenario for making sales– Suppose that the probability of making a sale is 0.3pp p y g– Then

• p(1) = 0.3 is the probability of success on the 1st try• p(2) = (1-p)p = 0.70.3 = 0.21 which is the probability of

failing on the 1st try (with probability 1-p) and succeeding on the 2nd (with probability p)( p y p)

• p(3) = (1-p)(1-p)p = 0.15 is the probability that the sale takes 3 tries, and so forth

A d l f h di ib i h b f– A random sample from the distribution represents the number of attempts needed to make the sale