pycon 2015- bayesian statistics made simple

8/18/2019 PyCon 2015- Bayesian Statistics Made Simple

1/145

Bayesian Statistics

Made Simple Allen B. Downey

Olin College

sites.google.com/site/simplebayes

http://sites.google.com/site/simplebayeshttp://sites.google.com/site/simplebayes


2/145


3/145

Follow along at home

sites.google.com/site/simplebayes

http://sites.google.com/site/simplebayeshttp://sites.google.com/site/simplebayes


4/145

The plan

From Bayes's Theorem to Bayesian inference.

A computational framework.

ork on e!ample problems.


5/145

Goals

By the en"# you shoul" be rea"y to$

%ork on similar problems.

%&earn more on your own.


6/145

Think Bayes

This tutorial is base" on my

book#

Think Bayes

Bayesian Statistics in Python

ublishe" by O'(eilly )e"ia

an" a*ailable un"er aCreati*e Commons license from

thinkbayes.com


7/145

Bayes’s Theorem

+igh on my list of "esert islan" algorithms$

,.-ulers metho"

.Bayess theorem

0.1aplan2)eier estimation3

4ot on the list$ 56"elsincompleteness theorem.


8/145

Probability

p7A8$ the probability that A occurs.

p7A9B8$ the probability that A occurs# gi*en that

B has occurre".

p7A an" B8 : p7A8 p7B9A8


9/145

Bayes's Theorem

By "efinition of con;oint probability$

p7A an" B8 : p7A8 p7B9A8 : 7,8

p7B an" A8 : p7B8 p7A9B8

-


10/145

Bayes's Theorem


11/145

Bayes's Theorem

One way to think about it$

Bayes's Theorem is an algorithm to get

from p7B9A8 to p7A9B8.

=seful if p7B9A8# p7A8 an" p7B8 are easier

than p7A9B8.

O( ...


12/145

Diachronic interpretation

+$ +ypothesis

D$ Data

5i*en p7+8# the probability of the hypothesisbefore you saw the "ata.

Fin" p7+9D8# the probability of the hypothesisafter you saw the "ata.


13/145

A cookie problem

>uppose there are two bowls of cookies.Bowl ?, has ,@ chocolate an" 0@ *anilla.

Bowl ? has @ of each.

Fre" picks a bowl at ran"om# an" then picks acookie at ran"om. The cookie turns out to be

*anilla.

hat is the probability that Fre" picke" fromBowl ?,

from ikipe"ia


14/145

ookie problem

+$ +ypothesis that cookie came from Bowl ,.

D$ Cookie is *anilla.

5i*en p7+8# the probability of the hypothesisbefore you saw the "ata.

Fin" p7+9D8# the probability of the hypothesisafter you saw the "ata.


15/145


p7+9D8 : p7+8 p7D9+8 / p7D8

p7+8$ prior

p7D9+8$ con"itional likelihoo" of the "ata

p7D8$ total likelihoo" of the "ata


16/145


p7+9D8 : p7+8 p7D9+8 / p7D8

p7+8$ prior : ,/

p7D9+8$ con"itional likelihoo" of the "ata : 0/

p7D8$ total likelihoo" of the "ata : /


17/145


p7+9D8 : 7,/870/8 / 7/8 : 0/

p7+8$ prior : ,/

p7D9+8$ con"itional likelihoo" of the "ata : 0/

p7D8$ total likelihoo" of the "ata : /


18/145

A little int!ition

p7+8$ prior : @E

p7+9D8$ posterior : @E

Ganilla cookie was more likely un"er +.

>lightly increases our "egree of belief in +.


19/145

omp!tation

mf represents a robability )ass Function

)aps from possible *alues to probabilities.

Diagram by yuml.me


20/145

"nstall test

+ow many of you got install_test.py running

Don't try to fi! it nowH

Instea"...


21/145

Partner !p

%If you "on't ha*e a working en*ironment# fin"

a neighbor who "oes.

%-*en if you "o# try pair programmingH

%Take a minute to intro"uce yourself.

%Juestions Ask your partner first 7please8.


22/145

"cebreaker

hat was your first computer

hat was your first programming language

hat is the longest time you ha*e spent fin"ing

a stupi" bug


23/145

Start yo!r engines

,.Kou clone" Bayes)a"e>imple# right

2.cd into that "irectory.0.>tart the ython interpreter.

$ python>>> from thinkbayes import Pmf


24/145

#r "Python

1.cd into Bayes)a"e>imple..>tart Iython.

0.Create a new notebook.

$ ipython notebook

from thinkbayes import Pmf


25/145

Pm$

from thinkbayes import Pmf

# make an empty Pmf

d6 = Pmf()

# outcomes of a six-sided die

for x in [1,,!,",,6$% d6.&et(x, 1)


26/145

Pm$

d6.Print()

d6.'ormaie()

d6.Print()

d6.*andom()


27/145

The Bayesian $ramework

,8 Buil" a mf that maps from each hypothesis

to a prior probability# p7+8.

8 )ultiply each prior probability by thelikelihoo" of the "ata# p7D9+8.

08 4ormaliLe# which "i*i"es through by the totallikelihoo"# p7D8.


28/145

Prior

pmf = Pmf()

pmf.&et(+o 1+, .)

pmf.&et(+o +, .)


29/145

%pdate

p(/ania 0 o 1) = !"

p(/ania 0 o ) = "

pmf.2ut(+o 1+, .3)pmf.2ut(+o +, .)


30/145

&ormalie

pmf.'ormaie()

.6 # return 4aue is p(5)

print pmf.Prob(+o 1+).6


31/145

()ercise

hat if we select another cookie# an" its

chocolate

The posterior 7after the first cookie8 becomesthe prior 7before the secon" cookie8.


32/145

()ercise

hat if we select another cookie# an" its

chocolate

pmf.2ut(+o 1+, .)pmf.2ut(+o +, .)

pmf.'ormaie()

pmf.Print()o 1 ."!

o .3!


33/145

S!mmary

Bayes's Theorem#

Cookie problem#

mf class.


34/145

The dice problem

I ha*e a bo! of "ice that contains a 2si"e" "ie#

a 2si"e" "ie# an 2si"e" "ie# a ,2si"e" "ie

an" a @2si"e" "ie.

>uppose I select a "ie from the bo! at ran"om#

roll it# an" get a . hat is the probability that I

rolle" each "ie


35/145

*ypothesis s!ites

A suite is a mutually e!clusi*e an" collecti*ely

e!hausti*e set of hypotheses.

(epresente" by a &uite that maps

hypothesis M probability.


36/145

S!ite

cass &uite(Pmf)%

*epresents a suite of hypotheses and

their probabiities.

def 77init77(sef, hypos)% 8nitiaies the distribution.

for hypo in hypos%

sef.&et(hypo, 1)

sef.'ormaie()


37/145

S!ite

def 9pdate(sef, data)%

9pdates the suite based on data.

for hypo in sef./aues()% ike = sef.Likelihood (data, hypo)

sef.2ut(hypo, ike)

sef.'ormaie()

self.&ikelihoo"


38/145

S!ite

:ikeihood is an abstract metho".

Chil" classes inherit 9pdate#

pro*i"e :ikeihood.


39/145

+ikelihood

Outcome$ %hat is the likelihoo" of this outcome on a

si!2si"e" "ie

%On a ten2si"e" "ie%On a four2si"e" "ie


40/145

+ikelihood

# hypo is the number of sides on the die

# data is the outcome

cass 5ice(&uite)%

def :ikeihood(sef, data, hypo)%

# rite this method;

rite your solution in dice.py


41/145

+ikelihood

# hypo is the number of sides on the die

# data is the outcome

cass 5ice(&uite)%


if hypo < data%

return ese%

return 1.hypo


42/145

Dice

# start ith eua priors

suite = 5ice([", 6, >, 1, $)

# update ith the data

suite.9pdate(6)

suite.Print()


43/145


44/145

Dice

for ro in [>, 3, 3, , "$%

suite.9pdate(ro)

suite.Print()


45/145

Dice

osterior "istribution$

" .

6 .

> .?

1 .>

.!>


46/145

S!mmary

Dice problem#

&ikelihoo" function#

>uite class.


47/145

http$//images.forwallpaper.com/files/images//N,a/N,a0e,N//coffee2break2yet.;pg


48/145

Trains

The trainspotting problem$%Kou belie*e that a freight carrier operates

between ,@@ an" ,@@@ locomoti*es with

consecuti*e serial numbers.%Kou spot locomoti*e ?0,.%+ow many locomoti*es "oes the carrier

operate

)o"ify train.py to compute your answer.


49/145

Trains

%If there are m trains# what is the chance of

spotting train ?n

%hat "oes the posterior "istribution look

like

%+ow woul" you summariLe it


50/145

Train

print suite.2ean()

print suite.2aximum:ikeihood()

print suite.@redibe8nter4a(?)


51/145

Trains

%hat if we spot more trains

%hy "i" we "o this e!ample


52/145

Trains

%ractice using the Bayesian framework# an"

figuring out &ikelihoo"78.

%-!ample that uses sparse "ata.

%Its a non2tri*ial# real problem.


53/145

Tanks

The 5erman tank problem.en.wikipe"ia.org/wiki/5ermanPtankPproblem

http://en.wikipedia.org/wiki/German_tank_problemhttp://en.wikipedia.org/wiki/German_tank_problem


54/145

Good time $or ,!estions


55/145

A (!ro problem

QWhen spun on edge 250 times, a Belgian one-

euro coin came up heads 140 times and tails

110. 't loo!s "ery suspicious to me,' said Barry

Blight, a statistics lecturer at the #ondon Schoolo$ %conomics. '$ the coin &ere uniased, the

chance o$ getting a result as e(treme as that

&ould e less than )*.' Q

From QThe 5uar"ianQ


56/145

A (!ro problem

)ac1ay asks# But do these data gi"e e"idence

that the coin is iased rather than $air

Assume that the coin has probability ( of

lan"ing hea"s.

7Forget that ( is a probabilityR ;ust think of it as

a physical characteristic.8


57/145

A (!ro problem

-stimation$ Base" on the "ata 7,@ hea"s# ,,@

tails8# what is (

+ypothesis testing$ hat is the probability that

the coin is fair


58/145

(!ro

e can use the >uite template again.

e ;ust ha*e to figure out the likelihoo"

function.


59/145

+ikelihood

# hypo is the prob of heads (1-1)

# data is a strinA, either +B+ or +C+

cass Duro(&uite)%


# one more, pease;

)o"ify euro.py to compute your answer.


60/145

+ikelihood

# hypo is the prob of heads (1-1)

# data is a strinA, either +B+ or +C+

cass Duro(&uite)%


x = hypo 1.

if data == +B+% return x

ese%

return 1-x


61/145

Prior

hat "o we belie*e about ( before seeing the

"ata

>tart with something simpleR we'll come backan" re*iew.

=niform prior $ any *alue of ( between @E an"

,@@E is e


62/145

Prior

suite = Duro(ranAe(, 11))


63/145


64/145

%pdate

>uppose we spin the coin once an" get hea"s.

suite.9pdate(+B+)

hat "oes the posterior "istribution look like

+int$ what is p7!:@E 9 D8


65/145


66/145

%pdate

>uppose we spin the coin again# an" get hea"s

again.

suite.9pdate(+B+)



67/145


68/145

%pdate

>uppose we spin the coin again# an" get tails.

suite.9pdate(+C+)


+int$ what's p7!:,@@E 9 D8


69/145


70/145

%pdate

After ,@ spins# N hea"s an" 0 tails$

for outcome in +BBBBBBBCCC+%

suite.9pdate(outcome)


71/145


72/145

%pdate

An" finally# after ,@ hea"s an" ,,@ tails$

e4idence = +B+ E 1" F +C+ E 11

for outcome in e4idence% suite.9pdate(outcome)


73/145


74/145

Posterior

%4ow what%+ow "o we summariLe the information in the

posterior >uite


75/145

Posterior

5i*en the posterior "istribution# what is the

probability that ! is @E

suite.Prob()

An" the answer is... @.@,

+mm. )aybe that's not the right


76/145

Posterior

+ow about the most likely *alue of !

pmf.2aximum:ikeihood()

An" the answer is E.


77/145

Posterior

Or the e!pecte" *alue

suite.2ean()

An" the answer is .E.


78/145

Posterior

Cre"ible inter*al

suite.@redibe8nter4a(?)


79/145

Posterior

The th percentile is ,.

The th percentile is ,.

These *alues form a @E cre"ible inter*al.

>o can we say$ QThere's a @E chance that ( is

between , an" ,Q


80/145

Fre,!entist response

Thank you smbc2comics.com


81/145

Bayesian response

Kes# ( is a ran"om *ariable#

Kes# 7,# ,8 is a @E cre"ible inter*al#

Kes# ( has a @E chance of being in it.

ro$ Bayesian stats are amenable to "ecision

analysis.

Con$ The prior is sub;ecti*e.


82/145

The prior is s!b-ecti.e

(emember the prior

e chose it pretty arbitrarily# an" reasonable

people might "isagree.

Is ( as likely to be ,E as @E

5i*en what we know about coins# I "oubt it.


83/145

Prior

+ow shoul" we capture backgroun" knowle"ge

about coins

Try a triangle prior.


84/145


85/145

Posterior

hat "o you think the posterior "istributions

look like

I was going to put an image here# but then I5oogle" QposteriorQ. 4e*er min".


86/145

Swamp the prior


87/145

Swamp the prior

ith enough "ata#reasonable people con*erge.

But if any p7+i8 : @# no "ata

will change that.

Swamp the prior


88/145

Swamp the prior

riors can be arbitrarily low#but a*oi" @.

>ee wikipe"ia.org/wiki/

Cromwell'sPrule

eseech you, in the o&els

o$ /hrist, thin! it possile thatyou may e mista!en.

http://en.wikipedia.org/wiki/%20Cromwell's_rulehttp://en.wikipedia.org/wiki/%20Cromwell's_rulehttp://en.wikipedia.org/wiki/%20Cromwell's_rulehttp://en.wikipedia.org/wiki/%20Cromwell's_rule


89/145

S!mmary o$ estimation

,.Form a suite of hypotheses# +i.

.Choose prior "istribution# p7+i8.

0.Compute likelihoo"s# p7D9+i8.

.Turn off brain.

.Compute posteriors# p7+i9D8.


90/145

*ypothesis testing

(emember the original


91/145

*ypothesis testing

D is e*i"ence in fa*or of + if

p7+9D8 S p7+8

which is true if

p7D9+8 S p7D9+8or e


92/145

*ypothesis testing

This term

p7D9+8 / p7D9+8

is calle" the likelihoo" ratio# or Bayes factor.

It measures the strength of the e*i"ence.


93/145

*ypothesis testing

F$ hypothesis that the coin is fair

B$ hypothesis that the coin is biase"

p7D9F8 is easy.p7D9B8 is har" because B is un"erspecifie".


94/145

Bogosity

Tempting$ we got ,@ hea"s out of @ spins#

so B is the hypothesis that ( : ,@/@.

But#,.Doesn't seem right to use the "ata twice.

.By this process# almost any "ata woul" be

e*i"ence in fa*or of B.


95/145

/e need some r!les

,.Kou ha*e to choose your hypothesis before

you see the "ata.

.Kou can choose a suite of hypotheses# but

in that case we a*erage o*er the suite.


96/145


97/145

+ikelihood

def G4eraAe:ikeihood(suite, data)% tota =

for hypo, prob in suite.8tems()%

ike = suite.:ikeihood(data, hypo) tota F= prob E ike

return tota


98/145

*ypothesis testing

F$ hypothesis that ( : @E.

B$ hypothesis that ( is not @E# but might be

any other *alue with e


99/145

Prior

fair = Duro()

fair.&et(, 1)


100/145

Prior

bias = Duro()

for x in ranAe(, 11)%

if x ;= %

bias.&et(x, 1) bias.'ormaie()


101/145

Bayes $actor

data = 1", 11

ike7fair = G4eraAe:ikeihood(fair, data)

ike7bias = G4eraAe:ikeihood(bias, data)

ratio = ike7bias ike7fair


102/145

*ypothesis testing

(ea" euro2.py.

4otice the new representation of the "ata# an"

correspon"ing &ikelihoo" function.

(un it an" interpret the results.


103/145

*ypothesis testing

An" the answer is$

p7D9B8 : . U ,@2N

p7D9F8 : . U ,@2N

&ikelihoo" ratio is about @.N.

>o this "ataset is e*i"ence against B.


104/145

Fair comparison0

%)o"ify the co"e that buil"s biasR try out a"ifferent "efinition of B an" run again.

bias = Duro()

for x in ranAe(, "?)% bias.&et(x, x)

for x in ranAe(1, 11)%

bias.&et(x, 1-x)

bias.'ormaie()


105/145

oncl!sion

%The Bayes factor "epen"s on the "efinitionof B.

%Depen"ing on what Vbiase"W means# the"ata might be e*i"ence for or against B.

%The e*i"ence is weak either way 7between

@. an" 8.


106/145

S!mmary

-uro problem#Bayesian estimation#

Bayesian hypothesis testing.


107/145

h t t

p $ / / n a o l i t o . " e * i a n t a r t . c o m / a r t / T e a 2 T i m e 2 0 , @ D 0 F

N N

/ord problem $or geeks


108/145

p g

A&IC-$ hat "i" you get on the math >AT

BOB$ N@

A&IC-$ Oh# well I got a N@. I guess that means I'm

smarter than you.

4A((ATO($ (eally hat is the probability that Alice is

smarter than Bob

Ass!me1 de$ine1 ,!anti$y


109/145

1 1 , y

Assume$ each person has some probability# ( # ofanswering a ran"om >AT


110/145

y

Treat ( as a ran"om tart with a prior "istribution.

=p"ate it.

Compare posterior "istributions.

Prior0


111/145

Distribution of raw scores.

+ikelihood


112/145

def Likelihood(self, data, hypo) ! " hypo score " data ra# " self.e!am.e%erse(score)

yes, no " ra#, self.e!am.ma!_score & ra# like " !''yes ' (1&!)''no return like

Posterior


113/145

Pm$ProbGreater


114/145

def PmfProbreater(pmf1, pmf2) ***eturns the prob that a %alue from pmf1

is +reater than a %alue from pmf2.***

Pm$ProbGreater


115/145

def PmfProbreater(pmf1, pmf2) ***eturns the prob that a %alue from pmf1

is +reater than a %alue from pmf2.***

terate throu+h all pairs of %alues. -heck #hether the %alue from pmf1 is +reater. dd up total probability of successful pairs.

Pm$ProbGreater


116/145

def PmfProbreater(pmf1, pmf2)

for !1, p1 in pmf1.tems() for !2, p2 in pmf2.tems()

/ 0LL 3 45

Pm$ProbGreater


117/145

def PmfProbreater(pmf1, pmf2)

total " 6.6 for !1, p1 in pmf1.tems()

for !2, p2 in pmf2.tems() if !1 > !2 total 7" p1 ' p2

return total

And the answer is222


118/145

Alice$ N@

Bob$ N@

robability that Alice isQsmarterQ$ ,E


119/145

Modeling


120/145

Modeling

%This result is base" on the simplification that

all >AT


121/145

Modeling

%For most real worl" problems# there are

se*eral reasonable mo"els.

%The best choice "epen"s on your goals.

%)o"eling errors often "ominate.

Modeling


122/145

Modeling

Therefore$

%Don't mistake the map for the territory.

%Don't sweat appro!imations smaller thanmo"eling errors.

%Iterate.

3ecess4


123/145

h t t p $ / / i m a g e s . w

i k i a . n o

c o o k i e . n e t /PP

c b @ , @ , , F @ , 0 C @ N / r e c e s s

/ i m a g e s

/ B / B c / ( e c e s sP

' i cP

f o rP

t h eP

I n t e r n e t . p n g

ase st!dy


124/145

ase st!dy

roblem$ stu"ents sign up to participate in acommunity ser*ice pro;ect. >ome fraction#


125/145

ase st!dy

If we can get a smaller sample of stu"entswhere we know who participate" an" who

reporte"# we can use that to impro*e the

estimates of < an" r.

An" we can use that to compute the posterior

"istribution of the number of stu"ents who

participate".

.ol!nteer2py


126/145

probs " numpy.linspace(6, 1, 161)

hypos " 89 for : in probs for r in probs hypos.append((:, r))

suite " ;olunteer(hypos)

.ol!nteer2py


127/145

/ students #ho si+ned up and reported data " 1


128/145

class ;olunteer(thinkbayes.3uite)

def Likelihood(self, data, hypo) if len(data) "" 2 return self.Likelihood1(data, hypo) elif len(data) "" ? return self.Likelihood2(data, hypo) else

raise ;alue@rror()

.ol!nteer2py


129/145

def Likelihood1(self, data, hypo) :, r " hypo p " : ' r si+ned_up, reported " data yes " reported no " si+ned_up & reported

like " p''yes ' (1&p)''no

return like

.ol!nteer2py


130/145

def Likelihood2(self, data, hypo)

:, r " hypo si+ned_up, participated, reported " data yes " participated no " si+ned_up & participated

like1 " :''yes ' (1&:)''no

yes " reported no " participated & reported

like2 " r''yes ' (1&r)''no

return like1 ' like2


131/145


132/145

S!mmary


133/145

S!mmary

%The Bayesian approach is a "i*i"e an"con


134/145

Think Bayes

This tutorial is base" on mybook#

Think Bayes

Bayesian Statistics ade Simple


an" a*ailable un"er a

Creati*e Commons license fromthinkbayes.com

ase st!dies


135/145

ase st!dies

%-uro%>AT%(e" line

%rice is (ight%Boston Bruins%aintball%Gariability hypothesis

%1i"ney tumor growth%5eiger counter %=nseen species

Think Stats


136/145

Think Stats

Kou might also likeThink Stats, 2nd edition

%(ploratory ata nalysis


an" a*ailable un"er a

Creati*e Commons license from

thinkstats.com

More reading


137/145

More reading

)ac1ay#n$ormation +heory, n$er

ence, and #earning lgori

thmsFree DF.

More reading

http://www.inference.phy.cam.ac.uk/itila/http://www.inference.phy.cam.ac.uk/itila/http://www.inference.phy.cam.ac.uk/itila/http://www.inference.phy.cam.ac.uk/itila/http://www.inference.phy.cam.ac.uk/itila/http://www.inference.phy.cam.ac.uk/itila/


138/145

More reading

Da*i"son2ilon#Bayesian ethods

$or 3ac!ers

On 5ithub.

More reading 5not $ree6

https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackershttps://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackershttps://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackershttps://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers


139/145

More reading 5not $ree6

+owson an" =rbach# Scienti$ic easoning +heBayesian pproach

>i*ia# ata nalysis Bayesian +utorial

5elman et al# Bayesian ata nalysis

/here does this $it0


140/145


=sual approach$% Analytic "istributions.%)ath.

%)ulti"imensional integrals.%4umerical metho"s 7)C)C8.



141/145


roblem$%+ar" to get starte".%+ar" to "e*elop solutions incrementally.

%+ar" to "e*elop un"erstan"ing.

My theory


142/145

My theory

%>tart with non2analytic "istributions.%=se backgroun" information to choose

meaningful priors.

%>tart with brute2force solutions.%If the results are goo" enough an" fast

enough# stop.%Otherwise# optimiLe 7where analysis is one

kin" of optimiLation8.%=se your reference implementation for

regression testing.

&eed help0


143/145

&eed help0

I am always looking for interesting pro;ects.%>abbatical [une @, to August @,.

Thank yo!4


144/145

Thank yo!4

lease fill out the sur*ey at

tinyurl.com/bayes@@

http://tinyurl.com/bayes0409http://tinyurl.com/bayes0409


145/145

pycon 2015- bayesian statistics made simple

Documents