IntroductiontoAdvancedProbabilityforGraphicalModels

CSC412ByElliotCreager

ThursdayJanuary11,2018

PresentedbyJonathanLorraine

*ManyslidesbasedonKaustav Kundu’s, KevinSwersky’s,Inmar Givoni’s,DannyTarlow’s,JasperSnoek’s slides,SamRoweis ‘sreviewofprobability, Bishop’sbook,andsomeimagesfromWikipedia

Outline

• Basics• Probabilityrules• Exponentialfamilymodels• Maximumlikelihood• ConjugateBayesianinference(timepermitting)

WhyRepresentUncertainty?

• Theworldisfullofuncertainty– “Whatwilltheweatherbeliketoday?”– “WillIlikethismovie?”– “Isthereapersoninthisimage?”

• We’retryingtobuildsystemsthatunderstandand(possibly)interactwiththerealworld

• Weoftencan’tprovesomethingistrue,butwecanstillaskhowlikelydifferentoutcomesareoraskforthemostlikelyexplanation

• Sometimesprobabilitygivesaconcisedescriptionofanotherwisecomplexphenomenon.

WhyUseProbabilitytoRepresentUncertainty?

• Writedownsimple,reasonablecriteriathatyou'dwantfromasystemofuncertainty(commonsensestuff),andyoualwaysgetprobability.

• CoxAxioms(Cox1946);SeeBishop,Section1.2.3

• Wewillrestrictourselvestoarelativelyinformaldiscussionofprobabilitytheory.

Notation• A randomvariableX representsoutcomesorstatesoftheworld.

• Wewillwritep(x)tomeanProbability(X=x)• Samplespace:thespaceofallpossibleoutcomes(maybediscrete,continuous,ormixed)

• p(x)istheprobabilitymass(density)function– Assignsanumbertoeachpointinsamplespace– Non-negative,sums(integrates)to1– Intuitively:howoftendoesxoccur,howmuchdowebelieveinx.

JointProbabilityDistribution• Prob(X=x,Y=y)– “ProbabilityofX=xandY=y”– p(x,y)

ConditionalProbabilityDistribution• Prob(X=x|Y=y)– “Probability ofX=xgivenY=y”– p(x|y)=p(x,y)/p(y)

MarginalProbabilityDistribution• Prob(X=x),Prob(Y=y)– “Probability ofX=x”– p(x)=\Sum_{y}p(x,y)=\Sum{y}p(x|y)p(y)

TheRulesofProbability

• SumRule(marginalization/summingout):

• Product/ChainRule:

),...,,(...)(

),()(

2112 3

Nx x x

y

xxxpxp

yxpxp

N

∑∑ ∑

∑

=

=

),...,|()...|()(),...,()()|(),(

111211 −=

=

NNN xxxpxxpxpxxpxpxypyxp

Bayes’Rule

• Oneofthemostimportantformulasinprobabilitytheory

• Thisgivesusawayof“reversing”conditionalprobabilities

• Readas”Posterior=likelihood*prior/evidence”

∑==

')'()'|(

)()|()()()|()|(

xxpxypxpxyp

ypxpxypyxp

Independence

• Tworandomvariablesaresaidtobeindependent iff theirjointdistributionfactors

• Tworandomvariablesareconditionallyindependentgivenathirdiftheyareindependentafterconditioningonthethird

)()()()|()()|(),( ypxpypyxpxpxypyxp ===

zzxpzypzxpzxypzyxp ∀== )|()|()|(),|()|,(

ContinuousRandomVariables• Outcomesarerealvalues.Probabilitydensityfunctionsdefinedistributions.– E.g.,

• Continuousjointdistributions:replacesumswithintegrals,andeverythingholds– E.g.,Marginalizationandconditionalprobability

∫∫ ==yy

yPyzxPzyxPzxP )()|,(),,(),(

⎭⎬⎫

⎩⎨⎧ −−= 2

2 )(21exp

21),|( µ

σσπσµ xxP

SummarizingProbabilityDistributions

• Itisoftenusefultogivesummariesofdistributionswithoutdefiningthewholedistribution(E.g.,meanandvariance)

• Mean:

• Variance:

dxxpxxxEx

)(][ ∫ ⋅==

dxxpxExxx

)(])[()var( 2∫ ⋅−=

=E[x2 ]−E[x]2

ExponentialFamily

• Familyofprobabilitydistributions• Manyofthestandarddistributionsbelongtothisfamily– Bernoulli,binomial/multinomial,Poisson,Normal(Gaussian),beta/Dirichlet,…

• Sharemanyimportantproperties– e.g. Theyhaveaconjugateprior(we’llgettothatlater.ImportantforBayesianstatistics)

Definition• Theexponentialfamilyofdistributionsoverx,givenparameterη (eta)isthesetofdistributionsoftheform

• x-scalar/vector,discrete/continuous• η – ‘naturalparameters’• u(x)– somefunctionofx(sufficientstatistic)• g(η)– normalizer• h(x)– basemeasure(oftenconstant)

)}(exp{)()()|( xugxhxp Tηηη =

1)}(exp{)()( =∫ dxxuxhg Tηη

SufficientStatistics

• Vaguedefinition:calledsobecausetheycompletelysummarizeadistribution.

• Lessvague:theyaretheonlypartofthedistributionthatinteractswiththeparametersandarethereforesufficienttoestimatetheparameters.

• Perhapsthenumberoftimesacoincameupheads,orthesumofvaluesmagnitudes.

Example1:Bernoulli

• Binaryrandomvariable-• p(heads)=µ• Cointoss

xxxp −−= 1)1()|( µµµ

}1,0{∈X]1,0[∈µ

Example1:Bernoulli

xxxp −−= 1)1()|( µµµ

}1

exp{ln)1(

)}1ln()1(lnexp{

xu

xx

⎟⎠

⎞⎜⎝

⎛−

−=

−−+=

µµ

µµ

)()(11)(

1ln

)(1)(

ηση

ησµµµ

η η

−=

+==⇒⎟⎟

⎠

⎞⎜⎜⎝

⎛

−=

=

=

−

ge

xxuxh

)}(exp{)()()|( xugxhxp Tηηη =

)exp()()|( xxp ηηση −=

Example2:Multinomial• p(valuek)=µk

• Forasingleobservation– dietoss– SometimescalledCategorical

• Formultipleobservations– integercountsonNtrials– Prob(1cameout3times,2cameoutonce,…,6cameout7timesifItossedadie20times)

1],1,0[1

=∈ ∑=

M

kkk µµ

∏∏ =

=M

k

xk

kk

Mk

xNxxP

11 !

!)|,...,( µµ

∑=

=M

kk Nx

1

Example2:Multinomial(1observation)

}lnexp{1∑=

=M

kkkx µ

xxx=

=

)(1)(

uh

)}(exp{)()()|( xugxhxp Tηηη =

∏=

=M

k

xkMkxxP

11 )|,...,( µµ

)exp()|( xx Tp ηη =

Parametersarenotindependentduetoconstraintofsumming to1,there’saslightlymoreinvolvednotation toaddressthat,seeBishop2.4

Example3:Normal(Gaussian)Distribution

• Gaussian(Normal)

⎭⎬⎫

⎩⎨⎧ −−= 2

2 )(21exp

21),|( µ

σσπσµ xxp

Example3:Normal(Gaussian)Distribution

• µisthemean• σ2 isthevariance• Canverifythesebycomputingintegrals.E.g.,

⎭⎬⎫

⎩⎨⎧ −−= 2

2 )(21exp

21),|( µ

σσπσµ xxp

€

x ⋅ 12πσ

exp −12σ 2 (x −µ)2

⎧ ⎨ ⎩

⎫ ⎬ ⎭ dx = µ

x→−∞

x→∞

∫

Example3:Normal(Gaussian)Distribution

• MultivariateGaussian

€

P(x |µ,∑) = 2π ∑−1/ 2 exp −12(x −µ)T ∑−1(x −µ)

⎧ ⎨ ⎩

⎫ ⎬ ⎭

Example3:Normal(Gaussian)Distribution

• MultivariateGaussian

• x isnowavector• µisthemeanvector• Σ isthecovariancematrix

⎭⎬⎫

⎩⎨⎧ −∑−−∑=∑ −− )()(21exp2),|( 12/1

µµπµ xxxp T

ImportantPropertiesofGaussians

• Allmarginals ofaGaussianareagainGaussian• AnyconditionalofaGaussianisGaussian• TheproductoftwoGaussiansisagainGaussian

• EventhesumoftwoindependentGaussianRVsisaGaussian.

• Beyondthescopeofthistutorial,butveryimportant:marginalizationandconditioningrulesformultivariateGaussians.

Gaussianmarginalizationvisualization

ExponentialFamilyRepresentation

}21exp{)

4exp()2()2(

}21

21exp{

21

)(21exp

21),|(

2222

212

1

221

222

22

22

⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡ −−=

=−

++−

=

⎭⎬⎫

⎩⎨⎧ −−=

−

xx

xx

xxp

σσµ

ηη

ηπ

µσσ

µσσπ

µσσπ

σµ

)}(exp{)()()|( xugxhxp Tηηη =

)(xh )(ηg Tη )(xu

Example:MaximumLikelihoodFora1DGaussian

• SupposewearegivenadatasetofsamplesofaGaussianrandomvariableX,D={x1,…,xN}andtoldthatthevarianceofthedataisσ2

Whatisourbestguessofµ?*Needtoassumedataisindependentandidenticallydistributed(i.i.d.)

x1 x2 xN…

Example:MaximumLikelihoodFora1DGaussian

Whatisourbestguessofµ?• Wecanwritedownthelikelihoodfunction:

• Wewanttochoosetheµthatmaximizesthisexpression– Takelog,thenbasiccalculus:differentiatew.r.t.µ,setderivativeto0,solveforµtogetsamplemean

∏ ∏= = ⎭

⎬⎫

⎩⎨⎧ −−==µ

N

i

N

i

ii xxpdp1 1

22 )(

21exp

21),|()|( µ

σσπσµ

€

µML =1N

xii=1

N∑

Example:MaximumLikelihoodFora1DGaussian

x1 x2 xN…µML

σML

MaximumLikelihood

MLestimationofmodelparametersforExponentialFamily

p(D |η) = p(x1,..., xN ) = h(xn )∏( )g(η)N exp{ηT u(xnn∑ )}

∂ln(p(D |η))

∂η= ..., set to 0, solve for ∇g(η)

∑=

=∇−N

nnML xu

Ng

1)(1)(ln η

• Caninprinciplebesolvedtogetestimateforeta.• ThesolutionfortheMLestimatordependsonthedataonlythroughsumoveru,whichisthereforecalledsufficientstatistic•Whatweneedtostoreinordertoestimateparameters.

BayesianProbabilities

• isthelikelihood function• isthepriorprobability of(orourpriorbelief over)θ– ourbeliefsoverwhatmodelsarelikelyornotbeforeseeinganydata

• isthenormalizationconstantorpartitionfunction

• istheposteriordistribution

– Readjustmentofourpriorbeliefsinthefaceofdata

)()()|()|(

dppdpdp θθ

θ =

∫= θθθ dPdpdp )()|()(

)|( θdp)(θp

)|( dp θ

Example:BayesianInferenceFora1DGaussian

• SupposewehaveapriorbeliefthatthemeanofsomerandomvariableXisµ0 andthevarianceofourbeliefisσ02

• WearethengivenadatasetofsamplesofX,d={x1,…,xN}andsomehowknowthatthevarianceofthedataisσ2

Whatistheposteriordistributionover(ourbeliefaboutthevalueof)µ?

Example:BayesianInferenceFora1DGaussian

x1 x2 xN…

Example:BayesianInferenceFora1DGaussian

x1 x2 xN… µ0

σ0

Priorbelief

Example:BayesianInferenceFora1DGaussian

• Rememberfromearlier

• isthelikelihoodfunction

• isthepriorprobability of(orourpriorbelief over)µ

)()()|()|(

dppdpdp µµ

=µ

)|( µdp

)(µp

∏ ∏= = ⎭

⎬⎫

⎩⎨⎧ −−==µ

N

i

N

i

ii xxPdp1 1

22 )(

21exp

21),|()|( µ

σσπσµ

⎭⎬⎫

⎩⎨⎧

µ−µ−=µµ 202

0000 )(

21exp

21),|(

σσπσp

Example:BayesianInferenceFora1DGaussian

),|()|()()|()|(

NNDppDpDp

σµµ=µ

µµ∝µ

Normal

€

µN =σ 2

Nσ 02 +σ 2 µ0 +

Nσ 02

Nσ 02 +σ 2 µML

1σN2 =

1σ 02 +

Nσ 2

where

Example:BayesianInferenceFora1DGaussian

x1 x2 xN… µ0

σ0

Priorbelief

Example:BayesianInferenceFora1DGaussian

x1 x2 xN… µ0

σ0

PriorbeliefµML

σML

MaximumLikelihood

Example:BayesianInferenceFora1DGaussian

x1 x2 xNµN

σN

PriorbeliefMaximumLikelihood

PosteriorDistribution

ConjugatePriors• NoticeintheGaussianparameterestimationexamplethatthefunctionalformoftheposteriorwasthatoftheprior(Gaussian)

• Priorsthatleadtothatformarecalled‘conjugatepriors’

• Foranymemberoftheexponentialfamilythereexistsaconjugatepriorthatcanbewrittenlike

• Multiplybylikelihoodtoobtainposterior(uptonormalization)oftheform

• Noticetheadditiontothesufficientstatistic• ν istheeffectivenumberofpseudo-observations.

}exp{)(),(),|( χνηηνχνχη ν Tgfp =

)})((exp{)(),,|(1

νχηηνχη ν +∝ ∑=

+N

nn

TN xugDp

ConjugatePriors- Examples

• BetaforBernoulli/binomial• Dirichlet forcategorical/multinomial• NormalformeanofNormal• Andmanymore...

• Whataresomepropertiesoftheconjugatepriorforthecovariance(orprecision)matrixofanormaldistribution?