Upload File

graphical models - svivek · • example: hidden markov model –naïve bayes classifier is also a...

  • Home
  • Documents
  • Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence
31
CS 6355: Structured Prediction Graphical Models 1

Upload: lydiep

Post on 26-Aug-2018

223 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

CS6355:StructuredPrediction

GraphicalModels

1

Page 2: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Sofar…Wediscussedsequencelabelingtasks:• HMM:HiddenMarkovModels• MEMM:MaximumEntropyMarkovModels• CRF:ConditionalRandomFieldsAllthesemodelsusealinearchainstructuretodescribetheinteractionsbetweenrandomvariables.

2

yt-1 yt

xt

yt-1 yt

xt

yt-1 yt

xt

HMM MEMM CRF

Page 3: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Thislecture

Graphicalmodels– Directed:BayesianNetworks– Undirected:MarkovNetworks(MarkovRandomField)

• Representations• Inference• Learning

3

Page 4: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

GraphicalModels

• Alanguagetorepresentprobabilitydistributionsovermultiplerandomvariables– Directedorundirectedgraphs

• Encodesconditionalindependenceassumptions• Orequivalently,encodesfactorizationofjointprobabilities.

• Generalmachinery for– Algorithmsforcomputingmarginalandconditionalprobabilities

• Recallthatwehavebeenlookingatmostprobablestatessofar• Exploitinggraphstructure

– An“inferenceengine”

– Canintroducepriorprobabilitydistributions• Becauseparametersarealsorandomvariables

4

Page 5: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Decomposejointprobabilityviaadirectedacyclicgraph– Nodesrepresentrandomvariables– Edgesrepresentconditionaldependencies– Eachnodeisassociatedwithaconditionalprobabilitytable

BayesianNetwork

ExamplefromRussellandNorvig 5

Page 6: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Decomposejointprobabilityviaadirectedacyclicgraph– Nodesrepresentrandomvariables– Edgesrepresentconditionaldependencies– Eachnodeisassociatedwithaconditionalprobabilitytable

BayesianNetwork

ExamplefromRussellandNorvig

P(B,E,A,J,M)=P(B)P(E)P(A|B,E)P(J|A)P)M|A)

Compactrepresentationofthejointprobabilitydistribution

6

Page 7: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

• Localindependencies:Anodeisindependentwithitsnon-descendantsgivenitsparents.

• Topologicalindependencies:Anodeisindependentofallothernodesgivenitsparents,childrenandchildren’sparents—thatisgivenitsMarkovBlanket.

• Globalindependencies:D-separation

IndependenceAssumptionsofaBN

7

(Xi ? NonDescendants(Xi)|Parents(Xi))

(Xi ? Xj |MB(Xi)) for all j 6= i

ExamplefromDaphneKoller

Page 8: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

• Localindependencies:Anodeisindependentwithitsnon-descendantsgivenitsparents.

• Topologicalindependencies:Anodeisindependentofallothernodesgivenitsparents,childrenandchildren’sparents—thatisgivenitsMarkovBlanket.

• Globalindependencies:D-separation

IndependenceAssumptionsofaBN

Wheredotheindependenceassumptionscomefrom?

8

(Xi ? NonDescendants(Xi)|Parents(Xi))

(Xi ? Xj |MB(Xi)) for all j 6= i

ExamplefromDaphneKoller

Page 9: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

• Localindependencies:Anodeisindependentwithitsnon-descendantsgivenitsparents.

• Topologicalindependencies:Anodeisindependentofallothernodesgivenitsparents,childrenandchildren’sparents—thatisgivenitsMarkovBlanket.

• Globalindependencies:D-separation

IndependenceAssumptionsofaBN

Wheredotheindependenceassumptionscomefrom?

Domainknowledge

9

(Xi ? NonDescendants(Xi)|Parents(Xi))

(Xi ? Xj |MB(Xi)) for all j 6= i

ExamplefromDaphneKoller

Page 10: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

BayesianNetwork

• Example:HiddenMarkovModel– NaïveBayesclassifierisalsoasimpleBayesnet

• SometimesBayesnetscannotrepresenttheindependencerelationswewantconveniently.– Eg:Segmentinganimagebyassigningalabeltoeachpixel

• Say,wewantadjacentlabelstoinfluenceeachotherTwoproblems:1. Whatistherightdirectionofarrows?

2. Foranychoiceofthearrows,strangedependenciesshowup.X8 isindependentofeverythinggivenitsMarkovblanket(othercirclednodeshere)

10

Page 11: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

BayesianNetwork

• Example:HiddenMarkovModel– NaïveBayesclassifierisalsoasimpleBayesnet

• SometimesBayesnetscannotrepresenttheindependencerelationswewantconveniently.– Eg:Segmentinganimagebyassigningalabeltoeachpixel

• Say,wewantadjacentlabelstoinfluenceeachother

ExamplefromKevinMurphy 11

Page 12: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

BayesianNetwork

• Example:HiddenMarkovModel– NaïveBayesclassifierisalsoasimpleBayesnet

• SometimesBayesnetscannotrepresenttheindependencerelationswewantconveniently.– Eg:Segmentinganimagebyassigningalabeltoeachpixel

• Say,wewantadjacentlabelstoinfluenceeachother

ExamplefromKevinMurphy

Twoproblems:1. Whatisthecorrectdirectionofarrows?

2. Foranychoiceofthearrows,strangedependenciesshowup.X8 isindependentofeverythinggivenitsMarkovblanket(othercirclednodeshere)

12

Page 13: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

FromdirectedtoundirectedNetworks

SometimesBayesnetscannotrepresenttheindependencerelationswewantconveniently.

– Eg:Segmentinganimagebyassigningalabeltoeachpixel• Say,wewantadjacentlabelstoinfluenceeachother

ExamplefromKevinMurphy 13

Page 14: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

UndirectedGraphicalModels

• Anotherwayofdefiningconditionalindependence

• Generalstructure– Nodesarerandomvariables– Edges(hyper-edges)definedependencies

• Thenodesinacomplete subgraph formaclique.

a.k.a MarkovRandomFields/MarkovNetworks

14

8cliques:{A},{B},{C},{D}{AB},{BC},{CD},{AD}

Page 15: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

UndirectedGraphicalModels

• Anotherwayofdefiningconditionalindependence

• Generalstructure– Nodesarerandomvariables– Edges(hyper-edges)definedependencies

• Thenodesinacomplete subgraph formaclique.

a.k.a MarkovRandomFields/MarkovNetworks

15

P (x) =1

Z

Y

c2Cliques

fc(xc)

8cliques:{A},{B},{C},{D}{AB},{BC},{CD},{AD}

P (A,B,C,D) =1

Zf1(A,B)f2(B,C)

f3(C,D)f4(A,D)

ThisisaGibbsdistributionifallfactorsarepositive

Page 16: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

UndirectedGraphicalModels

• Anotherwayofdefiningconditionalindependence

• Generalstructure– Nodesarerandomvariables– Edges(hyper-edges)definedependencies

• Thenodesinacomplete subgraph formaclique.

a.k.a MarkovRandomFields/MarkovNetworks

Thejointprobabilitydecouplesovercliques.Everycliquexcassociatedwithapotential function f(xc)

16

P (x) =1

Z

Y

c2Cliques

fc(xc)

8cliques:{A},{B},{C},{D}{AB},{BC},{CD},{AD}

P (A,B,C,D) =1

Zf1(A,B)f2(B,C)

f3(C,D)f4(A,D)

ThisisaGibbsdistributionifallfactorsarepositive

Page 17: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

• Localindependencies:Anodeisindependentofallothernodesgivenitsneighbors.

• Globalindependencies:IfX,Y,Zaresetsofnodes,XisconditionallyindependentofYgivenZifremovingallnodesofCremovesallpathsfromAtoB

IndependenceAssumptionsofaMRF

17

Page 18: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

IndependenceAssumptionsofaMRF

Wheredotheindependenceassumptionscomefrom?

18

• Localindependencies:Anodeisindependentofallothernodesgivenitsneighbors.

• Globalindependencies:IfX,Y,Zaresetsofnodes,XisconditionallyindependentofYgivenZifremovingallnodesofCremovesallpathsfromAtoB

Page 19: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

IndependenceAssumptionsofaMRF

Wheredotheindependenceassumptionscomefrom?

Domainknowledge

19

• Localindependencies:Anodeisindependentofallothernodesgivenitsneighbors.

• Globalindependencies:IfX,Y,Zaresetsofnodes,XisconditionallyindependentofYgivenZifremovingallnodesofCremovesallpathsfromAtoB

Page 20: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

MRFtoFactorgraph

20

Z: Calledthepartitionfunction, sumoverallassignmentstotherandomvariables

Normalize:

where

f(xc,µ)isoftenwrittenasexp(µT xc)Log-linearmodel

Page 21: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Factorgraphs

21

Z: Calledthepartitionfunction,sumoverallassignmentstotherandomvariables

Normalize:

where

Factorgraph:Makesthefactorizationexplicit,factors insteadofcliques

?

f(xc,µ)isoftenwrittenasexp(µT xc)Log-linearmodel

Whichcliques?

Page 22: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Factorgraphs

22

Z: Calledthepartitionfunction,sumoverallassignmentstotherandomvariables

Normalize:

where

Factorgraph:Makesthefactorizationexplicit,factors insteadofcliques

Factors

Factors

Factors

f(xc,µ)isoftenwrittenasexp(µT xc)Log-linearmodel

?

Page 23: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Factorgraphs

1 2 3

4 5

23

Z: Calledthepartitionfunction,sumoverallassignmentstotherandomvariables

Normalize:

where

Factorgraph:Makesthefactorizationexplicit,factors insteadofcliques

Factors

Factors

Factors

f(xc,µ)isoftenwrittenasexp(µT xc)Log-linearmodel

P (x) =1

Z

fa(x1, x2, x4)fb(x2, x3, x5)fc(x4, x5)

?

Page 24: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

CommentsaboutMRFs

• Connectiontostatisticalphysics– IdenticaltoBoltzmanndistributioninenergybasedmodels– Probabilityofasystemexistinginastate:

• Ifx isdependentonallitsneighbors:– Ifxcanbeinoneoftwostates(binary),Ising model– Ifxcanbeinoneofmorethantwostates(multiclass),Pottsmodel

Z: Zustandssumme,“sumoverstates”,morecommonlycalledthepartitionfunction

24

Page 25: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

CommentsaboutMRFs

• Connectiontostatisticalphysics– IdenticaltoBoltzmanndistributioninenergybasedmodels– Probabilityofasystemexistinginastate:

• Ifx isdependentonallitsneighbors:– Ifxcanbeinoneoftwostates(binary),Ising model– Ifxcanbeinoneofmorethantwostates(multiclass),Pottsmodel

Energyofcliquecexistinginstatexc

Z: Zustandssumme,“sumoverstates”,morecommonlycalledthepartitionfunction

25

Page 26: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

CommentsaboutMRFs

• Connectiontostatisticalphysics– IdenticaltoBoltzmanndistributioninenergybasedmodels– Probabilityofasystemexistinginastate:

• Ifx isdependentonallitsneighbors:– Ifxcanbeinoneoftwostates(binary),Ising model– Ifxcanbeinoneofmorethantwostates(multiclass),Pottsmodel

Energyofcliquecexistinginstatexc

Z: Zustandssumme,“sumoverstates”,morecommonlycalledthepartitionfunction

26

Page 27: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

BayesianNetworksvs.MarkovNetworks

• Bothnetworksrepresent– Asetofconditionalindependencerelations– i.e,askeletonthatshowshowajointprobabilitydistributionis

factorized

• Bothnetworkshavetheoremsaboutequivalencebetweenconditionalindependenceand jointprobabilityfactorization

• GivenaBayesnet,thereisnotalwaysanequivalentMarkovrandomfield,andviceversa– Withsomecaveats– SeethechapteronundirectedgraphicalmodelsinKoller and

Friedman’sbook

27

Page 28: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Inferenceingraphicalmodels

Ingeneral,computeprobabilityofasubsetofstates– P(xA),forsomesubsetsofrandomvariablesxA

• Note:Sofar,wehavegenerallyconsideredthesomethinglikeargmaxx P(x)

• Exactinference• “Approximate”inference

28

1 2 3

4 5(moreonthisinfutureclasses)

Page 29: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Inferenceingraphicalmodels

Ingeneral,computeprobabilityofasubsetofstates– P(xA),forsomesubsetsofrandomvariablesxA

• Note:Sofar,wehavegenerallyconsideredtheequivalentofargmaxx P(x)

• Exactinference• “Approximate”inference

29

1 2 3

4 5(moreonthisinfutureclasses)

Page 30: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Inferenceingraphicalmodels

Ingeneral,computeprobabilityofasubsetofstates– P(xA),forsomesubsetsofrandomvariablesxA

• Exactinference– Variableelimination

• Marginalizebysummingoutvariablesina“good”order• ThinkaboutwhatwedidforViterbi

– Beliefpropagation(exactonlyforgraphswithoutloops)• Nodespassmessagestoeachotherabouttheirestimateofwhattheneighbor’sstateshouldbe

– Generallyefficientfortrees,sequences(andmaybeothergraphstoo)

• “Approximate”inference30

1 2 3

4 5(moreonthisinfuturelectures)

Whatmakesanorderinggood?

Page 31: Graphical Models - svivek · • Example: Hidden Markov Model –Naïve Bayes classifier is also a simple Bayes net • Sometimes Bayes nets cannot represent the independence

Inferenceingraphicalmodels

Ingeneral,computeprobabilityofasubsetofstates– P(xA),forsomesubsetsofrandomvariablesxA

• Exactinference• “Approximate”inference– MarkovChainMonteCarlo

• GibbsSampling/Metropolis-Hastings– Variational algorithms

• Frameinferenceasanoptimizationproblem,perturbittoanapproximateoneandsolvetheapproximateproblem

– LoopyBeliefpropagation• RunBPandhopeitworks!

– Thenot-so-goodnews:Approximateinferenceisalsointractable!31

1 2 3

4 5

NP-hardingeneral,worksforsimplegraphs

(moreonthisinfuturelectures)


Spam Filtering with Naïve Bayes – Which Naïve Bayes?cobweb.cs.uga.edu/.../CSCI6900...Presentation2.pdf · Title: Spam Filtering with Naïve Bayes – Which Naïve Bayes? Author:

Naïve Bayes Classifiers

MLE and MAP•Maximum a posteriori (MAP) estimate •Naïve Bayes •Various Naïve Bayes models •model 1: Bernoulli Naïve Bayes •model 2: Multinomial Naïve Bayes •model 3:

Classification: Logistic Regressionalvin/courses/ugml2016/02b.pdf · Logistic Regression Example Contrasting Naïve Bayes and Logistic Regression Naïve Bayes easier Naïve Bayes

Naïve Bayesmgormley/courses/10601-s18/slides/... · 1. Bernoulli Naïve Bayes: –for binary features 2.Gaussian Naïve Bayes: –for continuousfeatures 3.MultinomialNaïve Bayes:

The Naïve Bayes Classifier - svivek · •The naïve Bayes Classifier •Learning the naïve Bayes Classifier •Practical concerns 2. Today’s lecture •The naïve Bayes Classifier

Logistic Regression - svivek · Logistic Regression is the discriminative version. This lecture •Logistic regression •Connection to Naïve Bayes •Training a logistic regression

Naïve Bayes Classifier

Klasifikasi Metagenom dengan Metode Naïve Bayes Classifier … · Klasifikasi Metagenom dengan Metode Naïve Bayes Classifier Metagenome Classification Using Naïve Bayes Classifier

Bayesianmethods&Naïve( Bayes( Lecture18

Naïve Bayes Text Classification

5. Aufgabenblatt Naïve Bayes Klassifikation Abgabe: 07.02 ... · Naïve Bayes Klassifikation • Implementiert den Naïve-Bayes-Klassifikatorals PL-SQL-Funktion(Vorlage s. Webseite)

Naïve Bayes, HMM

Naïve Bayes based Model

Naïve Bayes - GitHub Pagesaritter.github.io/courses/5522_slides/naive_bayes.pdf · Naïve Bayes for Digits Naïve Bayes: Assume all features are independent effects of the label

kNN & Naïve Bayes

Naïve Bayes Classfication

Metode klasifikasi Naïve Bayes

Discrete Naïve Bayes

Naïve Bayes: refinements

Bayes Classifier and Naïve Bayes - Oregon State …classes.engr.oregonstate.edu/eecs/winter2011/cs434/notes/bayes-6.pdfBayes Classifier and Naïve Bayes ... Probabilistic Classification

23: Naïve Bayes - Stanford Universityweb.stanford.edu/.../lectures/23_naive_bayes_blank.pdf21 “Brute Force Bayes” 24b_brute_force_bayes 32 Naïve Bayes Classifier 24c_naive_bayes

More Naïve Bayes

Naïve Bayes and Logistic Regression · Naïve Bayes and Logistic Regression Naïve Bayes •Generative Model "̂=argmax)*(")(-|") •Features assumed to be independent Logistic Regression

Naïve Bayes 𝑖 𝜶 - Kangwoncs.kangwon.ac.kr/.../2015_MachineLearning/07_naive_bayes.pdf · 2016. 6. 17. · Naïve Bayes •Bayes rule을적용하면모든데이터에대하여고려해야함

Naïve Bayes Learning

NAÏVE BAYES CLASSIFIER

Naïve Bayes - santini.sesantini.se/teaching/ml/2016/Lect_06/06_NaiveBayes.pdf · Lecture 6 - Self-Study: Naive Bayes 19 Naïve Bayes: discussion ! Naïve Bayes works surprisingly

Bayes and Naïve Bayes Classifiers

Naïve Bayes

Naïve Bayes Classifier · Naïve Bayes Classifier 17 • Bayes Classifier with additional “naïve” assumption: – Features are independent given class: – More generally: •

Naïve Bayes Classification

Classification: Naïve Bayes

Naïve Bayes (Continued)

Bayesian inference, Naïve Bayes model - Svetlana Lazebnikslazebni.cs.illinois.edu/fall16/lec13_bayesian_inference.pdf · Bayesian inference, Naïve Bayes model ... Bayes Rule •