bayesian networks€¦ · • all you (statistically) need to know about x 1 ... sinus flu allergy...

Bayesian Networks

George Konidarisgdk@cs.duke.edu

Spring 2016

RecallJoint distributions:

• P(X1, …, Xn). • All you (statistically) need to know about X1 … Xn.• From it you can infer P(X1), P(X1 | Xs), etc.

!! Raining Cold Prob.

True True 0.3True False 0.1False True 0.4False False 0.2

Joint Distributions Are UsefulClassification

• P(X1 | X2 … Xn)!!Co-occurrence

• P(Xa, Xb)!!Rare event detection• P(X1, …, Xn)

thing you want to know

things you know

how likely are these two things together?

Modeling Joint DistributionsGets large fast

• 2n entries for n binary RVs.!!Independence!

• A bit too strong.• Rarely holds.

!Conditional independence.

• Good compromise.

Conditional IndependenceA and B are conditionally independent given C if:

• P(A | B, C) = P(A | C)• P(A, B | C) = P(A | C) P(B | C)

!(recall independence: P(A, B) = P(A)P(B))!!

This means that, if we know C, we can treat A and B as if they were independent.!

A and B might not be independent otherwise!

ExampleConsider 3 RVs:

• Temperature• Humidity• Season

!

Temperature and humidity are not independent. !!But, they might be, given the season: the season explains both, and they become independent of each other.

Bayes NetsA particular type of graphical model:

• A directed, acyclic graph.• A node for each RV.

!!!!!!!

Given parents, each RV independent of non-descendants.

T H

S

Bayes Net!!!!!!!JPD decomposes:!!!So for each node, store conditional probability table (CPT):

P (x1, ..., xn) =Y

i

P (xi|parents(xi))

T H

S

P (xi|parents(xi))

ExampleSuppose we know:

• The flu causes sinus inflammation.• Allergies cause sinus inflammation.• Sinus inflammation causes a runny nose.• Sinus inflammation causes headaches.

Example

Sinus

Flu Allergy

Nose Headache

Example

Sinus

Flu Allergy

Nose

Headache

Flu PTrue 0.6False 0.4

Allergy PTrue 0.2False 0.8

Nose Sinus P

True True 0.8False True 0.2True False 0.3False False 0.7

Headache Sinus PTrue True 0.6False True 0.4True False 0.5False False 0.5

Sinus Flu Allergy PTrue True True 0.9False True True 0.1True True False 0.6False True False 0.4True False False 0.2False False False 0.8True False True 0.4False False True 0.6

joint: 32 (31) entries

Naive Bayes

S

W1 W2 W3 Wn…

P(S)

P(W1|S) P(W2|S) P(W3|S) P(Wn|S)

(spam filter!)

UsesThings you can do with a Bayes Net:

• Inference: given some variables, posterior?• (might be intractable: NP-hard)

• Learning (fill in CPTs)• Structure Learning (fill in edges)

!!

Generally:• Often few parents.• Inference cost often reasonable.• Can include domain knowledge.

Inference

Sinus

Flu Allergy

Nose Headache

What is:P(f | h)?

Inference!!!!!We know from definition of Bayes net:

P (f |h) = P (f, h)

P (h)=

PSAN P (f, h, S,A,N)P

SANF P (h, S,A,N, F )

P (h) =X

SANF

P (h, S,A,N, F )

P (h) =X

SANF

P (h|S)P (N |S)P (S|A,F )P (F )P (A)

Variable EliminationSo we have:!!!! … we can eliminate variables one at a time:(distributive law)

P (h) =X

SANF

P (h|S)P (N |S)P (S|A,F )P (F )P (A)

P (h) =X

SN

P (h|S)P (N |S)X

AF

P (S|A,F )P (F )P (A)

P (h) =X

S

P (h|S)X

N

P (N |S)X

AF

P (S|A,F )P (F )P (A)

Variable EliminationGenerically:

• Query about Xi and Xj.• Write out P(X1 … Xn) in terms of P(Xi | parents(Xi))• Sum out all variables except Xi and Xj• Answer query using joint distribution P(Xi, Xj)

!Good news:

• Potentially exponential reduction in computation.• Polynomial for trees.

!Bad news:

• Picking variables in optimal order NP-Hard.• For some networks, no elimination.

Spam Filter (Naive Bayes)

S

W1 W2 W3 Wn…

P(S)

P(W1|S) P(W2|S) P(W3|S) P(Wn|S)

Want P(S | W1 … Wn)

Naive Bayes

P (S|W1, ...,Wn) =P (W1, ...,Wn|S)P (S)

P (W1, ...,Wn)

given

P (W1, ...,Wn|S) =Y

i

P (Wi|S)(from the

Bayes Net)

Bayes NetsPotentially very compressed but exact.

• Requires careful construction!! VS!Approximate representation.

• Hope you’re not too wrong!!!

Many, many applications in all areas.