mean field inference in dependency networks: an empirical study
DESCRIPTION
Mean Field Inference in Dependency Networks: An Empirical Study. Daniel Lowd and Arash Shamaei University of Oregon. Learning and Inference in Graphical Models. We want to learn a probability distribution from data and use it to answer queries. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/1.jpg)
Mean Field Inference in Dependency Networks: An Empirical Study
Daniel Lowd and Arash ShamaeiUniversity of Oregon
![Page 2: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/2.jpg)
2
Learning and Inference inGraphical Models
We want to learn a probability distribution from data and use it to answer queries.
Applications: medical diagnosis, fault diagnosis, web usage analysis, bioinformatics, collaborative filtering, etc.
A
B C
Answers!Data Model
Learning Inference
![Page 3: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/3.jpg)
3
One-Slide Summary
1. In dependency networks, mean field inference is faster than Gibbs sampling, with similar accuracy.
2. Dependency networks are competitive with Bayesian networks.
A
B C
Answers!Data Model
Learning Inference
![Page 4: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/4.jpg)
4
Outline
• Graphical models:Dependency networks vs. others– Representation– Learning– Inference
• Mean field inference in dependency networks• Experiments
![Page 5: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/5.jpg)
5
Dependency NetworksRepresents a probability distribution over {X1, …, Xn} as a
set of conditional probability distributions.
Example:
€
{P1(X1|X3),P2(X2|X1,X3),P3(X3|X1,X2)}
X1
X2 X3
[Heckerman et al., 2000]
![Page 6: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/6.jpg)
6
Comparison of Graphical Models
Bayesian Network
Markov Network
Dependency Network
Allow cycles? N Y YEasy to learn? Y N YConsistent distribution?
Y Y N
Inferencealgorithms
…lots… …lots… Gibbs,MF (new!)
![Page 7: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/7.jpg)
7
Learning Dependency Networks
For each variable Xi, learn conditional distribution,
€
Pi(X i | X−i)
B=?
<0.2, 0.8>
<0.5, 0.5> <0.7, 0.3>
false
falseC=?
true
true
€
PA (A |B,C) =C=?
<0.7, 0.3> <0.4, 0.6>
falsetrue
€
PB (B |C) =
[Heckerman et al., 2000]
![Page 8: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/8.jpg)
8
Approximate Inference Methods
• Gibbs sampling: Slow but effective• Mean field: Fast and usually accurate• Belief propagation: Fast and usually accurate
A
B C
Answers!Model
![Page 9: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/9.jpg)
9
Gibbs SamplingResample each variable in turn, given its neighbors:
Use set of samples to answer queries. e.g.,
Converges to true distribution, given enough samples (assuming positive distribution).
€
x i(t+1) ~ P X i | X−i = x−i
( t )( )
€
P(B = True) =# of samples where B = True
total # of samples
Previously, the only method used to compute probabilities in DNs.
![Page 10: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/10.jpg)
10
Mean FieldApproximate P with simpler distribution Q:
To find best Q, optimize reverse K-L divergence:
Mean field updates converge to local optimum:
€
P(X1,X2,...,Xn ) ≈Q(X1)Q(X2)L Q(Xn )
€
KL(Q ||P) = Q(X = x)logP(X = x)Q(X = x)x
∑
€
Q(t )(X i) ∝ exp EX − i ~Q ( t−1) (X − i )
logP(X i | X−i)[ ]( )
Works for DNs! Never before tested!
![Page 11: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/11.jpg)
11
Mean Field in Dependency Networks
1. Initialize each Q(Xi) to a uniform distribution.2. Update each Q(Xi) in turn:
3. Stop when marginals Q(Xi) converge.
€
Q(t )(X i) ∝ exp EX − i ~Q ( t−1) (X − i )
logPi(X i | X−i)[ ]( )
If consistent, this is guaranteed to converge.If inconsistent, this always seems to converge in practice.
![Page 12: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/12.jpg)
12
Empirical Questions
Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy?
Q2. How do DNs compare to BNs in inference speed and accuracy?
![Page 13: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/13.jpg)
13
Experiments
• Learned DNs and BNs on 12 datasets• Generated queries from test data– Varied evidence variables from 10% to 90%– Score using average CMLL per variable
(conditional marginal log-likelihood):
€
CMLL(x,e) =1
| X |logP(X i = x i | E = e)
i∑
![Page 14: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/14.jpg)
14
Results: Accuracy in DNs
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0
0.1
0.2
0.3
0.4
0.5
0.6
DN.MF DN.Gibbs
Neg
ative
CM
LLN
egati
ve C
MLL
![Page 15: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/15.jpg)
15
Results: Timing in DNs (log scale)
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0.01
0.1
1
10
100
DN.MF DN.Gibbs
Infe
renc
e Ti
me
(s)
![Page 16: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/16.jpg)
16
MF vs. Gibbs in DNs,run for equal time
Evidence # of MF wins % wins
10% 9 75%
20% 10 83%
30% 10 83%
40% 9 75%
50% 10 83%
60% 10 83%
70% 11 92%
80% 11 92%
90% 12 100%
Average 10.2 85%
In DNs, MF usually more accurate, given equal time.
![Page 17: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/17.jpg)
17
Results: Accuracy
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
DN.MF DN.Gibbs BN.MF BN.Gibbs BN.BP
Neg
ative
CM
LL
![Page 18: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/18.jpg)
18
Gibbs: DN vs. BNEvidence DN wins Percent wins
10% 3 25%20% 1 8%30% 1 8%40% 3 25%50% 5 42%60% 7 58%70% 10 83%80% 10 83%90% 11 92%
Average 5.7 47%
With more evidence, DNs are more accurate.
![Page 19: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/19.jpg)
19
Experimental Results
Q1. In DNs, how does MF compare to Gibbs sampling in speed and accuracy?
A1. MF is consistently faster with similar accuracy, or more accurate with similar speed.
Q2. How do DNs compare to BNs in inference speed and accuracy?
A2. DNs are competitive with BNs – better with more evidence, worse with less evidence.
![Page 20: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/20.jpg)
20
Conclusion
• MF inference in DNs is fast and accurate, especially with more evidence.
• Future work:– Relational dependency networks
(Neville & Jensen, 2007)
– More powerful approximations
Source code available: http://libra.cs.uoregon.edu/
![Page 21: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/21.jpg)
21
Results: Timing (log scale)
NLTCS
MSNBC
KDDCup 2000Plan
tsAudio
Netflix
Jester
MSWeb
Book
WebKB
Reuter
s-52
20 Newsgr
oups0.01
0.1
1
10
100
1000
DN.MF DN.Gibbs BN.MF BN.Gibbs BN.BP
Infe
renc
e Ti
me
(s)
![Page 22: Mean Field Inference in Dependency Networks: An Empirical Study](https://reader035.vdocuments.site/reader035/viewer/2022081520/56816831550346895ddddc05/html5/thumbnails/22.jpg)
22
Learned Models
1. Learning time is comparable.2. DNs usually have higher pseudo-likelihood (PLL)3. DNs sometimes have higher log-likelihood (LL)