physical fluctuomatics 7th~10th belief propagation
DESCRIPTION
Physical Fluctuomatics 7th~10th Belief propagation. Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University [email protected] http://www.smapip.is.tohoku.ac.jp/~kazu/. Textbooks. - PowerPoint PPT PresentationTRANSCRIPT
Physics Fluctuomatics (Tohoku University) 1
Physical Fluctuomatics7th~10th Belief propagation
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
Physics Fluctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 8.Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese), Chapters 6-9.
Physics Fluctuomatics (Tohoku University) 3
What is an important point in computational complexity?
How should we treat the calculation of the summation over 2N configuration?
FT, FT, FT,
211 2
,,,x x x
NN
xxxf
}}
} ;,,,
F){or Tfor(
F){or Tfor( F){or Tfor(
;0
21
2
1
L
N
xxxfaax
xx
a
N fold loops
If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.
Markov Chain Monte Carlo MethodBelief Propagation Method This Talk
Physics Fluctuomatics (Tohoku University) 4
Probabilistic Model and Belief Propagation
Probabilistic Information Processing
Probabilistic Models
Bayes Formulas
Belief Propagation
J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).
Bayesian Networks
Physics Fluctuomatics (Tohoku University) 5
Mathematical Formulation of Belief Propagation
Similarity of Mathematical Structures between Mean Field Theory and Bepief PropagationY. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and Practice (MIT Press, 2001).
Generalization of Belief PropagationS. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energyapproximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
Interpretations of Belief Propagation based on Information GeometryS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).
Physics Fluctuomatics (Tohoku University) 6
Generalized Extensions of Belief Propagation based on Cluster Variation Method
Generalized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).
Key Technology is the cluster variation method in Statistical PhysicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951).T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).
Physics Fluctuomatics (Tohoku University) 7
Belief Propagation in Statistical Physics
In graphical models with tree graphical structures, Bethe approximation is equivalent to Transfer Matrix Method in Statistical Physics and give us exact results for computations of statistical quantities.
In Graphical Models with Cycles, Belief Propagation is equivalent to Bethe approximation or Cluster Variation Method.
Bethe Approximation
Trandfer Matrix Method
(Tree Structures)Belief Propagation
Cluster Variation Method(Kikuchi Approximation)
Generalized Belief Propagation
Physics Fluctuomatics (Tohoku University) 8
Applications of Belief PropagationsImage ProcessingK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).
Low Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).
CDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).
Satisfability ProblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 (2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).
Physics Fluctuomatics (Tohoku University) 9
Strategy of Approximate Algorithm in Probabilistic Information Processing
It is very hard to compute marginal probabilities exactly except some tractable cases.
What is the tractable cases in which marginal probabilities can be computed exactly?Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?
FT, FT, FT,
21112 3
,,,x x x
LL
xxxPxP
Physics Fluctuomatics (Tohoku University) 10
Graphical Representations of Tractable Probabilistic Models
A B C D E
),(),(),(),( EDWDCWCBWBAW DECDBCAB
),( BAWAB ),( CBWBC ),( DCWCD ),( EDWDE
A B C D E
),( BAWAB ),( CBWBC ),( DCWCD ),( EDWDE
B C DX X X=
=
Physics Fluctuomatics (Tohoku University) 11
Graphical Representations of Tractable Probabilistic Models
A B C D EA B C D E
A BA B C D E
B C D EX
Physics Fluctuomatics (Tohoku University) 12
Graphical Representations of Tractable Probabilistic Models
A B C D EA B C D E
A BA B C D E
B C D EX
A B
B C D E A
B C D E
Physics Fluctuomatics (Tohoku University) 13
Graphical Representations of Tractable Probabilistic Models
A B C D EA B C D E
A BA B C D E
B C D EX
A B
B C D E A
B C D E
A B
Physics Fluctuomatics (Tohoku University) 14
Graphical Representations of Tractable Probabilistic Models
A B C D EA B C D E
A BA B C D E
B C D EX
A B
B C D E A
B C D E
A B
A B C D EB C D E
Physics Fluctuomatics (Tohoku University) 15
Graphical Representations of Tractable Probabilistic Models
A B C D EB C D E
Physics Fluctuomatics (Tohoku University) 16
Graphical Representations of Tractable Probabilistic Models
B C D E
C D EX
A B C D EB C D E
A B C
Physics Fluctuomatics (Tohoku University) 17
Graphical Representations of Tractable Probabilistic Models
B C D E
C D EX
C D E B
C D E
A B C D EB C D E
A B C
A B C X
Physics Fluctuomatics (Tohoku University) 18
Graphical Representations of Tractable Probabilistic Models
B C D E
C D EX
C D E B
C D E
B C
A B C D EB C D E
A B C
A B C X
Physics Fluctuomatics (Tohoku University) 19
Graphical Representations of Tractable Probabilistic Models
B C D E
C D EX
C D E B
C D E
B C
A B C D EB C D E
A B C
A B C
B C D EC D E
X
Physics Fluctuomatics (Tohoku University) 20
Graphical Representations of Tractable Probabilistic Models
A B C D EA B C D E
Physics Fluctuomatics (Tohoku University) 21
Graphical Representations of Tractable Probabilistic Models
A B C D EB C D E
A B C D EA B C D E
Physics Fluctuomatics (Tohoku University) 22
Graphical Representations of Tractable Probabilistic Models
A B C D EB C D E
B C D EC D E
A B C D EA B C D E
Physics Fluctuomatics (Tohoku University) 23
Graphical Representations of Tractable Probabilistic Models
A B C D EB C D E
B C D EC D E
A B C D EA B C D E
C D ED E
Physics Fluctuomatics (Tohoku University) 24
Graphical Representations of Tractable Probabilistic Models
A B C D EB C D E
B C D EC D E
A B C D EA B C D E
C D ED E
D EE
Physics Fluctuomatics (Tohoku University) 25
Graphical Representations of Tractable Probabilistic Models
A B C E E
),(),(),(),(),( FEWEDWECWCBWCAW EFDECEBCAC
),( BAWAB ),( CBWBC ),( DCWCD ),( EDWDE
C C DX X X=
=
F
),( FEWEF
EX
A
B
EC
D
F
Physics Fluctuomatics (Tohoku University) 26
Graphical Representations of Tractable Probabilistic Models
A
B
ECA B C D E F
D
F
Physics Fluctuomatics (Tohoku University) 27
Graphical Representations of Tractable Probabilistic Models
A
B
ECA B C D E F
D
F
A
B
EC
D
F
B C D E F
A
A
C
A
C
Physics Fluctuomatics (Tohoku University) 28
Graphical Representations of Tractable Probabilistic Models
A
B
ECA B C D E F
D
F
A
B
EC
C D E F
D
FA
B
EC
B C D E F
D
F
B
CBB
C
Physics Fluctuomatics (Tohoku University) 29
Graphical Representations of Tractable Probabilistic Models
A
B
ECA B C D E F
D
F
A
B
EC
C D E F
D
FA
B
EC
B C D E F
D
F
ECD E F
D
F
Physics Fluctuomatics (Tohoku University) 30
Graphical Representations of Tractable Probabilistic Models
A
B
ECA B C D E F
D
F
A
B
EC
C D E F
D
FA
B
EC
B C D E F
D
F
ECD E F
D
F
ECE F
D
F
Physics Fluctuomatics (Tohoku University) 31
Graphical Representations of Tractable Probabilistic Models
A
B
ECA B C D E F
D
F
A
B
EC
C D E F
D
FA
B
EC
B C D E F
D
F
ECD E F
D
F
ECE F
D
F
EF F
Physics Fluctuomatics (Tohoku University) 32
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A
B
ECA B C D F
E}Pr{
D
F
Physics Fluctuomatics (Tohoku University) 33
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A
B
ECA B C D F
E}Pr{
D
F
A
B
ECA B C E
D E
FF
D
=
Physics Fluctuomatics (Tohoku University) 34
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A
B
ECA B C D F
E}Pr{
D
F
A
B
ECA B C E
D E
FF
D
=
= ECE
D E
F
Physics Fluctuomatics (Tohoku University) 35
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A
B
ECA B C D F
E}Pr{
D
F
A
B
ECA B C E
D E
FF
D
=
= ECE
D E
FEC
D
F
=
Physics Fluctuomatics (Tohoku University) 36
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages A
B
ECA B D F
EC },Pr{
D
F
A
CA E
D E
FF
D
=
= ECE
D E
F=
ECB
CB
A
C B
CA
B
EC
D
F
Physics Fluctuomatics (Tohoku University) 37
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages
},Pr{ ECA
B
EC
D
F
=}Pr{E EC
D
F
=
EC
A
B
ECC
EC
D
F
A
B
EC
D
F
C
C
ECE },Pr{}Pr{
Recursion Formulas for Messages
Physics Fluctuomatics (Tohoku University) 38
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages
A
B
ECC
EC
EC
D
FE
E
FEC
D
FE
E
D
EC
D
F
E
EC
A
B
ECC
A
C
A
B
ECC
B
C
A
A
C
A
C B
CBB
C
E
DE
DD
E
FE
FF
A
B
EC
D
FStep 1
Step 2
Step 3
Physics Fluctuomatics (Tohoku University) 39
Graphical Representations of Tractable Probabilistic Models
Graphical Representation of Marginal Probability in terms of Messages
Step 1
Step 2
Step 3
A
B
EC
D
F
A
B
EC
D
F
A
B
EC
D
F},Pr{ EC
A
B
EC
D
F=
}Pr{E EC
D
F=
}Pr{BB
C
=
},Pr{ CAA
B
EC=
Physics Fluctuomatics (Tohoku University) 40
Belief Propagation
ab
1
cd
2
3 4
56
2221}2,1{11
21
,,,,,,,,,,
xWxWxxWxWxWxxP
CBA dcbadcba
a3xc5x
b4xd6x
Probabilistic Models with no Cycles
Physics Fluctuomatics (Tohoku University) 41
Belief Propagation
1
2 21}2,1{ , xxW
b41
1, xWB b
1
a3
1, xWA a
d
26
2, xWC c
c5
2
2, xWD d
2221}2,1{
11
21
,,,,,
,,,,,
xWxWxxWxWxW
xxP
C
BA
dcba
dcba
ab
1
cd
2
3 4
56
Probabilistic Model on Tree Graph
Physics Fluctuomatics (Tohoku University) 42
Probabilistic Model on Tree Graph
ab
1
cd
2
3 4
56
22522621}2,1{114113
2121}2,1{
,
,,,,,,
xMxMxxWxMxM
xxPxxP
dc,b,a,
dcba
a
a 1113 , xWxM A b
b 1114 , xWxM B
c
c 2225 , xWxM C d
D xdWxM 2226 ,
Physics Fluctuomatics (Tohoku University) 43
Belief Propagation
ab
1
cd
2
3 4
56
1
1
11411321}2,1{
1121}2,1{221
,
,,,
x
x a bBA
xMxMxxW
xWxWxxWxM ba
Probabilistic Model on Tree Graph
a
a 1113 , xWxM A
b
b 1114 , xWxM B
Physics Fluctuomatics (Tohoku University) 44
Belief Propagation for Probabilistic Model on Tree Graph
},{
},{ ,1Prji
jiji xxWZ
xX
No Cycles!!
1X
2X 3X
1kX
kX
2kX
3kX
1kX
Physics Fluctuomatics (Tohoku University) 45
Belief Propagation for Probabilistic Model on Square Grid Graph
E: Set of all the links
Eji
jiijL xxWxxxPxP},{
21 ,,,,
Physics Fluctuomatics (Tohoku University) 46
Belief Propagation for Probabilistic Model on Square Grid Graph
Physics Fluctuomatics (Tohoku University) 47
Belief Propagation for Probabilistic Model on Square Grid Graph
Physics Fluctuomatics (Tohoku University) 48
Marginal Probability
1 3 4
,,,,, 432122x x x x
NN
xxxxxPxP
Physics Fluctuomatics (Tohoku University) 49
Marginal Probability
1 3 4x x x xN
2
1 3 4
,,,,, 432122x x x x
NN
xxxxxPxP
Physics Fluctuomatics (Tohoku University) 50
Marginal Probability
1 3 4x x x xN
2 2
1 3 4
,,,,, 432122x x x x
NN
xxxxxPxP
Physics Fluctuomatics (Tohoku University) 51
Marginal Probability
3 4
,,,,,, 432121}2,1{x x x
NN
xxxxxPxxP
Physics Fluctuomatics (Tohoku University) 52
Marginal Probability
3 4x x xN
3 4
,,,,,, 432121}2,1{x x x
NN
xxxxxPxxP
1 2
Physics Fluctuomatics (Tohoku University) 53
Marginal Probability
3 4x x xN
3 4
,,,,,, 432121}2,1{x x x
NN
xxxxxPxxP
1 2 1 2
Physics Fluctuomatics (Tohoku University) 54
Belief Propagation for Probabilistic Model on Square Grid Graph
1
21}2,1{22 ,x
xxPxP
Physics Fluctuomatics (Tohoku University) 55
Belief Propagation for Probabilistic Model on Square Grid Graph
1
21}2,1{22 ,x
xxPxP
14
5
3
2
6
8
7
Physics Fluctuomatics (Tohoku University) 56
Belief Propagation for Probabilistic Model on Square Grid Graph
21 7
6
8
1
21}2,1{22 ,x
xxPxP
14
5
3
2
6
8
7
Physics Fluctuomatics (Tohoku University) 57
Belief Propagation for Probabilistic Model on Square Grid Graph
21 7
6
8
1
21}2,1{22 ,x
xxPxP
Message Update Rule
14
5
3
2
6
8
7
3
2 1
5
41x
12
Physics Fluctuomatics (Tohoku University) 58
Belief Propagation for Probabilistic Model on Square Grid Graph
21
3
4
5
3
2 1
5
413M
14M
15M
1x1
21M2
1 2
1
1151141132112
1151141132112
221 ,
,
z z
z
zMzMzMzzW
zMzMzMxzWxM
MM
Fixed Point Equations for Messages
Physics Fluctuomatics (Tohoku University) 59
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Physics Fluctuomatics (Tohoku University) 60
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 61
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
0M0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 62
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
01 MM
0M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 63
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
12
01
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 64
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
12
01
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics (Tohoku University) 65
Fixed Point Equation and Iterative Method
Fixed Point Equation ** MM
Iterative Method
23
12
01
MM
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics (Tohoku University) 66
Belief Propagation for Probabilistic Model on Square Grid Graph
Four Kinds of Update Rule with Three Inputs and One Output
Physics Fluctuomatics (Tohoku University) 67
Interpretation of Belief Propagation based on Information Theory
0ln)(
x
xxx P
QQPQD
x
xx 1)( ,0 QQ
ZQF
ZQQxxWQPQD
QF
Ejijiij
ln][
lnln)(,ln)(]|[
][
},{
xxxxx
0 PQDPQ xx
ZPFQQFQ
ln][1][min
x
x
x Eji
jiij xxWZ},{
,
Eji
jiijV xxWZ
xxxP},{
||21 ,1,,,
Free Energy
Kullback-Leibler Divergence
Physics Fluctuomatics (Tohoku University) 68
Interpretation of Belief Propagation based on Information Theory
ZQFPQD ln
xxxx
xxxx
xxx
xx
xx xx
xx
QQWQ
QQWQ
QQxWQQF
E
E
E
ln)(ln
ln)(ln)(
ln)(ln)(
\
0ln)(
x
xxx P
QQPQD
Free EnergyKL Divergence
E
WZ
P
xx 1
x\x
x
x
)(
)(
Q
Q
Physics Fluctuomatics (Tohoku University) 69
Interpretation of Belief Propagation based on Information Theory
ZQFPQD ln
Ejijjiiijij
Viii
Ejiijij
Ejiijij
QQQQQQ
WQ
WQQF
},{
},{
},{
lnln,ln,
ln
,ln,
ln)(
,ln,
xxx
Bethe Free Energy
Free EnergyKL Divergence
E
WZ
P
xx 1
x\x
xx )()( QQ
ix
ii QxQ\x
x)()(
Physics Fluctuomatics (Tohoku University) 70
Interpretation of Belief Propagation based on Information Theory
FPQDQQ
minargminarg
,iji QQ
ZQQFPQD iji ln,Bethe
ijiQQQ
QQFPQDiji
,minargminarg Bethe,
1,
iji QQ
Ejijjiiijij
Viii
Ejiijijiji
QQQQQQ
QQWQQQF
},{
},{Bethe
lnln,ln,
ln,ln,,
Physics Fluctuomatics (Tohoku University) 71
Interpretation of Belief Propagation based on Information Theory
Ejiijij
Viii
Vi ijijiji
ijiiji
QQFQQL
},{
,
BetheBethe
1,1
,
,,
1, ,,,minarg Bethe,
ijiijiijiQQ
QQQQQQFiji
Lagrange Multipliers to ensure the constraints
Physics Fluctuomatics (Tohoku University) 72
Interpretation of Belief Propagation based on Information Theory
Bijijij
Viii
Vi ijijiji
Ejijjiiijij
Viii
Ejiijij
Ejiijij
Viii
Vi ijijijiijiiji
QQQQ
QQQQQQ
QQWQ
QQQQFQQL
1,1,
lnln,ln,
ln,ln,
1,1
,,,
,
},{
},{
},{
,BetheBethe
0,Bethe
ijiii
QQLxQ
Extremum Condition
0,, Bethe
ijijiij
QQLxxQ
Interpretation of Bethe Approximation (7)
FGfP yxyxyx g ,,,
ikikiiii x
ixQ )(
1||1exp },{,
)()(exp
,,
},{,},{, jjijijii
jiijjiij
aa
xxWxxQ
Extremum Condition 0,Bethe
ijiii
QQLxQ 0,
, Bethe
iji
jiij
QQLxxQ
73Physics Fluctuomatics (Tohoku
University)
ik
iiki
ii xMZ
xQ 1
}{\}{\,1,
ijljjljiij
jikiik
ijjiij xMxxWxM
ZxxQ
)()(exp\
},{, ijik
ikijii xMx
Physics Fluctuomatics (Tohoku University) 74
Interpretation of Belief Propagation based on Information Theory
FGfP yxyxyx g ,,,14 2
5
13M
14M
15M
12M
3
26M14
5
13M
14M
15M
12W3
2
6
27M
8
7
28M
115114
1131121
111
xMxM
xMxMZ
xQ
2282272262112
11511411312
2112
,
1,
xMxMxMxxW
xMxMxMZ
xxQ
Extremum Condition 0,Bethe
ijiii
QQLxQ 0,
, Bethe
iji
jiij
QQLxxQ
Physics Fluctuomatics (Tohoku University) 75
Interpretation of Belief Propagation based on Information Theory
FGfP yxyxyx g ,,,14 2
5
13M
14M
15M
12M
3
26M14
5
13M
14M
15M
12W3
2
6
27M
8
7
28M
,121 QQ
115114
1131121
111
xMxM
xMxMZ
xQ
2282272262112
11511411312
2112
,
1,
xMxMxMxxW
xMxMxMZ
xxQ
1514
1312
21
,
MM
MW
M
Message Update Rule
Physics Fluctuomatics (Tohoku University) 76
Interpretation of Belief Propagation based on Information Theory
15141312
15141312
21 ,
,
MMMW
MMMWM
1
3
4 2
5
13M
14M
15M
21M
14
5
3
2
6
8
7
2a
14 2
5
3
=
Message Passing Rule of Belief Propagation
Physics Fluctuomatics (Tohoku University)
Graphical Representations for Probabilistic Models
Probability distribution with two random variables is assigned to a edge.
),(},Pr{ 21}2,1{2211 xxfxXxX 1 2
)(}Pr{ 1111 xfxX 1 Node
Edge
Probability distribution with one random variable is assigned to a graph with one node.
),,(},,Pr{ 321}3,2,1{332211 xxxfxXxXxX 31
2Hyper-edge
77
Physics Fluctuomatics (Tohoku University)
Hyper-graph
Bayesian Network and Graphical Model
),(),(},,Pr{ 32}3,2{21}2,1{332211 xxfxxfxXxXxX
321
4
),(),(),(},,Pr{
13}1,3{32}3,2{21}2,1{
332211
xxfxxfxxfxXxXxX
Tree
31
2Cycle
More practical probabilistic models are expressed in terms of a product of functions and is assigned to chain, tree, cycle or hyper-graph representation.
1 2 3Chain
),(),(),(},,,Pr{
41}4,1{31}3,1{21}2,1{
44332211
xxfxxfxxfxXxXxXxX
312
45),,(),,(
},,,,Pr{
543}5,4,3{321}3,2,1{
5544332211
xxxfxxxfxXxXxXxXxX
78
Physics Fluctuomatics (Tohoku University) 79
Graphical Representations of Tractable Probabilistic Models
),,(),,(),,(),,( IHEWGFDWEDCWCBAW EHIDFGCDEABC
),,( CBAWABC ),,( EDCWCDE
X=C
D
EA
CB
D
F GX
E H
I
=
A
B C
D
E
F G
H
I
),,( IHEWEHI),,( GEDWDEG
X
H IF GA B
Physics Fluctuomatics (Tohoku University) 80
Graphical Representations of Tractable Probabilistic Models
A B F G H I
EDC },,Pr{
A
B C
D
E
F G
H
I
C
D
E A
B C F G
D
E H
I
C
D
E A
B C F G
D
E H
I
A
B C
D
E
F G
H
I
=
= x x x =
D E H IF GA B
Physics Fluctuomatics (Tohoku University) 81
Graphical Representations of Tractable Probabilistic Models
A B D E F G H I
C}Pr{
A
B C
D
E
F G
H
I
C
D
EA
B C
C
D
EA
B C F G
D
E H
I=
= x =
A
B C
D
E
Physics Fluctuomatics (Tohoku University) 82
Graphical Representations of Tractable Probabilistic Models
},,Pr{ EDCC
D
E A
B C F G
D
E H
Ix x x
C
D
EA
B C x}Pr{C
C
D
E
F G
D
E H
IA
B Cx x xC
D
EA
B C x D E
C
D
E
F G
D
E H
Ix x
C
D
ED E
D E
C
D
E
F G
H
I
D E
EDCC },,Pr{}Pr{
Physics Fluctuomatics (Tohoku University) 83
Belief Propagation on Hypergraph Representations in terms of Cactus Tree
C
D
ED E
C
D
E
F G
H
I
F G
D
E H
I
A
B C
F G
D
E H
I
F G
H I
A B A
B C
A
B C
D
E}Pr{C
A
B C
D
E
F G
H
I
Update Flow of Messages in computing the marginal probability Pr{C}
Physics Fluctuomatics (Tohoku University) 84
Interpretation of Belief Propagationfor Hypergraphs based on Information Theory
or :, VEE
We consider hypergraphs which satisfy
Cactus Tree
HypergraphV: Set of all the nodesE: Set of all the hyperedges
Physics Fluctuomatics (Tohoku University) 85
Interpretation of Belief Propagation based on Information Theory
0ln)(
x
xxx P
QQPQD
x
xx 1)( ,0 QQ
ZQF
ZQQWQPQD
QF
E
ln][
lnln)(ln)(]|[
][
xxxxxx
0 PQDPQ xx
ZPFQQFQ
ln][1][min
x
x
x
xE
WZ
E
V xWZ
xxxP
1,,, ||21
Free Energy
Kullback-Leibler Divergence
Physics Fluctuomatics (Tohoku University) 86
Interpretation of Belief Propagation based on Information Theory
ZQFPQD ln
xxxx
xxxx
xxx
xx
xx xx
xx
QQWQ
QQWQ
QQxWQQF
E
E
E
ln)(ln
ln)(ln)(
ln)(ln)(
\
0ln)(
x
xxx P
QQPQD
Free EnergyKL Divergence
E
WZ
P
xx 1
x\x
x
x
)(
)(
Q
Q
E i xiiii
Viii
E
E
i
xQxQQQ
WQ
QQWQQF
lnln
ln
ln
ln)(ln
x
x
xx
xx
xx
xxxx
Physics Fluctuomatics (Tohoku University) 87
Interpretation of Belief Propagation based on Information Theory
ZQFPQD ln
Free EnergyKL Divergence
E
WZ
P
xx 1
x\x
xx )()( QQ
ix
ii QxQ\x
x)()(
Bethe Free Energy
Physics Fluctuomatics (Tohoku University) 88
Interpretation of Belief Propagation based on Information Theory
FPQDQQ
minargminarg
ix
ii QxQ\
x
x
ZQQFPQD iji ln,Bethe
QQFPQD iQQQ i
,minargminarg Bethe,
1
x
xQxQix
ii
E i xiiii
Viii
Ei
i
xQxQQQ
QQWQQQF
lnln
lnln,Bethe
x
x
xx
xx
Physics Fluctuomatics (Tohoku University) 89
Interpretation of Belief Propagation based on Information Theory
EVi xiii
Vi i xiii
ii
QxQ
QxQ
QQFQQL
i
i
11
,,
\,
BetheBethe
x
x
x
x
1 ,,minarg\
Bethe,
xx
xx QxQQxQQQFiii x
iix
iiiQQ
Lagrange Multipliers to ensure the constraints
Physics Fluctuomatics (Tohoku University) 90
Interpretation of Belief Propagation based on Information Theory
Eij
Vi xiii
Vi i x xiiii
E Vi xii
Viii
xii
E
EVi xiii
Vi i x xiiiiii
QxQQxQx
QQQQ
xQxQWQ
QxQ
QxQxQQFQQL
ii i
i
i
i
i i
11
lnln
lnln
11
,,
\,
\,BetheBethe
xx
x
x
x
x
xx
xx
xx
x
x
0,Bethe
QQLxQ iii
Extremum Condition
0,Bethe
QQLQ ix
Physics Fluctuomatics (Tohoku University) 91
Interpretation of Belief Propagation based on Information Theory
i
iiii
ii xMZ
xQ
\1
j jjjj xMxW
ZxQ
\\
1
Extremum Condition 0,Bethe
QQLxQ iii 0,
, Bethe
QQL
xxQ ijiij
ix
ii QxQ\
x
x
ij jjjj
iiii xMxW
ZZxM
\ \\\
Physics Fluctuomatics (Tohoku University) 92
Summary
Belief Propagation and Message Passing RuleInterpretation of Belief Propagation in the stand point of Information Theory
Future Talks
11th Probabilistic image processing by means of physical models 12th Bayesian network and belief propagation in statistical inference
Physics Fluctuomatics (Tohoku University) 93
Practice 9-1
a b c d
XY ya,b,c,d,xPyxP ,,
a
AXA xaWxM , b
BXB xbWxM ,
c
CYC ycWyM , d
DYD ydWyM ,
ydWycWyxWxbWxaWya,b,c,d,xP DCXYBA ,,,,,,
We consider a probability distribution P(a,b,c,d,x,y) defined by
Show that marginal Probability
is expressed by
yMyMyxWxMxMyxP DCXYXBXAXY 22,,
Physics Fluctuomatics (Tohoku University) 94
Practice 9-2
1151141131121
111 xMxMxMxMZ
xQ
228227226211211511411312
2112 ,1, xMxMxMxxWxMxMxMZ
xxQ
2
228227226211212
1112 ,
x
xMxMxMxxWZZxM
By substituting
2
211211 ,x
xxQxQto , derive the following equation.
Physics Fluctuomatics (Tohoku University) 95
Practice 9-3
Make a program to solve the nonlinear equation x=tanh(Cx) for various values of C. Obtain the solutions for C=0.5, 1.0, 2.0 numerically. Discuss how the iterative procedures converge to the fixed points of the equations in the cases of C=0.5, 1.0, 2.0 by drawing the graphs of y=tanh(Cx) and y=x.
23
12
01
tanhtanhtanh
CxxCxxCxx