グラフィカルモデルにおける確率伝搬法の数理 …kazu/tutorial...22 december, 2007...

22 December, 200722 December, 2007 OCAM2007 (Osaka)OCAM2007 (Osaka) 11

グラフィカルモデルにおける確率伝搬法の数理グラフィカルモデルにおける確率伝搬法の数理Mathematical Structure of Belief Propagation in Mathematical Structure of Belief Propagation in

Graphical Probabilistic ModelsGraphical Probabilistic Models

東北大学東北大学大学院情報科学研究科大学院情報科学研究科応用情報科学専攻応用情報科学専攻田中田中和之和之

(Kazuyuki Tanaka, Tohoku University)(Kazuyuki Tanaka, Tohoku University)[email protected]@smapip.is.tohoku.ac.jp

http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/


ContentsContents

1.1. IntroductionIntroduction2.2. Graphical ModelGraphical Model3.3. Belief PropagationBelief Propagation4.4. Interpretation of Belief PropagationInterpretation of Belief Propagation5.5. Belief Propagation for Gaussian Graphical ModelBelief Propagation for Gaussian Graphical Model6.6. Application to Bayesian Image AnalysisApplication to Bayesian Image Analysis7.7. Statistical Performance for Bayesian Image AnalysisStatistical Performance for Bayesian Image Analysis8.8. Concluding RemarksConcluding Remarks


Graphical Model in Probabilistic Graphical Model in Probabilistic Information ProcessingInformation Processing

Probabilistic Probabilistic Information ProcessingInformation Processing

BayesBayes FormulaFormula

Belief PropagationBelief Propagation

J. Pearl: Probabilistic Reasoning in Intelligent Systems: J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbocoding and decoding: Turbo--codes, IEEE Trans. Comm., codes, IEEE Trans. Comm., 4444(1996).(1996).

Bayesian NetworkBayesian Network

Graphical ModelGraphical Model


Belief PropagationBelief Propagation

Belief Propagation is equivalent to Belief Propagation is equivalent to BetheBethe ApproximationApproximationY. Y. KabashimaKabashima and D. and D. SaadSaad: : EurophysEurophys. . LettLett. 1998. . 1998. Generalized Belief Propagation was proposed based on Cluster VarGeneralized Belief Propagation was proposed based on Cluster Variation iation MethodMethodJ. S. J. S. YedidiaYedidia, W. T. Freeman and Y. Weiss: IEEE Transactions on , W. T. Freeman and Y. Weiss: IEEE Transactions on Information Theory, 2005.Information Theory, 2005.Interpretation of Generalized Belief Propagation based on InformInterpretation of Generalized Belief Propagation based on Information ation GeometryGeometryS. Ikeda, T. Tanaka and S. S. Ikeda, T. Tanaka and S. AmariAmari: Neural Computation 2004.: Neural Computation 2004.


Application of Belief PropagationApplication of Belief Propagation

LDPC codes in Error Correcting CodesLDPC codes in Error Correcting CodesY. Kabashima and D. Saad: J. Phys. A, Y. Kabashima and D. Saad: J. Phys. A, 3737, 2004, Topical Review. , 2004, Topical Review.

CDMA Multiuser Detection in Mobile Phone CommunicationCDMA Multiuser Detection in Mobile Phone CommunicationY. Kabashima: J. Phys. A, Y. Kabashima: J. Phys. A, 3636, 2003., 2003.

SatisfabilitySatisfability (SAT) Problems in Computation Theory(SAT) Problems in Computation TheoryM. M. MezardMezard, G. , G. ParisiParisi, R. , R. ZecchinaZecchina: Science, : Science, 297297, 2002., 2002.

Image ProcessingImage ProcessingK. Tanaka: J. Phys. A, K. Tanaka: J. Phys. A, 3535, 2002, Topical Review., 2002, Topical Review.

Probabilistic Inference in AIProbabilistic Inference in AIK. Tanaka: IEICE Trans. on Information and Systems, 2005.K. Tanaka: IEICE Trans. on Information and Systems, 2005.


ContentsContents


22 December, 2007 OCAM2007 (Osaka) 7

Probabilistic Model and Graphical Model

)()(),( 221121 xxxxP ΦΦ=

Probability distributions with two random variables is assigned to a graph representation with two nodes and one link.

x1 x2Non-Linked Two Nodes

),(),( 211221 xxxxP Φ= x1 x2

)()( 111 xxP Φ= x1 Node

Linked Two Nodes

Probability distribution with one random variables is assigned to some graph representations with a node.

Independent Case


Probabilistic Model and Graphical Model

),(),(),,( 32232112321 xxxxxxxP ΦΦ=

),(),(),(),(),,,,(

5115411431132112

54321

xxxxxxxxxxxxxP

ΦΦΦΦ= x3

x2 x1 x4

x5),(),(),(),,(

133132232112

321

xxxxxxxxxP

ΦΦΦ=Tree

x3x1

x2Cycle

Function consisting of a product of functions with two variables can be assigned to a graph representation.

x1 x2 x3Chain


ContentsContents



Graphical Representation for Tractable Models

Tractable Model

⎟⎟⎠

⎞⎜⎜⎝

⎛Φ⎟⎟

⎠

⎞⎜⎜⎝

⎛Φ⎟⎟

⎠

⎞⎜⎜⎝

⎛Φ=

ΦΦΦ

∑∑∑

∑∑∑

432

2 3 4

),(),(),(

),(),(),(

411431132112

411431132112

xxx

x x x

xxxxxx

xxxxxx ∑∑∑2 3 4x x x

Intractable Model∑∑∑ ΦΦΦ

2 3 4

),(),(),( 244243343223x x x

xxxxxx

Tree Graph

Cycle Graph

It is possible to calculate each summation independently.

It is hard to calculate each summation independently.

x2

x4 x3

x1

∑∑∑2 3 4x x x

x3

x2

x4


Belief Propagation for Tree Graphical Model

3

2 1

5

4

3

2 1

5

4

13→M

14→M

15→MAfter taking the summations over red nodes 3,4 and 5, the function of nodes 2 and 1 can be expressed in terms of some messages.

44 344 2144 344 2144 344 21)(

5115

)(

4114

)(

31132112

5115411431132112

115

5

114

4

113

3

3 4 5

),(),(),(),(

),(),(),(),(

xM

x

xM

x

xM

x

x x x

xxxxxxxx

xxxxxxxx

→→→

⎟⎟⎠

⎞⎜⎜⎝

⎛Φ⎟⎟

⎠

⎞⎜⎜⎝

⎛Φ⎟⎟

⎠

⎞⎜⎜⎝

⎛ΦΦ=

ΦΦΦΦ

∑∑∑

∑∑∑


Belief Propagation for Tree Graphical Model

3

2 1

5

4

By taking the summation over all the nodes except node 2, message from node 1 to node 2 can be expressed in terms of all the messages incoming to node 1 except the own message.

3

2 1

5

4

13→M14→M

15→M∑=

1x

121→M

2

Summation over all the nodes except 2


Belief Propagation for Graphical Model on Tree

( ) ( ) ( ) ( ) ( )∑ ++→−→−→−++→ Φ=kx

kkkkkkkkkkkkkkkk xxxMxMxMxM 11,32111 ,

Graph has no cycle.

It passes through each node only once.

1X

2X 3X

1−kX

kX

2−kX

3−kX

1+kX

Message Passing RuleMessage Passing Rule


Belief Propagation for Graphical Model with Cycles

( ) ( ) ( ) ( ) ( )∑ ++→−→−→−++→ Φ≅kx

kkkkkkkkkkkkkkkk xxxMxMxMxM 11,32111 ,

1X

2X 3X

1−kX

kX

2−kX

3−kX

1+kX

Message Passing Rule It passes through each node repeatedly.It passes through each node repeatedly.

It seems to be It seems to be tree locally.tree locally.


Loopy Belief Propagation for Graphical Model in Image Processing

We consider a graphical model represented in terms of the square lattice.Square lattice includes a lot of cycles.Belief propagation are applied to the calculation of statistical quantities as an approximate algorithm.

Every graph consisting of a node and its four neighbouring nodes can be regarded as a tree graph.

Loopy Belief Propagation

3

2 1

5

4∑←1x

12

2 1 4

5

32 1 4

5

3



( )MMrrr

Ψ←

( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )∑∑∑

→→→

→→→

→ Φ

Φ←

1 2

1

1151141132112

1151141132112

221 ,

,

z z

z

zMzMzMzz

zMzMzMxzxM

41

3

2

5

Message Passing Rule in Loopy Belief Propagation

Averages, variances and covariances of the graphical model are expressed in terms of messages.

3

2 1

5

4∑←1z

12


Iteration Procedure

Fixed Point Equation ( )*xx Ψ=*

Iteration

( )( )( )

M

23

12

01

xxxxxx

Ψ←Ψ←Ψ←

0x1x

1x

0

xy =

)(xy Ψ=

y

x*x


Belief Propagation and Graphical Model with CyclesBelief Propagation and Graphical Model with Cycles

( ) ( ) ( )∏∈

Ω Φ==Nij

jiij xxxxxPxP ,,,, 21 Lr

( ) ( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )∑

∑

→→→→

→→→→

Ω

≅

≡

1

115114113112

115114113112

\2111 ,,,

z

xx

zMzMzMzMxMxMxMxM

xxxPxPi

rL

２１

３

４

５

{ }Ω=Ω ,,2,1 L

links the all of Set:N

２１

３

４

５( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )∑∑∑

→→→

→→→

→ Φ

Φ=

1 2

1

1151141132112

1151141132112

221 ,

,

z z

z

zMzMzMzz

zMzMzMxzxM

Fixed Point Equation

Marginal Probability



We have four kinds of message passing rules for each node.

Each massage passing rule includes 3 incoming messages and 1 outgoing message

Visualizations of Passing Messages


ContentsContents



Interpretation of Belief Propagation (1)

[ ] ( )( ) 0ln)( ≥⎟⎟

⎠

⎞⎜⎜⎝

⎛≡∑ x

xxx P

QQPQD ( ) ⎟⎠

⎞⎜⎝

⎛=≥ ∑

xxx 1)( ,0 QQQ

( ) ( )

ZQF

ZQQxxWQPQD

QF

Nijjiij

ln][

lnln)(,ln)(]|[

][

+=

++= ∑∑∑∈ 4444444 34444444 21

xxxxx

( ) ( ) [ ] 0=⇒= PQDPQ xx

( ) ZPFQQFQ

ln][1][min −==⎭⎬⎫

⎩⎨⎧

=∑x

x

( )∑∏∈

≡x Nij

jiij xxWZ ,

( ) ( )∏∈

=Nij

jiijL xxWZ

xxxP ,1,,, 21 L

Free Energy

Kullback-Leibler Divergence



[ ] [ ] ( )ZQFPQD ln+=

[ ] ( ) ( )

{ }( ) ( )

( ) ( ) ( )xx

xxx

xxx

x

xx

xx

QQxxxxQ

QQxxQ

QQxxQQF

Nij x xjiijjiij

Nij x xjiij

xx

Nijjiij

i j

i j ji

ln)(,ln,

ln)(,ln)(

ln)(,ln)(

,\

∑∑∑∑

∑∑∑∑ ∑

∑∑∑

+Φ=

+Φ⎟⎟⎠

⎞⎜⎜⎝

⎛=

+Φ≡

∈

∈

∈

[ ] ( )( ) 0ln)( ≥⎟⎟

⎠

⎞⎜⎜⎝

⎛≡∑ x

xxx P

QQPQD

Free EnergyKL Divergence

( ) ( )∏∈

Φ=Nij

jiij xxZ

P ,1x

{ }∑≡

ji xx

jiij

Q

xxQ

,)(

),(

\xx



[ ] [ ] ( )ZQFPQD ln+=

[ ] ( ) ( )

( )

( ) ( )

( ) ( )

( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑

∑∑

∑∑∑

∑

∑∑∑

∈

Ω∈

∈

∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−+

+

Φ≈

+

Φ=

Nijjjiiijij

iii

Nijijij

Nijijij

QQQQQQ

QQ

Q

QQ

QQF

ξξξ ζ

ξ

ξ ζ

ξ ζ

ξξξξζξζξ

ξξ

ζξζξ

ζξζξ

lnln,ln,

ln

,ln,

ln)(

,ln,

xxx

Bethe Free Energy

Free EnergyKL Divergence( ) ( )∏

∈

Φ=Nij

jiij xxZ

P ,1x

{ }∑≡

ji xxjiij QxxQ

,)(),(

\xx

∑≡ix

ii QxQ\x

x)()(



[ ] FPQDQQ γγ

minargminarg ≅

( ) ( )∑=ς

ςξξ ,iji QQ

[ ] { }[ ] ZQQFPQD iji ln,Bethe +≅

[ ]{ }

{ }[ ]ijiQQQ

QQFPQDiji

,minargminarg Bethe,

⇒

( ) ( ) 1, == ∑∑∑ξ ςξ

ςξξ iji QQ

{ }[ ] ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑

∑∑∑∑∑

∈

Ω∈∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−+

+Φ≡

Nijjjiiijij

iii

Nijijijiji

QQQQQQ

QQQQQF

ξξξ ς

ξξ ς

ξξξξςξςξ

ξξςξςξ

lnln,ln,

ln,ln,,Bethe



{ }[ ] { }[ ]

( ) ( ) ( )

( ) ( )∑ ∑∑∑ ∑

∑ ∑∑ ∑

∈Ω∈

Ω∈ ∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−

≡

Nijijij

iii

i Njijiji

ijiiji

QQ

QQ

QQFQQL

i

1,1

,

,,

,

BetheBethe

ξ ζξ

ξ ς

ζξνξν

ζξξξλ

{ }{ }[ ] ( ) ( ) ( ) ( )

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

=== ∑∑∑∑ 1, ,,,minarg Bethe, ξ ςξς

ςξξςξξ ijiijiijiQQ

QQQQQQFiji

Lagrange Multipliers to ensure the constraints



{ }[ ] { }[ ] ( ) ( ) ( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( )∑ ∑∑∑ ∑∑ ∑∑ ∑

∑ ∑∑∑∑

∑∑∑∑∑

∑ ∑∑∑ ∑

∑ ∑∑ ∑

∈Ω∈Ω∈ ∈

∈

Ω∈∈

∈Ω∈

Ω∈ ∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−+

+=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−≡

Nijijij

iii

i Njijiji

Nijjjiiijij

iii

Nijijij

Nijijij

iii

i Njijijiijiiji

QQQQ

QQQQQQ

QQWQ

QQ

QQQQFQQL

i

i

1,1,

lnln,ln,

ln,ln,

1,1

,,,

,

,BetheBethe

ξ ζξξ ζ

ξξξ ζ

ξξ ζ

ξ ζξ

ξ ς

ζξνξνζξξξλ

ξξξξζξζξ

ξξζξζξ

ζξνξν

ζξξξλ

( ) { }[ ] 0,Bethe =∂

∂iji

ii

QQLxQ

Extremum Condition

( ) { }[ ] 0,, Bethe =

∂∂

ijijiij

QQLxxQ



144 2

5

13→M

14→M

15→M

12→M

33

26→M144

5

13→M

14→M

15→M

12Φ33

2

6

27→M

88

77

28→M

( ) ( ) ( )

( ) ( )115114

1131121

111

xMxM

xMxMZ

xQ

→→

→→

×

= ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )2282272262112

11511411312

2112

,

1,

xMxMxMxx

xMxMxMZ

xxQ

→→→

→→→

Φ×

=

ExtremumCondition( ) { }[ ] 0,Bethe =

∂∂

ijiii

QQLxQ ( ) { }[ ] 0,

, Bethe =∂

∂iji

jiij

QQLxxQ



144 2

5

13→M

14→M

15→M

12→M

33

26→M144

5

13→M

14→M

15→M

12W33

2

6

27→M

88

77

28→M

( ) ( )∑=ς

ζξξ ,121 QQ

( ) ( ) ( )

( ) ( )115114

1131121

111

xMxM

xMxMZ

xQ

→→

→→

×

= ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )2282272262112

11511411312

2112

,

1,

xMxMxMxxW

xMxMxMZ

xxQ

→→→

→→→

×

=

( )( ) ( )

( ) ( )ςς

ςξς

ξ

ς

1514

1312

21

,

→→

→

→

×

Φ∝ ∑MM

MM

Message Update Rule



( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )∑∑∑

→→→

→→→

→ Φ

Φ=

ξ ς

ς

ςςςξς

ςςςξςξ

15141312

15141312

21 ,

,

MMM

MMMM

1

33

44 2

5

13→M

14→M

15→M

21→M

144

5

33

2

6

88

77

∑2x

L

144 2

5

33

=

Message Passing Rule of Belief Propagation

It corresponds to Bethe approximation in the statistical mechanics.


ContentsContents



Gaussian Graphical Model

( )

formula. integral Gauss ldimensiona-multi theusingby

calculated becan

average and

lnenergy Free

fff d

Z

∫−

ρ

( ) ( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−= ∑∑

∈Ω∈ Nijji

iii ffgf

Z22

21

21exp1 αβρ f

( )+∞∞−∈ ,if

( ) gCIfff 11 )( −−+=∫ αβρ dCovariant MatrixCovariant Matrix

( )( )

( )⎪⎩

⎪⎨

⎧∈−=

=otherwise0

14

Nijji

ji C


Kullback-Leibler Divergence of Gaussian Graphical Model

[ ] ( ) ( )( ) [ ] ( )ZFzdzzQzQQD lnln +=⎟⎟

⎠

⎞⎜⎜⎝

⎛≡ ∫

∞+

∞−ρ

ρρ r

r

rr

[ ] ( )( ) ( )( )( ) ( )∫

∑∑

+

−++−+−−=∈Ω∈

zdzQzQ

mmVVVgmVFNij

jijijii

iii

rrr ln

221

21 22 αβρ

( ) zdzQzm iirr

∫≡( ) ( ) zdzQmzV iii

rr∫ −≡ 2

( )( ) ( ) zdzQmzmzV jjiiijrr

∫ −−≡

Entropy TermEntropy Term

( ) ( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−= ∑∑

∈Ω∈ Nijji

iii ffgf

Zgf 22

21

21exp1,, αββαρ rr


Belief Propagation

( ) ( ) ( )∫ −≡ zdzQzffQ iiiirrδ

( ) ( ) ( )( ) ( )⎟

⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Nij jii

jiij

iii fQfQ

ffQfQQ

,f

Trial Function

( ) ( ) ( ) ( )∫ −−≡ zdzQzfzfffQ jjiijiijrr δδ,

Tractable Form


Belief Propagation

( ) ( ) ( )( ) ( )⎟

⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Nij jii

jiij

iii fQfQ

ffQfQQ

,f

( ) ( ) ( )

( ) ( )( ) ( )⎟

⎠⎞

⎜⎝⎛ −−−=

−≡

−

∫

γγγγγγ

γ

γγγγ

π

δ

mfmf

zdzQzffQ

rrrr

rrrrr

1T

21exp

det2

1 AA

Trial Function

Marginal Distribution of GGM is also GGM

( ) iii V=γA ( ) ijij V=γA( ) iimm =γ

r


Belief Propagation

[ ] { }[ ]( )( ) ( )( )

( ) ( )( )∑∑

∑∑

∈Ω∈

∈Ω∈

⎟⎠⎞

⎜⎝⎛ ++⎟

⎠⎞

⎜⎝⎛ +−

−++−+−−=

=

Nijij

ii

Nijjijiji

iiii

ijii

V

mmVVVgmV

VVmFF

Adet2ln2112ln

2113

221

21

,,

2

22

ππ

αβ

ρ

( )( ) ( )

( ) ( )⎟⎠⎞

⎜⎝⎛ −−−= −

γγγ mfmffQ rrrrr 1T

21exp

det2

1γγ

γγ

γγπ

AA

( ) ( ) ( )( ) ( )⎟

⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Nij jii

jiij

iii fQfQ

ffQfQQ

,f

Bethe Free Energy in GGMBethe Free Energy in GGM


Belief Propagation

( )

( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛−⎟⎟

⎠

⎞⎜⎜⎝

⎛⎟⎠⎞

⎜⎝⎛ ++=⎟⎟

⎠

⎞⎜⎜⎝

⎛−

01

0110

1001

41

43

1

01

0

ααβVV

V

{ }[ ]{ }

0340,,

11 =⎟⎟

⎠

⎞⎜⎜⎝

⎛−++⇒=

∂

∂∑

∈

−−

Nijj jjji

ijiii

i

ijii iVVVV

iVV

VVmFαβ

{ }[ ]00

,,1

=⎟⎟⎠

⎞⎜⎜⎝

⎛+−⇒=

∂

∂−

jVVVV

iV

VVmF

jjji

ijii

ij

ijii α

( )1VVV jiij ≡= Vii and Vij do not depend on pixel i and link ij

( )0VVii ≡

( )

( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛13

81

1

0

αVV0=β

exact. is mrgm rr 12 )( −+= CI ασ

{ }[ ] ( ) ( ) 00,,

=−−−⇒=∂

∂∑∈ icj

jiiii

ijii mmgmmVVmF

αβ


Loopy Belief Propagation and TAP Free EnergyLoopy Belief Propagation ( )

( )( )0

41121

5223

2

+→

+−=

++−=

α

ααα

αα

OVVVV

VVV

jiji

jiij

{ }[ ]( )( ) ( )( )

( ) ( )∑∑

∑∑

∈Ω∈

∈Ω∈

+−⎟⎠⎞

⎜⎝⎛ +−

−+++−−=

Nijji

ii

Nijjiji

iiii

ijii

OVVV

mmVVgmV

VVmF

42

22

212ln

211

21

21

,,

ααπ

αβ

( ) iiiV=γA ( ) ijij

V=γA

TAP TAP Free Free EnergyEnergy

( ) 01 =−− −

ijijAα

Mean Field Free EnergyMean Field Free Energy


Generalized Belief PropagationCluster: Set of nodes

( ) ( ){ }

∑∈<

−−≡C',''

' 1γγγγ

γμγμ

1 2

3 4

1 2

3 4

1

3

2

4

Ω B

Ω∪= BC

414,114,434,334,323,223,212,112

>>>>>>>>

Example: System consisting of 4 nodes

( ) ( ) ( ) ( ) 114342312 −==== μμμμ( ) ( ) ( ) ( ) 14321 ==== μμμμ

{ }BBBCBBBB

∈∀∈∀==

∉⇒∈∀∈∀>>⇒=

' and for '' ' and :

' and ' Subcluster

γγγγααγγγγ

αγαγγγα

IU

I

I


Selection of B in LBP and GBP

Ω

B1 2 3

4 5 6

7 8 9

1 2

4 5

1

4

2

5

2 3

5 6

3

6

7 8

4

7

5

8 8 9

6

9

1 2

4 5

2 3

5 64 5

7 8

5 6

8 9

B

LBP (Bethe Approx.）

GBP (Square Approx.

in CVM)


Selection of B and C in Loopy Belief Propagation

BΩ

LBP (Bethe Approx.）

CThe set of Basic ClustersThe set of Basic Clusters The Set of Basic Clusters The Set of Basic Clusters

and Their and Their SubclustersSubclusters


Selection of B and C in Generalized Belief Propagation

BΩ

GBP (Square Approximation in CVM）

CThe set of Basic ClustersThe set of Basic Clusters The Set of Basic Clusters The Set of Basic Clusters

and Their and Their SubclustersSubclusters


Generalized Belief Propagation

[ ] { }[ ]( )( ) ( )( )

( ) ( )( )∑

∑∑

∈

∈Ω∈

⎟⎠⎞

⎜⎝⎛ +−

−++−+−−=

=

C

jijijiijNij

iiiii

ijii

mmVVVgmV

VVmFF

γγ

γπγμ

αβ

ρ

Adet2ln211

221

21

,,

22

( ) ( ){ }

( )γμ

γγγγ

−

∈∏=

C

QQ ff

( ) ( ) ( )( ) ( )

( ) ( )⎟⎠⎞

⎜⎝⎛ −−−=−≡ −∫ γγγγγ

γγγγγγ

πδ mfAmf

Azzzff 1

21exp

det2

1 TdQQ

Trial Function

Marginal Distribution of GGM is also GGM

( ) iiiV=γA

( ) ijijV=γA


{ }[ ]{ }

( )( ){ }

00,,

,

1

,,=++⇒=

∂

∂∑∑

∈≤

−

∈<< Ciii

Bijji

ijii

VVVmF

γγγγ

γγγ

γμαβ A

Generalized Belief Propagation{ }[ ] ( ) ( )

{ }00

,,

,,=−−−⇒=

∂

∂∑

∈<< Cijjjiii

i

ijii mmgmmVVmF

γγγ

αβ

{ }[ ] ( )( ){ }

00,,

,,

1 =+−⇒=∂

∂∑

∈<<

−

Cjiij

ij

ijii

VVVmF

γγγγγγμα A

{ } NijiVV iji ∈Ω∈≡ ,,V( )VΨV =

m is exactgm rr 12 )( −+= CI ασ


ContentsContents



Bayes Formula and Bayesian Network

Posterior Probability

}Pr{}Pr{}|Pr{}|Pr{

BAABBA =

Bayes Rule

Prior Probability

Event A is given as the observed data.Event B corresponds to the original information to estimate. Thus the Bayes formula can be applied to the estimation of the original information from the given data.

A

BBayesian Network

Data-Generating Process


Image Restoration by Probabilistic Model

Original Image

Degraded Image

Transmission

Noise

444 3444 21

444 8444 764444444 84444444 76

4444444 84444444 76

Likelihood Marginal

PriorLikelihood

Posterior

}ageDegradedImPr{}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

=

Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability.

Bayes Formula


Image Restoration by Probabilistic Model

Degraded

Image

i

fi: Light Intensity of Pixel iin Original Image

),( iii yxr =r

Position Vector of Pixel i

gi: Light Intensity of Pixel iin Degraded Image

i

Original

Image

The original images and degraded images are represented by f = {fi} and g = {gi}, respectively.


Probabilistic Modeling of Image Restoration

444 8444 764444444 84444444 76

4444444 84444444 76

PriorLikelihood

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{


∝

∏=

Ψ====N

iii fg

1),(}|Pr{

}Image Original|Image DegradedPr{

fFgG

fg

Random Fieldsfi

gi

fi

gior

Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels.


Probabilistic Modeling of Image Restoration

444 8444 764444444 84444444 76

4444444 84444444 76

PriorLikelihood

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{


∝

∏Φ===NN:

),(}Pr{ }Image OriginalPr{

ijji fffF

f

Random Fields

Assumption 2: The original image is generated according to a prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels.

i j

Product over All the Nearest Neighbour Pairs of Pixels


Prior Probability for Binary Image

== >p p p−

21 p−

21i j Probability of

NeigbouringPixel

∏Φ==NeighbourNearest :

),(}Pr{ij

ji fffFi j

It is important how we should assume the function Φ(fi,fj) in the prior probability.

)0,1()1,0()0,0()1,1( Φ=Φ>Φ=Φ

We assume that every nearest-neighbour pair of pixels take the same state of each other in the prior probability.

1,0=if


Prior Probability for Binary Image

Prior probability prefers to the configuration with the least number of red lines.

Which state should the center pixel be taken when the states of neighbouring pixels are fixed to the white states?

？

>

== >p p p−

21 p−

21i j Probability of

Nearest NeigbourPair of Pixels


Prior Probability for Binary ImagePrior Probability for Binary Image

Which state should the center pixel be taken when the states of neighbouring pixels are fixed as this figure?

？-？== >

p p

> >=

Prior probability prefers to the configuration with the least number of red lines.


What happens for the case of large umber of pixels?

p 0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0lnp

Disordered State Critical Point(Large fluctuation)

small p large p

Covariance between the nearest neghbourpairs of pixels

Sampling by Marko chain Monte Carlo

Ordered State

Patterns with both ordered statesand disordered states are often generated near the critical point.


Pattern near Critical Point of Prior Probability

0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0ln p

similar

small p large p

Covariance between the nearest neghbourpairs of pixels

We regard that patterns generated near the critical point are similar to the local patterns in real world images.


Bayesian Image Analysis

{ } { } { }{ }

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛Φ⎟

⎟⎠

⎞⎜⎜⎝

⎛Ψ∝

=

======

∏∏= NeighbourNearest:1

),(),(

PrPrPr

Pr

ijji

N

iii ffgf

gGfFfFgG

gGfF

fg

{ }fF =Pr { }fFgG ==Pr gOriginalImage

Degraded Image

Prior Probability


Degradation Process

Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.

B：Set of all the nearest neighbourpairs of pixels

Ω：Set of All the pixels


Estimation of Original Image

We have some choices to estimate the restored image from posterior probability.

In each choice, the computational time is generally exponential order of the number of pixels.

}|Pr{maxargˆ gG === iiz

i zFfi

2

}|Pr{minargˆ⎟⎟

⎠

⎞

⎜⎜

⎝

⎛==−= ∑

zgGzFiii zf

iζ

ζ

}|Pr{maxargˆ gGzFfz

===

Thresholded Posterior Mean (TPM) estimation

Maximum posterior marginal (MPM) estimation

Maximum A Posteriori (MAP) estimation

∑ =====if

ii fF\

}|Pr{}|Pr{f

gGfFgG

(1)

(2)

(3)


Statistical Estimation of Hyperparameters

∑ =====z

zFzFgGgG }|Pr{},|Pr{},|Pr{ αββα

( )},|Pr{max arg)ˆ,ˆ(

,βαβα

βαgG ==

f g

Marginalized with respect to F

}|Pr{ αfF = },|Pr{ βfFgG == gOriginal Image

Marginal Likelihood

Degraded ImageΩy

x

},|Pr{ βαgG =

Hyperparameters α, β are determined so as to maximize the marginal likelihood Pr{G=g|α,β} with respect to α, β.


Maximization of Marginal Likelihood by EM Algorithm

∑ =====z

zFzFgGgG }|Pr{},|Pr{},|Pr{ ασσαMarginal Likelihood

( ) },|,Pr{ln}',',|Pr{

,',',

∑ =====z

gGzFgGzF

g

σασα

σασαQ

( ) ( )( )

( ) ( )( )( )

( ) ( )( ).,,maxarg1,1 :Step-M

},|,Pr{ln)}(),(,|Pr{

,, :Step-E

,ttQtt

tt

ttQ

σασασα

σασα

σασα

σα←++

====← ∑z

gGzFgGzF

E-step and M-Step are iterated until convergence:EM (Expectation Maximization) Algorithm

Q-Function


Prior Probability in Probabilistic Image Processing

{ } ( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−== ∑

∈Nijji ff

ZfF 2

211Pr α

αexp

Prior

rr

0005.0=α 0030.0=α0001.0=α

Samples are generated by MCMC.

Markov Chain Monte Carlo Method

{ }Ω=Ω ,,2,1 L



Degradation Process

Additive White Gaussian Noise

( )2,0~ σNfg ii −

{ } ( )∏Ω∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−===

iii gffFgG 2

22 21exp

2

1Prσπσ

rrrr

Histogram of Gaussian Random Numbers

nr Noise Gaussianfr

Image Original gr Image Degraded


( ) ( ) gzd,gzPz

m

mm

gm rrrrrM

rr

CII

22

1

,,,ασ

σασα+

=≡

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

= ∫Ω

Degradation Process and Prior

Degradation Process

Prior Probability Density Function

( )+∞∞−∈ ,, ii gf

( ) ( ) ( )∏∈

⎟⎠⎞

⎜⎝⎛ −−=

Nijji ff

ZfP 2

2exp1 α

αα

PR

r

( ) ( )∏Ω∈

⎟⎠⎞

⎜⎝⎛ −−=

iii gffgP 2

221exp

21,

σσπσ

rr

( ) ( ) ( )( )σα

ασσα

,

,,,

gP

fPfgPgfP r

rrrrr

=Posterior Probability Density Function

( )2,0~ σNn

nfg

i

iii +=

{ }Ω=Ω ,,2,1 L


( )( )

( )⎪⎩

⎪⎨

⎧∈−=

=otherwise0

14

Nijji

ji C

Multi-Dimensional Gaussian Integral Formula


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

=

Ωg

gg

gM

r 2

1

( ) ( )( ) ( )( ) ( ) ( )

gtt

gtt

tt rr

CIC

CIC

2T

2

21 1Tr11

σασα

σα+Ω

++Ω

=+ −

( ) ( )( ) ( )( )

( ) ( )( ) ( )( )

gtt

ttgtt

tt rr22

242T

2

22 1Tr11

CI

CCI

I

σα

σα

σα

σσ+Ω

++Ω

=+

( ) ( )( )( )

( ) ( )( ).,,,maxarg1,1,

gttQtt rσασασασα

←++

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100( )tσ

( )tα

gr ( )gmf rrr,ˆ,ˆˆ σα=


Belief Propagation

Input

Output

BP EM

Update Rule of BP

２１

３

４

５



( ) ( )( )( )

( ) ( )( ).,,,maxarg1,1,


←++

gr

( )gmf rrr,ˆ,ˆˆ σα=

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

Loopy Belief Propagation

Exact

0006000ˆ335.36ˆ.=

=

LBP

LBP

ασ

0007130ˆ624.37ˆ.=

=

Exact

Exact

ασ


ContentsContents



Statistical Performance by Sample Average

( ) ∑∑= =

→ −Ω

≡3

1

2

1

2161,k l

kkl fmMrrσα

( )*αfPr

11→gr

Prior Probability

1fr

2fr

3fr

21→gr

12→gr

22→gr

13→gr

23→gr

Degradation Process

( )*2,σfgPrr

( )*1,σfgPrr

( )*3,σfgPrr

11→mr

12→mr

21→mr

22→mr

31→mr

32→mr

( )σ,gfP ,11 α→rr








Statistical Performance Analysis

( ) ( ) { }( ) { } { }∫ ∫

∫

===−Ω

=

=−Ω

≡

fdgdfFfFgGfgm

gdgGfgmM

rrrrrrrrrrr

rrrrrr

Pr,Pr,,1

,Pr,,1,

**2

**2

ασσα

σασασα

( )σα ,,gm rr

g

( )*αfPr gr

Prior Probability Degradation

Process

( )*,σfgPrrf

r

( )**,σ,gfP αrr



Statistical Performance Analysis

( ) ( ) { } { }

( ) ⎟⎠⎞

⎜⎝⎛ +⎟

⎟

⎠

⎞

⎜⎜

⎝

⎛

+−

+Ω+

+Ω=

===−Ω

=

−

∫

CICCI

I

CI

I

CII

2**1*2

22**

2

2*

**2

Tr1

Tr1

Pr,Pr,,1,

σααασσα

ασ

σ

ασσασα fdgdfFfFgGfgmMrrrrrrrrrrr

( ) ( ) ( )∏∈

⎟⎠⎞

⎜⎝⎛ −−=

Nijji ff

ZfP 2

2exp1 α

αα

PR

r

( ) ( )∏Ω∈

⎟⎠⎞

⎜⎝⎛ −−=

iii gffgP 2

221exp

21,

σσπσ

rrMulti-DimentionalGaussian Integral Formula

( )( )

( )⎪⎩

⎪⎨

⎧∈−=

=otherwise0

14

Bijji

ji C

( )

g

gmr

rr

CII

2

,,

ασ

σα

+=

Nishimori (2000)



( ) ( )( ) ( ) ( )( )( ) ( ) ( )( ) ( ) ( ) ( )( )

( ) ( )( ) ( ) ( )( )( ) ( ) gdgPgttmgttmB

ttVttVtVtV

Bijji

rrrr ∑∫∈

++−+++

++−++=

+−+

**2

10

10

,,1,1,1,121,141,14

1414

σασασα

σασα

αα

( ) ( ) ( ) ( )( )

( ) ( )( )( ) ( ) gdgPggttm

ttVt

Bijii

rrr ∑∫∈

−++Ω

+

=+

**2

02

,,1,11,1

σασα

σασ

( ) ( )( )( )

( ) ( )( ) ( )∫←++ gdgPgttQtt rrr **

,,,,,maxarg1,1 σασασασα

σα


Statistical Behaviour of EM (Expectation Maximization) Algorithm

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

=

Ωg

gg

gM

r 2

1

( ) ( )( ) ( ) ( ) ( ) 22*

2**

2

21

)()(Tr1Tr11

CIC

CIC

ttCI

tttt

σαα

σα

σα

σα+

+Ω

−+Ω

=+ −

( ) ( )( ) ( )

( ) ( )( ) ( ) 22*

2**42

2

22

)()(Tr1

)(Tr11

CICIC

CII

tttt

tttt

σαα

σασα

σα

σσ+

+Ω

−+Ω

=+

( ) ( )( )

( )( ) ( )( ) ( )∫←

++

gdgPgttQ

ttrrr **

,,,,,maxarg

1,1

σασασα

σα

σα

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100( )tσ

( )tα ( )tα

( )tσ

( )( ) 1000

0001.0040

001.0*

*

==

=

=

σασ

α

Numerical Experiments for Standard Image

( )( ) 1000

0001.0040*

==

=

σασ

Statistical Behaviour of EM Algorithm



( )( ) ( ) ( )( ) ( ) ( ) ( )( )

( ) ( )( ) ( ) ( )( )( ) ( ) gdgPgttmgttmB

ttVttVt

Bijji

rrrr ∑∫∈

++−+++

++−++=

**2

10

,,1,1,1,12

1,141,141

σασασα

σασαα

( ) ( ) ( ) ( )( )

( ) ( )( )( ) ( ) gdgPggttm

ttVt

Bijii

rrr ∑∫∈

−++Ω

+

=+

**2

02

,,1,11,1

σασα

σασ

( ) ( )( )( )

( ) ( )( ) ( )∫←++ gdgPgttQtt rrr **


σα

( ) ggm rrr

CII

2,,ασ

σα+

=

( )( )( )( ) ( )( ) ⎟⎟

⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛−⎟⎟

⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛++=⎟⎟

⎠

⎞⎜⎜⎝

⎛−

01

0110

1001

41

,43

,,

1

201

0

αασσασα

σαVV

V



( ) ( )( ) ( ) ( )( ) ( ) ( ) ( )( ) ( ) ( ) ( )( )

( ) ( )( ) ( ) ( )( )( ) ( ) gdgPgttmgttmB

ttVttVtVtV

Bijji

rrrr ∑∫∈

++−+++

++−++=+−+

**2

1010

,,1,1,1,121,141,141414

σασασα

σασααα

( ) ( ) ( ) ( )( )

( ) ( )( )( ) ( ) gdgPggttm

ttVt

Bijii

rrr ∑∫∈

−++Ω

+

=+

**2

02

,,1,11,1

σασα

σασ ( ) ggm rrr

CII

2,,ασ

σα+

=

( )( )( )( )

( )( )( )( )

( )( ) ( )( )( )

( )( )( )( ) ( )( )( )

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎠

⎞⎜⎜⎝

⎛

−−−

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎠

⎞⎜⎜⎝

⎛

−+−+

⎟⎟⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛

−

0001

0100101001010010

,,2,

21

1000010000100001

,,2,

,41

41

00100001

,,

1

2020

1

2020

0

02

1

0

σασασαα

σασασα

σαα

σ

σασα

VVV

VVV

V

VV


0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100


gr

( ) ( )( )( )

( ) ( )( ) ( )∫←++ gdgPgttQtt rrr **


σα

( )tα

( )tσ

( )( ) 1000

0001.0040

0007.0*

*

==

=

=

σασ

α

( )( ) 1000

0001.0040*

==

=

σασ

0006980ˆ983.39ˆ

0005640ˆ542.38ˆ

.

.

====

GBP

GBP

LBP

LBP

ασασ

Statistical Behaviourof EM Algorithm

Numerical Experiments for Standard Image

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

( )tσ

( ) ( )( )( )

( ) ( )( ).,,,maxarg1,1,


←++

( )tα


ContentsContents



SummarySummary

Formulation of belief propagation for Formulation of belief propagation for graphical models has been summarized.graphical models has been summarized.Probabilistic image processing by using Probabilistic image processing by using Gaussian graphical model has been shown Gaussian graphical model has been shown as the most basic example.as the most basic example.It has been explained how to construct a It has been explained how to construct a belief propagation algorithm for image belief propagation algorithm for image processing.processing.


SMAPIP ProjectSMAPIP ProjectSMAPIP Project

MEXT Grant-in Aid for Scientific Research on Priority AreasMEXT Grant-in Aid for Scientific Research on Priority Areas

Period: 2002 –2005Head Investigator: Kazuyuki TanakaPeriod: 2002 –2005Head Investigator: Kazuyuki Tanaka

Member:K. Tanaka, Y. Kabashima,H. Nishimori, T. Tanaka, M. Okada, O. Watanabe, N. Murata, ......


DEX-SMI ProjectDEXDEX--SMI ProjectSMI Project

http://dex-smi.sp.dis.titech.ac.jp/DEX-SMI/http://dexhttp://dex--smi.sp.dis.titech.ac.jp/DEXsmi.sp.dis.titech.ac.jp/DEX--SMISMI//

情報統計力学 GOGO

DEX-SMI GOGO

MEXT Grant-in Aid for Scientific Research on Priority Areas

Period: 2006 –2009Head Investigator: Yoshiyuki KabashimaPeriod: 2006 –2009Head Investigator: Yoshiyuki Kabashima

Deepening and Expansion of Statistical Mechanical Informatics


ReferencesReferencesK. Tanaka and D. M. K. Tanaka and D. M. TitteringtonTitterington: Statistical Trajectory of : Statistical Trajectory of

Approximate EM Algorithm for Probabilistic Image Processing, Approximate EM Algorithm for Probabilistic Image Processing, J. Phys. A, J. Phys. A, 4040 (2007).(2007).

K. Tanaka: Generalized Belief Propagation Formula in K. Tanaka: Generalized Belief Propagation Formula in Probabilistic Information Processing based on Gaussian Probabilistic Information Processing based on Gaussian Graphical Model, IEICE Transactions on Information and Graphical Model, IEICE Transactions on Information and Systems, Systems, J88J88--DD--IIII (2005)(2005)

K. Tanaka, H. K. Tanaka, H. ShounoShouno, M. Okada and D. M. , M. Okada and D. M. TitteringtonTitterington: : Accuracy of the Accuracy of the BetheBethe Approximation for Approximation for HyperparameterHyperparameterEstimation in Probabilistic Image Processing, J. Phys. A, Estimation in Probabilistic Image Processing, J. Phys. A, 3737(2004).(2004).

K. Tanaka: Probabilistic Inference by Means of Cluster VariationK. Tanaka: Probabilistic Inference by Means of Cluster VariationMethod and Linear Response Theory, IEICE Transactions on Method and Linear Response Theory, IEICE Transactions on Information and Systems, Information and Systems, E86E86--DD (2003). (2003).

K. Tanaka: StatisticalK. Tanaka: Statistical--Mechanical Approach to Image Processing Mechanical Approach to Image Processing (Topical Review), J. Phys. A, (Topical Review), J. Phys. A, 3535 (2002).(2002).


本講演の参考図書本講演の参考図書

田中和之著田中和之著: : 確率モデルによる画像処理技術入門確率モデルによる画像処理技術入門, , 森森北出版北出版, 2006., 2006.田中和之編著田中和之編著: : 臨時別冊・数理科学臨時別冊・数理科学SGCSGCライブラリ「確ライブラリ「確率的情報処理と統計力学率的情報処理と統計力学 ------様々なアプローチとその様々なアプローチとそのチュートリアル」チュートリアル」, , サイエンス社，サイエンス社，2006.2006.

グラフィカルモデルにおける確率伝搬法の数理 …kazu/tutorial...22 december, 2007...

Documents