probabilistic image processing and bayesian networkkazu/tanaka-titechlecture2005/... · kazu/...

17-18 October, 2005 Tokyo Institute of Technology 1

Probabilistic image processing and Bayesian networkKazuyuki Tanaka

Graduate School of Information Sciences,Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/

ReferencesReferencesK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002).K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004).


Bayesian Network and Belief Propagation

Probabilistic Information Processing

Probabilistic Model

Bayes Formula

Belief Propagation

J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44(1996).

Bayesian Network

17-18 October, 2005 3Tokyo Institute of Technology

Formulation Formulation of Belief Propagationof Belief Propagation

Link between Link between belief propagation belief propagation andand statistical statistical mechanics.mechanics.Y. Kabashima and D. Saad, Belief propagation vs. TAP for decodinY. Kabashima and D. Saad, Belief propagation vs. TAP for decoding g corrupted messages, corrupted messages, Europhys. Lett.Europhys. Lett. 4444 (1998). (1998). M. Opper and D. Saad (eds), M. Opper and D. Saad (eds), Advanced Mean Field Methods Advanced Mean Field Methods ------Theory andTheory andPracticePractice (MIT Press, 2001).(MIT Press, 2001).

Generalized belief propagationGeneralized belief propagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing freeJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free--energy energy approximations and generalized belief propagation algorithms, IEapproximations and generalized belief propagation algorithms, IEEE EE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005).(2005).

Information geometrical interpretation Information geometrical interpretation of belief propagationof belief propagationS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free eneS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and rgy, and information geometry, Neural Computation, information geometry, Neural Computation, 1616 (2004).(2004).


Extension of Belief PropagationExtension of Belief Propagation

Generalized Belief PropagationGeneralized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing freeJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free--energy energy approximations and generalized belief propagation algorithms, IEapproximations and generalized belief propagation algorithms, IEEE EE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005).(2005).

Generalized belief propagation is equivalent Generalized belief propagation is equivalent to the cluster variation method in statistical to the cluster variation method in statistical mechanicsmechanicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 8181 (1951).(1951).T. Morita: Cluster variation method of cooperative phenomena andT. Morita: Cluster variation method of cooperative phenomena and its its generalization I, J. Phys. Soc. Jpn, generalization I, J. Phys. Soc. Jpn, 1212 (1957).(1957).


Application of Belief Application of Belief PropagationPropagation

Image ProcessingImage ProcessingK. Tanaka: StatisticalK. Tanaka: Statistical--mechanical approach to image processing (Topical Review), J. mechanical approach to image processing (Topical Review), J. Phys. A, Phys. A, 3535 (2002).(2002).A. S. Willsky: Multiresolution Markov Models for Signal and ImagA. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, e Processing, Proceedings of IEEE, Proceedings of IEEE, 9090 (2002).(2002).

Low Density Parity Check CodesLow Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics ofY. Kabashima and D. Saad: Statistical mechanics of lowlow--density paritydensity parity--check codes check codes (Topical Review), J. Phys. A, (Topical Review), J. Phys. A, 3737 (2004). (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and lowand low--density density parityparity--check codes, IEEE Transactions on Information Theory, check codes, IEEE Transactions on Information Theory, 5050 (2004).(2004).

CDMA Multiuser Detection AlgorithmCDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief of belief propagation, J. Phys. A, 36 (2003).propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate Belief propagation, density T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Trastatistical neurodynamics for CDMA multiuser detection, IEEE Transactions on nsactions on Information Theory, Information Theory, 5151 (2005).(2005).

SSatisfability atisfability PProblemroblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics meO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase thods and phase transitions in optimization problems, Theoretical Computer Scientransitions in optimization problems, Theoretical Computer Science, ce, 265265 (2001).(2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic soluM. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random tion of random satisfability problems, Science, satisfability problems, Science, 297297 (2002).(2002).


ContentsContents1. Introduction2. Belief Propagation3. Belief Propagation and Cluster Variation Method4. Belief Propagation for Gaussian Graphical Model5. Cluster Variation Method for Gaussian Graphical Model6. Bayesian Image Analysis and Gaussian Graphical Model7. Image Segmentation8. Concluding Remarks


How should we treat the calculation of the summation over 2N configuration?

( )∑ ∑ ∑= = =1,0 1,0 1,0

211 2

,,,x x x

NN

xxxW LL

It is very hard to calculate exactly except some special cases.It is very hard to calculate exactly except some special cases.

Formulation for approximate algorithmAccuracy of the approximate algorithm

Belief Propagation


Tractable Model

Factorizable

Not Factorizable

Probabilistic models with no loop are tractable.

Probabilistic models with loop are not tractable.

( )

⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∑∑∑∑

∑∑∑∑

dcbadcba

dcba

),(),(),(),(

),(),(),(,

xDxCxBxA

xDxCxBxAa b c d

a

b

cd

ab

cd

( )∑∑∑∑a b c d

dcba xW ,,,,


Probabilistic model on a graph with no loop

( )( ) ( ) ),(),(,),(,

,,,,,

12 yWyWyxWxWxWyxP

DCBA dcbadcba

≡

( ) ( )∑∑∑∑∑≡a b c d

dcbax

yxPyP ,,,,,2

ab

1

cd

2

3 4

56

Marginal probability of the node 2



( )( )∑≡

→

aa xW

xM

A ,13

1

a3

1

ab

12

3 4

( ) ( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( )∑

∑ ∑∑

∑∑∑

→→

→

=

⎟⎠

⎞⎜⎝

⎛⎟⎠

⎞⎜⎝

⎛=

≡

x

xBA

xBA

xMxMyxW

xWxWyxW

xWxWyxWyM

141312

12

1221

,

,,,

,,,

ba

a b

ba

ba

b41

( )( )∑≡

→

bb xW

xM

B ,14

ab

1

cd

2

3 4

56


Probabilistic model on a graph with no loop( ) ( ) ( ) ),(),(,),(,,,,,, 12 yWyWyxWxWxWyxP DCBA dcbadcba ≡

ab

1

cd

2

3 4

56

( ) ( )

( )

( ) ( )

( )

( )

( ) ( ) ( )yMyMyM

xWxWyxWyWyW

yWyWxWxWyxW

,x,y,,,PyP

yM

xBA

yM

D

yM

C

xDCBA

x

212625

12

12

2

212625

),(),(,),(),(

),(),(),(),(,

→→→=

⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜

⎝

⎛

⎟⎠

⎞⎜⎝

⎛⎟⎠

⎞⎜⎝

⎛

⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜

⎝

⎛

=

⎟⎠

⎞⎜⎝

⎛⎟⎠

⎞⎜⎝

⎛⎟⎠

⎞⎜⎝

⎛⎟⎠

⎞⎜⎝

⎛=

≡

→→→

∑ ∑∑∑∑

∑ ∑∑∑∑

∑∑∑∑∑

4444444 34444444 214342143421 badc

dcba

a b c d

badc

dcba

dcba

φ=dcba III



Message from the node 1 to the node 2 can be expressed in terms of the product of message from all the neighbouring nodes of the node 1 except one from the node 2.

( ) ( ) ( ) ( )yMyMyMyP 2126252 →→→=

Marginal probability can be expressed in terms of the product of messages from all the neighbouring nodes of node 2.

ab

1

cd

2

3 4

56

( ) ( ) ( ) ( )∑ →→→ =y

yMyMyxWxM 26251212 ,

( ) ( ) ( ) ( )∑ →→→ =x

xMxMyxWyM 14131221 ,


Belief Propagation on a graph with no loop

{ } ( )∏−

=++==

1

111, ,1Pr

N

iiiii xxW

ZxX

( ) ( )

( ) ( ) ( ) ( )∑

∑∑ ∑ ∏

++→−→−→−

=++++→

=

⎟⎟⎠

⎞⎜⎜⎝

⎛=

k

k

xkkkkkkkkkkkkk

x x x

k

iiiiikkk

xxWxMxMxM

xxWxM

11,321

111,11

,

,1 2

L

1X2X 3X

1−kX

kX

2−kX

3−kX

1+kX


Probabilistic Model on a Graph with Loops

( ) ( )∏∈

=Nij

jiijL xxWZ

xxxP ,1,,, 21 L

( ) ( ){ }∑=

1\2111 ,,,

xLxxxPxP

xL

Marginal Probability

( )∑∏∈

≡x Nij

jiij xxWZ ,Ω

( ) ( ){ }∑=

21,\212112 ,,,,

xxLxxxPxxP

xL


Message Passing Rule of Belief Propagation

Fixed Point Equations for Massage

( )MMrrr

Φ=

( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )∑∑

∑

→→→

→→→

→ =

ξ ς

ς

ςςςξς

ςςςξςξ

15141312

15141312

21 ,

,

MMMW

MMMWM

1

33

44 2

5

13→M

14→M

15→M

21→M


Approximate Representation of Marginal Probability

( ) ( )

( ) ( ) ( ) ( )( ) ( ) ( ) ( )∑

∑

→→→→

→→→→=

≡

1

1

115114113112

115114113112

\11

x

x

xMxMxMxMxMxMxMxM

PxPx

x144 2

5

13→M

14→M

15→M

12→M

33

Fixed Point Equations for Messages( )MM

rrrΦ=


Fixed Point Equation and Iterative Method

Fixed Point Equation ( )** MMrrr

Φ=Iterative Method

( )( )( )

M

rr

rr

rr

23

12

01

MM

MM

MM

Φ←

Φ←

Φ←

0M1M

1M

0

xy =

)(xy Φ=

y

x*M


Kullback-Leibler divergence and Free Energy

[ ] ( )( ) 0ln)( ≥⎟⎟

⎠

⎞⎜⎜⎝

⎛≡∑ x

xxx P

QQPQD ( ) ⎟⎠

⎞⎜⎝

⎛=≥ ∑

xxx 1)( ,0 QQQ

( ) ( )

ZQF

ZQQxxWQPQD

QF

Nijjiij

ln][

lnln)(,ln)(]|[

][

+=

++= ∑∑∑∈ 4444444 34444444 21

xxxxx

( ) ( ) [ ] 0=⇒= PQDPQ xx

( ) ZPFQQFQ

ln][1][min −==⎭⎬⎫

⎩⎨⎧

=∑x

x

( )∑∏∈

≡x Nij

jiij xxWZ ,

( ) ( )∏∈

=Nij

jiijL xxWZ

xxxP ,1,,, 21 L

Free Energy


Free Energy and Cluster Variation Method

[ ] [ ] ( )ZQFPQD ln+=

[ ] ( ) ( )

{ }( ) ( )

( ) ( ) ( )xx

xxx

xxx

x

xx

xx

QQxxWxxQ

QQxxWQ

QQxxWQQF

Nij x xjiijjiij

Nij x xjiij

xx

Nijjiij

i j

i j ji

ln)(,ln,

ln)(,ln)(

ln)(,ln)(

,\

∑∑∑∑

∑∑∑∑ ∑

∑∑∑

+=

+⎟⎟⎠

⎞⎜⎜⎝

⎛=

+≡

∈

∈

∈

[ ] ( )( ) 0ln)( ≥⎟⎟

⎠

⎞⎜⎜⎝

⎛≡∑ x

xxx P

QQPQD

Free EnergyKL Divergence

( ) ( )∏∈

=Nij

jiij xxWZ

P ,1x

{ }∑≡

ji xx

jiij

Q

xxQ

,)(

),(

\xx


Free Energy and Cluster Variation Method

[ ] [ ] ( )ZQFPQD ln+=

[ ] ( ) ( )

( )

( ) ( )

( ) ( )

( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑

∑∑

∑∑∑

∑

∑∑∑

∈

Ω∈

∈

∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−+

+

≈

+

=

Nijjjiiijij

iii

Nijijij

Nijijij

QQQQQQ

QQ

WQ

QQ

WQQF

ξξξ ζ

ξ

ξ ζ

ξ ζ

ξξξξζξζξ

ξξ

ζξζξ

ζξζξ

lnln,ln,

ln

,ln,

ln)(

,ln,

xxx

Bethe Free Energy

Free EnergyKL Divergence( ) ( )∏

∈

=Nij

jiij xxWZ

P ,1x

{ }∑≡

ji xxjiij QxxQ

,

)(),(\x

x

∑≡ix

ii QxQ\x

x)()(


Basic Framework of Cluster Variation Method

[ ] FPQDQQ γγ

minargminarg ≅

( ) ( )∑=ς

ςξξ ,iji QQ

[ ] { }[ ] ZQQFPQD iji ln,Bethe +≅

[ ]{ }

{ }[ ]ijiQQQ

QQFPQDiji

,minargminarg Bethe,

⇒

( ) ( ) 1, ==∑∑∑ξ ςξ

ςξξ iji QQ

{ }[ ] ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )∑ ∑∑∑∑

∑∑∑∑∑

∈

Ω∈∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−+

+≡

Nijjjiiijij

iii

Nijijijiji

QQQQQQ

QQWQQQF

ξξξ ς

ξξ ς

ξξξξςξςξ

ξξςξςξ

lnln,ln,

ln,ln,,Bethe



{ }[ ] { }[ ]

( ) ( ) ( )

( ) ( )∑ ∑∑∑ ∑

∑∑∑ ∑

∈Ω∈

Ω∈ ∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−

≡

Nijijij

iii

i Njijiji

ijiiji

QQ

QQ

QQFQQL

i

1,1

,

,,

,

BetheBethe

ξ ζξ

ξ ς

ζξνξν

ζξξξλ

{ }{ }[ ] ( ) ( ) ( ) ( )

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

=== ∑∑∑∑ 1, ,,,minarg Bethe, ξ ςξς

ςξξςξξ ijiijiijiQQ

QQQQQQFiji

Lagrange Multipliers to ensure the constraints



{ }[ ] { }[ ] ( ) ( ) ( )

( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( )∑ ∑∑∑ ∑∑∑∑ ∑

∑ ∑∑∑∑

∑∑∑∑∑

∑ ∑∑∑ ∑

∑∑∑ ∑

∈Ω∈Ω∈ ∈

∈

Ω∈∈

∈Ω∈

Ω∈ ∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−+

+=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

⎟⎟⎠

⎞⎜⎜⎝

⎛−−≡

Nijijij

iii

i Njijiji

Nijjjiiijij

iii

Nijijij

Nijijij

iii

i Njijijiijiiji

QQQQ

QQQQQQ

QQWQ

QQ

QQQQFQQL

i

i

1,1,

lnln,ln,

ln,ln,

1,1

,,,

,

,BetheBethe

ξ ζξξ ζ

ξξξ ζ

ξξ ζ

ξ ζξ

ξ ς

ζξνξνζξξξλ

ξξξξζξζξ

ξξζξζξ

ζξνξν

ζξξξλ

( ) { }[ ] 0,Bethe =∂

∂iji

ii

QQLxQ

Extremum Condition

( ) { }[ ] 0,, Bethe =

∂∂

ijijiij

QQLxxQ


Approximate Marginal Probability in Cluster Variation Method

27→M144 2

5

13→M

14→M

15→M

12→M

33

26→M

144

5

13→M

14→M

15→M

12W33

2

6

27→M

88

77

28→M

( ) ( ) ( )

( ) ( )115114

1131121

111

xMxM

xMxMZ

xQ

→→

→→

×

= ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )2282272262112

11511411312

2112

,

1,

xMxMxMxxW

xMxMxMZ

xxQ

→→→

→→→

×

=

ExtremumCondition( ) { }[ ] 0,Bethe =

∂∂

ijiii

QQLxQ ( ) { }[ ] 0,

, Bethe =∂

∂iji

jiij

QQLxxQ


Cluster Variation Method and Belief Propagation

27→M144 2

5

13→M

14→M

15→M

12→M

33

26→M

144

5

13→M

14→M

15→M

12W33

2

6

27→M

88

77

28→M

( ) ( )∑=ς

ζξξ ,121 QQ

( ) ( ) ( )

( ) ( )115114

1131121

111

xMxM

xMxMZ

xQ

→→

→→

×

= ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )2282272262112

11511411312

2112

,

1,

xMxMxMxxW

xMxMxMZ

xxQ

→→→

→→→

×

=

( )( ) ( )

( ) ( )ςς

ςξς

ξ

ς

1514

1312

21

,

→→

→

→

×

∝∑MM

MWM

Message Update Rule


Gaussian Graphical Model

( )

formula. integral Gauss ldimensiona-multi theusingby

calculated becan quantities lstatisticaother the

and average The xxx d∫ ρ

{ }Ω∈= ixix

( ) ( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−= ∑∑

∈Ω∈ Nijji

iii xxgx

Z22

21

21exp1 αβρ x

( )+∞∞−∈ ,ix

( ){ }

( )

( )( )⎪

⎪⎩

⎪⎪⎨

⎧

∈

=+

=

∑∈

otherwise 0 /

/1

Nij

jiNijj

ij βα

βα

H

( ) gHxxx 1−=∫ dρmatrix : Ω×ΩH



( )( ) ( ) ( ) ( )

( ) ( ) ( ) ( )∑∑

∑

→→→

→→→

→ =

ξ ς

ς

ςςςξς

ςςςξςξ

15141312

15141312

21 ,

,

MMMW

MMMWM

1

33

44 2

5

13→M

14→M

15→M

21→M ( ) ⎟⎠⎞

⎜⎝⎛ −−≡ →→

→

→

2

21exp

2

)(

jijiji

jiM

μςλπ

λ

ς



1

33

44 2

5

( )1313 , →→ λμ( )1414 , →→ λμ

( )1515 , →→ λμ( )2121 , →→ μλ

( )151413

15141321

→→→

→→→→ ++++

+++=

λλλαβλλλβαλ

151413

151514141313121

→→→

→→→→→→→ +++

+++=

λλλβλμλμλμβμ g

Fixed-Point Equations

⎟⎟⎠

⎞⎜⎜⎝

⎛Φ=⎟⎟

⎠

⎞⎜⎜⎝

⎛λμ

λμ

r

rr

r

r

Natural Iteration



1

33

44 2

5

( )1313 , →→ λμ( )1414 , →→ λμ

( )1515 , →→ λμ( )2121 , →→ μλ

( )151413

15141321

→→→

→→→→ ++++

+++=

λλλαβλλλβαλ

151413

151514141313121

→→→

→→→→→→→ +++

+++=

λλλβλμλμλμβμ g

( ) ⎟⎠⎞

⎜⎝⎛ −−≡ →→

→

→

21

1

21exp

2

)(

jijiji

ji

x

xM

μλπ

λ

( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( )∑ →→→→

→→→→=

1

115114113112

11511411311211

xxMxMxMxM

xMxMxMxMxP


Kullback-Leibler Divergence of Gaussian Graphical Model

[ ] ( ) ( )( ) [ ] ZQFdQQQD lnln +=⎟⎟

⎠

⎞⎜⎜⎝

⎛≡ ∫

∞+

∞−z

zzz

ρρ

[ ] ( )( )

( )( ) ( ) ( )∫∑

∑

+−++−+

−−=

∈

Ω∈

zzz dQQmmVVV

gmVQF

jijijiijNij

iiiii

ln221

21

2

2

α

β

( ) zz dQzm ii ∫≡( ) ( ) zz dQmzV iii ∫ −≡ 2

( )( ) ( ) zz dQmzmzV jjiiij ∫ −−≡

Entropy Term

( ) ( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−= ∑∑

∈Ω∈ Nijji

iii xxgx

Z22

21

21exp1 αβρ x


Cluster Variation Method

( ) ( ) ( )∫ −≡ zz dQzxxQ iiii δ

( ) ( ) ( )( ) ( )⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Nij jii

jiij

iii xQxQ

xxQxQQ

,x

Trial Function

( ) ( ) ( ) ( )∫ −−≡ zz dQzxzxxxQ jjiijiij δδ,

Tractable Form


Cluster Variation Method for Gaussian Graphical Model

( ) ( ) ( )( ) ( )⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Nij jii

jiij

iii xQxQ

xxQxQQ

,x

( ) ( ) ( )

( ) ( )( ) ( )⎟

⎠⎞

⎜⎝⎛ −−−=

−≡

−

∫

γγγγγ

γγ

γγγγ

π

δ

mxAmxA

zzzxx

1T

21exp

det2

1

dQQ

Trial Function

Marginal Distribution of GGM is also GGM

( ) iiiV=γA ( ) ijij

V=γA( ) iim=γm



[ ] { }[ ]( )( ) ( )( )

( ) ( ) ( )( )∑∑

∑∑

∈Ω∈

∈Ω∈

⎟⎠⎞

⎜⎝⎛ ++⎟

⎠⎞

⎜⎝⎛ +−

−++−+−−=

=

Nijij

ii

Nijjijiji

iiii

ijii

Vi

mmVVVgmV

VVmFQF

Adet2ln2112ln

211

221

21

,,

2

22

ππμ

αβ

( )( ) ( )

( ) ( )⎟⎠⎞

⎜⎝⎛ −−−= −

γγγγγ

γγγγ

πmxAmx

Ax 1T

21exp

det2

1Q

( ) ( ) ( )( ) ( )⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Nij jii

jiij

iii xQxQ

xxQxQQ

,x

Bethe Free Energy in GGMBethe Free Energy in GGM



( ) ( ){ }

1

141−

∈

−

⎟⎟⎠

⎞⎜⎜⎝

⎛+−−= ∑

Nijjiiiji i

V Aαβμ

( )jiij VVV 241121 αα

++−=

{ }[ ] ( ) ( ){ }

00,,

=−−−⇒=∂

∂∑∈Nijj

jiiii

ijii mmgmm

VVmFαβ

{ }[ ] ( ){ }

0340,, 11 =−++⇒=

∂

∂∑∈

−−

Nijjiiiji

i

ijii AV

VVmFAαβ

{ }[ ] ( ) 00,, 1 =−−⇒=

∂

∂ −

ijijij

ijii

VVVmF

Aα

⎟⎟⎠

⎞⎜⎜⎝

⎛≡

jij

ijiij VV

VVA


Iteration Procedure

Fixed Point Equation ( )*VΨV =*

Iteration

( )( )( )

M

)2()3(

)1()2(

)0()1(

VΨVVΨVVΨV

←

←

←

)0(M)1(V

)1(V

0

xy =

)(xy Ψ=

y

x*V


Cluster Variation Method and TAP Free Energy

Loopy Belief Propagation ( )( )

( )0

41121

5223

2

+→

+−=

++−=

α

ααα

αα

OVVVV

VVV

jiji

jiij

{ }[ ]( )( ) ( )( )

( ) ( )∑∑

∑∑

∈Ω∈

∈Ω∈

+−⎟⎠⎞

⎜⎝⎛ +−

−+++−−=

Nijji

ii

Nijjiji

iiii

ijii

OVVV

mmVVgmV

VVmF

42

22

212ln

211

21

21

,,

ααπ

αβ

( ) iiiV=γA ( ) ijij

V=γA

TAP Free TAP Free EnergyEnergy

( ) 01 =−− −

ijijAα

Mean Field Free EnergyMean Field Free Energy


Bayesian Image Analysis

Original Image Degraded Image

Transmission

Noise

{ } { } { }{ }

444 3444 21

44 844 7644444 844444 76

44444 344444 21

Likelihood Marginal

yProbabilit PrioriA Processn Degradatio

yProbabilit PosterioriA Image Degraded

Image OriginalImage OriginalImage DegradedImage DegradedImage Original

PrPrPr

Pr =


Bayesian Image AnalysisDegradation Process ( )+∞∞−∈ ,, ii gf

( ) ( )∏Ω∈

⎟⎠⎞

⎜⎝⎛ −−=

iii gfP 2

221exp

21,

σσπσfg

( )2,0~ σNn

nfg

i

iii += Ω

Original Image Degraded Image

Transmission

Additive White Gaussian Noise



A Priori Probability ( )+∞∞−∈ ,, ji gf

( ) ( ) ( )∏∈

⎟⎠⎞

⎜⎝⎛ −−=

Nijji ff

ZP 2

PR 21exp1 α

ααf

Standard Images

Generate

Similar?



( )+∞∞−∈ ,, ji gf

( ) ( ) ( )( )

( )

( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−×

=

=

∑∑∈Ω∈ Nij

jii

ii ffgf

Z

PPP

P

222

POS

21

21exp

,,1

,,

,,

ασ

σα

σαασ

σα

g

gffg

gf

A Posteriori Probability



Bayesian Image Analysis( )+∞∞−∈ ,, ji gf

( ) ( )

( ) ( )

( ) ( )∏

∑∑

∈

∈Ω∈

=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−×

=

Nijjiij

Nijji

iii

ffWZ

ffgf

ZP

,,,

1

21

21exp

,,1,,

POS

222

POS

σα

ασ

σασα

g

ggf

( ) ( ) ( ) ( ) ⎟⎠⎞

⎜⎝⎛ −−−−−−≡ 22

22

2 21

81

81exp, jijjiijiij ffgfgfffW α

σσ





Ωy

x

{ }Ω∈= ifif { }Ω∈= igig

fg

( )αfP ( )σ,fgP gOriginal Image Degraded Image

( ) ( ) ( )( )σα

ασσα

,,

,,g

ffggf

PPP

P =

( ) ( ) iiiii dffPfdPff ∫∫+∞

∞−== σασα ,,,,ˆ gfgf

A Priori Probability


Degraded Image

Pixels


Hyperparameter Determination by Maximization of Marginal Likelihood

( ) ( ) ( ) ( ) fffgfgfg dPPdPP ∫∫ == ασσασα ,,,,

( )( )

( )σασασα

,max argˆ,ˆ,

gP=

MarginalizationMarginalization

g( )σα ,gP{ }Ω∈= ifif

Original Image

Marginal Likelihood{ }Ω∈= igig

Degraded Image

Ωy

x

( )∫= fgf dPff ii σα ˆ,ˆ,ˆ

f g( )αfP ( )σ,fgP g( ) ( ) ( )ασσα ffggf PPP ,,, =


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

( ) ( ) ( ) fffgg dPPP ∫= ασσα ,,Marginal Likelihood

{ }Ω∈= igigIncomplete Data

Ωy

x

( ) ( ) ( ) fgfgfg dPPQ ∫= ',',ln,,,,',' σασασασα

( ) ( ) 0,,',''

,0,,','' ','','

=⎥⎦⎤

⎢⎣⎡∂∂

=⎥⎦⎤

⎢⎣⎡∂∂

==== σσαασσαα

σασασ

σασαα

gg QQ

( ) ( ) 0, ,0, =∂∂

=∂∂ σα

ασα

σgg PP

Equivalent

Q-Function


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

( ) ( ) ( ) fffgg dPPP ∫= ασσα ,,Marginal Likelihood Ω

y

x

( ) ( ) ( ) fgfgfg dPPQ ∫= ',',ln,,,,',' σασασασαQ-Function

( ) ( )( ) ( ) ( )( ) ( )( ) ( )( )

( )( ) ( )( ).,','maxarg1,1 :Step-M

.',',ln,,,',' :Step-E

','ttQtt

dPttPttQ

σασασα

σασασασα

βα←++

← ∫ fgfgf

Iterate the following EM-steps until convergence:EM Algorithm

A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).


OneOne--Dimensional SignalDimensional Signal

EM Algorithm

i

i

i

0 127 255

0 127 255

0 127 255

100

0

200

100

0

200

100

0

200

if

ig

if

Original Signal

Degraded Signal

Estimated Signal

40=σ


Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model

Original ImageOriginal Image Degraded ImageDegraded Image

MSE: 1529MSE: 1529

MSE: 1512MSE: 1512

EM Algorithm with Belief Propagation


Exact Results of Gaussian Graphical Model

( ) ( ) ( ) ( )

∫

∑∑

⎟⎠⎞

⎜⎝⎛ −−−

⎟⎠⎞

⎜⎝⎛ −−−

=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−−−==

∈Ω∈

fCffgf

Cffgf

ggf

d

ffgfZ

PNij

jii

ii

T22

T22

222

POS

21

21exp

21

21exp

21

21exp

,,1,,

ασ

ασ

ασσα

σα

( ) ( )( ) ( ) ⎟

⎠⎞

⎜⎝⎛

+−

+=

Ωg

CICg

CICg 2

T2 2

1expdet2det,

ασα

ασπασαP

( ) gCIf12ˆ −

+= ασ

Multi-dimensional Gauss integral formula

Ωy

x

( )( )

( )σασασα

,max argˆ,ˆ,

gP=


Comparison of Belief Propagation with Comparison of Belief Propagation with Exact Results in Gaussian Graphical ModelExact Results in Gaussian Graphical Model

( )2ˆ||

1MSE ∑Ω∈

−Ω

=i

ii ff

--5.214445.2144437.91937.9190.0007590.000759315315ExactExact

--5.192015.1920136.30236.3020.0006110.000611327327Belief Belief PropagationPropagation

MSEMSE α σ ( )σα ˆ,ˆln gP

--5.175285.1752834.97534.9750.0006520.000652236236ExactExact



40=σ

40=σ ( )( )

( )σασασα

,max argˆ,ˆ,

gP=


Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model

Original ImageOriginal Image

MSE:315MSE:315MSE: 325MSE: 325

MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411

MSE: 1512MSE: 1512

Degraded ImageDegraded Image Belief PropagationBelief Propagation

LowpassLowpass FilterFilter Median FilterMedian Filter

Exact

Wiener Filter

( )2ˆ||

1MSE ∑Ω∈

−Ω

=i

ii ff


Original ImageOriginal Image

MSE236MSE236MSE: 260MSE: 260

MSE: 372MSE: 372 MSE: 244MSE: 244MSE: 224MSE: 224

MSE: 1529MSE: 1529

Degraded ImageDegraded Image Belief PropagationBelief Propagation

LowpassLowpass FilterFilter Median FilterMedian Filter

Exact

Wiener Filter

( )2ˆ||

1MSE ∑Ω∈

−Ω

=i

ii ff

Image Restoration by Gaussian Image Restoration by Gaussian Graphical ModelGraphical Model


Extension of Belief PropagationExtension of Belief Propagation

Generalized Belief PropagationGeneralized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing freeJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free--energy energy approximations and generalized belief propagation algorithms, IEapproximations and generalized belief propagation algorithms, IEEE EE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005).(2005).

Generalized belief propagation is equivalent Generalized belief propagation is equivalent to the cluster variation method in statistical to the cluster variation method in statistical mechanicsmechanicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 8181 (1951).(1951).T. Morita: Cluster variation method of cooperative phenomena andT. Morita: Cluster variation method of cooperative phenomena and its its generalization I, J. Phys. Soc. Jpn, generalization I, J. Phys. Soc. Jpn, 1212 (1957).(1957).


Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model

( )2ˆ||

1MSE ∑Ω∈

−Ω

=i

ii ff

--5.211725.2117237.90937.9090.0007580.000758315315Generalized Generalized

Belief Belief PropagationPropagation

--5.214445.2144437.91937.9190.0007590.000759315315ExactExact



--5.172565.1725634.97134.9710.0006520.000652236236Generalized Generalized


--5.175285.1752834.97534.9750.0006520.000652236236ExactExact



40=σ

40=σ

( )( )

( )σασασα

,max argˆ,ˆ,

gP=


Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

( )( )

2

,,, ˆ∑

Ω∈−

Ω=

yxyxyx ff

||1MSE

548548(5x5)(5x5)

445445(5x5)(5x5)315315

Generalized Generalized Belief Belief

PropagationPropagation864864(3x3)(3x3)Wiener Wiener

FilterFilter315315ExactExact

486486(3x3)(3x3)Median Median FilterFilter

413413(5x5)(5x5)327327Belief Belief

PropagationPropagation388388(3x3)(3x3)LowpassLowpass

FilterFilter

MSEMSEMSEMSE40=σ

GBPGBP

(3x3) (3x3) LowpassLowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener


Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

( )( )

2

,,, ˆ∑

Ω∈−

Ω=

yxyxyx ff

||1MSE

372372(5x5)(5x5)

244244(5x5)(5x5)236236

Generalized Generalized Belief Belief

PropagationPropagation703703(3x3)(3x3)Wiener Wiener

FilterFilter236236ExactExact

331331(3x3)(3x3)Median Median FilterFilter

224224(5x5)(5x5)260260Belief Belief

PropagationPropagation241241(3x3)(3x3)LowpassLowpass

FilterFilter

MSEMSEMSEMSE40=σ

GBPGBP

(5x5) (5x5) LowpassLowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener


Image Segmentation by Image Segmentation by Gauss Mixture ModelGauss Mixture Model

( ) ( )∏Ω∈

=i

iaP γγa

( )( ) ( )

( )( )∏Ω∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−=

iii

ii

afaa

P 222

1exp2

1,, μσσπ

σμaf

( ) ( )( ) ( )( ) ( ) ( ) ( ) 204321

,1924 ,1923,1272 ,641

========

σσσσμμμμ

( )( ) ( )

( )( )

( )( )( )∏∑

∑

Ω∈ =⎟⎟⎠

⎞⎜⎜⎝

⎛ −−=

=

i

K

k

i

kkf

kk

PP

P

12

2

2exp

2

,,

,,

σμ

σπγ

aγaσμaf

γσμfGauss Mixture Model

( )( ) ( )

( )γσμfγaσμaf

γσμfa

,,,,

,,,

PPP

P

=


Image Segmentation by Combining Image Segmentation by Combining Gauss Mixture Model with Potts Model Gauss Mixture Model with Potts Model

( ) ( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎟⎠

⎞⎜⎜⎝

⎛= ∏∏

∈Ω∈ Bijaa

ii ji

aZ

P ,PR

exp1 δαγγ

γa

( )( ) ( )

( )( )∏Ω∈

⎟⎟⎠

⎞⎜⎜⎝

⎛−−=

iii

ii

afaa

P 222

1exp2

1,, μσσπ

σμaf

( ) ( )( ) ( )( ) ( ) ( ) ( ) 204321

,1924 ,1923,1272 ,641

========

σσσσμμμμ

( )( ) ( )

( )γσμfγaσμaf

γσμfa

,,,,

,,,

PPP

P

=

Belief PropagationBelief Propagation

{ }Ω∈= iaia

{ }Ω∈= ifif

{ }Ω∈= iaiˆaPotts Model


Image SegmentationImage Segmentation

Original Image Histogram Gauss Mixture Model

Gauss Mixture Model and

Potts Model


( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( ) 0101.05 ,4.145 ,8.2245

3982.04 ,7.114 ,4.16843375.03 ,6.233 ,6.13030711.02 ,0.182 ,2.422

1831.01 ,7.21 ,7.121

===============

γσμγσμγσμγσμγσμ


Motion DetectionMotion Detection

SegmentationAND

Detection

ba −

cb −

Gauss Mixture Model and Potts Model with Belief PropagationGauss Mixture Model and Potts Model with Belief Propagation

Segmentation

a

b

c


SummarySummaryFormulation of belief propagationFormulation of belief propagationAccuracy of belief propagation in Bayesian Accuracy of belief propagation in Bayesian image analysis by means of Gaussian image analysis by means of Gaussian graphical model (Comparison between the graphical model (Comparison between the belief propagation and exact calculation)belief propagation and exact calculation)Some applications of Bayesian image Some applications of Bayesian image analysis and belief propagationanalysis and belief propagation


Related ProblemRelated Problem

( ) ( ) ( ) fgffggff ddPP∫ − ασσα ,,,ˆ 2

Statistical Performance Spin Glass Theory

H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001.

f g( )αfP ( )σ,fgP g( ) ( ) ( )ασσα ffggf PPP ,,, =

( ) ( )∫= fgfg dPff ii σασα ,,,,ˆ

probabilistic image processing and bayesian networkkazu/tanaka-titechlecture2005/... · kazu/...

Documents