kazuyuki tanaka gsis, tohoku university, sendai, japan kazu

1

Statistical Mechanical Approach to Probabilistic Information Processing:

Cluster Variation Method and Belief Propagation

Kazuyuki TanakaGSIS, Tohoku University, Sendai, Japan

http://www.smapip.is.tohoku.ac.jp/~kazu/

CollaboratorsMuneki Yasuda (GSIS, Tohoku University, Japan)

Sun Kataoka (GSIS, Tohoku University, Japan)D. M. Titterington (Department of Statistics, University of Glasgow, UK)

13 March, 2013 WPI, Tohoku University, Sendai

2

Outline

1. Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation

2. Bayesian Image Modeling by Generalized Sparse Prior

3. Noise Reductions by Generalized Sparse Prior4. Concluding Remarks


3

Probabilistic Model and Belief Propagation

Probabilistic Information Processing

Probabilistic Models

Bayes Formulas

Belief Propagation=Bethe Approximation

Bayesian NetworksMarkov Random Fields


Eji

jiij ffUP},{

,f

jkjkj

jfjiij

iji

fMffU

fM

,

Constant

V: Set of all the nodes (vertices) in graph GE: Set of all the links (edges) in graph G

ji

),( EVG

Message=Effective Field |},{ Ekjkj

WPI, Tohoku University, Sendai 4

Mathematical Formulation of Belief Propagation

Similarity of Mathematical Structures between Mean Field Theory and Belief PropagationY. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and 　 Practice (MIT Press, 2001).

Generalization of Belief PropagationS. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energyapproximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).

Interpretations of Belief Propagation based on Information GeometryS. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).

13 March, 2013


Generalized Extensions of Belief Propagation based on Cluster Variation Method

Generalized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).

Cluster Variation MethodR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951).T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957).

13 March, 2013


Applications of Belief PropagationsImage ProcessingK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).

Low Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics of 　 low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).

CDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).

Satisfability ProblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 　 (2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).

13 March, 2013


Interpretation of Belief Propagation based on Information Theory

ff

fQffUQQF

Ejiji ln,ln)(

},{

ZQFPQQPQD ln][ln)(

f

fff

Bethe Free Energy

KL Divergence

Eji

jiij ffUZ

P},{

,1f

}{\

)()(if

ii QfQf

f

13 March, 2013

},{\

)(),(jfif

jiij QffQf

f

Eji jjii

jiij

Viii fQfQ

ffQfQQ

},{ )()(),(

)(f

Entropy

Marginal ProbabilityOne-Body Distribution Two-Body Distribution

Trial

8

Supervised Learning of Pairwise Markov Random Fields by Loopy Belief PropagationPrior Probability of natural images is assumed to be the following pairwise Markov random fields:

8

Eji

ji ffUP},{

}Pr{ ffF

1,,1,0 qfi


Supervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields:

9

Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation

9

1

0

1

00,,

},{

1

0

1

00

1

0,00,

0,

1

0

1

0

1

0

1

0,

},{

,

,

ln1

1 ,

21ln ,

q

if

q

jfjiijjimmnnm

iji

q

if

q

jfjiijjim

q

ifiiimmm

inm

q

if

q

ifinim

q

m

q

njnimnmji

Ejiji

ffQffKK

ffQff

fQfiKK

qfff

ffKffUffUP

f

Histogram from Supervised Data


M. Yasuda, S. Kataoka and K.Tanaka, J. Phys. Soc. Jpn, Vol.81, No.4, Article No.044801, 2012.

10

Supervised Learning of Pairwise Markov Random Fields by Loopy Belief PropagationSupervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields:

10

Ejiji

Ejiji ffffUP

},{

45.0

},{ 21exp~lnexp f


11

Bayesian Image Modeling by Generalized Sparse PriorAssumption: Prior Probability is given as the following Gibbs distribution with the interaction between every nearest neighbour pair of pixels:

11

Ejiji

pff

pZpPp

},{21exp

,1,},|Pr{

ffF

p=0: q-state Potts modelp=2: Discrete Gaussian Graphical Model

1,,1,0 qfi


12

Bayesian Image Modeling by Generalized Sparse Prior

q=16p=0.2


Eji

pji pfPff

E },{,1

f ,ln1 pZ

V

Ejiji

pff

pZpPp

},{21exp

,1,},|Pr{

ffF

Loopy Belief Propagation

13

Bayesian Image Modeling by Generalized Sparse Prior: Conditional Maximization of Entropy

)( )(ln)(maxarg

},|Pr{

},{)( z Eji

p

jiPEuPzzPP

up

zzz

fF

zf

13

Eji

pji ffup

uppZ

uppfPupfF

},{,

21exp

,,1

,,},|Pr{

Lagrange Multiplier

Loopy Belief Propagation


K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

14

Bayesian Image Modeling by Generalized Sparse Prior:Conditional Maximization of Entropy

Lena

Mandrill

Critical Point by Linear Response in LBPq=16

p=0.2



15

Prior Analysis by LBP and Conditional Maximization of Entropy in Generalized Sparse Prior

15

Eji

p

ji ffupuppZ

uppPup},{

,21exp

,,1,,},|Pr{

ffF

** ,, uup

Prior Probability

q=16 p=0.2 q=16 p=0.5

Eji

pji ff

Eu

},{

*** 1



16

Prior Analysis by LBP and Conditional Maximization of Entropy in Generalized Sparse Prior

16

Log-Likelihood for p when the original image f* is given

Eji

p

ji ffE

u},{

*** 1

*

},{

***

**

,,ln,21

},|Pr{ln

uppZffup

up

Eji

p

ji

fF

q=16

},|Pr{lnmaxarg *** uppp

fF

LBP

Free Energy of Prior



17

Noise Reductions by Generalized Sparse Prior

Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.

Prior

Processn Degradatio

Posterior

}Image OriginalPr{

}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

0

17

Viii fg 2

2 )(2

1exp

},|Pr{

fFgG


18

Noise Reductions by Generalized Sparse Prior

z

z

zf z

zzz

gGfF

Viii

Eji

p

ji

P VPgz

EuPzzPP

up

22

},{

)( )()(

)( )(ln)(maxarg

},,,|Pr{

Posterior Probability

PriorProcessn Degradatio

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

18

Eji

p

jiVi

ii ffCgfBCBpZ

CBpPup

},{

2

21

21exp

,,,1

,,,},,,|Pr{

g

gfgGfF Lagrange Multipliers



19

Noise Reductions by Generalized Sparse Priors

and Loopy Belief Propagation

p=0.5p=0.2

Original Image Degraded Image

RestoredImage

p=113 March, 2013 WPI, Tohoku University, Sendai


20

Noise Reductions by Generalized Sparse Priors

and Loopy Belief Propagation

p=0.5p=0.2

Original Image Degraded Image

RestoredImage

p=113 March, 2013 WPI, Tohoku University, Sendai


21

SummaryFormulation of Bayesian image modeling for image processing by means of generalized sparse priors and loopy belief propagation are proposed.Our formulation is based on the conditional maximization of entropy with some constraints.In our sparse priors, although the first order phase transitions often appear, our algorithm works well also in such cases.


22

References1. S. Kataoka, M. Yasuda, K. Tanaka and D. M. Titterington: Statistical Analysis of the

Expectation-Maximization Algorithm with Loopy Belief Propagation in Bayesian Image Modeling, Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.50-63,2012.

2. M. Yasuda and K. Tanaka: TAP Equation for Nonnegative Boltzmann Machine: Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.192-209, 2012.

3. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Analysis of Gaussian Image Inpainting Problems, Journal of the Physical Society of Japan, Vol.81, No.2, Article No.025001, 2012.

4. M. Yasuda, S. Kataoka and K.Tanaka: Inverse Problem in Pairwise Markov Random Fields using Loopy Belief Propagation, Journal of the Physical Society of Japan, Vol.81, No.4, Article No.044801, pp.1-8, 2012.

5. M. Yasuda, Y. Kabashima and K. Tanaka: Replica Plefka Expansion of Ising systems, Journal of Statistical Mechanics: Theory and Experiment, Vol.2012, No.4, Article No.P04002, pp.1-16, 2012.

6. K. Tanaka, M. Yasuda and D. M. Titterington: Bayesian image modeling by means of generalized sparse prior and loopy belief propagation, Journal of the Physical Society of Japan, Vol.81, No.11, Article No.114802, 2012.

7. M. Yasuda and K. Tanaka: Susceptibility Propagation by Using Diagonal Consistency, Physical Review E, Vol.87, No.1, Article No.012134, 2013.


kazuyuki tanaka gsis, tohoku university, sendai, japan kazu

Documents

tohoku university

basis of belief propagation

approximate belief propagation

belief propagationy

information theory

university of glasgow

information geometrys

national tsing hua university