kazuyuki tanaka gsis, tohoku university, sendai, japan smapip.is.tohoku.ac.jp/~kazu

1

Bayesian Image Modeling by Generalized Sparse Markov Random Fields and

Loopy Belief Propagation

Kazuyuki TanakaGSIS, Tohoku University, Sendai, Japan

http://www.smapip.is.tohoku.ac.jp/~kazu/

CollaboratorsMuneki Yasuda (GSIS, Tohoku University, Japan)

Sun Kataoka (GSIS, Tohoku University, Japan)D. M. Titterington (Department of Statistics, University of Glasgow, UK)

20 March, 2013 SPDSA2013, Sendai, Japan

2

Outline

1. Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation

2. Bayesian Image Modeling by Generalized Sparse Prior

3. Noise Reductions by Generalized Sparse Prior4. Concluding Remarks


3

Probabilistic Model and Belief Propagation

Probabilistic Information Processing

Probabilistic Models

Bayes Formulas

Belief Propagation=Bethe Approximation

Bayesian NetworksMarkov Random Fields


Eji

jiij ffUP},{

,f

jkjkj

jfjiij

iji

fMffU

fM

,

Constant

V: Set of all the nodes (vertices) in graph GE: Set of all the links (edges) in graph G

ji

),( EVG

Message=Effective Field |},{ Ekjkj

4

Supervised Learning of Pairwise Markov Random Fields by Loopy Belief PropagationPrior Probability of natural images is assumed to be the following pairwise Markov random fields:

4

Eji

ji ffUP},{

}Pr{ ffF

1,,1,0 qfi


Supervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields:

5

Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation

5

1

0

1

00,,

},{

1

0

1

00

1

0,00,

0,

1

0

1

0

1

0

1

0,

},{

,

,

ln1

1 ,

21ln ,

q

f

q

fjiijjimmnnm

iji

q

f

q

fjiijjim

q

fiiimmm

inm

q

f

q

finim

q

m

q

njnimnmji

Ejiji

i j

i j

i

i i

ffPffKK

ffPff

fPfiKK

qfff

ffKffUffUP

f

Histogram from Supervised Data


M. Yasuda, S. Kataoka and K.Tanaka, J. Phys. Soc. Jpn, Vol.81, No.4, Article No.044801, 2012.

6

Supervised Learning of Pairwise Markov Random Fields by Loopy Belief PropagationSupervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields:

6

Ejiji

Ejiji ffffUP

},{

45.0

},{ 21exp~lnexp f


7

Bayesian Image Modeling by Generalized Sparse PriorAssumption: Prior Probability is given as the following Gibbs distribution with the interaction between every nearest neighbour pair of pixels:

7

Ejiji

pff

pZpPp

},{21exp

,1,},|Pr{

ffF

p=0: q-state Potts modelp=2: Discrete Gaussian Graphical Model

1,,1,0 qfi


8

Prior in Bayesian Image Modeling

8

Ejiji

pff

pZpPp

},{21exp

,1,},|Pr{

ffF

**),( upu

Eji

ji

pff

Eu

},{

*** 1

In the region of 0<p<0.3504…, the first order phase transition appears and the solution * does not exist.

Eji

ji pPffpup

,

,),(f

f



},|Pr{lnmaxarg **

pfF

q=16

9

Bayesian Image Modeling by Generalized Sparse Prior: Conditional Maximization of Entropy

)( )(ln)(maxarg

},|Pr{

},{)( z Eji

p

jiPEuPzzPP

up

zzz

fF

zf

9

Eji

pji ffup

uppZ

uppfPupfF

},{,

21exp

,,1

,,},|Pr{

Lagrange Multiplier



K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012.

10

Prior Analysis by LBP and Conditional Maximization of Entropy in Bayesian Image Modeling

10

Eji

p

ji ffupuppZ

uppPup},{

,21exp

,,1,,},|Pr{

ffF

** ,, uup

Ejijiji

p

ji

jiji

fji

ijkjkij

ApffPffEu

AAup

CpffP

EjiffAMj

p

},{},{

},{

\

,,1),(

messagesby LBPin ,, marginals Compute

)},{( 21exp

f

M

M

Prior Probability

q=16 p=0.2 q=16 p=0.5

Eji

pji ff

Eu

},{

*** 1

Repeat until A converges


LBP

11

Prior Analysis by LBP and Conditional Maximization of Entropy in Generalized Sparse Prior

11

Log-Likelihood for p when the original image f* is given

Eji

p

ji ffE

u},{

*** 1

*

},{

***

**

,,ln,21

},|Pr{ln

uppZffup

up

Eji

p

ji

fF

q=16

},|Pr{lnmaxarg *** uppp

fF

LBP

Free Energy of Prior



12

Degradation Process in Bayesian Image Modeling

Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.

Prior

Processn Degradatio

Posterior

}Image OriginalPr{

}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

0

12

Viii fg 2

2 )(2

1exp

},|Pr{

fFgG


13

Noise Reductions by Generalized Sparse Prior

z

z

zf z

zzz

gGfF

Viii

Eji

p

ji

P VPgz

EuPzzPP

up

22

},{

)( )()(

)( )(ln)(maxarg

},,,|Pr{

Posterior Probability

PriorProcessn Degradatio

Posterior

}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

13

Eji

p

jiVi

ii ffCgfBCBpZ

CBpPup

},{

2

21

21exp

,,,1

,,,},,,|Pr{

g

gfgGfF Lagrange Multipliers



14

Noise Reduction Procedures Based on LBP and Conditional Maximization of Entropy

14

Repeat until C converge

Eji f fjijiji

fji

ijkjkijjiji

i j

j

CpffPffEu

CC

ffCMCpffP

C

p

p

},{},{

\},{

,,1

21exp,,

converges untilRepeat

M

i

i j

j

fiiii

Eji f fjiijji

iijiij

fjijj

ijkjkij

CBpfPgfV

B

CBpffPffE

u

CBpfPCBpffP

ijVi

ffCgfBj

p

p

,,,1

,,,,1

messagesby ,,, and ,,,, marginals Compute

21

||21exp

2

2

},{

\

g

g

Lgg

LL

Repeat until u, B and L converge

CBpfPpf

CupuuB

iii ,,,maxarg,ˆˆ, ,ˆ ,ˆ

gg


}ˆ,ˆ,|Pr{maxargˆ

ˆ2ln21ˆ,,ln

ˆ,,ˆ,,ln

}ˆ,ˆ,|Pr{ln2

upp

VuppZ

uppZ

up

p

gG

g

gG

Marginals and Free Energy in LBP

Repeat until C and M convergeInput p and data g

15

Noise Reductions by Generalized Sparse Priors

and Loopy Belief Propagation

p=0.5p=0.2

Original Image Degraded Image

RestoredImage

p=120 March, 2013 SPDSA2013, Sendai, Japan


16

Noise Reductions by Generalized Sparse Priors

and Loopy Belief Propagation

p=0.5p=0.2

Original Image Degraded Image

RestoredImage

p=120 March, 2013 SPDSA2013, Sendai, Japan


17

SummaryFormulation of Bayesian image modeling for image processing by means of generalized sparse priors and loopy belief propagation are proposed.Our formulation is based on the conditional maximization of entropy with some constraints.In our sparse priors, although the first order phase transitions often appear, our algorithm works well also in such cases.


18

References1. S. Kataoka, M. Yasuda, K. Tanaka and D. M. Titterington: Statistical Analysis of the

Expectation-Maximization Algorithm with Loopy Belief Propagation in Bayesian Image Modeling, Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.50-63,2012.

2. M. Yasuda and K. Tanaka: TAP Equation for Nonnegative Boltzmann Machine: Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.192-209, 2012.

3. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Analysis of Gaussian Image Inpainting Problems, Journal of the Physical Society of Japan, Vol.81, No.2, Article No.025001, 2012.

4. M. Yasuda, S. Kataoka and K.Tanaka: Inverse Problem in Pairwise Markov Random Fields using Loopy Belief Propagation, Journal of the Physical Society of Japan, Vol.81, No.4, Article No.044801, pp.1-8, 2012.

5. M. Yasuda, Y. Kabashima and K. Tanaka: Replica Plefka Expansion of Ising systems, Journal of Statistical Mechanics: Theory and Experiment, Vol.2012, No.4, Article No.P04002, pp.1-16, 2012.

6. K. Tanaka, M. Yasuda and D. M. Titterington: Bayesian image modeling by means of generalized sparse prior and loopy belief propagation, Journal of the Physical Society of Japan, Vol.81, No.11, Article No.114802, 2012.

7. M. Yasuda and K. Tanaka: Susceptibility Propagation by Using Diagonal Consistency, Physical Review E, Vol.87, No.1, Article No.012134, 2013.


kazuyuki tanaka gsis, tohoku university, sendai, japan smapip.is.tohoku.ac.jp/~kazu

Documents