em algorithm with markov chain monte carlo method for bayesian image analysis

10 October, 200710 October, 2007 University of GlasgowUniversity of Glasgow 11

EM Algorithm withEM Algorithm withMarkov Chain Monte Carlo Method forMarkov Chain Monte Carlo Method for

Bayesian Image AnalysisBayesian Image Analysis

EM Algorithm withEM Algorithm withMarkov Chain Monte Carlo Method forMarkov Chain Monte Carlo Method for

Bayesian Image AnalysisBayesian Image Analysis

Kazuyuki TanakaKazuyuki TanakaGraduate School of Information Sciences,Graduate School of Information Sciences,

Tohoku UniversityTohoku Universityhttp://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/

Collaborators: D. M. Titterington (Department of Statistics, University of Glasgow)


ContentContentss

1.1. IntroductionIntroduction

2.2. Gaussian Graphical Model Gaussian Graphical Model and EM Algorithmand EM Algorithm

3.3. Markov Chain Monte Carlo Markov Chain Monte Carlo MethodMethod

4.4. Concluding RemarksConcluding Remarks


ContentContentss






MRF and Statistical InferenceMRF and Statistical Inference

Geman and Geman (1986): IEEE Transactions on PAMIGeman and Geman (1986): IEEE Transactions on PAMIImage Processing for Image Processing for Markov Random Fields (MRF)Markov Random Fields (MRF) (Simulated Annealing, Line Fields) (Simulated Annealing, Line Fields)

How can we estimate hyperparameters in the degradation process and in the prior model only from observed data?

•EM Algorithm

In the EM algorithm, we have to calculate some statistical quantities in the posterior and the prior models. •Belief Propagation

•Markov Chain Monte Carlo Method


Statistical Analysis in EM AlgorithmStatistical Analysis in EM Algorithm

J. Inoue and K. Tanaka: Phys. Rev. E 2002, J. Phys. A 2003J. Inoue and K. Tanaka: Phys. Rev. E 2002, J. Phys. A 2003Statistical Behaviour of EM Algorithm for MRFStatistical Behaviour of EM Algorithm for MRF

(Graphical Models on Complete Graph)(Graphical Models on Complete Graph)

K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: J. Phys. A 2004J. Phys. A 2004

Hyperparameter Estimation by using Belief Propagation Hyperparameter Estimation by using Belief Propagation (BP) for Gaussian Graphical Model in Image Processing(BP) for Gaussian Graphical Model in Image Processing

It is possible to estimate statistical behaviour of EM algoritIt is possible to estimate statistical behaviour of EM algorithm with belief propagation analytically.hm with belief propagation analytically.

K. Tanaka and D. M. Titterington: J. Phys. A 2007K. Tanaka and D. M. Titterington: J. Phys. A 2007Statistical Trajectory of Approximate EM Algorithm for Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image ProcessingProbabilistic Image Processing


ContentContentss





10 October, 2007 University of Glasgow 7

Bayesian Image Restoration

Original Image Degraded Image

transmission

Noise

Likelihood Marginal

yProbabilit PrioriA Process nDegradatio

yProbabilit PosterioriA

Image Degraded

Image OriginalImage OriginalImage Degraded

Image DegradedImage Original

Pr

PrPr

Pr


Bayes Formula and Probabilistic Image Processing

,,2,1

g fP ,σfgP

g

g

g

g

2

1

Original Image Degraded Image

α,σgP

αfP,σfgP,gfP

,

Prior Probability

PosteriorProbability

Degradation Process

Pixel

f

f

f

f

2

1

links the all of Set:N


Prior Probability in Probabilistic Image Processing

Bijji ff

Z2

Prior)(

2

1exp

)(

1Pr

fF

0005.0 0030.00001.0

Samples are generated by MCMC.

B: Set of All the Nearest Neighbour pairs of Pixels

: Set of all the nodes

Markov Chain Monte Carlo Method


Degradation Process

Additive White Gaussian Noise

2,0~ Nfg ii

iii gf 2

22 2

1exp

2

1Pr

fFgG

Histogram of Gaussian Random Numbers

n NoiseGaussian f Image Original g Image Degraded


gzd,gzPz

m

m

m

gm

CI

I2

2

1

,,,

Degradation Process and Prior

Degradation Process

Prior Probability Density Function

,, ii gf

Bijji ff

ZfP 2

PR 2exp

1

iii gffgP 2

22

1exp

2

1,

,

,,,

gP

fPfgPgfP

Posterior Probability Density Function

2,0~ Nn

nfg

i

iii

,,2,1 links theall ofSet :B

otherwise0

1

4

Bij

ji

ji C

Multi-Dimensional Gaussian Integral Formula


Maximization of Marginal Likelihood by EM Algorithm

zdzPzgPgP

,,Marginal Likelihood zdgzPgzPgQ

,,ln',',,',',

.,,maxarg1,1

.,,ln,,,,

,ttQtt

zdgzPttgzPttQ

:Step-M

:Step-E

Iterate the following EM-steps until convergence:

EM Algorithm

Q-Function

A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data

via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

i

ii ttPgftf

gf ))(),(,|()()1(|| 22

.,)(),(,maxarg)1(),1(,

gttQtt

Bij

jiBij

ji ttPfftPffff

gff ))(),(,|()())1(|()( 22

)1(

||)(ln2

)1(PR

t

Zt

=

Extremum Condisions of Q(,|(t),(t),g) w.r.t. and



g

g

g

2

1

g

gCI

Cg

CI

C2

T2

21 1

Tr1

1tttt

tt

gCI

Cg

CI

I22

242T

2

22 1

Tr1

1tt

tt

tt

tt

.,)(),(,maxarg)1(),1(,

gttQtt

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

t

t

g

gmf

,ˆ,ˆˆ


Statistical Behaviour of EM (Expectation Maximization) Algorithm

g

g

g

2

1

g1

22*

2**

2

2

))()((

)(Tr

1

)()(

)(Tr

1)1(

CI

CC

CI

C

tt

I

tt

tt

22*

2**42

2

2

))()((

)()()(Tr

1

))()((

)(Tr

1)1(

CI

CIC

CI

I

tt

tt

tt

tt

ggg dPttQ

tt

**

,,,)(),(,maxarg

)1(),1(

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

t

t t

t

1000

0001.00

40

001.0*

*

Numerical Experiments for Standard Image

1000

0001.00

40*

Statistical Behaviour of EM Algorithm


ContentContentss







i

ii ttPgftf

gf ))(),(,|()()1(|| 22

.,)(),(,maxarg)1(),1(,

gttQtt

Bij

jiBij

ji ttPfftPffff

gff ))(),(,|()())1(|()( 22

)1(

||)(ln2

)1(PR

t

Zt

=

Markov Chain Monte Carlo



Bij

jiij ffP ),(),,|( gf

ii

ikk

cjjiij

cjjiij

cjjiij

ikffi ffff

ff

w),(),(

),(

)|'(\

,ff

fi(t) fi(t+1)

wi(f(t+1)|f(t))

),,|()|(

),,|()|(

gfff

gfff

Pw

Pw

i

i

ijff jj \

i

i

ci

Basic Step


)()1()0(

)()1()0(

)()1()0(

)()()(

)2()2()2(

)1()1()1(

LLL fff

fff

fff

Frequency

fi

L

lff

fiii

ili

i

L

ffffPfP

1,

\21

)(1

),,|,,,,,()(

f

g

Marginal Probabilities can be estimated from histograms.



)()1()0(

)()1()0(

)()1()0(

)()()(

)2()2()2(

)1()1()1(

LLL fff

fff

fff


i

ii ttPgftf

gf ))(),(,|()(||

1)1( 2

1

2 ))(),(,|()(||

2)1(

Bijji ttPff

Bt

f

gf

1

)()0(,),0()0(),()0( )()()2()2()1()1(

tt

LL ffffff

EM

MCMC



0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

Non-Synchronized Updateg

.,,,maxarg1,1,

gttQtt

1000

0001.00

40*


t

t

20 Samples

Input

Output

MCMC =50

EM

MCMC (=50)

MCMC (=1)

Exact

EMInput

Output

MCMC


0.0000

0.0002

0.0004

0.0006

0.0008

0.0010

0.0012

0 20 40 60 80 100


Non-Synchronized Updateg

.,,,maxarg1,1,

gttQtt

1000

0001.00

40

0010.0*

*


t

t

20 Samples

Input

Output

MCMC =50

EM

MCMC (=50)

MCMC (=1)

Exact

EMInput

Output

MCMC


ContentContentss






SummarySummary

We construct EM algorithms by means of We construct EM algorithms by means of Markov Chain Monte Carlo method and Markov Chain Monte Carlo method and compare them with some exact calculations.compare them with some exact calculations.

Input

Output

Exact EM

Input

Output

MCMC =50

EM EMInput

Output

MCMC


New Project 1

fi(t) fi(t+1)

wi(f(t+1)|f(t))

ii

ci

Basic Step

EMInput

Output

MCMC

Can we derive the trajectory of EM algorithm by solving the master equations for any step t in the case of ?

i

ii ttPgftf

gf ))(),(,|()(||

1)1( 2

1

2 ))(),(,|()(||

2)1(

Bijji ttPff

Bt

f

gf

EM


New Project 1

i

ci

iitit

tt

dwpwp

pp

fffffff

ff

)|()()|()(

)()(1

Bij

ijiBij

ji PffttPffff

fgf )()())(),(,|()( 22

Transition Probability

i

tiii

ii PgfttPgfff

fgf )()())(),(,|()( 22

From the solution of master equation, we calculate

These are included in the EM update rules.

i

kkcj

jiijik

ffi ffw ),()|'(\

,ff


New Project 2

Can we replace the calculation of statistical quantities in the prior probability by the MCMC?

EM

i

ii ttPgftf

gf ))(),(,|()()1(|| 22

Bij

jiBij

ji ttPfftPffff

gff ))(),(,|()())1(|()( 22

Input

Output

MCMC

EM


New Project 3New Project 3

K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: J. Phys. A 2004J. Phys. A 2004

Hyperparameter Estimation by using Belief Propagation Hyperparameter Estimation by using Belief Propagation (BP) for Gaussian Graphical Model in Image Processing(BP) for Gaussian Graphical Model in Image Processing

K. Tanaka and D. M. Titterington: J. Phys. A 2007K. Tanaka and D. M. Titterington: J. Phys. A 2007Statistical Trajectory of Approximate EM Algorithm for Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image ProcessingProbabilistic Image Processing

Our previous works in EM algorithm and Loopy Belief Propagation


New Project 3

.,,,maxarg1,1,

gttQtt

g

f̂

Loopy Belief Propagation

Exact

0006000ˆ

335.36ˆ

.

LBP

LBP

0007130ˆ

624.37ˆ

.

Exact

Exact

MSE:327

MSE:315

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100 t

tLoopy BP

Exact


New Project 3

g

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

t

t

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

gdgPgttQtt

**

,,,,,maxarg1,1

.,,,maxarg1,1,

gttQtt

t

t

1000

0001.00

40

0007.0*

*

1000

0001.00

40*

0005640ˆ

542.38ˆ

.

LBP

LBP

Statistical Behaviour of EM Algorithm


Loopy BP

Exact

Loopy BP

Exact


New Project 3New Project 3

Can we update both messages and Can we update both messages and hyperparameters in the same step?hyperparameters in the same step?Can we calculate the statistical trajectory?Can we calculate the statistical trajectory?

Input

Output

BP EM EM

Input

Output

BP

More Practical Algorithm

em algorithm with markov chain monte carlo method for bayesian image analysis

Documents

em algorithmin

em algorithmj

following emsteps

em algorithmqfunctiona

image processingit

mrf graphical models

statistical inferencegeman

statistical quantities