.

The Relative Entropy Rate of Two Hidden Markov

Processes

Or Zuk

Dept. of Phys. Of Comp. Systems

Weizmann Inst. Of Science

Rehovot, Israel

2

Overview

Introduction Distance Measures and Relative Entropy rate Results: Generalization from Entropy Rate. Future Directions

3

Introduction

Hidden Markov Processes are relevant: Error Correction (Markovian source +noise) Signal Processing, Speech recognition Experimental physics -telegraph noise, TLS+noise,

quantum jumps. Bioinformatics -biological sequences, gene

expression

t

Transmission

Noise 10%

Markov chain HMP47000 48000 49000

0

50

100

co

un

ts

time (ms)

Quantum jumps

0 200 400 600 800 1000

7.0

8.0

9.0

R (

Mo

hm

)

Mesoscopic wires

4

HMP - Definitions

Markov Process:

X – Markov Process

Mλ – Transition Matrix

mλ(i,j) = Pr(Xn+1 = j| Xn = i)

Hidden Markov Process :Y – Noisy Observation of XRλ – Noise/Emission Matrix

rλ(i,j) = Pr(Yn = j| Xn = i)

Mλ

RλRλ

Xn Xn+1

Yn+1Yn

Models are denoted by λ and µ.

5

Example: Binary HMP

0 1

p(1|0)

p(0|1)

p(1|1)

p(0|0)

)1|1()1|0(

)0|1()0|0(

pp

pp

0 1

q(0|0) q(1|0)q(0|1)

q(1|1)

)1|1()1|0(

)0|1()0|0(

qq

qq

Transition Emission

6

Example: Binary HMP (Cont.) A simple, Symmetric Binary HMP :

M = R =

All properties of the process depend on two parameters, p and . Assume w.l.og. p, < ½

pp

pp

1

1

1

1

7

Overview

Introduction Distance Measures and Relative Entropy rate Results: Generalization from Entropy Rate. Future Directions

8

Distance Measures for Two HMPs

Why important ? Often, one learns a HMP from data. It is important

to know how different is the learned model from the true model.

Sometimes, many HMPs may represent different sources (e.g. different authors, different protein families etc.), and we wish to know which sources are similar.

What distance measure to use? Look at joint distributions of N consecutive Y

symbols Pλ(N) and Pµ

(N) .

9

Relative Entropy (RE) Rate

Notation : Relative Entropy for finite (N-symbol) distributions:

Take the limit to get the RE-rate:

jiiji YYYY ,...,,][ 1

Alternative definition, using conditional relative entropy:

10

Relative Entropy (RE) Rate

First proposed for HMPs by [Juang&Rabiner 85]. Not a norm (not symmetric, no triangle inequality). Still it has several natural interpretations:

-If one generates data from λ, and gives likelihood score to µ, then D(λ || µ) is the average likelihood-loss per symbol (compared to the optimal model λ).

-If one compresses data generated λ, assuming erroneously it was generated by µ, then one ‘looses’ on average D(λ || µ) per symbol.

For Markov chains, D(λ || µ) is easily given by:

11

Relative Entropy (RE) Rate

For HMPs, D(λ || µ) is difficult to compute. So far only bounds [Silva&Narayanan] or approximation algorithms

[Li et al. 05, Do 03, Mohammad&Tranter 05] are known. D(λ || µ) generalizes the concept of the Shannon entropy

rate, using:

H(λ) = log s – D(λ || u)

Where u is the uniform model, s is the alphabet size of Y. The entropy rate H for an HMP is a Lyapunov Exponent,

which is hard to compute generally. [Jacquet et al 04] What is known (for H) ? Lyapunov exponent representation,

analyticity, asymptotic expansions in different Regimes. Generalize results and techniques to the RE-rate.

12

Why is calculating D(λ || µ) difficult?

Markov Chains:

-All states with the same no. of flips have the same prob.

Polynomial number of types (probs).

2N

XX

X

Y 2N

X

2N

Y

HMPs :Many Markov chains, {X} contributes to the same Y. Different {Y}s have different probs.

Exponential number of types (probs). Method of types does not work here.

13

Overview

Introduction Distance Measures and Relative Entropy rate Results: Generalization from Entropy Rate. Future Directions

14

RE-Rate and Lyapunov Exponents

What is Lyapunov exponent? Arises in Dynamical Systems, Control Theory, Statistical

Physics etc. Measures the stability of the system. Take two (square) matrices A,B. Choose each time at

random A (with prob. p) or B (w.p. 1-p). Look at the norm:

(1/N) log ||ABBBAABAB…BA||

The limit:

-Exists a.s. [Furstenberg&Kesten 60]

-Called Top Lyaponov Exponent.

-Independent of Matrix Norm chosen. HMP entropy rate is given as a Lyaponov Exponent

[Jacquet et al. 04]

15

RE-Rate and Lyapunov Exponents

What about RE-rate? Given as the difference of two Lyapunov Exponents:

-The G’s are random matrices, which are simply obtained from M and R using the forward equations.

-Different matrices appear in the two Lyapunov exponents, but the probabilities selecting the matrices are the same.

16

Analyticity of the RE-Rate

Is the RE-rate continuous, ‘smooth’, or even analytic in the parameters governing the HMPs?

For Lyapunov exponents: Known analyticity in the matrix entries [Rulle 79], and their probabilities [Peres 90,91] separately.

For HMP entropy rate, analyticity was recently shown by [Han&Marcus 05].

17

Analyticity of the RE-Rate

Using both results, we are able to show:

Thm: The RE-rate is analytic in the HMPs parameters.

Analyticity is shown only in the interior of the parameters domain (i.e. strictly positive probabilities).

Behavior on the boundaries is more complicated. Sometimes analyticity remains on the boundaries (and beyond). Sometimes we encounter singularities. Full characterization is still lacking [Marcus&Han 05].

18

RE-Rate Taylor Series Expansion

While in general the RE-rate is not known, there are specific parameters values for which it is easily given in closed-form (e.g. for Markov-Chains). Perhaps we can ‘expand’ around these values, and get asymptotic results near them.

Similar approach was used for Lyapunov exponents [Derrida], and for HMP entropy rate [Jacquet et al. 04, Weizmann&Ordenlich 04, Zuk et al. 05] giving first-order asymptotics in various regimes.

19

Different Regimes – Binary Case

p -> 0 , p -> ½ ( fixed)

-> 0 , -> ½ (p fixed)

We concentrate on the ‘High-SNR regime’ -> 0, and

‘almost-memoryless regime’ p-> ½.

p

00 ½

½

For High-SNR (η= λ,µ) :

Solution can be given as a power-series in :

20

RE-Rate Taylor Series Expansion

In [Zuk,Domany,Kanter&Aizenman 06] we give a procedure for calculating the full Taylor-Series Expansion for the HMP entropy rate, in the ‘High SNR’, and ‘almost memoryless’ regime.

Main observation: Finite systems give the correct RE rate up to a given order:

Was discovered using computer experiments (symbolic computation in Maple).

Stronger result holds for the entropy rate (orders ‘settle’ for N ≥ (k+3)/2)

Does not hold for any regime. For some regimes (e.g. p->0), even first order never settles.

21

Proof Outline (with M. Aizenman)

X

Y

(k+3)/2

H(p,) up to O(k)

Two main Ideas:

A. To distinguish between noise at different site

2 3….j

….

B. When m=0, the observation Ym=Xm,

conditioning back to the past is ‘blocked’

m=0

k+2

H(λ)

D(λ||µ)

22

Overview

Introduction Distance Measures and Relative Entropy rate Results: Generalization from Entropy Rate. Future Directions

23

RE-Rate Taylor Series Expansion

First order :

Higher orders were computed for the binary symmetric case. Similar results for the ‘almost-memoryless’ regime. Radius of convergence seems larger for the latter

expansion, albeit no rigorous results are known.

24

Future Directions

o Study other regimes. (e.g. two ‘close’ models).o Behavior of the EM algorithm. o Generalizations (e.g. different alphabets sizes,

continuous case).o Physical realization of HMPs (mesoscopic systems,

quantum jumps) o Domain of Analyticity - Radius of convergence.

25

Thanks

o Eytan Domany (Weizmann Inst.)o Ido Kanter (Bar-Ilan Univ.)o Michael Aizenman (Princeton Univ.)o Libi Hertzberg (Weizmann Inst.)