Improving analysis and performance of modern error-correction schemes:

a physics approach

Phys.Rev.Lett. 93, 198702 (2004)IT workshop, San Antonio 10/2004CNLS workshop, Santa Fe 01/2005Phys.Rev.Lett. 95, 228701 (2005)arxiv.org/abs/cond-mat/0506037

arxiv.org/abs/cs.IT/0507031IT workshop, Allerton 09/2005

arxiv.org/abs/cs.IT/0601070arxiv.org/abs/cs.IT/0601113

Misha Chertkov (Theory Division, LANL)

Vladimir Chernyak (Department of Chemistry, Wayne State)Misha Stepanov (Theory Division, LANL)

Bane Vasic (Department of ECE, University of Arizona)

arxiv.org/abs/cond-mat/0601487arxiv.org/abs/cond-mat/0603189

Analyzing error-floor for LDPC codes Understanding Belief-Propagation:Loop Calculus

MC,VC

towards improving Belief Propagation

UoC,04/10/06

Menu:(first part)

• Analogous vs Digital &• Analogous Error-Correction &• Digital Error-Correction &• LDPC, Tanner graph, Parity Check &• Inference, Maximum-Likelihood, MAP &• MAP vs Belief Propagation (sum-product) & • BP is exact on the tree &• Error-correction Optimization &• Shannon-Transition &• Error-floor &

Introduction

• Instanton method – the idea &• Instanton-amoeba (efficient numerical method) &• Test code: (155,64,20) LDPC &• Instantons for the Gaussian channel (Results) &• BER: Monte-Carlo vs Instanton &

• Conclusions &• Path Forward &

Instanton: proof of principles test

Analogous vs digital

Analogous Digital

continuoushard to copy

discreteeasy to copy

0111100101

camera picturemusic on tape

typed text computer file

real number better/worse

integer number yes/no

menu

Error-correction for analogous

One iteration 4 iteration

16 iteration clean menu

menu

menu

L

L

N

N

Digital Error-Correction

Coding

Decoding

N > L R=L/N - code rate

)|()|( )(

1

)()()( ini

N

i

outi

inout xxpxxP

22exp)|(

222

syxs

yxp

channelwhite

Gaussian symmetricexam

ple

Nxxx ,,1

)()()( inoutin xxxnoise

menu

Low Density Parity Check Codes

menu

N=10variable nodes

M=N-L=5 checking nodes

Parity check matrix

0

0

0

0

0

10

9

8

7

6

5

4

3

2

1

v

v

v

v

v

v

v

v

v

v

H mod 2

Tanner graph

M

ii

ii x

1

1,

112

“spin” variables -

- set of constraintsM

Ni

,,1

,,1

(linear coding)

Parity check matrix(155,64,20) code

Tanner graph(155,64,20) code

menu

Inference

Given the detected (real) signal ---

To find the most probable (integer) pre-image ---

outx

inx

)|~(argmax ][~ outinallowedx xxPdecodingin

Maximum-Likelihood (ML) Decoding

menu

Decoding (optimal)

N

kkk

M

ii hhFhZ

11}{

exp1,)(exp)(

)( )(outxh “magnetic” field

log-likelihood

)|( )()( inout xxP

constraints

“free energy”

“partition function”

(symbol to symbol) Maximum-A-Posteriori (MAP) decoding (close to optimal)

)()( hmsignhoutput j

Efficient but Expensive:requires operationsL2

hhFhm

)()(

“magnetization”=a-posteriori log-likelihood

Stat Mech interpretation was suggested byN. Sourlas (Nature ‘89)

To notice – spin glass (replica) approach for random codes:e.g. Rujan ’93, Kanter, Saad ’99; Montanari, Sourlas ’00; Montanari ’01; Franz, Leone, Montanari, Ricci-Tersenghi ‘02 menu

Sub-optimal but efficient decoding

i

jii

j

jj

i

jii

j

jj

hm

h

tanhtanhtanh

tanhtanh

1

1 Belief Propagation (BP=sum-product) Gallager’63;Pearl ’88;MacKay ‘99

=solving Eqs. on the graph

it

i

i

ji

ti

j

jt

j

h

h

)(

)(1)1( tanhtanh

Iterative solution of BP= Message Passing (MP)

Q*m*N steps instead of Q - number of MP iterations

m - number of checking nodes contributing a variable node

L2

What about efficiency? Why BP is a good replacement for MAP?

* (no loops!)

menu

Tree -- no loops -- approximation

}{}{

1

}{

}{}{

1

}{

11}{

exp1,)(

exp1,)(

exp1,)(exp)(

kkk

iij

kkk

iij

N

kkk

M

ii

hhY

hhX

hhFhZ

j

j

i

jii

j

jj h tanhtanh 1

2/)/ln(

)()(2

1)exp(

)()(2

1)exp(

jjj

i

ji

i

jiiiii

j

jj

i

ji

i

jiiiii

j

jj

XY

YXYXhY

YXYXhX

MAP

BP

Belief Propagation is optimal (i.e. equivalentto Maximum-A-Posteriori decoding) on a tree (no loops)

Analogy: Bethe lattice (1937)

Gallager ’63; Pearl ’88; MacKay ’99Yedidia, Freeman, Weiss ‘01 menu

Bit Error Rate (BER)

)1|()( )()()(

outouti

outi xPxmxdB

measure of unsuccessful decoding

Probability of making an error in the bit “i”

{+1} is chosen for the initial code-word

probability density for givenmagnetic field/noise realization

(channel)

Digital error-correction scheme/optimizationDigital error-correction scheme/optimization

1. describe the channel/noise --- External2. suggest coding scheme3. suggest decoding scheme4. measure BER/FER5. If BER/FER is not satisfactory (small enough) goto 2

menu

From R. Urbanke, “Iterative coding systems”

SNR, s

BE

R, B

Shannon transition/limit

menu

Error floor (finite size & BP-approximate)

Error floor prediction for some regular (3,6) LDPC Codes using a 5-bit decoder. From T. Richardson “Error floor for LDPC codes”, 2003 Allerton conference Proccedings.

No-go zone for brute-force Monte-Carlo numerics.

Estimating very low BER is the major bottleneck

in coding theory/practice

menu

Our (current) objective:

For given (a) channel (b) coder (c) decoderto estimate BER by means ofanalytical and/or semi-analytical methods.

Hint:

BER is small and it is mainly formed at some very special“bad” configurations of the noise/”magnetic field”

Instanton approach is the right way to identifythe “bad” configurations and thus to estimate BER!

menu

Instanton Method

Laplace methodSaddle-point method

Steepest descent

1noise

2noise

...noise

errors

no errors

Error-surface (ES)

Point at the ESclosest to zero

menu

BER = d(noise) Weight(noise)instanton config.

of the noiseBER Weight

instanon config of the noise

Point at the ESclosest to zero

Parity check matrix(155,64,20) code

Tanner graph(155,64,20) code

menu

Found with numericalinstanton-amoeba scheme

instanton-amoeba menu

Instantons for (155,64,20) code: Gaussian channel

076.10210

4622 efl 203.10

79

8062 efl 298.10188

4422 efl

Phys. Rev. Lett -- Nov 25, 2005

menu

menu

We suggested amoeba-instanton method for efficient numerical evaluation of BER in the regime of high SNR (error floor). The main idea: error-floor is controlled

by only a few most damaging configurations of the noise (instantons).

Conclusions (for the first part – error floor analysis)

Results of the amoeba-instanton are successfully validatedagainst brut-force Monte-Carlo (in the regime of moderate

SNR)

menu

Path Forward Extend the amoeba-instanton test • to study the error-floor • to develop universal computational tool-box for the error-floor analysis Other codes Other decoding schemes (e.g. number of iterations)

Other channels (e.g. magnetic recording and fiber-optics specific)

Major challenge !!!! – to improve BP qualitativelyNew decoding ?!

New coding ?! Efficient (channel specific) LDPC optimization

Inter-symbol interference + noise (2d and 3d + error-correction)

Distributed coding, Network codingCombinatorial optimization menu

Understanding Belief Propagation

Questions:

• Why it works so well … even when it should not? -- BP is gauge fixing condition

• Can one constructs a full solution (MAP) from BP? -- yes one can!/loop series

Making use of the loop calculus/series

Improving BP – approximate algorithms • LDPC decoding• SATisfiability resolution• Data reconstruction• Clustering• etc

Answers:arxiv.org/abs/cond-mat/0601487arxiv.org/abs/cond-mat/0603189

first slide

Vertex Model

aXa

afZ

}{

Xa

aafZp 1)(

Partition function

Probability

},{ ia

otherwise

if ii

ii,0

,,,1)(

Reduction to bipartite graph (error-correction):

i i

ii

ii

ii q

hf exp1,exp1,)(

),(

),,(

),,,,,,(

1

13122

1814121

453423181412

baab

Ising variables on edges

}{edgesX

improving BP

Bethe Free Energy --- Variational Approach

Generalization ofYedidia, Freeman,Weiss ‘01

0bF

Constraints(introduce in minimizationthrough Lagrange multipliers)

Belief Propagation (Bethe-Peierls) equations

acacca

acacaaa

aaaaa

aa bbbbfbFacaa

lnlnln),(

self-energy entropy entropy correction

improving BP

)()()(:;,

1)()(:;,

1)(,)(0:;,

\\ccaaacac

acacaa

acacaa

bbbacca

bbacca

bbacca

accaca

aca

Loop series

abb --- beliefs (prob.) calculated within BP !

BetheFZ ln0 •BP is special, not only without loops!•Gauge invariant representation!

=C

improving BP

•integral representation• algebraic representation• gauge representation

Three alternative derivations:

Loop series (derivation #1)

“vertex”

“propagator”

ab --- gauge degrees of freedom (at our disposal !)

},,{' 2112

improving BP

a cb

cbbcaa

aaa ffZ

),(' 2

1)()(

coshsinhcoshsinh1

cosh

sinhcoshsinhcosh1*1

cbbcbc

cbcbcbbcbcbcbccbbc

V

V

cbbc

ababbaabaaaa

cbbc

aa

cb

fP

VPZ

)sinh(cosh)()(

cosh2),('

1

),(

Loop series (derivation #2)

' ),(

~ a cb

bca VPZ

.

""

**1

contribcolored

** … *

Expand the “vertex” (edge) term

Calculate resulting terms one-by-one

• Each node enters the product only once• Node is colored if it contains at least one colored edge

Gauge fixing condition:

To forbid “loose end contribution” for any node !!

ab --- gauge degrees of freedom (at our disposal !)

improving BP

Loop series (derivation #3)

fixing the gauge!!to kill loops

Belief Propagation !!

equations

Loop series has just been derived!!

improving BP

Future work

•Approximate algorithms --- leading loop, next after leading,.. --- apply to LDPC decoding --- different graphs, lattices• Generalization --- Ising Potts (longer alphabets) --- continuous alphabets (XY,Heisenberg,Quantum models) first slideimproving BP

Conclusions ( for the second part – Understanding/Improving BP)

• Loopy BP works well because BP is nothing but GAUGE FIXING condition

• Simple finite series --- LOOP SERIES --- for MAP is constructed in terms of BP solution

Instantons on the tree (semi-analytical)

menu

PRL 93, 198702 (2004) ITW 2004, San Antonio

m=2, l=3, n=3 m=3, l=5, n=2

Instanton-amoeba (efficient-numerical scheme)

)1|)(1()1|)(1(max~ **

efui uPuPB

0))((

)()(

*0

**

um

uluu

To minimize BER with respect to the unit vector !!

error-surface

unite vector in the noise space

Minimization method of our choice is simplex-minimization (amoeba)

)(

)1|)(1(maxarg

*

*

efef

uef

ull

uPu

menuinstanton-amoeba for Tanner code

Different noise models for different channels

)|()|( )(

1

)()()( ini

N

i

outi

inout xxpxxP

ii

ii x

xp

xp

sh

ssxp

)1|(

)1|(log

2

1

22exp)1|(

2

222

White

Gaussian

)(111 )()( uluxx outin Linear

)|()|( xypyxp Symmetric

simplifications

Laplacian

i

ii

i

i

ii xp

xp

sh

ssxp

0,1

02,1

2,1

)1|(

)1|(log

2

1

exp)2/()1|(

menu

Rational structure of instanton (computational tree analysis/explanation) min-sum

4 iterations

based on Wiberg ‘96

Phys.Rev.Lett. 95, 228701 (2005)

Minimize effective actionkeeping the condition

menu

Bit-Error-Rate: Gaussian channel

menu

Instantons for (155,64,20) code: Laplacian channel

6.7efl 0.8efl 0.8efl

menuIT workshop, Allerton 09/2005

Instantons as medians of pseudo-codewords

menuPRL -- Nov 25, 2005

Bit-Error-Rate: Laplacian channel

menu