andrea montanari and ruediger urbanke tifr tuesday, january 6th, 2008 phase transitions in coding,...
TRANSCRIPT
![Page 1: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/1.jpg)
Andrea Montanari and Ruediger UrbankeTIFR
Tuesday, January 6th, 2008
Phase Transitions in Coding, Communications, and Inference
![Page 2: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/2.jpg)
Outline
1) Thresholds in coding, the large size limit
(definition and density evolution characterization)
2) The inversion of limits (length to infty vs size to infty) 3) Phase transitions in measurements (compressed sensing versus message passing, dense versus sparse matrices)
4) Phase transitions in collaborative filtering (the low-rank matrix model)
![Page 3: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/3.jpg)
Model
Shannon ’48
binary symmetric channel
capacity: R≤1-h(ε)
binary erasures channel
capacity: R≤1-ε
![Page 4: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/4.jpg)
Channel Coding
code
decoding
C={000, 010, 101, 111}
n ... blocklength
xMAP(y)=argmaxX in C p(x | y)
xiMAP(y)=argmaxXi p(xi |y)
![Page 5: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/5.jpg)
Factor Graph Representation of Linear Codes
(7, 4) Hamming code
every linear code
Tanner, Wiberg, Koetter, Loeliger, Frey
parity-check matrix
![Page 6: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/6.jpg)
Low-Density Parity Check Codes
(3, 4)-regular codes
Gallager ‘60
number of edges is linear in n
![Page 7: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/7.jpg)
Ensemble
![Page 8: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/8.jpg)
Variations on the Theme
irregular LDPC ensembleregular RA ensembleirregular MN ensembleirregular RA ensembleARA ensembleturbo code
degree distributions as well as structure
protographirregular LDGM ensemble
(Luby, Mitzenmacher, Shokrollahi, Spielman, and Stehman)Divsalar, Jin, and McEliece Jin, Khandekar, and McEliece Abbasfar, Divsalar, KungBerrou and GlavieuxThorpe, Andrews, DolinarDavey, MacKay
![Page 9: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/9.jpg)
Message-Passing Decoding -- BEC
?
?
00
0
?
?
?
0+?0+? =??
0
0
?
?
?
??
0=00?
?
0
0
0
?0
?
decoded
decoded
0+00+0 =00
![Page 10: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/10.jpg)
Message-Passing Decoding -- BSCGallager Algorithm
![Page 11: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/11.jpg)
Asymptotic Analysis: Computation Graph
probability that computation graphof fixed depth becomes tree
tends to 1 as n tends to infinity
![Page 12: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/12.jpg)
Asymptotic Analysis: Density Evolution -- BEC
x
1-(1-x)r-1
x x
ε (1-(1-x)r-1)l-1
ε
Luby,Mitzenmacher, Shokrollahi,
Spielman, and Steman ‘97
![Page 13: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/13.jpg)
Asymptotic Analysis: Density Evolution -- BEC
ε
phase transition: εBP so that xt → 0 for ε< εBP
xt → x∞>0 for ε> εBP
![Page 14: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/14.jpg)
Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm
phase transition: εBP so that xt → 0 for ε< εBP
xt → x∞>0 for ε> εBP
xt =ε (1-p+(xt-1))+(1-ε) p-(xt-1)
p+(x)=((1+(1-2x)r-1)/2) l-1 p-(x)=((1-(1-2x)r-1)/2) l-1
![Page 15: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/15.jpg)
Asymptotic Analysis: Density Evolution -- BP
![Page 16: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/16.jpg)
Inversion of Limits
size versus number of iterations
![Page 17: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/17.jpg)
Density Evolution Limit
![Page 18: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/18.jpg)
Density Evolution Limit
![Page 19: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/19.jpg)
“Practical” Limit
![Page 20: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/20.jpg)
“Practical” Limit
![Page 21: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/21.jpg)
The Two Limits
Easy: (Density Evolution Limit)
Hard(er): (“Practical Limit”)
![Page 22: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/22.jpg)
Binary Erasure Channel
DE Limit
“Practical” Limit
implies
![Page 23: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/23.jpg)
What about “General” Case
expansion
probabilistic methods
Korada and U.
![Page 24: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/24.jpg)
Expansion
Miller and Burshtein: Random element of LDPC(l, r, n) ensemble is expander with
expansion close to 1-1/l with high probability
expansion ~ 1-1/l
![Page 25: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/25.jpg)
Why is Expansion Useful?
![Page 26: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/26.jpg)
Setting: Channel
![Page 27: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/27.jpg)
Setting: Ensemble
![Page 28: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/28.jpg)
Setting: Algorithm
![Page 29: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/29.jpg)
Aim: Show for this setting that ...
DE Limit
“Practical” Limit
implies
![Page 30: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/30.jpg)
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
![Page 31: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/31.jpg)
Linearized Decoding Algorithm
![Page 32: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/32.jpg)
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
![Page 33: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/33.jpg)
Combine with Density Evolution
![Page 34: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/34.jpg)
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
![Page 35: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/35.jpg)
Correlation and Interaction
0 1
1 000Expected growth:
(r-1) 2 ε?< 1
Problem: interaction correlation
(r-1)
2 ε
![Page 36: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/36.jpg)
Correlation and Interaction
![Page 37: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/37.jpg)
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
![Page 38: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/38.jpg)
Witness
![Page 39: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/39.jpg)
Witness
![Page 40: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/40.jpg)
Witness
![Page 41: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/41.jpg)
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
![Page 42: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/42.jpg)
Monotonicity
![Page 43: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/43.jpg)
Randomizing the Noise Outside
randomizing noise outside the witness increases the probability of error
FKG
→
←⁄
≤
![Page 44: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/44.jpg)
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
![Page 45: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/45.jpg)
Expansion
random graph has expansion close to expansion of a treewith high probability
⇒this limits interaction
0 1
1 000
![Page 46: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/46.jpg)
References
For a list of references see:http://ipg.epfl.ch/doku.php?id=en:courses:2007-2208:mct
![Page 47: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/47.jpg)
Results
![Page 48: Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference](https://reader031.vdocuments.site/reader031/viewer/2022013101/56649f095503460f94c1d764/html5/thumbnails/48.jpg)
Open Problems
0.0
0.4
0.3
0.2
0.1
0.2 0.4 0.6 0.8
Pb
channel entropy