TheCutoff Rate and Other Limits:

Passing the Impassable

Richard E. BlahutUniversity of Illinois

UIUC

04/18/23 1

Shannon’s Ideal Channel

1

1Q

Example: Binary Memoryless Channel

• Stationary• Discrete• Memoryless

0 1, , Ky bb K

|Qk j

0 1, , Jx a aK

0 0

1 104/18/23 2

0 1 0 0 1 1 0 1 0 1 0 …1 1 1 0 0 1 0 1 1 0 1 …0 1 1 1 0 0 1 0 1 1 1 …1 0 1 …....

1 1 1 1 1 0 0 0 1 0 0 …

… 0 0 0 11 1

100, 000n C

50, 0002 2k

0 0 …0 0 …0 0 …0 0 …...

1 1 …

… 0 0 0… 0 0 1… 0 1 0… 0 1 1

… 1 1 1

50, 000k

50, 0002 2k

A Large Code

04/18/23 3

+ ++

A convolutional encoder

04/18/23 4

Information theory asserts existence of good codes

Coding theory wants practical codes and decoders

There are binary codes22kn

50,000100,000 22 15,005102 2 10kn

04/18/23 5

Brief History of Codes

•Algebraic Block Codes 1948Reed-Solomon codes (1960)

•Convolutional Codes 1954Sequential decoding (1951)Viterbi algorithm (1967)

•Euclidean Trellis Codes 1982•Turbo Codes 1993

Gallager (LDPC) codes (1960)

04/18/23 6

Decoders

Maximum LikelihoodMaximum Block PosteriorMaximum Symbol PosteriorTypical SequenceIterative PosteriorMinimum DistanceBounded Distance

04/18/23 7

My View

1) Channel Capacity

2) Cutoff Rate

3) Critical Rate

C

0R

critR

C0RcritR0

Distance-basedcodes

Likelihood-basedcodes

Posterior-basedcodes

Rate

Polar codes04/18/23 8

For any fixed there is a sequence of codes for which exponentially in blocklength.

This sequence does not approach

Channel Error Exponent

( ) ( )nE R o nep e

Fact #2 Every code satisfies

Fact #1 Codes exist such that

,R C

C

*( )E R

( )nE Rep e

( ) ( )nE R o nep e

0ep

04/18/23 9

critR 0R C

g

E(R)

( )rmE ReP e

R

*( ) ( )nE R o n

ep e

( )E R

04/18/23 10

2k

10050

nk

10251

nk

10452

nk

10653

nk

Pickone

Pickone

Pickone

Pickone

A sequence of codes drawn from a set of ensembles

04/18/23 11

Channel Capacity

||

|

max log

k jj k j

p jj k j

QC p Q

p Q

Channel Critical Rate1/2

| |*crit | |

1/2| |

max log

k j k jj k j k j

p j kj k j k j

k

Q QR p Q Q

p Q Q

Channel Cutoff Rate2

1/20 |max log j k j

p k jR p Q

04/18/23 12

Binary Hypotheses Testing

0 0 00 01 0( 1)

1 1 10 11 1( 1)

: , , ,

: , , ,

K

K

H q q q

H q q q

K

K

q

q

Type 1 Error

Type 2 Error

0

1

( ) ( )

( ) ( )

10 1

10 1

( ) log

nD o n

nD o n

k kk

k kk

kk

kk

e

e

q qq

q q

pD p

q

q q

q q

p q

0qq

1q

(100)q

(001)q (010)q

3P

04/18/23 13

Binary Hypotheses TestingChange Notation

( )e r

r

0

1

( ) ( )

( )

e r D

r D

q q

q q

1s

s

( ) ( )

( )

ne r o n

nr o n

e

e

1/(1 ) /(1 )0 11/(1 ) /(1 )0 1

s s sk k

k s s sk k

q qq

q q

04/18/23 14

Bounds on ( )E R and ep ( ) ( )nE R o n

e epUpper Bounds on

Sphere Packing Bound0

1

: is the transmitted codeword

: is not the transmitted codewordm

m

H

H

c

c

Minimum Distance Bound0

1

: of and , is the transmitted codeword

:of and , is the transmitted codewordm m m

m m m

H

H

c c c

c c c

Lower Bounds onRandom Coding BoundExpurgated Bound

( )E R

( )E R

04/18/23 15

Bhattacharyya Distance

1/2 1/2( , ) log ( | ) ( | )B m m m my

d Q y Q y c c c c

Hamming Distance Euclidean Distance

04/18/23 16

1704/18/23

is quadratic near

Let be a sequence with

Then with

2( ) ( )E R C R( )E R C

( ) 0f n

so 2( )nf n

A Code Sequence Approaching Capacity

np ee with 12

if CRn

Rn CRn( )C R f nn

2( ) 0nf npe e

np ee

04/18/23 18

•Capacity: C•Shannon (1948)

•Cutoff Rate:•Jacobs & Berlekamp (1968)•Massey (1981)•Arikan (1985/1988)

•Error Exponent: •Gallager (1965)•Forney (1968)•Blahut (1972)

( )E R

0R

04/18/23 19

Gallager (1965)

11/(1 )

|0

( ) max max logs

s

j k js p k j

E R sR p Q

Forney (1968)

( ')

( )

' 1| |

nE R

e

n E R R R

L

dE dER RdR dR

p e

N e

Blahut (1972)

ˆ( ) max min

ˆ| ;

Qp R

R

E R D Q Q

Q I p Q R

Q

Q where

||

|

ˆˆ ˆ( ) log k j

j k jj k k j

QD Q Q p Q

Q is the Kullback divergence

04/18/23 20

Forney’s List Decoding

Likelihood Function ( | )mp y c

Senseword

Codeword mc

y

Likelihood Ratio

'

( | )

( | )

m

mm m

p

p

y c

y c

( )

( )List

1

nE Re

n E R R R

p e

N e

ss

crit

( ( ) )crit critList

if

n E R R R

R R

N e04/18/23 21

Sequential Decoding

• • Exponential waiting time • Work exponential in time• Pareto Distribution

with• Work unbounded if

' '[ ( ) ]~ n E R R RLN e

0/R R

0 / 1R R

Sequential decoding fails if 0R R

Is maximum likelihood decoding sequential decoding?

04/18/23 22

Two Pareto parameters and

Pareto Distribution

11

12

22 1 2

2

kx

k

1( / )( / )( )0

( / ) for Pr1 for

k k x x kp xx k

k x x kX xx k

k

04/18/23 23

Start with an exponential distribution

ye yp yy

0( )0 0

If is exponential, then is a Pareto distribution

Y YX ke

The Origin of a Pareto Distribution

04/18/23 24

The Origins of Graph-Based Codes

Brillouin

deBrogle Shannon

Battail (1987) Hagenauer (1989)

Berrou et al (1993)

04/18/23 25

Coding Beyond the Cutoff Rate

Parallel – Pinsker

Hybrid – Jelinek

Turbo – Berrou/Glavieux

LDPC – Gallager/Tanner/Wiberg

Polar - Arikan

04/18/23 26

The Massey Distraction (1981)

?

0

1

2

3

?

0

1

2

3

00

01

10

11

?

00

01

10

11

1 1

0

1

0

1

1

0

1

0

1

1

?

?

QEC BEC

(4)0

4log

1 3R

QEC

2 BEC

(2)0

2 42 2log log

1 1 3R

(4) 2(1 )C

(2)2 2(1 )C

04/18/23 27

Performance Measures

Bit Error Ratevs.

Message Error Rate

04/18/23 28

The Arikan Retraction

04/18/23 29

Splitting and Combining the BSC

04/18/23 30

E R( )

C

( )E R

R

Splitting and the Error Exponent

/ 2C/ 4C2( )E R 1( )E R ( )E R

04/18/23 31

The Arkan Redistraction*

0U

1U

2U

3U

4U

5U

6U

7U

0X

1X

2X

3X

4X

5X

6X

7X

0Y

1Y

2Y

3Y

4Y

5Y

6Y

7Y

Q

Q

Q

Q

Q

Q

Q

Q

*Rhetorical04/18/23 32