coding theory ml decoding of linear codes. trellis

Coding TheoryML decoding of linear codes. Trellis representations.

Implementation of ML decoding

Irina Bocharova, Tartu, 2021, of 36

Agenda

1 Maximum-likelihood (ML)-decoding of linear codes.

2 Trellis representations.

3 The Viterbi decoding algorithm.


ML decoding of linear codes

• Let u be a message sequence and c = uG is a codewordof a linear code C determined by its generator matrix G .Denote by s the corresponding sequence of modulationsignals. For example, if binary phase shift keying (BPSK)is used then 0→ −1, 1→ 1.

• Let r = s+ n be the output signal of the communicationchannel. For example, for BSC s = c and n = e is abinary error sequence, summation is performed modulo 2.For the AWGN channel, s is a sequence of real numbersdependent on the modulation system and n is a sequenceof independent noise samples, where each sample is aGaussian variable with zero mean and variance N0/2.


ML decoding of linear codesDenote by c the decoder output. The optimaldecoder minimizes the probability of decoding error

Pe =∑r

P(r)∑c6=c

P (c 6= c|r),

P(r) is independent of decoding procedure then theoptimal decoder minimizes P (c 6= c|r) for all r ormaximizes

P (c = c|r) = P(r|c)P(c)P(r)

,

if P(c) is the same for all c then the optimal decoderis ML decoder maxc∈C P(r|c).



Denote by d Hamming distance between r and c andn be their length then∑

j

logP(rj |cj) = d log p + (n − d) log(1− p)

= d log

(p

1− p

)+ n log(1− p),

For p < 0.5, log(

p1−p

)< 0 then

the ML decoder for a BSC chooses a codeword c

which minimizes the Hamming distance d .



For the AWGN channel:the ML decoder chooses a codeword c = (c1, ..., cn)which minimizes the squared Euclidean distancebetween r = (r1, ..., rn) and s = (s1, ..., sn), wheresi = 2ci − 1, that is,

d2E= ||r−s||2 = ||r||2−2(r, s)+ ||s||2 =

n∑i=1

(ri−si)2

or maximizes the scalar product

(r, s) =n∑

i=1

(ri , si).


ML decoding for a linear code

• Exhaustive search over all codewords (both forthe BSC and the AWGN channel)• Exhaustive search over error combinations (for

the BSC)

An upper bound on ML decoding complexity

complexity ≤ min{2k, 2n−k}

or complexity exponent

log2 complexity

n≤ min{R , 1− R},

where R = k/n.Irina Bocharova, Tartu, 2021, of 36

Syndrome decoding

Consider the BSC. Assume that the transmittedcodeword is c and the received vector r = c+ e,where e = (e1, e2, ..., en) is the error vector.The number of corrected error vectors does notdepend on a particular code and is equal 2n−k = 2r .Due to linearity of the code, decision regions arecongruent. They di�er by a shift in the Hammingspace. Their size can be computed as 2n/2k .ML-decoding guaranties correction of all 2r errorvectors.Trivial solution: search over 2r error vectors,r = n− k . For each vector e compute (r+ e)HT andchoose vector c = r + e which gives the zero vector.


Syndrome decoding

Consider syndrome vector

ξ = rHT.

Sinceξ = rHT = (c+ e)HT = eHT,

there is a one-to-one correspondence between 2r

vectors e and 2r syndromes ξ. We use a table whichkeeps for each ξ the smallest weight e such thatξ = eHT.


Syndrome decoding

G =

1 1 0 1 0 10 1 0 0 1 10 0 1 1 0 1

(1)

An equivalent code is

G =

1 0 0 1 1 00 1 0 0 1 10 0 1 1 0 1

(2)

H =

1 0 1 1 0 01 1 0 0 1 00 1 1 0 0 1

(3)


Syndrome decoding

syndrome error0 0 0 0 0 0 0 0 00 0 1 0 0 0 0 0 10 1 0 0 0 0 0 1 00 1 1 0 1 0 0 0 01 0 0 0 0 0 1 0 01 0 1 0 0 1 0 0 01 1 0 1 0 0 0 0 01 1 1 0 1 0 1 0 0


Trellis representation. ML decoding by using code trellis

Trellis or trellis diagram is graph with the followingproperties:• Graph nodes are split into nonintersecting sets

called levels. Nodes of a given level areconnected only with nodes of the neighboringlevels.

• The initial and the last levels contain only onenode each

• Only movement from a level with smallernumber to a level with larger number is allowed.

• Graph edges are labeled by metrics("distances"). Path metric is equal to the sumof edge metrics.


Trellis representation.

0

1

2

3

4

5

6

7

8

a01

a02

a13a14

a15

a24a25

a36a37a46a47

a56

a57

a68

a78


Trellis representation. ML decoding

For BSC:If we have trellis representation of the code, then ML decodingis reduced to searching for the shortest path in the trelliswhose edges are labeled by Hamming distances betweensymbols of the received sequence and symbols of codewords.For AWGN channel: labels are Euclidean distances or scalarproducts between the received symbols and symbols ofcodewords.Trivial solution is exhaustive search over all paths of the trellis.In the above trellis there are 10 paths. To compute pathmetrics we need to perform 3 additions (30 additions in total).To �nd the shortest path we need to perform 9 comparisonsand selections.


Dynamic programming. The Viterbi algorithm

For node where paths merge, without loss ofoptimality we can choose the path with the bestmetric and discard other paths.For example, we compare a01 + a15 and a02 + a25. Atthe next levels to these sums we add the same terms.Thus, without loss of optimality, we can choose theminimal of two sums and assign it to node 5.In total, it is enough to perform 13 additions and 7comparisons and selections.


Dynamic programming. The Viterbi algorithm

• Initialization. Metric of the initial node is set to 0.

• Loop over levels

• for each node of the level do� Find metric of paths entering this node as the sum of

the proceeding node metrics and metrics of the edges

connecting them with the processed node

� Choose the candidate path with the minimum

metric, assign its metric to the processed node

� Form the path to the processed node as the chosen

candidate path to the previous node followed by edge

connecting it to the processed node

• End

• End


Minimal code trellis

We consider so-called nonsectionalized trellis. Itcontains one code symbol on each edge, the numberof levels is equal to n + 1. Example. Consider (3, 2)code determined by a generator matrix

G =

(1 1 00 1 1

)(4)

0

1

0

11

0

0

1


0

1

0

11

0

0

1

0, 1, 1 is transmitted by −1, 1, 1r = (−0.83, 0.68,−1.09)

0.83

−0.83

−0.68

0.680.68

−0.68

1.09

−1.09


0

1

0

11

0

0

1

r = (−0.83, 0.68,−1.09)

0.83

−0.83

−0.68

0.680.68

−0.68

1.09

−1.09


Minimal trellis. Constructing minimal trellis via generator matrix

Consider a sequence ψ = (ψ0, ψ1, ..., ψn), where ψi ,i = 0, 1, ..., n denotes the number of nodes at thei -th level of the trellis. The vector ψ is calledcomplexity pro�le of the code trellis.A trellis with complexity pro�le ψ is called minimaltrellis if complexity pro�le ψ

′of any other trellis of

the same code satis�es the inequality

ψ′

i ≥ ψi , i = 0, 1, ..., n.

Theorem

Any code has minimal trellis representation. All

minimal trellises coincide up to node numbering at

each level of the trellis.Irina Bocharova, Tartu, 2021, of 36

Minimal trellis. Constructing minimal trellis via generator matrix

Let x = (x1, x2, ..., xn) be an arbitrary binarysequence. Denote by b(x) the number of position ofthe �rst nonzero element in x and by e(x) thenumber of the last nonzero element in x .Elements in positions b(x), ..., e(x)− 1 we callactive.The number of active elements in x is calledspan of x .For example, x = (0, 0, 1, 0, 1, 1, 1, 0, 0, 0), b(x) = 3,e(x) = 7, span is equal 4.A generator matrix is said to be in minimal span formif all row starts and all row ends are di�erent.


Constructing minimal trellis via generator matrix

Assume that generator matrix rows are sorted inorder of increasing their starts.

G =

1 0 1 1 0 11 0 1 0 1 01 1 0 1 0 0

(5)

By linear row operations this matrix can be reducedto the minimal span form. By adding the 1st row tothe 2d and 3d rows, after permuting the 2d and the3d rows we obtain

G =

1 0 1 1 0 10 1 1 0 0 10 0 0 1 1 1

. (6)



Now we add the 2d row to the �rst and the 3d rowto the 2d row. We obtain the matrix in the MSF

G =

1 1 0 1 0 00 1 1 1 1 00 0 0 1 1 1

(7)


Constructing minimal trellises

G =

1 0 1 1 0 11 0 1 0 1 01 1 0 1 0 0

G =

1 0 1 1 0 10 1 1 0 0 10 0 0 1 1 1

G =

1 1 0 1 0 00 1 1 1 1 00 0 0 1 1 1



u1 u2u1 u2u1 u3u2 u3

0 0

1

00

01

10

11

00

01

10

11

00

01

10

11

0

1

00

1

0

1

1

0

0

0

1

1

0

1

1

0

1

0

0

1

0

1

1

0

0

1


Minimal trellis via generator matrix

Theorem

Trellis obtained via generator matrix in the MSF is

minimal.

• The number of states at each level of the trellis isdetermined by the number active rows. Since all rowsstart in di�erent positions all linear combinations of rowsup to this level are di�erent.

• Paths with di�erent past paths lead to the di�erentnodes. Only paths which are linear combinations ofnonactive rows can have the same continuation.

• At each level paths passing through di�erent nodes donot have the same continuation (future path).


Minimal trellis via generator matrix

Time-varying schemeof modulo 2 adders

ui , i = 1, 2, 3

ci , i = 1, ..., 6


Minimal trellis via parity-check matrix

H =

1 1 0 0 1 10 0 1 0 1 10 1 0 1 0 1

(8)

• Syndrome for sequences of length 1 can beequal to one of vectors (0, 0, 0) or (1, 0, 0)

• Syndromes of length 2 sequences belong to theset (0, 0, 0), (1, 0, 0), (1, 0, 1) and (0, 0, 1)

• The maximum number of syndromes is 2r = 8



Constructing minimal trellises from parity-checkmatrix

H =

1 1 0 0 1 10 0 1 0 1 10 1 0 1 0 1



000 000

100

000

001

100

101

000

001

110

111

000

001

110

111

000

111

0000

1

0

1

0

1

0

0

1

1

0

11

0

0

11

0

0

1

1

0

0

1

Figure: Syndrome trellis

Theorem: Syndrome trellis is minimal.


Minimal trellis via parity-check matrix

Theorem

Syndrome trellis is minimal.

Proof. Consider a codeword c = (cp, cf ). Partialsyndromes

ξp = cpHT

andξf = c

fHT

coincide, that is, ξp = ξf . Thus, all codewords withthe same continuation c

f leave the same nodedetermined by ξp.


Minimal trellises

Minimal trellis is unique up to the node numbering.Minimal trellises of equivalent codes can havedi�erent complexity. A trivial upper bound on themaximum trellis complexity for any code is

ψmax = maxiψi ≤ min

{2k , 2n−k

}.


Examples

Table: Lower and upper bounds on state complexity(log

2ψmax)

Code Lower bound Upper boundBCH(32,26,4) 5 6BCH(64,45,8) 12 19BCH(64,51,6) 12 13BCH(64,57,4) 6 7BCH(76,50,9) 13 26BCH(127,99,9) 15 28


Quasi-cyclic block codesA linear block code C of length n = ml over a �nite �eldGF (q) is called a quasi-cyclic code of index l if for everycodeword c ∈ C there exists a number l such that thecodeword obtained by l cyclic shifts is also a codeword in C.That is,

c = (c0, . . . , cn−1) ∈ C → c

′= (cn−l , . . . , c0, . . . cn−l−1) ∈ C.

The binary (6, 3) code with generator matrix

G =

1 1 0 1 0 00 0 1 1 0 10 1 0 0 1 1

(9)

is a quasi-cyclic code with l = 2. To ease the visualization wecan write the shifts as blocks,

G =

11 01 0000 11 0101 00 11

(10)


Examples

Table: Search results and upper bounds on state complexity(log

2ψmax)

Code Search result Upper bound(24,12,8) 8 12(48,24,12) 16 24(92,46,16) 26 46(52,26,10) 12 26


coding theory ml decoding of linear codes. trellis

Documents