1 near shannon limit performance of low density parity check codes david j.c. mackay and radford m....
Post on 19-Dec-2015
223 views
TRANSCRIPT
1
Near Shannon Limit Performance of Low Density Parity Check Codes
David J.C. MacKay and Radford M. Neal,
Electronics Letters 29th Vol. 32 No. 28, August 1996.
2
Outline
Features of LDPC Codes History of LDPC Codes Some Fundamentals A Simple Example
Properties of LDPC Codes How to construct?
Decoding Concept of Message Passing Sum Product Algorithm Concept of Iterative Decoding Channel Transmission Decoding Algorithm Decoding Example
Performance Cost
3
Shannon Limit (1/2)
Shannon Limit:
Describes the theoretical maximum information
transfer rate of the communication channel (channel
capacity), for a particular levels of noise.
Given:
A noisy channel with channel capacity C
Information transmitted at a rate R
4
Shannon Limit (2/2)
If R < C, there exist codes that allow the probability of
error at the receiver to be made arbitrarily small.
Theoretically, it is possible to transmit information nearly without
error at any rate below a limiting rate, C.
If R > C, all codes will have a probability of error greater
than a certain positive minimal level.
This level increases as the rate increases.
Information cannot be guaranteed to be transmitted reliably
across a channel at rates beyond the channel capacity.
5
Features of LDPC Codes
Low-Density Parity-Check Codes (LDPC Codes) is an error
correcting code.
A method of transmitting a message over a noisy transmission channel.
Approaching Shannon capacity
For example, 0.3 dB from Shannon limit (1999).
An closer design from (Chung:2001), 0.0045 dB away from capacity.
Linear decoding complexity in time.
Suitable for parallel implementation.
6
History of LDPC Codes
Also known as Gallager codes, by whom devel
oped the LDPC concept in his doctoral dissertati
on at MIT in 1960.
Long time being ignored due to requirement of hi
gh complexity computation.
In 90’s, Rediscovered by MacKay and Richardso
n/Urbanke.
7
Some Fundamentals
The structure of a linear block code is described by the generator matrix G or the parity-check matrix H.
H is sparse. Very few 1's in each row and column.
Regular LDPC codes: Each column of H had a small weight (e.g. 3), and the weight per
row was also uniform. H was constructed at random subject to these constraints.
Irregular LDPC codes: The number of 1's per column or row is not constant. Usually irregular LDPC codes outperforms regular LDPC codes.
8
A Simple Example (1/6)
Variable Node: box with an '=' sign. Check Node: box with a '+' sign. Constraints:
All lines connecting to a variable node have the same value, and all values connecting to a check node must sum, modulo two, to zero (they must sum to an even number).
www.wikipedia.org
9
A Simple Example (2/6)
There are 8 possible 6 bit strings which correspond to valid codewords: 000000, 011001, 110010, 111100, 101011, 100101, 001110, 010111.
This LDPC code fragment represents a 3-bit message encoded as 6 bits. The redundancy is been used to aid in recovering from channel errors.
10
A Simple Example (3/6)
The parity-check matrix representing this graph fragment is:
11
A Simple Example (4/6) Consider the valid codeword: 101011
Been transmitted across a binary erasure channel, and received with
the 1st and 4th bit erased: ?01?11
Belief Propagation is particularly simple for the binary erasure channel.
Consists of iterative constraint satisfaction.
12
A Simple Example (5/6) Consider the erased codeword: ?01?11
In this case: The first step of belief propagation is to realize that the 4th bit
must be 0 to satisfy the middle constraint. Now that we have decoded the 4th bit, we realize that the 1st bit
must be a 1 to satisfy the leftmost constraint.
13
A Simple Example (6/6) Thus we are able to iteratively decode the
message encoded with our LDPC Code. We can validate this result by multiplying the
corrected codeword r by the parity-check matrix H: Because the outcome z (the syndrome) of this operation is
the 3 x 1 zero vector, we have successfully validated the resulting codeword r.
(mod 2)
14
Properties of LDPC Codes
The structure of a linear block code is completely described by the generator matrix G or the parity-check matrix H. r = Gm
r: codeword, G: generator matrix, m: input message
HG = 0 Hr = 0
A Low-Density Parity-Check Code is a code which has a very sparse, random parity check matrix H. Typically, column weight is around 3 or 4.
15
How to construct? (1/4)
Construction 1A: An M by N matrix is created at random with:
weight per column t (e.g., t = 3).
weight per row as uniform as possible.
overlap between any two columns no greater than 1. No cycle graph.
16
How to construct? (2/4)
Construction 2A: Up to M/2 of the columns: designated weight 2 columns
Such that there is zero overlap between any pair of columns.
The remaining columns: made at random with weight 3.
The weight per row: as uniform as possible.
Overlap between any two columns of the entire matrix no greater than 1.
17
How to construct? (3/4)
Construction 1B and 2B: A small number of columns are deleted from a matrix
produced by Constructions 1A and 2A.
The bipartite graph has no short cycles of length less than some length l.
18
How to construct? (4/4)
Above constructions do not ensure all the rows of the matrix are linearly independent. The M by N matrix created is the parity matrix of a
linear code with rate at least R = K/N, where K = N - M.
The generator matrix of the code can be created by Gaussian elimination.
19
Decoding
Decoding problem is to find the most probable v
ector x iteratively, such that:
Hx mod 2 = 0
Gallager's algorithm may be viewed as an appro
ximate Belief Propagation (BP) algorithm.
Message Passing (MP).
20
Concept of Message Passing (1/4)
How to know the number of soldiers stand in
line?
Each soldier plus 1 to the number hearing from the
neighbor on one side, then pass the result to the
neighbor on the opposite side.
21
Concept of Message Passing (2/4)
In the beginning, every soldier knows that: There exists at least one soldier (himself).
Intrinsic Information.
Figure 1. Each node represent a soldier & Local rule: +1
22
Concept of Message Passing (3/4)
Start from the leftmost and rightmost soldiers.Extrinsic Information.
Figure 2. Extrinsic Information Flow.
23
Concept of Message Passing (4/4)
The total number = [left + right] + [oneself]
Overall Information =
Extrinsic Information + Intrinsic Information.
Figure 3. Overall Information Flow.
24
Sum Product Algorithm (1/4)
During decoding, apply Sum Product Algorithm to
derive: Extrinsic Information (Extrinsic Probability)
Overall Information (Overall Probability)
In this paper, channel transmission: BPSK through AWGN.
Binary Phase-Shift Keying.
Additive White Gaussian Noise.
25
Sum Product Algorithm (2/4)
A simple example: If local rule:
26
Sum Product Algorithm (3/4) Form 1. valid codeword for rule:
(Overall Probability with m2 = 0):
P10P20P30 + P11P20P31
(Extrinsic Probability with m2 = 0):
P10P30 + P11P31
27
Sum Product Algorithm (4/4)
Likelihood Ratio:
, where
( with m1 or m2 is similar )
28
Concept of Iterative Decoding (1/6) A simple example:
If received a codeword with Likelihood Ratio:
( “10110” is an invalid codeword )
29
Concept of Iterative Decoding (2/6) A simple example:
Calculate Extrinsic Probability by Check Nodes:
30
Concept of Iterative Decoding (3/6) A simple example:
We then obtain the Overall Probability of 1st. Round:
( “10010” is an valid codeword )
31
Concept of Iterative Decoding (4/6) A simple example:
2nd. Round :
32
Concept of Iterative Decoding (5/6) A simple example:
2nd. Round :
33
Concept of Iterative Decoding (6/6) A simple example:
We then obtain the Overall Probability of 2nd. Round:
( “10010” is an valid codeword )
34
Channel Transmission
BPSK through AWGN:
A Gaussian channel with binary input ±a and additive
noise of variance σ2 = 1.
t: BPSK-modulated signal,
AWGN channel:
Posterior probability:
35
Decoding Algorithm (1/6)
We refer to the elements of x as bits, to the rows
of H as checks, and denote:
the set of bits n that participate in check m by N(m):
the set of checks in which bit n participates by M(n):
a set N(m) with bit n excluded by N(m)\n
36
Decoding Algorithm (2/6)
37
Decoding Algorithm (3/6)
N(1) = {1, 2, 3, 6, 7, 10}, N(2) = {1, 3, 5, 6, 8, 9}, … etc
M(1) = {1, 2, 5}, M(2) = {1, 4, 5}, … etc N(1)\1 = {2, 3, 6, 7, 10}, N(2)\3 = {1, 5, 6, 8, 9}, … etc
M(1)\1 = {2, 5}, M(2)\4 = {1, 5}, … etc
38
Decoding Algorithm (4/6) The algorithm has two parts, in which quantities qmn and rmn associate
d with each non-zero element in the H matrix are iteratively updated:
qxmn: the probability that bit n of x is x, given the information obtained via check
s other than check m.
rxmn: the probability of check m being satisfied if bit n of x is considered fixed at
x, and other bits have a separable distribution given by the probabilities
.
The algorithm would produce the exact posterior probabilities of all the bits of t
he bipartite graph defined by the matrix H.
39
Decoding Algorithm (5/6) Initialization:
q0mn and q1
mn are initialized to the values f0n and f1
n
Horizontal Step:
Define:
For each m, n:
Set:
40
Decoding Algorithm (6/6) Vertical Step:
For each n and m, and for x = 0, 1 we update:
We can also update the “pseudoposterior probabilities” q0n and q
1n, given by:
, where αmn is chosen such that q0mn+ q1
mn = 1
41
Decoding Example (1/10)
42
Decoding Example (2/10)
43
Decoding Example (3/10) BPSK through AWGN:
Simulated a Gaussian channel with binary input ±a and additive
noise of variance σ2 = 1.
t: BPSK-modulated signal,
AWGN channel:
44
Decoding Example (4/10) BPSK through AWGN:
Posterior probability:
45
Recall: Decoding Algorithm Input: The Posterior Probabilities pn(x).
Initialization: Let qmn(x) = pn(x).
1. Horizontal Step:
(a). Form the δq matrix from qmn(0) - qmn(1) (at sparse non-zero location).
(b). For each nonzero location (m, n), let δrmn be the product of δq matrix elements along its row, excluding the (m, n) position.
(c). Let rmn (1) = (1 -δrmn ) / 2, rmn (0) = (1 +δrmn ) / 2
2. Vertical Step:
For each nonzero location (m, n) let qmn(0) be the product along its column, excluding the (m, n) position, times pn(0).
Similarly for qmn(1). Then normalize.
46
Decoding Example (5/10) Initialization: Let qmn(x) = pn(x).
47
Decoding Example (6/10) Iteration 1: Horizontal Step:
(a)
(b)
48
Decoding Example (7/10) Iteration 1: Horizontal Step:
(c)
49
Decoding Example (8/10) Iteration 1: Vertical Step:
(a)
(b)
50
Decoding Example (9/10) Iteration 1: After Vertical Step:
Hc mod 2 ≠ 0
Recall: update pseudoposterior probabilities qn(1) and qn(0), given by:
51
Decoding Example (10/10) After two more iterations:
However, a failure is declared if some maximum number of iterations (e.g., 100) occurs without a valid decoding.
Hc mod 2 = 0Successfully decoded
52
Performance (1/4) Compares the performance of LDPC codes with text
book codes and with state of the art codes.
Textbook codes: The curve (7, 1/2) shows the performance of a rate 1/2 convolutional c
ode with constraint length 7
de facto standard for satellite communications.
The curve (7, 1/2)C shows the performance of the concatenated code
composed of the same convolutional code and a Reed-Solomon code.
53
Performance (2/4) State of the art:
The curve (15,1/4)C shows the performance of concat
enated code developed at JPL based on a constraint l
ength 15, rate 1/4 convolutional code.
Extremely expensive and computer intensive.
The curve Turbo shows the performance of the rate 1/
2 Turbo code.
54
Performance (3/4) LDPC codes: from left to right, the codes had the following
parameters (N; K; R): (29507; 9507; 0:322) (Construction 2B);
(15000; 5000; 0:333) (Construction 2A);
(14971; 4971; 0:332) (Construction 2B);
(65389; 32621; 0:499) (Construction 1B);
(19839; 9839; 0:496) (Construction 1B);
(13298; 3296; 0:248) (Construction 1B);
(29331; 19331; 0:659) (Construction 1B).
55
Performance (4/4)
Figure LDPC codes’ performance over Gaussian channel compared with
that of standard textbook codes and state of art codes.
56
Cost (1/2) In a brute force approach, the time to create the code scales
as N3, where N is the block size.
Encoding time scales as N2
But encoding involves only binary arithmetic so for the block lengths
studied here it takes considerably less time than the simulation of the
Gaussian channel.
It may be possible to reduce encoding time using sparse
matrix techniques.
57
Cost (2/2) Decoding involves approximately 6Nt floating point
multiplies per iteration, so the total number of operations
per decoded bit (assuming 20 iterations) is about 120t/R,
independent of block length.
(6Nt * 20 ) / K = 120t * (N/K) = 120t / R
For the codes presented here, this is about 800
operations.
58
Thank you
Sincerely thanks for your listening.
59
How It Works
60
Probabilities Used
61
Vertical Step I: Single Tier Approximate Subset
62
Vertical Step II: Multi Tier Approximate Subset
63
Horizontal Step