A Tutorial on Low Density Parity-Check Codes

Tuan Ta
The University of Texas at Austin
Abstract Low density parity-check codes are one of the hottest topics in coding theory nowadays.
Equipped with very fast encoding and decoding algorithms (probabilistically,) LDPC are very attractive both
theoretically and practically. In this paper, I present a throughout view of LDPC. I discuss in detail LDPCs
popular decoding algorithm: belief propagation. I also present some noticeable encoding algorithms including
Richardson and Urbankes lower triangular modification that gives rise to a linear encoding time. I show that
irregular LDPC perform better than regular LDPC thus are more desirable.
Index Termsbelief propagation, density evolution, Gallager codes, irregular codes, low density paritycheck codes

is introduced as an example to illustrate the

behavior of a linear block code. I also discuss the
use of Tanner graph as equivalence to a parity-check
matrix for linear block codes. Section II is
concluded by the definition of LDPC. In section III,
I discuss how to decode an LDPC using belief
propagation algorithm. Both the hard-decision
decoder and the soft-decision decoder are presented.
An example is given to illustrate the use of the
algorithm. Section IV talks about how to encode an
LDPC in linear time. Two approaches are presented,
the accumulate approach, and Richardson and
Urbankes lower triangular modification approach.
Section V discusses irregular LDPC and shows that
irregular codes perform better than regular codes.
Three designs are considered: Lubys, Richardsons
and Chungs. Section V shows that it is possible for
irregular LDPC to come as close as 0.0045 dB to
Shannon capacity.

I. INTRODUCTION
OW density parity-check code (LDPC) is an
error correcting code used in noisy
communication channel to reduce the probability of
loss of information. With LDPC, this probability
can be reduced to as small as desired, thus the data
transmission rate can be as close to Shannons limit
as desired.
LDPC was developed by Robert Gallager in his
doctoral dissertation at MIT in 1960 [1]. It was
published 3 years latter in MIT Press. Due to the
limitation in computational effort in implementing
the coder and decoder for such codes and the
introduction of Reed-Solomon codes, LDPC was
ignored for almost 30 years. During that long
period, the only notable work done on the subject
was due to R. Michael Tanner in 1981 [2] where he
generalized LDPC codes and introduced a graphical
representation of the codes later called Tanner
graph. Since 1993, with the invention of turbo
codes, researchers switched their focus to finding
low complexity code which can approach Shannon
channel capacity. LDPC was reinvented with the
work of Mackay [3], [4] and Luby [5]. Nowadays,
LDPC have made its way into some modern
applications such as 10GBase-T Ethernet, WiFi,
WiMAX, Digital Video Broadcasting (DVB).
Before discussing LDPC, I present a brief review
of error correcting code in section II. I pay the most
attention to linear block code. (7, 4) Hamming code

II. ERROR CORRECTING CODES

A. Introduction
In communication, errors can occur due to a lot
of reasons: noisy channel, scratches on CD or DVD,
power surge in electronic circuits, etc. It is often
desirable to detect and correct those errors. If no
additional information is added to the original
message, errors can turn a legal message (bit
pattern) into another legal bit pattern. Therefore,
redundancy is used in error-correcting schemes. By
1

adding redundancy, a lot of bit patterns will become

illegal. A good coding scheme will make an illegal
pattern caused by errors to be closer to one of the
legal patterns than others.
A metric used to measure the closeness
between two bit patterns is Hamming distance. The
Hamming distance between two bit patterns is the
number of bits that are different. For example, bit
pattern 1100 and 0100 differ by one bit (the 1st bit),
thus have Hamming distance of one. Two identical
bit patterns have Hamming distance of zero.
A parity bit is an additional bit added to the bit
pattern to make sure that the total number of 1s is
even (even parity) or odd (odd parity). For example,
the information message is 01001100 and even
parity is used. Since the number of 1s in the
original message is 3, a 1 is added at the end to
give the transmitted message of 010011001. The
decoder counts the number of 1s (normally done by
exclusive OR the bit stream) to determine if an error
has occurred. A single parity bit can detect (but not
correct) any odd number of errors. In the previous
example, the code rate (number of useful
information bits/total number of bits) is 8/9. This is
an efficient code rate, but the effectiveness is
limited. Single parity bit is often used in scenarios
where the likelihood of errors is small and the
receiver is able to request retransmission.
Sometimes it is used even when retransmission is
not possible. Early IBM Personal Computers
employ single bit parity and simple crash when an
error occurs [6].

have 3 parity equations, which can identify up to

23 = 8 error conditions. One condition identifies no
error, so seven would be left to identify up to seven
places of single error. Therefore, we can detect and
correct any single error in a 7-bit word. With 3
parity bits, we have 4 bits left for information. Thus
this is a (7, 4) block code. In a (7, 4) Hamming
code, the parity equations are determined as follow:
The first parity equation checks bit 4, 5, 6, 7
The second parity equation checks bit 2, 3, 6, 7
The third parity equation checks bit 1, 3, 5, 7
This rule is easy to remember. The parity
equations use the binary representation of the
location of the error bit. For example, location 5 has
the binary representation of 101, thus appears in
equation 1 and 3. By applying this rule, we can tell
which bit is wrong by reading the value of the
binary combination of the result of the parity
equations, with 1 being incorrect and 0 being
correct. For example, if equation 1 and 2 are
incorrect and equation 3 is correct, we can tell that
the bit at location 6 (110) is wrong.
At the encoder, if location 3, 5, 6, 7 contain the
original information and location 1, 2, 4 contain the
parity bits (locations which are power of 2) then
using the first parity equation and bits at location 5,
6, 7, we can calculate the value of the parity bit at
location 4 and so on.
(7, 4) Hamming code can be summarized in the
following table [7]
TABLE 1

B. Linear block code

If a code uses n bits to provide error protection to
k bits of information, it is called a (n, k) block code.
Often times, the minimum Hamming distance d
between any two valid codewords is also included
to give a (n, k, d) block code. An example of block
codes is Hamming code. Consider the scenario
where we wish to correct single error using the
fewest number of parity bits (highest code rate).
Each parity bit gives the decoder a parity equation
to validate the received code. With 3 parity bits, we

(7, 4) HAMMING CODE FULL TABLE

1

p1

p2

d1

p3

d2

d3

d4

Equation
corresponds to p1

Equation
corresponds to p2

Equation
corresponds to p3

Bit number

1
1

G = 0
0

0
0

In table 1, p denotes parity bits, d denotes data

bits.
Removing the parity bits, we have
TABLE 2
(7, 4) HAMMING CODE ABBREVIATED TABLE

d1

d2

d3

d4

p1

p2

p3

by
For parity bit locations: use associated data
bits (from Table 2)
For data bit locations: put a 1 for the position
of the data bit, the rest are 0s
For the generator matrix of (7, 4) Hamming code
above, bit location 1 (1st row) is a parity bit, thus we
use row 1 from table 2 (1101). Bit location 5 (5th
row) is a data bit, and bit 5 is data bit number 2,
thus we set bit 2 to 1 (0100).
If the message is
1
0
p=
1

1

(1)

Given the received codeword y, the syndrome

vector is
z = Hy

(2)

then the codeword will be

If z = 0 then the received codeword is error-free,

else the value of z is the position of the flipped bit.
For the (7, 4) Hamming code, the parity-check
matrix is

1 0 1 0 1 0 1
H = 0 1 1 0 0 1 1
0 0 0 1 1 1 1

(4)

H is driven straight from Table 1. G is obtained

For larger dimension block codes (larger n and

k), a matrix representation of the codes is used. This
representation includes a generator matrix, G, and a
parity-check matrix, H. Given a message p, the
codeword will be the product of G and p with
entries modulo 2:
c = Gp

1 0 1
0 1 1
0 0 0

1 1 1
1 0 0

0 1 0
0 0 1

1
1

c = Gp = 0
0

0
0

(3)

and the generator matrix is

1
2 0

3 1
1
1
0 1 1
0

1 = 2 = 0
1
0 0 0
1
0 1 0
1 1
1 1
0 0 1

1
0
0
1
1

0
1
0
1
0

If no bit is flipped during transmission, in other

words, y = c. Then the syndrome vector is

0
1

1 0 1 0 1 0 1 1 2 0

z = Hy = 0 1 1 0 0 1 1 0 = 4 = 0
0 0 0 1 1 1 1 0 2 0

1
1

shown below, with check nodes on the left and

message nodes on the right.
c1
c2
c1+c3+c5+c7= f
0
c2+c3+c6+c7= f
0
c4+c5+c6+c7= f
0

If the 6th bit is flipped,

0
1

1

y = 0
0

0
1

c3
c4
c5
c6
c7

Figure 1: Tanner graph for (7, 4) Hamming code

As seen above, there is an edge connects a check
node with a message node if the message node is
included in the check nodes equation. From a
Tanner graph, we can deduce a parity-check matrix
by putting a 1 at position (i, j) if there is an edge
connecting fi and cj. The code defined by a Tanner
graph (or a parity-check matrix) is the set of vectors
c = (c1,,cn) such that HcT = 0. In other words, the
code forms the null space of H.

then
0
1

1 0 1 0 1 0 1 1 2 0

z = Hy = 0 1 1 0 0 1 1 0 = 3 = 1
0 0 0 1 1 1 1 0 1 1

0
1

D. Low Density Parity-check Code (LDPC)

Any linear code has a bipartite graph and a
parity-check matrix representation. But not all linear
code has a sparse representation. A n m matrix is
sparse if the number of 1s in any row, the row
weight wr, and the number of 1s in any column, the
column weight wc, is much less than the dimension
(wr << m, wc << n). A code represented by a sparse
parity-check matrix is called low density paritycheck code (LDPC). The sparse property of LDPC
gives rise to its algorithmic advantages. An LDPC
code is said to be regular if wc is constant for every
column, wr is constant for every row and
n
wr = wc . An LDPC which is not regular is called
m
irregular.

Reading z from the bottom up (higher position

first), we see the flipped bit is indeed 6 (110).
C. Tanner graph
A very useful way of representing linear block
codes is using Tanner graph. Tanner graph is a
bipartite graph, which means the graph is separated
into two partitions. These partitions are called by
different names: subcode nodes and digit nodes,
variable nodes and check nodes, message nodes and
check nodes. I will call them message nodes and
check nodes from now on. Tanner graph maps
directly to the parity-check matrix H of a linear
block code, with check nodes represent the rows of
H. The Tanners graph of (7, 4) Hamming code is
4

to be correct for them. For example,

message node c2 receives a 1 (y2=1), so it
sends a message containing 1 to check nodes
f1 and f2. Table 3 illustrates this step.

III. DECODING
Different authors come up independently with
more or less the same iterative decoding algorithm.
They call it different names: the sum-product
algorithm, the belief propagation algorithm, and the
message passing algorithm. There are two
derivations of this algorithm: hard-decision and
soft-decision schemes.

2.

A. Hard-decision Decoder

In the second step, every check nodes

calculate a response to their connected
message nodes using the messages they
receive from step 1. The response message
in this case is the value (0 or 1) that the
check node believes the message node has
based on the information of other message
nodes connected to that check node. This
response is calculated using the parity-check
equations which force all message nodes
connect to a particular check node to sum to
0 (mod 2).
In Table 3, check node f1 receives 1 from
c4, 0 from c5, 1 from c8 thus it believes c2
has 0 (1+0+1+0=0), and sends that
information back to c2. Similarly, it receives
1 from c2, 1 from c4, 1 from c8 thus it
believes c5 has 1 (1+1+1+1=0), and sends 1
back to c5.
At this point, if all the equations at all
check nodes are satisfied, meaning the
values that the check nodes calculate match
the values they receive, the algorithm
terminates. If not, we move on to step 3.

Figure 2: Belief propagation example code

In [8], Leiner uses a (4, 8) linear block code to
illustrate the hard-decision decoder. The code is
represented in Figure 2, its corresponding paritycheck matrix is
0
1
H=
0

1
1
0
0

0
1
1
0

1
0
0
1

1
0
0
1

0
1
1
0

0
0
1
1

1
0
1

3.

In this step, the message nodes use the

messages they get from the check nodes to
decide if the bit at their position is a 0 or a 1
by majority rule. The message nodes then
send this hard-decision to their connected
check nodes. Table 4 illustrates this step. To
make it clear, let us look at message node c2.
It receives 2 0s from check nodes f1 and f2.
Together with what it already has y2 = 1, it
decides that its real value is 0. It then sends
this information back to check nodes f1 and
f2.

4.

Repeat step 2 until either exit at step 2 or a

certain number of iterations has been passed.

(5)

An error free codeword of H is

c = [1 0 0 1 0 1 0 1]T. Suppose we receive
y = [1 1 0 1 0 1 0 1]T. So c2 was flipped.
The algorithm is as follow:
1. In the first step, all message nodes send a
message to their connected check nodes. In
this case, the message is the bit they believe
5

In this example, the algorithm terminates

right after the first iteration as all paritycheck equations have been satisfied. c2 is
corrected to 0.

the messages are the conditional probability that the

received bit is a 1 or a 0 given the received vector y.
Let Pi = Pr [ci = 1 | y ] be the conditional
probability that ci is a 1 given the value of y. We
have Pr[ci = 0 | y ] = 1 Pi .
Let qij(l ) be the message sent by message node ci

TABLE 3
CHECK NODES ACTIVITIES FOR HARD-DECISION
DECODER FOR CODE OF FIGURE 2
check
nodes
f1

f2

f3

f4

to check node fj at round l. Every message contains

a pair qij(l ) (0) and qij(l ) (1) which stands for the
amount of belief that yi is 0 or 1,
qij( l ) (0) + qij( l ) (1) = 1 . In particular, qij( 0 ) (1) = Pi and

activities
receive

c2 1

c4 1

c5 0

c8 1

send

0 c2

0 c4

1 c5

0 c8

receive

c1 1

c2 1

c3 0

c6 1

send

0 c1

0 c2

1 c3

0 c6

receive

c3 0

c6 1

c7 0

c8 1

send

0 c3

1 c6

0 c7

1 c8

receive

c1 1

c4 1

c5 0

c7 0

send

1 c1

1 c4

0 c5

0 c7

qij( 0 ) (0) = 1 Pi .
Similarly, let r ji(l ) be the message sent by check
node fj to message node ci at round l. Every message
contains a pair r ji(l ) (0) and r ji(l ) (1) which stands for
the amount of belief that yi is 0 or 1. We also have
r ji(l ) (0) + r ji( l ) (1) = 1 . r ji(l ) (0) is also the probability
that there are an even number of 1s on all other
message nodes rather than ci.
First, let consider the probability that there are an
even number of 1s on 2 message nodes. Let q1 be
the probability that there is a 1 at message node c1
and q2 be the probability that there is a 1 at message
node c2. We have

TABLE 4
MESSAGE NODES DECISIONS FOR HARDDECISION DECODER FOR CODE OF FIGURE 2
message
nodes

yi

messages from
check nodes

decision

c1

f2 0

f4 1

c2

f1 0

f2 0

c3

f2 1

f3 0

c4

f1 0

f4 1

c5

f1 1

f4 0

c6

f2 0

f3 1

c7

f3 0

f4 0

c8

f1 1

f3 1

Pr[c1 c 2 = 0] = q1 q 2 + (1 q1 )(1 q 2 )
= 1 q1 q2 + 2q1q2
1
(2 2q1 2q 2 + 4q1 q 2 )
2
1
= [1 + (1 2q1 )(1 2q2 )] = q (6)
2

Now consider the probability that there are an

even number of 1s on 3 message nodes, c1, c2 and
c3. Note that 1 q is the probability that there are an
odd number of 1s on c1 and c2.
Pr[(c1 c2 ) c3 = 0]

B. Soft-decision Decoder
The soft-decision decoder operates with the same
principle as the hard-decision decoder, except that

1
= [1 + (1 2(1 q ))(1 2q3 )]
2
6

1
= [1 + (1 2q1 )(1 2q2 )(1 2q3 )]
2

As seen above, this algorithm uses a lot of

multiplications which are costly to implement.
Another approach is to use logarithmic likelihood
ratio. Let

(7)

In general

1 1 n
Pr[c1 ... c n = 0] = + (1 2q i )
2 2 i =1

Pr[ci = 0 | y ] 1 Pi
=
Li =
Pi
Pr[ci = 1 | y ]

(8)

(15)

Therefore, the message that fj sends to ci at round

Pr[ci = 0 | y ]

l i = ln Li = ln
c
Pr[
=
1
|
y
]
i

l is
1 1
r ji( l ) (0) = + (1 2q i('lj1) (1))
2 2 i 'V j i

(9)

r ji( l ) (1) = 1 r ji(l ) (0)

(10)

Li is the likelihood ratio and li is the log

likelihood ratio at message node ci. Using log ratio
turns multiplications into additions which are much
cheaper to implement in hardware.
With log likelihood ratio, we have

where Vj is the set of all message nodes connected

to check node fj.
The message that ci sends to fj at round l is

q (ijl ) (0) = k ij (1 Pi )
q (ijl ) (1) = k ij Pi

j 'Ci j

j 'Ci j

( l 1)
j 'i

( l 1)
j 'i

( 0)

Pi =

(11)

(1)

(12)
(l )

m ij

1 Pi
= ln (l )
= ln
q ij (1)
Pi
q ij(l ) (0)

= li +

(17)

r j('li1) (0)
(l 1) (1)
j 'Ci j r j 'i

( l 1)
j 'i

(18)

j 'Ci j

qij(0) + qij(1) = 1

From (9) and (10), the message that fj sends to ci

at round l is

At each message node, the following calculations

are made

Q i(l ) (0) = k i (1 Pi ) r ji( l ) (0)

(13)

jCi

m (jil ) = ln

Q i (1) = k i Pi r ji (1)
(l )

(l )
ji

r ( 0)
r ji(l ) (1)

= ln

(14)

jCi

Qi(l ) is the effective probability of 0 and 1 at

1 1
+ (1 2q (i 'lj1) (1))
2 2 i 'V j i
1 1
(1 2q (i 'lj1) (1))
2 2 i 'V j i

1+

message node ci at round l. If Q (1) > Q (0) then

(l )
i

1
1 + Li

From (11) and (12), the message that ci sends to

fj at round l is

where Ci is the set of all check nodes connected to

message node ci.
The constant kij is chosen so that

(l )

(16)

(l )
i

i 'V j i

= ln

the estimation at this point is ci = 1, otherwise ci =

0. If this estimation satisfies the parity-check
equations then the algorithm terminates. Else, the
algorithm runs through a predetermined number of
iterations.

tanh(

tanh(

i 'V j i

mi('lj1)
2
mi('lj1)
2

)
(19)
)

We have (19) because from (18),

e
7

mi ' j

1 q i ' j (1)
qi ' j (1)

(20)

Thus

2.
q i ' j (1) =

1
1+ e

and

1 2qi ' j (1) =

mi ' j

mi ' j

1
+1

= tanh(

mi ' j
2

(22)

Equation (13), (14) turn into

(l )
i

= ln

Q i(l ) (0)
Q i( l ) (1)

=l

(0)
i

+ m

(l )
ji

is the

calculated by a method called density

evolution. For a message alphabet of size q,
Pe (l ) can be expressed by means of q 1

(23)

jCi

coupled recursive functions.

If l i( l ) > 0 then ci = 0 else ci = 1.

3.

In practice, belief propagation is executed for a

maximum number of iterations or until the passed
likelihoods are closed to certainty, whichever comes
first. A certain likelihood is li = , where Pi = 0
for li = and Pi = 1 for li = -.
One very important aspect of belief propagation
is that its running time is linear to the code length.
Since the algorithm traverses between check nodes
and message nodes, and the graph is sparse, the
number of traversals is small. Moreover, if the
algorithm runs a fixed number of iterations then
each edge is traversed a fixed number of times, thus
the number of operations is fixed and only depends
on the number of edges. If we let the number of
check nodes and message nodes increases linearly
with the code length, the number of operations
performed by belief propagation also increases
linearly with the code length.

There exists a channel parameter * with the

following property: if < * then
lim Pe (l ) = 0 , else if > * then there
l

exists a constant () > 0 such that

Pe (l ) > () for all l 1. Here 2 is the
noise variance in the channel. In other
words, * sets the limit to which belief
propagation decodes successfully.
IV. ENCODING
If the generator matrix G of a linear block code is
known then encoding can be done using equation
(1). The cost (number of operations) of this method
depends on the Hamming weights (number of 1s)
of the basis vectors of G. If the vectors are dense,
the cost of encoding using this method is
proportional to n2. This cost becomes linear with n
if G is sparse.
However, LDPC is given by the null space of a
sparse parity-check matrix H. It is unlikely that the
generator matrix G will also be sparse. Therefore
the straightforward method of encoding LDPC
would require number of operations proportional to
n2. This is too slow for most practical applications.
Therefore it is desirable to have encoding
algorithms that run in linear time. This section will
look at two approaches to achieve that goal.

C. Performance of Belief Propagation

A parameter to measure the performance of
belief propagation is the expected fraction of
incorrect messages passed at the lth iteration,
Pen (l ) . In [9], Richardson and Urbanke show that
1.

Pe (l )

expected fraction of incorrect messages

passed at round l assuming that the graph
does not contain any cycle of length 2l or
less. The assumption is to ensure that the
decoding neighborhoods become tree-like
so that the messages are independent for l
rounds [10]. The value Pe (l ) can be

(21)

mi ' j

lim Pen (l ) = Pe (l ), where

For any > 0, the probability that the actual

fraction of incorrect messages passed among
any instance at round l that lies outside
( Pen (l ) , Pen (l ) + ) converges to zero
exponentially fast with n.

A. Accumulate approach
The first approach modifies LDPC code so it has
8

an inherited fast encoding algorithm. In this case, degree d having all its connected check nodes
we assign a value to each check node which is equal erased is pd. This probability is conditioned on the
to the sum of all message nodes that are connected event that the degree of the message node is d.
to it. (It would be more appropriate to talk about Since the graph is created randomly, the probability
information nodes and redundant nodes, but for that a message node has all its connected check
consistence of notation, I will use message nodes nodes erased is M d p d , which is a constant
d
and check nodes.) The number of summations
needed to calculate the value of a check node is independent of the length of the code. Therefore, no
bounded by the number of message nodes connected algorithm can recover the value of that message
to a check node. This is a constant when the code is node.
sparse. The message consists of the message nodes B. Lower triangular modification approach
appended by the values of the check nodes. To
In [11], Richardson and Urbanke propose an
illustrate the difference between this modified encoding algorithm that has effectively linear
version of LDPC and the original version, consider running time for any code with a sparse parity-check
Figure 2. If Figure 2 represents an original LDPC matrix. The algorithm consists of two phases:
then c1, c2, c3, c4 are information bits and c5, c6, c7, preprocessing and encoding.
c8 are parity bits which have to be calculated from
In the preprocessing phase, H is converted into
c1, c2, c3, c4 by solving the parity-check equations in the form shown in Figure 3 by row and column
f1, f2, f3, f4. The code rate is 4/8 = 1/2. Now if permutations.
Figure 2 represents a modified LDPC, then all of c1,
c2, c3, c4 ,c5, c6, c7, c8 are information bits; while f1,
n-k
g
k-g
f2, f3, f4 are redundant bits calculated from c1,,c8.
f1 is connected to c2, c4, c5, c8 so f1 = c2 + c4 + c5 +
1
c8 and so on. The codeword in this case is
1
T
1
[c1 c8 f1 f4] . The code rate is 8/12 = 2/3.
k-g
1
Although this approach gives a linear encoder, it k
1
causes a major problem at the decoder. In case the
g
channel is erasure, the value of the check nodes
might be erased. On the contrary, the check nodes of
the original LDPC are dependencies, not values,
n
thus they are not affected by the channel. In other
Figure 3: Parity-check matrix in approximately
words, a check node defines a relationship of its
lower triangular form
connected message nodes. This relationship comes
straight from the parity-check matrix. The fact that
In matrix notation,
in modified LDPC, the values of check nodes can be
A B T
erased creates a lower bounded for the convergence
H=
(24)

of any decoding algorithm. In [r4], Shokrollahi

C D E
proves the existence of such lower bound. Suppose
that the channel is erasure with the erasure where T has a lower triangular form with all
probability p. Then an expected p-fraction of the diagonal entries equal to 1. Since the operation is
message nodes and an expected p-fraction of the done by row and column permutations and H is
check nodes will be erased. Let Md be the fraction sparse, A, B, C, D, E, T are also sparse. g, the gap,
of message nodes of degree d (connected with d measures how close H can be made, by row and
check nodes.) The probability of a message node of column permutations, to a lower triangular matrix.
9

TABLE 5
COMPUTING p1 USING RICHARDSON AND URBANKES ENCODING ALGORITHM

Operation
AsT
T1[AsT]
E[T1AsT]
CsT
[ET1AsT]+ [CsT]
1(ET1AsT + CsT)

Comment
Multiplication by sparse matrix
Back-substitution, T is lower triangular
Multiplication by sparse matrix
Multiplication by sparse matrix
Addition
Multiplication by gg matrix

Complexity
O(n)
O(n)
O(n)
O(n)
O(n)
O(n+g2)

TABLE 6
COMPUTING p2 USING RICHARDSON AND URBANKES ENCODING ALGORITHM

Operation
AsT
Bp1T
[AsT] + [Bp1T]
T1(AsT + Bp1T)

Comment
Multiplication by sparse matrix
Multiplication by sparse matrix
Addition
Back-substitution, T is lower triangular

Multiple H from the left by

I
ET 1

0
I

(ET1A + C)sT + (ET1B + D)p1T = 0 (29)

The procedure to find p1 and p2 is summarized in
Table 5 and 6.
Define: = ET1B + D and assume for the
moment that is nonsingular. Then

(25)

we get

A
B
T

ET 1 A + C ET 1B + D 0

p1T = 1(ET1A + C)sT

(26)

= 1(ET1AsT + CsT)

Let the codeword c = (s, p1, p2) where s is the

information bits, p1 and p2 are the parity-check bits,
p1 has length g, p2 has length k g.
By HcT = 0, we have
I
ET 1

(30)

First we compute AsT. Since A is sparse, this is

done in linear time O(n). Then we compute
T1[AsT] = yT. Since [AsT] = TyT and T is lower
triangular, by back-substitution we can compute yT
in linear time. The calculations EyT and CsT are
also done in O(n) as E, C are sparse. Now we have
(ET1AsT + CsT) = zT computed in O(n). Since
is gg, p1 is computed from (30) in O(n+g2).
From (28), p2T = T1(AsT + Bp1T). The steps to
calculate p2 are quite similar and are shown in Table
6. We see that p2 can be computed in O(n).
As seen in Table 5 and Table 6, c can be
computed in O(n+g2). Richardson and Urbanke

0 T
Hc
I

s
A
B
T

=
p1 = 0 (27)
1
1
ET A + C ET B + D 0 p
2
Therefore

AsT + Bp1T + Tp2T = 0

Complexity
O(n)
O(n)
O(n)
O(n)

(28)
10

prove in [11] that the gap g concentrates around its

expected value, n, with high probability. here is a
small constant. For a regular LDPC with row weight
wr = 6, column weight wc = 3, = 0.017. Therefore
even though mathematically the encoding algorithm
run in O(n2) (2 O(n2) to be precise), in practice the
encoder still runs in reasonable time for n =
100,000. In the same paper, Richardson and
Urbanke also show that for known optimized

corrected first, followed by slightly smaller degree

nodes, and so on.
Before getting into the details of how to
construct irregular codes, let us introduce some
notations. Let dl, dr be the maximum degrees of
message nodes and check nodes. Define the left
(right) degree of an edge to be the degree of the
message node (check node) that is connected to the
edge. i (i) is the fraction of edges with left (right)
degree i. Any LDPC graph is specified by the
sequences (1 ,..., d l ) and ( 1 ,..., d r ) . Further,

codes, the expected g is bounded by O( n ) thus the

encoder runs in O(n).

define
V. IRREGULAR CODES
It has been shown that irregular LDPC perform
better than regular LDPC [12], [13], [14]. The idea
was pioneered by Luby et al in [12]. He thinks of
finding coefficients for an irregular code as a game,
with the message nodes and check nodes as players.
Each player tries to choose the right number of
edges for them. A constraint of the game is that the
message nodes and the check nodes must agree on
the total number of edges. From the point of view of
the message nodes, it is best to have high degree
since the more information it has from the check
nodes, the more accurately it can judge what its
correct value should be. On the other hand, from the
point of view of the check nodes, it is best to have
low degree, since the lower the degree of a check
node, the more valuable the information it can
transmit back to the message nodes. These two
requirements must be appropriately balanced to
have a good code.
MacKay shows in [15], [16] that for regular
codes, it is best to have low density. However,
allowing irregular codes provides another degree of
freedom. In [12], Luby shows that having a wide
spread of degree is advantageous, at least for the
message nodes. The reason is message nodes with
high degree tend to correct their value faster. These
nodes then provide good information to the check
nodes, which subsequently provide better
information to the lower degree message nodes.
Therefore irregular graph has potential to provide a
wave effect where high degree message nodes are

( x) = i x i 1

(31)

( x) = i x i 1

(32)

and

to be the degree distribution of message nodes and

check nodes. Also, define pi to be the probability
that an incorrect message is passed in the ith
iteration.
Now consider a pair of message node and check
node (m, c) and let c be another check node of m
different than c. At the end of the ith iteration, c
will send m its correct value if there are an even
number (including 0) of message nodes other than
m sending c the incorrect bit. By an analogous
analysis to equation (8), the probability that c
receives an even number of errors is
1 + (1 2 p i ) d r 1
2

(33)

for the case of unvarying degrees of the nodes, and

1 + (1 2 p i )
2

(34)

for the case of varying degrees of the nodes, where

(x) is defined in (32).
A. Lubys design
In [12], Luby proves the iterative description of
pi .

11

So pi+1 = f(pi), therefore we want f(x) < x.

Another constraint to and is

dl

p i +1 = p 0 j .
j =1

j 1 1 + (1 2 p i )

p 0

t =bi , j t
j

1 (1 2 pi )

+ (1 p 0 )
t = bi , j
j

j 1 1 + (1 2 p i )

t
2

j 1 t

i
i

(38)

this task by examining the conditions at

x = p 0 / A for some integer A. By plugging those

values of x into (37), it creates a system of linear

inequalities. The algorithm finds any that satisfies
this linear system. As seen, Lubys approach cannot
determine the best sequence and . Instead, it
determines a good vector given a vector and a
desired code rate R.
Luby shows through simulations that the best
codes have constant , in other words, the check
nodes have the same degree. Some results from [r6]
is reproduced in Table 7.

(35)

Note that p0 is the error probability of the

channel. bi,j is given by the smallest integer the
satisfies
1 p 0 1 + (1 2 p i )

p0
1 (1 2 p i )

= (1 R )

Equation (38) makes sure that the total number

of left and right degrees are the same. Lubys
approach tries to find any sequence that satisfies
(38) and f(x) < x for x (0, p 0 ) . It accomplishes

j 1 t

1 (1 2 p i )
.

2 bi , j j +1

(36)

TABLE 7

The goal of this design is to find sequences

= (1 ,..., dl ) and = ( 1 ,..., d r ) that yield the

LUBYS IRREGULAR CODES

biggest value of p0 such that the sequence {pi}

decreases to 0. Define
dl

f ( x) = p 0 j .
j =1

t
j

j 1 1 + (1 2 x)

p
0

t =bi , j t

1 (1 2 x)

j 1 t

j 1 1 + (1 2 x)

+ (1 p 0 )

t = bi , j t

1 (1 2 x)
.

j 1 t

(37)

p* in Table 7 is the maximum value of p0

12

achieved by each code. All of the code above have

code rate R = 1/2. Previously, the best p* for
Gallagers regular codes with code rate 1/2 is
0.0517 [1].

degree. The reason why this optimization works is

because it suits Differential Evolution well. The
detail of Differential Evolution is discussed in [17].
Some results from [13] is reproduced in Table 8.

B. Richardsons design
In [13], Richardson, Shokrollahi and Urbanke
propose a design of irregular LDPC that can
approach Shannon channel capacity tighter than
turbo code. Their algorithm employs two
optimizations: tolerate the error floor for practical
purpose and carefully design quantization of density
evolution to match the quantization of messages
passed. The idea of the former optimization is that
in practice, we always allow a finite (but small)
probability of error . If we choose small enough
then it automatically implies convergence. The
latter optimization makes sure that the performance
loss due to quantization errors is minimized. Since
belief propagation is optimal, the quantized version
is suboptimal, therefore the simulation results can
be thought of as lower bound for actual values.
Richardsons algorithm starts with an arbitrary
degree distribution (, ). It sets the target error
probability and the maximum number of iterations
m. The algorithm searches for the maximum
admissible channel parameter such that belief
propagation returns a probability of error less than
after m iterations. Now slightly change the degree
distribution pair and runs the algorithm again and
check if a larger admissible channel parameter or a
lower probability of error is found. If yes then set
the current distribution pair to the new distribution
pair, else keep the original pair. This process is
repeated a large number of times. The basic of this
algorithm is that Richardson notices the existence of
stable regions where the probability of error does
not decrease much with the increase number of
iterations. This fact helps limit the search space of
the degree distribution thus shortens the running
time.
Another optimization in Richardsons algorithm
is the fact that he lets the degree to be a continuous
variable, and round it to return to real integer

TABLE 8
RICHARDSONS IRREGULAR CODES

dv is the maximum message node degree, for

each dv, the individual degree fraction is provided.
* is the channel parameter discussed in Section III
C. p* is the input bit error probability of a harddecision decoder. All codes have rate 1/2.
C. Chungs design
In his PhD dissertation, Chung introduces a
derivation of density evolution called discretized
density evolution. This derivation is claimed to
model exactly the behavior of discretized belief
propagation. In his letter [14], Chung introduces an
irregular code which is within 0.0045 dB of
Shannon capacity. This code has dv = 8000, which is
13

much greater than the maximum message node

degree studied by Luby and Richardson. Chungs
code is the closest code to Shannon capacity that
has been simulated. It further confirms that LDPC
indeed approaches channel capacity.

[6] Information
and
Entropy,
OpenCourseWare. Spring 2008.

MIT

[7] Wikipedia. Hamming(7,4). Accessed May 01,

2009.
[8] B. M. J. Leiner, LDPC Codes a brief
Tutorial, April 2005.

VI. CONCLUSION
This paper summaries the important concepts
regarding low density parity-check code (LDPC). It
goes through the motivation of LDPC and how
LDPC can be encoded and decoded. Different
modifications of the codes are presented, especially
irregular codes. I chose to leave out derivations of
regular codes such as MacKay codes [18], repeataccumulate codes [19] because they have become
less important with the advance of irregular codes.
This paper however has not mentioned how
LDPC is implemented in real hardware. For this, I
refer the readers to [20], where a decoder design
based on IEEE 802.11n standards with very high
throughput (900Mbps for FPGA, 2Gbps for ASIC
design) is discussed.

[9] T. Richardson and R. Urbanke, The capacity of

low-density parity check codes under messagepassing decoding, IEEE Trans. Inform. Theory,
vol. 47, pp. 599 618, 2001.
[10] A. Shokrollahi, LDPC Codes: An
Introduction, Digital Fountain, Inc., April 2,
2003.
[11] T. Richardson and R. Urbanke, Efficient
encoding of low-density parity-check codes,"
IEEE Trans. Inform. Theory, vol. 47, pp. 638
656, 2001.
[12] M. Luby, M. Mitzenmacher, A. Shokrollahi,
and D. Spielman, Improved Low-Density
Parity-Check Codes Using Irregular Graphs,"
IEEE Trans. Inform. Theory, vol. 47, pp. 585
598, 2001.

REFERENCES
[1] R. Gallager, Low density parity-check codes,
IRE Trans, Information Theory, pp. 21-28.
January 1962.

[13] T. Richardson, A. Shokrollahi, and R.

Urbanke, Design of capacity-approaching
irregular low-density parity-check codes," IEEE
Trans. Inform. Theory, vol. 47, pp. 619 637 ,
2001.

[2] R. M. Tanner, A recursive approach to low

complexity codes, IEEE Trans. Information
Theory, pp. 533-547, September 1981.

[14] S.-Y. Chung, D. Forney, T. Richardson, and

R. Urbanke, On the design of low-density
parity-check codes within 0.0045 dB of the
Shannon limit," IEEE Communication Letters,
vol. 5, pp. 58 60, 2001.

[3] D. Mackay and R. Neal, Good codes based on

very sparse matrices, Cryptography and
Coding, 5th IMA Conf., C. Boyd, Ed., Lecture
Aotes in Computer Science, pp. 100-111, Berlin,
Germany, 1995.

[15] D. J. C. MacKay, Good error correcting

codes based on very sparse matrices, IEEE
Trans. Inform. Theory, vol. 45, pp. 399431,
Mar. 1999.

[4] D. Mackay, Good error correcting codes based

on very sparse matrices, IEEE Trans.
Information Theory, pp 399-431, March 1999.
[5] N. Alon and M. Luby, A linear time erasureresilient code with nearly optimal recovery,
IEEE Trans. Information Theory, vol. 47, pp.
6238-656, February 2001.

[16] D. J. C. MacKay and R. M. Neal, Near

Shannon limit performance of low-density
parity-check codes, Electron. Lett., vol. 32, pp.
16451646, 1996.
14

[17] K. Price and R. Storn, Differential evolution

A simple and efficient heuristic for global
optimization over continuous spaces, J. Global
Optimiz., vol. 11, pp. 341359, 1997.

[19] D.Divsalar, H. Jin and R. McEliece, Coding

theorems for turbo-like codes, Proc. 36th
Annual Allerton Conf. on Comm., Control and
Conputingce, pp. 201-210. September 1998.

[18] D.
Mackay,
Information
Theory,
Interference, and Learning Algorithms,
Cambridge University Press 2003.

[20] Marjan Karkooti, Predrag Radosavljevic and

Joseph R. Cavallaro, Configurable, High
Throughput,
Irregular
LDPC
Decoder
Architecture:
Tradeoff
Analysis
and
Implementation, Rice Digital Scholarship
Archive, September 01, 2006.

15