Turbo and LDPC Codes:

Implementation, Simulation,
and Standardization
June 7, 2006

Matthew Valenti
Rohit Iyer Seshadri
West Virginia University
Morgantown, WV 26506-6109
[email protected]
 Tutorial Overview
 Channel capacity 1:15 PM
Valenti
 Convolutional codes
– the MAP algorithm
 Turbo codes
– Standard binary turbo codes: UMTS and cdma2000
– Duobinary CRSC turbo codes: DVB-RCS and 802.16
3:15 PM
 LDPC codes Iyer Seshadri
– Tanner graphs and the message passing algorithm
– Standard binary LDPC codes: DVB-S2
 Bit interleaved coded modulation (BICM) 4:30 PM
Valenti
– Combining high-order modulation with a binary capacity
approaching code.
 EXIT chart analysis of turbo codes
Turbo and LDPC Codes
6/7/2006 2/133
 Software to Accompany Tutorial
 Iterative Solution’s Coded Modulation Library (CML) is a library for
simulating and analyzing coded modulation.
 Available for free at the Iterative Solutions website:
– www.iterativesolutions.com
 Runs in matlab, but uses c-mex for efficiency.
 Supported features:
– Simulation of BICM
• Turbo, LDPC, or convolutional codes.
• PSK, QAM, FSK modulation.
• BICM-ID: Iterative demodulation and decoding.
– Generation of ergodic capacity curves (BICM/CM constraints).
– Information outage probability in block fading.
– Calculation of throughput of hybrid-ARQ.
 Implemented standards:
– Binary turbo codes: UMTS/3GPP, cdma2000/3GPP2.
– Duobinary turbo codes: DVB-RCS, wimax/802.16.
– LDPC codes: DVB-S2.

Turbo and LDPC Codes

6/7/2006 3/133
 Noisy Channel Coding Theorem
 Claude Shannon, “A mathematical theory of communication,” Bell
Systems Technical Journal, 1948.
 Every channel has associated with it a capacity C.
– Measured in bits per channel use (modulated symbol).
 The channel capacity is an upper bound on information rate r.
– There exists a code of rate r < C that achieves reliable communications.
• Reliable means an arbitrarily small error probability.

Turbo and LDPC Codes

6/7/2006 4/133
 Computing Channel Capacity
 The capacity is the mutual information between the
channel’s input X and output Y maximized over all
possible input distributions:
k p
C  max I ( X ; Y )
p( x )

R pax, yflog U
z
 max S
T z
p( x )
2
p( x, y)
p( x ) p( y)
dxdy V
W

Turbo and LDPC Codes

6/7/2006 5/133
 Capacity of AWGN
with Unconstrained Input
 Consider an AWGN channel with 1-dimensional input:
– y=x+n
– where n is Gaussian with variance No/2
– x is a signal with average energy (variance) Es
 The capacity in this channel is:
k p F
G I F IJ
H JK G
1 2 Es 1 2rEb
C  max I ( X ; Y )  log 2  1  log 2 1
p( x ) 2 No 2 H No K
– where Eb is the energy per (information) bit.
 This capacity is achieved by a Gaussian input x.
– This is not a practical modulation.

Turbo and LDPC Codes

6/7/2006 6/133
 Capacity of AWGN with
BPSK Constrained Input
 If we only consider antipodal (BPSK) modulation, then

X   Es
 and the capacity is:
l a fq
C  max I X ; Y
p( x )
maximized when
two signals are equally likely
 Ia
X; Y f p ( x ): p 1/ 2

 H (Y )  H ( N )

zaf


 p y log 2 p( y)dy 
1
2
b g
log 2 eNo

This term must be integrated numerically with

z

pY ( y)  pX ( y) pN ( y)  pX ( ) pN ( y   )d


Turbo and LDPC Codes

6/7/2006 7/133
 Capacity of AWGN w/ 1-D Signaling

It is theoretically d
p ac ity Boun
1.0
impossible to operate BPSK Ca
in this region.
d
ou n
B
city
Spectral Efficiency

ap a
Code Rate r

nC

It is theoretically
n no

0.5
possible to operate
Sh a

in this region.

-2 -1 0 1 2 3 4 5 6 7 8 9 10

Eb/No in dB
 Power Efficiency of Standard
Binary Channel Codes
d
p ac ity Boun
1.0
BPSK Ca Uncoded
BPSK

d
ou n
B

Iridium
city
Spectral Efficiency

1998
ap a
Code Rate r

nC

Pioneer
n no

Turbo Code 1968-72

0.5
Sh a

1993
LDPC Code IS-95
2001 1991 Odenwalder
Chung, Forney, Voyager Convolutional
Richardson, Urbanke 1977 Codes 1976

Galileo:BVD
Galileo:LGA 1992
1996
Mariner
1969
arbitrarily low
BER: Pb  10 5
-2 -1 0 1 2 3 4 5 6 7 8 9 10

Eb/No in dB
 Binary Convolutional Codes

D D Constraint Length K = 3

 A convolutional encoder comprises:

– k input streams
• We assume k=1 throughout this tutorial.
– n output streams
– m delay elements arranged in a shift register.
– Combinatorial logic (OR gates).
• Each of the n outputs depends on some modulo-2 combination of the k current
inputs and the m previous inputs in storage
 The constraint length is the maximum number of past and present
input bits that each output bit can depend on.
– K=m+1

Turbo and LDPC Codes

6/7/2006 10/133
 State Diagrams
 A convolutional encoder is a finite state machine, and can
be represented in terms of a state diagram.
Corresponding
output code bits
Input data bit
1/11 S1 = 10 1/10

0/00

S0 = 00 1/00 0/01 S3 = 11 1/01

0/11 S2 = 01 0/10

Since k=1,
2m = 4 total states 2 branches enter
and 2 branches leave
each state
Turbo and LDPC Codes
6/7/2006 11/133
 Trellis Diagram
 Although a state diagram is a helpful tool to understand
the operation of the encoder, it does not show how the
states change over time for a particular input sequence.
 A trellis is an expansion of the state diagram which
explicitly shows the passage of time.
– All the possible states are shown for each instant of time.
– Time is indicated by a movement to the right.
– The input data bits and output code bits are represented by a
unique path through the trellis.

Turbo and LDPC Codes

6/7/2006 12/133
 Trellis Diagram
Every branch every sequence of
corresponds to input and
output bits input data bits
a particular data bit corresponds to
and 2-bits of the for time L = 4
a unique path
code word through the trellis
1/01 1/01
S3
0/1 0/1 0/ 1
0 0 0
0
1/1

0
1/ 1

1/1
1 1
S2
0/0
1
0/0
1 0/0 0/0
1/ 0 1/ 0
0 0

0/1
0/1

0/1

0/1
S1

1
1
1

1
1
1/ 11 11 11
1/ 1/ 1/
0/00 0/00 0/00 0/00 0/00 0/00
S0
i=0 i=1 i=2 i=3 i=4 i=5 i=6

initial final state

new state after m=2
state
first bit is encoded tail bits
 Recursive Systematic
Convolutional (RSC) Codes

xi

D D ri D D

 An RSC encoder is constructed from a standard convolutional encoder

by feeding back one of the outputs.
 An RSC code is systematic.
– The input bits appear directly in the output.
 An RSC encoder is an Infinite Impulse Response (IIR) Filter.
– An arbitrary input will cause a “good” (high weight) output with high
probability.
– Some inputs will cause “bad” (low weight) outputs.

Turbo and LDPC Codes

6/7/2006 14/133
 State Diagram of RSC Code
 With an RSC code, the output labels are the same.
 However, input labels are changed so that each state has
an input “0” and an input “1”

1/11 S1 = 10 0/10

0/00

S0 = 00 0/00 1/01 S3 = 11 1/01

1/11 S2 = 01 0/10

 Messages labeling transitions that start from S1 and S2 are

complemented.
Turbo and LDPC Codes
6/7/2006 15/133
 Trellis Diagram of RSC Code

1/01
S3
0/1 0/1 0/ 1
0 0 0
0
0/1

0
0/ 1

0/1
1 1
S2
1/0
1
1/0
1 1/0 1/0
0/ 0 0/ 0
0 0

1/1
1/1

1/1

1/1
S1

1
1
1

1
1
1/ 11 11 11
1/ 1/ 1/
0/00 0/00 0/00 0/00 0/00 0/00
S0
i=0 i=1 i=2 i=3 i=4 i=5 i=6

m = 2 tail bits
no longer all-zeros
must be calculated by the encoder
 Convolutional Codewords
 Consider the trellis section at time t.
1/01 – Let S(t) be the encoder state at time t.
S3 S3
0/1
0
– When there are four states, S(t)  {S0, S1, S2, S3}
 Let u(t) be the message bit at time t.
– The encoder state S(t) depends on u(t) and S(t-1)
0
0/1

S2 1
1/0 S2
0/ 0  Depending on its initial state S(t-1) and the final state
0 S(t), the encoder will generate an n-bit long word
– x(t) = (x1, x2, …, xn)
1 /1

S1 S1
1

1/
11  The word is transmitted over a channel during time t,
0/00 and the received signal is:
S0 S0
– y(t) = (y1, y2, …, yn)
– For BPSK, each y = (2x-1) + n
 If there are L input data bits plus m tail bits, the overall
transmitted codeword is:
– x = [x(1), x(2), …, x(L), … x(L+m)]
 And the received codeword is:
– y = [ y(1), y(2), …, y(L), …, y(L+m)]
Turbo and LDPC Codes
6/7/2006 17/133
 MAP Decoding
 The goal of the maximum a posteriori (MAP) decoder is to determine
P( u(t)=1 | y ) and P( u(t)=0 | y ) for each t.
– The probability of each message bit, given the entire received codeword.
 These two probabilities are conveniently expressed as a log-likelihood
ratio:
P u (t )  1 | y 
 (t )  log
P u (t )  0 | y 

Turbo and LDPC Codes

6/7/2006 18/133
 Determining Message Bit Probabilities
from the Branch Probabilities
 Let pi,j(t) be the probability that the encoder made a
p3,3
S3 p3 S3 transition from Si to Sj at time t, given the entire
,2
received codeword.
– pi,j(t) = P( Si(t-1)  Sj(t) | y )
,3

S2
p1

p 1,2 S2
– where Sj(t) means that S(t)=Sj
p2
For each t,
,1

 P(S (t  1)  S (t ) | y )  1
p 2,0

S1 S1
i j
p 0,1 Si  S j
p0,0
S0 S0
 The probability that u(t) = 1 is
P (u (t )  1 | y )   P S (t  1)  S
S i  S j :u 1
i j (t ) | y 

 Likewise
P (u (t )  0 | y )   P S (t  1)  S
S i  S j :u  0
i j (t ) | y 

Turbo and LDPC Codes

6/7/2006 19/133
 Determining the Branch Probabilities
 Let i,j(t) = Probability of transition from state Si
3,3
3 3
,2
3 to state Sj at time t, given just the received
word y(t)
,3

2 2  i,j(t) = P( Si(t-1)  Sj(t) | y(t) )

1

 1,2
2
,1
 Let i(t-1) = Probability of starting at state Si at
 2,0

1 1 time t, given all symbols received prior to time t.

 0,1
0,0  i(t-1) = P( Si(t-1) | y(1), y(2), …, y(t-1) )
0 0
 j = Probability of ending at state Sj at time t,
given all symbols received after time t.
 j(t) = P( Sj(t) | y(t+1), …, y(L+m) )
 Then the branch probability is:
– pi,j(t) = i(t-1) i,j(t) j (t)
Turbo and LDPC Codes
6/7/2006 20/133
 Computing α
 α can be computed recursively.
3,3(t)
3(t-1) 3(t)  Prob. of path going through Si(t-1) and
terminating at Sj(t), given y(1)…y(t) is:
(t)
,3
1

• i(t-1) i,j(t)
 Prob. of being in state Sj(t), given y(1)…y(t)
1(t-1)
is found by adding the probabilities of the
two paths terminating at state Sj(t).
 For example,
 3(t)=1(t-1) 1,3(t) + 3(t-1) 3,3(t)
 The values of α can be computed for every
state in the trellis by “sweeping” through the
trellis in the forward direction.
Turbo and LDPC Codes
6/7/2006 21/133
 Computing 
 Likewise,  is computed recursively.
3,3(t+1)
3(t) 3
,2 (t
+1
3(t+1)  Prob. of path going through Sj(t+1) and
)
terminating at Si(t), given y(t+1), …, y(L+m)
2(t+1) – j(t+1) i,j(t+1)
 Prob. of being in state Si(t), given y(t+1), …,
y(L+m) is found by adding the probabilities of
the two paths starting at state Si(t).
 For example,
 3(t) = 2(t+1) 1,2(t+1) + 3(t+1) 3,3(t+1)
 The values of  can be computed for every
state in the trellis by “sweeping” through the
trellis in the reverse direction.
Turbo and LDPC Codes
6/7/2006 22/133
 Computing 
 Every branch in the trellis is labeled with:
 i,j(t) = P( Si(t-1)  Sj(t) | y(t) )
 Let xi,j = (x1, x2, …, xn) be the word generated by the encoder when
transitioning from Si to Sj.
 i,j(t) = P( xi,j | y(t) )
 From Bayes rule,
 i,j(t) = P( xi,j | y(t) ) = P( y(t) | xi,j ) P( xi,j ) / P( y(t) )
 P( y(t) )
– Is not strictly needed because will be the same value for the numerator
and denominator of the LLR (t).
– Instead of computing directly, can be found indirectly as a normalization
factor (chosen for numerical stability)
 P( xi,j )
– Initially found assuming that code bits are equally likely.
– In a turbo code, this is provided to the decoder as “a priori” information.

Turbo and LDPC Codes

6/7/2006 23/133
 Computing P( y(t) | xi,j )
 If BPSK modulation is used over an AWGN channel, the
probability of code bit y given x is conditionally Gaussian:
1   ( y  mx ) 2 
P( y | x)  exp 
2   2 2

mx  Es (2 x  1)
N0
2 
2

– In Rayleigh fading, multiply mx by a, the fading amplitude.

 The conditional probability of the word y(t)
n
P ( y | x)   p ( y i | xi )
i 1

Turbo and LDPC Codes

6/7/2006 24/133
 Overview of MAP algorithm
 Label every branch of the trellis with i,j(t).
 Sweep through trellis in forward-direction to compute i(t) at every
node in the trellis.
 Sweep through trellis in reverse-direction to compute j(t) at every
node in the trellis.
 Compute the LLR of the message bit at each trellis section:
P u (t )  1 | y 
 (t )  log
P u (t )  0 | y 
  (t  1)
S i  S j :u 1
i i, j (t )  j (t )
 log
  (t  1)
Si  S j :u  0
i i, j (t )  j (t )

 MAP algorithm also called the “forward-backward” algorithm (Forney).

Turbo and LDPC Codes
6/7/2006 25/133
 Log Domain Decoding
 The MAP algorithm can be simplified by performing in the
log domain.
– exponential terms (e.g. used to compute ) disappear.
– multiplications become additions.
– Addition can be approximated with maximization.
 Redefine all quantities:
 i,j(t) = log P( Si(t-1)  Sj(t) | y(t) )
 i(t-1) = log P( Si(t-1) | y(1), y(2), …, y(t-1) )
 j(t) = log P( Sj(t) | y(t+1), …, y(L+m) )
 Details of the log-domain implementation will be
presented later…

Turbo and LDPC Codes

6/7/2006 26/133
 Parallel Concatenated Codes
with Nonuniform Interleaving
 A stronger code can be created by encoding in parallel.
 A nonuniform interleaver scrambles the ordering of bits at the input
of the second encoder.
– Uses a pseudo-random interleaving pattern.
 It is very unlikely that both encoders produce low weight code words.
 MUX increases code rate from 1/3 to 1/2.
Systematic Output
Input RSC
#1 xi

MUX
Parity
Output
RSC
#2
Nonuniform
Interleaver

Turbo and LDPC Codes

6/7/2006 27/133
 Random Coding Interpretation
of Turbo Codes
 Random codes achieve the best performance.
– Shannon showed that as n, random codes achieve channel
capacity.
 However, random codes are not feasible.
– The code must contain enough structure so that decoding can be
realized with actual hardware.
 Coding dilemma:
– “All codes are good, except those that we can think of.”
 With turbo codes:
– The nonuniform interleaver adds apparent randomness to the
code.
– Yet, they contain enough structure so that decoding is feasible.

Turbo and LDPC Codes

6/7/2006 28/133
 Comparison of a Turbo Code
and a Convolutional Code
 First consider a K=12 convolutional code.
– dmin = 18
 d = 187 (output weight of all dmin paths)
 Now consider the original turbo code.
– C. Berrou, A. Glavieux, and P. Thitimasjshima, “Near Shannon limit error-
correcting coding and decoding: Turbo-codes,” in Proc. IEEE Int. Conf. on
Commun., Geneva, Switzerland, May 1993, pp. 1064-1070.
– Same complexity as the K=12 convolutional code
– Constraint length 5 RSC encoders
– k = 65,536 bit interleaver
– Minimum distance dmin = 6
– ad = 3 minimum distance code words
– Minimum distance code words have average information weight of only
fd  2

Turbo and LDPC Codes

6/7/2006 29/133
 Comparison of
Minimum-distance Asymptotes
0
10
Convolutional Code  Convolutional code:
CC free distance asymptote
Turbo Code
-2 TC free distance asymptote
d min  18
10
c d min  187
 E 
10
-4 Pb  187  Q 18 b 
 No 
BER

 Turbo code:

-6
10
d min  6
~
ad min w 3 2
~
cd min  d min

10
-8
k 65536
 E 
0.5 1 1.5 2 2.5 3 3.5 4  
Pb  9.2 10 5 Q 6 b 
No 
E b/No in dB 
 The Turbo-Principle
 Turbo codes get their name because the decoder uses
feedback, like a turbo engine.

Turbo and LDPC Codes

6/7/2006 31/133
 Performance as a Function of
Number of Iterations
10
0  K=5
– constraint length
10
-1  r = 1/2
1 iteration – code rate
10
-2  L= 65,536
– interleaver size
2 iterations
10
-3 – number data bits
 Log-MAP algorithm
BER

-4
10 6 iterations 3 iterations

-5
10 10 iterations

-6
10 18 iterations

-7
10
0.5 1 1.5 2
E b/No in dB
 Summary of Performance Factors
and Tradeoffs
 Latency vs. performance
– Frame (interleaver) size L
 Complexity vs. performance
– Decoding algorithm
– Number of iterations
– Encoder constraint length K
 Spectral efficiency vs. performance
– Overall code rate r
 Other factors
– Interleaver design
– Puncture pattern
– Trellis termination

Turbo and LDPC Codes

6/7/2006 33/133
 Tradeoff: BER Performance versus
Frame Size (Latency)
0
10
K=1024
-1 K=4096
10
K=16384  K=5
K=65536
-2
10 -2  Rate r = 1/2
10
 18 decoder iterations

-3
 AWGN Channel
10
-4
BER

10
-4
10

-5
10-6
10

-6
10

-8-7
10
10
0.5
0.5 1 1 1.5 1.5 2 2.5
2 3 2.5
Eb/No in dB
 Characteristics of Turbo Codes
 Turbo codes have extraordinary performance at low SNR.
– Very close to the Shannon limit.
– Due to a low multiplicity of low weight code words.
 However, turbo codes have a BER “floor”.
– This is due to their low minimum distance.
 Performance improves for larger block sizes.
– Larger block sizes mean more latency (delay).
– However, larger block sizes are not more complex to decode.
– The BER floor is lower for larger frame/interleaver sizes
 The complexity of a constraint length KTC turbo code is the
same as a K = KCC convolutional code, where:
– KCC  2+KTC+ log2(number decoder iterations)
Turbo and LDPC Codes
6/7/2006 35/133
 UMTS Turbo Encoder
Systematic
Input Output
Xk Xk

“Upper” Uninterleaved
RSC Parity Output
Zk
Encoder

Interleaved
“Lower” Parity
Interleaved
Interleaver Input RSC Z’k
X’k Encoder

 From 3GPP TS 25 212 v6.6.0, Release 6 (2005-09)

– UMTS Multiplexing and channel coding
 Data is segmented into blocks of L bits.
– where 40  L  5114
 UMTS Interleaver:
Inserting Data into Matrix
 Data is fed row-wise into a R by C matrix.
– R = 5, 10, or 20.
– 8  C  256
– If L < RC then matrix is padded with dummy characters.
In the CML, the UMTS interleaver is created by the function CreateUMTSInterleaver
Interleaving and Deinterleaving are implemented by Interleave and Deinterleave

X1 X2 X3 X4 X5 X6 X7 X8
X9 X10 X11 X12 X13 X14 X15 X16
X17 X18 X19 X20 X21 X22 X23 X24
X25 X26 X27 X28 X29 X30 X31 X32
X33 X34 X35 X36 X37 X38 X39 X40

Turbo and LDPC Codes

6/7/2006 37/133
 UMTS Interleaver:
Intra-Row Permutations
 Data is permuted within each row.
– Permutation rules are rather complicated.
– See spec for details.

X2 X6 X5 X7 X3 X4 X1 X8
X10 X12 X11 X15 X13 X14 X9 X16
X18 X22 X21 X23 X19 X20 X17 X24
X26 X28 X27 X31 X29 X30 X25 X32
X40 X36 X35 X39 X37 X38 X33 X34

Turbo and LDPC Codes

6/7/2006 38/133
 UMTS Interleaver:
Inter-Row Permutations
 Rows are permuted.
– If R = 5 or 10, the matrix is reflected about the middle row.
– For R=20 the rule is more complicated and depends on L.
• See spec for R=20 case.

X40 X36 X35 X39 X37 X38 X33 X34

X26 X28 X27 X31 X29 X30 X25 X32
X18 X22 X21 X23 X19 X20 X17 X24
X10 X12 X11 X15 X13 X14 X9 X16
X2 X6 X5 X7 X3 X4 X1 X8

Turbo and LDPC Codes

6/7/2006 39/133
 UMTS Interleaver:
Reading Data From Matrix
 Data is read from matrix column-wise.

X40 X36 X35 X39 X37 X38 X33 X34

X26 X28 X27 X31 X29 X30 X25 X32
X18 X22 X21 X23 X19 X20 X17 X24
X10 X12 X11 X15 X13 X14 X9 X16
X2 X6 X5 X7 X3 X4 X1 X8

 Thus:
– X’1 = X40 X’2 = X26 X’3 = X18 …
– X’38 = X24 X’2 = X16 X’40 = X8
Turbo and LDPC Codes
6/7/2006 40/133
 UMTS Constituent RSC Encoder

Systematic Output
(Upper Encoder Only)

Parity Output
(Both Encoders)

D D D

 Upper and lower encoders are identical:

– Feedforward generator is 15 in octal.
– Feedback generator is 13 in octal.

Turbo and LDPC Codes

6/7/2006 41/133
 Trellis Termination

XL+1 XL+2 XL+3

ZL+1 ZL+2 ZL+3

D D D

 After the Lth input bit, a 3 bit tail is calculated.

– The tail bit equals the fed back bit.
– This guarantees that the registers get filled with zeros.
 Each encoder has its own tail.
– The tail bits and their parity bits are transmitted at the end.

Turbo and LDPC Codes

6/7/2006 42/133
 Output Stream Format
 The format of the output steam is:
X1 Z1 Z’1 X2 Z2 Z’2 … XL ZL Z’L XL+1 ZL+1 XL+2 ZL+2 XL+3 ZL+3 X’L+1 Z’L+1 X’L+2 Z’L+2 X’L+3 Z’L+3

L data bits and 3 tail bits for 3 tail bits for

their associated upper encoder lower encoder
2L parity bits and their and their
(total of 3L bits) 3 parity bits 3 parity bits

Total number of coded bits = 3L + 12

L 1
Code rate: r  
3 L  12 3

Turbo and LDPC Codes

6/7/2006 43/133
 Channel Model
and LLRs

{0,1} {-1,1} y r
BPSK
Modulator

a n 2a
2

 Channel gain: a
– Rayleigh random variable if Rayleigh fading
– a = 1 if AWGN channel
 Noise 2 
1

3
E  E 
– variance is: 2r  b  2 b 
 No   No 

Turbo and LDPC Codes

6/7/2006 44/133
 SISO-MAP Decoding Block
This block is implemented in the CML by the SisoDecode function

u,i SISO u,o

MAP
c,i Decoder c,o

 Inputs:
 u,i LLR’s of the data bits. This comes from the other decoder r.
 c,i LLR’s of the code bits. This comes from the channel observations r.
 Two output streams:
 u,o LLR’s of the data bits. Passed to the other decoder.
 c,o LLR’s of the code bits. Not used by the other decoder.

Turbo and LDPC Codes

6/7/2006 45/133
 Turbo Decoding Architecture

“Upper”
r(Xk) MAP
r(Zk) Demux
Decoder

Interleave

“Lower” Deinnterleave X k
zeros MAP
Demux Decoder
r(Z’k)

 Initialization and timing:

– Upper u,i input is initialized to all zeros.
– Upper decoder executes first, then lower decoder.
 Performance as a Function of
Number of Iterations
 L=640 bits
0
BER of 640 bit turbo code in AW GN
10  AWGN channel
-1
 10 iterations
10
1 iteration
-2
10

-3
10
2 iterations
BER

-4
10

-5
10 3 iterations

-6
10 10 iterations

-7
10
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
Eb/No in dB
 Log-MAP Algorithm:
Overview
 Log-MAP algorithm is MAP implemented in log-domain.
– Multiplications become additions.
– Additions become special “max*” operator (Jacobi logarithm)
 Log-MAP is similar to the Viterbi algorithm.
– Except “max” is replaced by “max*” in the ACS operation.
 Processing:
– Sweep through the trellis in forward direction using modified
Viterbi algorithm.
– Sweep through the trellis in backward direction using modified
Viterbi algorithm.
– Determine LLR for each trellis section.
– Determine output extrinsic info for each trellis section.

Turbo and LDPC Codes

6/7/2006 48/133
 The max* operator
 max* must implement the following operation:
z  max( x, y )  ln 1  exp  y  x  
 max( x, y )  f c  y  x 
 max* ( x, y )
 Ways to accomplish this:
– C-function calls or large look-up-table. log-MAP

– (Piecewise) linear approximation.

– Rough correction value.
 0 if y - x  1.5
z  max( x, y )   constant-log-MAP
0.5 f y - x  1.5

– Max operator. z  max( x, y ) max-log-MAP

Turbo and LDPC Codes

6/7/2006 49/133
 The Correction Function
0.7

0.6 dec_type option in SisoDecode

=0 For linear-log-MAP (DEFAULT)
Constant-log-MAP = 1 For max-log-MAP algorithm
0.5 = 2 For Constant-log-MAP algorithm
= 3 For log-MAP, correction factor from
0.4
small nonuniform table and interpolation
= 4 For log-MAP, correction factor
uses C function calls
fc(|y-x|) 0.3

0.2
log-MAP

0.1

-0.1
0 1 2 3 4 5 6 7 8 9 10
|y-x|
 The Trellis for UMTS
 Dotted line = data 0
 Solid line = data 1
 00
S0 S0  Note that each node has one each of
 10 data 0 and 1 entering and leaving it.
S1 S1
 The branch from node Si to Sj has
S2 S2 metric ij

S3 S3
 ij  X k (i, j )uk ,i  X k (i, j )ck1,i  Z k (i, j )ck,2i

S4 S4
data bit associated
S5 S5 with branch Si Sj The two code bits labeling
with branch Si Sj
S6 S6

S7 S7

Turbo and LDPC Codes

6/7/2006 51/133
 Forward Recursion
 A new metric must be
 00
calculated for each node in the
’0 0
trellis using:
 10
’1 1
od id
 j  max*  ' i1  i1 j ,  ' i2  i2 j it
’2 2
 where i1 and i2 are the two
’3 3
states connected to j.
’4 4  Start from the beginning of the
’5 5
trellis (i.e. the left edge).
 Initialize stage 0:
’6 6
o = 0
’7 7 i = - for all i  0
Turbo and LDPC Codes
6/7/2006 52/133
 Backward Recursion
 A new metric must be
 00
calculated for each node in the
0 ’0
trellis using:
 10
1 ’1
od id
 i  max*  ' j  ij ,  ' j  ij
1 1 2 2
it
2 ’2
 where j1 and j2 are the two
3 ’3
states connected to i.
4 ’4  Start from the end of the trellis
5 ’5
(i.e. the right edge).
 Initialize stage L+3:
6 ’6
o = 0
7 ’7 i = - for all i  0
Turbo and LDPC Codes
6/7/2006 53/133
 Log-likelihood Ratio
 The likelihood of any one branch is:
 i   ij   j
 00
 0 0  The likelihood of data 1 is found by summing
 10 the likelihoods of the solid branches.
1 1  The likelihood of data 0 is found by summing
the likelihoods of the dashed branches.
 2 2  The log likelihood ratio (LLR) is:

F
bg GP X 1 I
 3 3
 X k  ln
HP X
k

k 0 JK
 4 4
n
 max*  i   ij   j
Si  S j : X k  1
s
5 5
 max* n  i ij  s
j
Si  S j : X k  0
6 6

 7 7

Turbo and LDPC Codes

6/7/2006 54/133
 Memory Issues
 A naïve solution:
– Calculate ’s for entire trellis (forward sweep), and store.
– Calculate ’s for the entire trellis (backward sweep), and store.
– At the kth stage of the trellis, compute  by combining ’s with stored ’s
and ’s .
 A better approach:
– Calculate ’s for the entire trellis and store.
– Calculate ’s for the kth stage of the trellis, and immediately compute  by
combining ’s with these ’s and stored ’s .
– Use the ’s for the kth stage to compute ’s for state k+1.
 Normalization:
– In log-domain, ’s can be normalized by subtracting a common term from
all ’s at the same stage.
– Can normalize relative to 0, which eliminates the need to store 0
– Same for the ’s

Turbo and LDPC Codes

6/7/2006 55/133
 Sliding Window Algorithm
 Can use a sliding window to compute ’s
– Windows need some overlap due to uncertainty in terminating
state.

use these initialization assume these states

values for  region are equally likely

calculate 
and  over
this region.

Turbo and LDPC Codes

6/7/2006 56/133
 Extrinsic Information
 The extrinsic information is found by subtracting the
corresponding input from the LLR output, i.e.
 u,i (lower) = u,o (upper) - u,i (upper)
 u,i (upper) = u,o (lower) - u,i (lower)
 It is necessary to subtract the information that is already
available at the other decoder in order to prevent “positive
feedback”.
 The extrinsic information is the amount of new information
gained by the current decoder step.

Turbo and LDPC Codes

6/7/2006 57/133
 Performance Comparison
0
BER of 640 bit turbo code
10
max-log-MAP
constant-log-MAP
-1
10 log-MAP

-2
10
Fading

-3
10
AWGN
BER

-4
10

-5
10

-6
10
10 decoder iterations
-7
10
0 0.5 1 1.5 2 2.5 3
Eb/No in dB
 cdma2000
 cdma2000 uses a rate ⅓ constituent encoder.
– Overall turbo code rate can be 1/5, 1/4, 1/3, or 1/2.
– Fixed interleaver lengths:
• 378, 570, 762, 1146, 1530, 2398, 3066, 4602, 6138, 9210, 12282, or 20730

Systematic
Output Xi

First Parity
Output Z1,i

Second Parity
Output Z2,i
Data Input
Xi D D D

Turbo and LDPC Codes

6/7/2006 59/133
 0
10

performance of
cdma2000 turbo code
in AWGN
-2 with interleaver length 1530
10
Bit Error Rate

-4
10

1/5 1/4 1/3 1/2

-6
10

-8
10
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
Eb/No in dB
 Circular Recursive Systematic
Convolutional (CRSC) Codes
1/01 1/01 1/01 1/01 1/01 1/01
S3 S3
0/1 0/1 0/1 0/1 0/ 1 0/ 1
0 0 0 0 0 0

0
0

0
0/1

0/1
1
0/1

0/1

0/1

0/1
1 1 1 1
S2 1/0 1/0 1/0
1 1/0 1/0 1/0 S2
0/ 0 0/ 0 0/ 0 0/ 0 0/0 0/0
0 0 0 0 0 0

1/1

1/1
1/1

1 /1

1/1

1 /1
S1 S1

1
1

1
11 11 11 11 11 11
1/ 1/ 1/ 1/ 1/ 1/
0/00 0/00 0/00 0/00 0/00 0/00
S0 S0
 CRSC codes use the concept of tailbiting.
– Sequence is encode so that initial state is same as final state.
 Advantage and disadvantages
– No need for tail bits.
– Need to encode twice.
– Complicates decoder.
Turbo and LDPC Codes
6/7/2006 61/133
 Duobinary codes

A
S1 S2 S3

W Y

 Duobinary codes are defined over GF(4).

– two bits taken in per clock cycle.
– Output is systematic and rate 2/4.
 Hardware benefits
– Half as many states in trellis.
– Smaller loss due to max-log-MAP decoding.
Turbo and LDPC Codes
6/7/2006 62/133
 DVB-RCS
 Digital Video Broadcasting – Return Channel via Satellite.
– Consumer-grade Internet service over satellite.
– 144 kbps to 2 Mbps satellite uplink.
– Uses same antenna as downlink.
– QPSK modulation.
 DVB-RCS uses a pair of duobinary CRSC codes.
 Ket parameters:
– input of N = k/2 couples
– N = {48,64,212,220,228,424,432,440,752,848,856,864}
– r={1/3, 2/5, 1/2, 2/3, 3/4, 4/5, 6/7}
 M.C. Valenti, S. Cheng, and R. Iyer Seshadri, “Turbo and LDPC
codes for digital video broadcasting,” Chapter 12 of Turbo Code
Applications: A Journey from a Paper to Realization, Springer, 2005.

Turbo and LDPC Codes

6/7/2006 63/133
 DVB-RCS:
Influence of DecodingAlgorithm
0
10  rate r=⅓
 length N=212
 8 iterations.
-1
10  AWGN.

-2
10
FER

-3
10

-4
10 Max-Log-MAP, unknown circulation state
log-MAP, unknown circulation state
log-MAP, known circulation state
-5
10
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
Eb/No in dB
 DVB-RCS:
0
Influence of Block Length
10  rate ⅓
N=48  max-log-MAP
N=64
-1  8 iterations
10 N=212
N=432  AWGN
N=752
-2
10
FER

-3
10

-4
10

-5
10
0 0.5 1 1.5 2 2.5 3 3.5
Eb/No in dB
 DVB-RCS:
0
Influence of Code Rate
10  N=212
r=6/7
 max-log-MAP
r=4/5
-1
r=3/4  8 iterations
10
r=2/3  AWGN
r=1/2
-2 r=2/5
10
r=1/3
FER

-3
10

-4
10

-5
10
0 1 2 3 4 5 6
Eb/No in dB
 802.16 (WiMax)
 The standard specifies an optional convolutional turbo code (CTC) for
operation in the 2-11 GHz range.
 Uses same duobinary CRSC encoder as DVB-RCS, though without
output W.
A
S1 S2 S3

 Modulation: BPSK, QPSK, 16-QAM, 64-QAM, 256-QAM.

 Key parameters:
– Input message size 8 to 256 bytes long.
– r = {1/2, 2/3, 3/4, 5/6, 7/8}

Turbo and LDPC Codes

6/7/2006 67/133
 Prelude to LDPC Codes:
Review of Linear Block Codes
 Vn = n-dimensional vector space over {0,1}

 A (n, k) linear block code with dataword length k, codeword length n

is a k-dimensional vector subspace of Vn

 A codeword c is generated by the matrix multiplication c = uG, where

u is the k-bit long message and G is a k by n generator matrix

 The parity check matrix H is a n-k by n matrix of ones and zeros,

such that if c is a valid codeword then, cHT = 0

 Each row of H specifies a parity check equation. The code bits in

positions where the row is one must sum (modulo-2) to zero

Turbo and LDPC Codes

6/7/2006 68/133
 Low-Density Parity-Check Codes
 Low-Density Parity-Check (LDPC) codes are a class of linear block
codes characterized by sparse parity check matrices H
– H has a low-density of 1’s

 LDPC codes were originally invented by Robert Gallager in the early

1960’s but were largely ignored until they were “rediscovered” in the
mid-1990’s by MacKay

 Sparseness of H can yield large minimum distance dmin and reduces

decoding complexity

 Can perform within 0.0045 dB of Shannon limit

Turbo and LDPC Codes

6/7/2006 69/133
 Decoding LDPC codes
 Like Turbo codes, LDPC can be decoded iteratively
– Instead of a trellis, the decoding takes place on a Tanner graph
– Messages are exchanged between the v-nodes and c-nodes
– Edges of the graph act as information pathways
 Hard decision decoding
– Bit-flipping algorithm
 Soft decision decoding
– Sum-product algorithm
• Also known as message passing/ belief propagation algorithm
– Min-sum algorithm
• Reduced complexity approximation to the sum-product algorithm
 In general, the per-iteration complexity of LDPC codes is less than it is
for turbo codes
– However, many more iterations may be required (max100;avg30)
– Thus, overall complexity can be higher than turbo

Turbo and LDPC Codes

6/7/2006 70/133
 Tanner Graphs
 A Tanner graph is a bipartite graph that describes the parity check
matrix H
 There are two classes of nodes:
– Variable-nodes: Correspond to bits of the codeword or equivalently, to
columns of the parity check matrix
• There are n v-nodes
– Check-nodes: Correspond to parity check equations or equivalently, to
rows of the parity check matrix
• There are m=n-k c-nodes
– Bipartite means that nodes of the same type cannot be connected (e.g. a
c-node cannot be connected to another c-node)
 The ith check node is connected to the jth variable node iff the (i,j)th
element of the parity check matrix is one, i.e. if hij =1
– All of the v-nodes connected to a particular c-node must sum (modulo-2)
to zero

Turbo and LDPC Codes

6/7/2006 71/133
 Example: Tanner Graph
for (7,4) Hamming Code

1 1 1 0 1 0 0
H  1 1 0 1 0 1 0
1 0 1 1 0 0 1

c-nodes
f0 f1 f2

v0 v1 v2 v3 v4 v5 v6

v-nodes

Turbo and LDPC Codes

6/7/2006 72/133
 More on Tanner Graphs
 A cycle of length l in a Tanner graph is a path of l distinct edges
which closes on itself
 The girth of a Tanner graph is the minimum cycle length of the
graph.
– The shortest possible cycle in a Tanner graph has length 4

c-nodes
f0 f1 f2

v0 v1 v2 v3 v4 v5 v6

v-nodes

Turbo and LDPC Codes

6/7/2006 73/133
 Bit-Flipping Algorithm:
(7,4) Hamming Code

f0 =1 f1 =1 f2 =0

y0 =1 y1 =1 y2 =1 y3 =1 y4 =0 y5 =0 y6 =1

Received code word

c0 =1 c1 =0 c2 =1 c3 =1 c4 =0 c5 =0 c6 =1

Transmitted code word

Turbo and LDPC Codes

6/7/2006 74/133
 Bit-Flipping Algorithm:
(7,4) Hamming Code

f0 =1 f1 =1 f2 =0

y0 =1 y1 =1 y2 =1 y3 =1 y4 =0 y5 =0 y6 =1

Turbo and LDPC Codes

6/7/2006 75/133
 Bit-Flipping Algorithm:
(7,4) Hamming Code

f0 =0 f1 =0 f2 =0

y0 =1 y1 =0 y2 =1 y3 =1 y4 =0 y5 =0 y6 =1

Turbo and LDPC Codes

6/7/2006 76/133
 Generalized Bit-Flipping Algorithm
 Step 1: Compute parity-checks
– If all checks are zero, stop decoding

 Step 2: Flip any digit contained in T or more failed check equations

 Step 3: Repeat 1 to 2 until all the parity checks are zero or a maximum
number of iterations are reached

 The parameter T can be varied for a faster convergence

Turbo and LDPC Codes

6/7/2006 77/133
 Generalized Bit Flipping:
(15,7) BCH Code
f0 =1 f1 =0 f2 =0 f3 =0 f4 =1 f5 =0 f6 =0 f7 =1

y0 =0 y1 =0 y2 =0 y3 =0 y4 =1 y5 =0 y6 =0 y7 =0 y8 =0 y9 =0 y10 =0 y11 =0 y12 =0 y13 =0 y14 =1

Received code word

c0 =0 c1 =0 c2 =0 c3 =0 c4 =0 c5 =0 c6 =0 c7 =0 c8 =0 c9 =0 c10 =0 c11 =0 c12 =0 c13 =0 c14 =0

Transmitted code word

Turbo and LDPC Codes

6/7/2006 78/133
 Generalized Bit Flipping:
(15,7) BCH Code
f0 =0 f1 =0 f2 =0 f3 =0 f4 =0 f5 =0 f6 =0 f7 =1

y0 =0 y1 =0 y2 =0 y3 =0 y4 =0 y5 =0 y6 =0 y7 =0 y8 =0 y9 =0 y10 =0 y11 =0 y12 =0 y13 =0 y14 =1

Turbo and LDPC Codes

6/7/2006 79/133
 Generalized Bit Flipping:
(15,7) BCH Code
f0 =0 f1 =0 f2 =0 f3 =0 f4 =0 f5 =0 f6 =0 f7 =0

y0 =0 y1 =0 y2 =0 y3 =0 y4 =0 y5 =0 y6 =0 y7 =0 y8 =0 y9 =0 y10 =0 y11 =0 y12 =0 y13 =0 y14 =0

Turbo and LDPC Codes

6/7/2006 80/133
 Sum-Product Algorithm:
Notation
 Q0 = P(ci =0|y, Si), Q1 =P(ci =1|y, Si)
 Si = event that bits in c satisfy the dv parity check equations involving ci
 qij (b) = extrinsic info to be passed from v-node i to c-node j
– Probability that ci =b given extrinsic information from check nodes and channel
sample yi
 rji(b) = extrinsic info to be passed from c-node j to v-node I
– Probability of the jth check equation being satisfied give that ci =b
 Ci = {j: hji = 1}
– This is the set of row location of the 1’s in the i th column
 Ci\j= {j’: hj’i=1}\{j}
– The set of row locations of the 1’s in the i th column, excluding location j
 Rj = {i: hji = 1}
– This is the set of column location of the 1’s in the j th row
 Rj\i= {i’: hji’=1}\{i}
– The set of column locations of the 1’s in the j th row, excluding location i
Turbo and LDPC Codes
6/7/2006 81/133
 Sum-Product Algorithm
Step 1: Initialize qij (0) =1-pi = 1/(1+exp(-2yi/ 2))
qij (1) =pi = 1/(1+exp(2yi/ 2 ))
qij (b) = probability that ci =b, given the channel sample

f0 f1 f2

q10
q00 q01 q02
q11 q q31 q32 q51 q62
20 q22 q40
v0 v1 v2 v3 v4 v5 v6
y0 y1 y2 y3 y4 y5 y6
y0 y1 y2 y3 y4 y5 y6
Received code word (output of AWGN)

Turbo and LDPC Codes

6/7/2006 82/133
 Sum-Product Algorithm
1 1
Step 2: At each c-node, update the r messages rji (0)     1  2qi ' j (1) 
2 2 i 'R j\i
rji (1)  1  rji (0)
rji (b) = probability that jth check equation is satisfied given ci =b
f0 f1 f2

r13
r23
r00 r01
r02 r15 r26
r22 r03
r11
rr2010
v0 v1 v2 v3 v4 v5 v6

Turbo and LDPC Codes

6/7/2006 83/133
 Sum-Product Algorithm
qij (0)  kij (1  pi )   rj 'i (0) 
Step 3: Update qij (0) and qij (1) j 'Ci \ j

qij (1)  kij ( pi )   rj 'i (1) 

j 'Ci \ j

f0 f1 f2

q10 q32
q00 q02 q62
q01 q31 q40 q51
q20
q11 q22

v0 v1 v2 v3 v4 v5 v6
y0 y1 y2 y3 y4 y5 y6

Qi (0)  kij (1  pi )  r ji (0)  Make hard decision

jCi
1 if Qi (1)  0.5
Qi (1)  kij ( pi )  rji (1)  cˆi  
jCi
0 otherwise

Turbo and LDPC Codes

6/7/2006 84/133
 Halting Criteria
 After each iteration, halt if:

cˆ H T  0

 This is effective, because the probability of an undetectable decoding

error is negligible

 Otherwise, halt once the maximum number of iterations is reached

 If the Tanner graph contains no cycles, then Qi converges to the true

APP value as the number of iterations tends to infinity

Turbo and LDPC Codes

6/7/2006 85/133
 Sum-Product Algorithm in Log Domain
 The sum-product algorithm in probability domain has two shortcomings
– Numerically unstable
– Too many multiplications

 A log domain version is often used for practical purposes

 P (ci  0 | y, Si ) 
 Qi  log   LLR of the ith code bit (ultimate goal of algorithm)
P ( c i  1| y , Si ) 
 
 qij = log (qij(0)/qij(1))extrinsic info to be passed from v-node i to c-node j

 rji = log(rji(0)/rji(1))extrinsic info to be passed from c-node j to v-node I

Turbo and LDPC Codes

6/7/2006 86/133
 Sum-Product Decoder
(in Log-Domain)
 Initialize:
– qij = i = 2yi/2 = channel LLR value
 Loop over all i,j for which hij = 1
– At each c-node, update the r messages:
   
rji    i ' j      i ' j  
  
 i 'R   
 j \i   i 'R j \i 
– At each v-node update the q message and Q LLR:
Qi  i   rji
jCi

qij  Qi  rji
– Make hard decision:
1 if Qi  0
cˆi  
0 otherwise
Turbo and LDPC Codes
6/7/2006 87/133
 Sum-Product Algorithm:
Notation
 ij = sign( qij )
 ij = | qij |
 (x) = -log tanh(x/2) = log( (ex+1)/(ex-1) )= -1(x)
6

4
 (x)

0
0 1 2 3 4 5 6
x

Turbo and LDPC Codes

6/7/2006 88/133
 Min-Sum Algorithm
 Note that:




     i ' j      min i ' j  min i ' j
 i'  i' i'

 So we can replace the r message update formula with

 
rji     i ' j  min i ' j
 i 'R  i 'R j\i
 j \i 

 This greatly reduces complexity, since now we don’t have to worry about
computing the nonlinear  function.

 Note that since  is just the sign of q,  can be implemented by using XOR
operations.

Turbo and LDPC Codes

6/7/2006 89/133
 BER of
-1
Different Decoding Algorithms
10

Code #1:
-2 MacKay’s construction 2A
10
AWGN channel
BPSK modulation
Min-sum
-3
10
BER

-4
10

-5
10

-6 Sum-product
10

-7
10
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Eb/No in dB
 Extrinsic-information Scaling
 As with max-log-MAP decoding of turbo codes, min-sum decoding of LDPC
codes produces an extrinsic information estimate which is biased.
– In particular, rji is overly optimistic.

 A significant performance improvement can be achieved by multiplying rji by

a constant , where <1.
 
rji     i ' j  min  i ' j

 i 'R  i 'R j \i
 j \i 
– See: J. Heo, “Analysis of scaling soft information on low density parity
check code,” IEE Electronic Letters, 23rd Jan. 2003.

– Experimentation shows that =0.9 gives best performance.

Turbo and LDPC Codes

6/7/2006 91/133
 BER of
-1
Different Decoding Algorithms
10

Code #1:
-2 MacKay’s construction 2A
10
AWGN channel
BPSK modulation
Min-sum
-3
10
BER

-4
10

-5 Min-sum
10 w/ extrinsic info scaling
Scale factor =0.9

-6 Sum-product
10

-7
10
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Eb/No in dB
 Regular vs. Irregular LDPC codes
 An LDPC code is regular if the rows and columns of H have uniform
weight, i.e. all rows have the same number of ones (dv) and all columns
have the same number of ones (dc)
– The codes of Gallager and MacKay were regular (or as close as possible)
– Although regular codes had impressive performance, they are still about 1 dB
from capacity and generally perform worse than turbo codes
 An LDPC code is irregular if the rows and columns have non-uniform
weight
– Irregular LDPC codes tend to outperform turbo codes for block lengths of
about n>105
 The degree distribution pair (λ, ρ) for a LDPC code is defined as
dv
 ( x)   x
i2
i
i 1

dc
 ( x)  
i 1
i x i 1

 λi, ρi represent the fraction of edges emanating from variable (check)

nodes of degree i
Turbo and LDPC Codes
6/7/2006 93/133
 Constructing Regular LDPC Codes:
MacKay, 1996
 Around 1996, Mackay and Neal described methods for constructing sparse H
matrices

 The idea is to randomly generate a M × N matrix H with weight dv columns

and weight dc rows, subject to some constraints

 Construction 1A: Overlap between any two columns is no greater than 1

– This avoids length 4 cycles

 Construction 2A: M/2 columns have dv =2, with no overlap between any pair
of columns. Remaining columns have dv =3. As with 1A, the overlap between
any two columns is no greater than 1

 Construction 1B and 2B: Obtained by deleting select columns from 1A and 2A

– Can result in a higher rate code

Turbo and LDPC Codes

6/7/2006 94/133
 Constructing Irregular LDPC Codes:
Luby, et. al., 1998
 Luby et. al. (1998) developed LDPC codes based on irregular LDPC
Tanner graphs
 Message and check nodes have conflicting requirements
– Message nodes benefit from having a large degree
– LDPC codes perform better with check nodes having low degrees
 Irregular LDPC codes help balance these competing requirements
– High degree message nodes converge to the correct value quickly
– This increases the quality of information passed to the check nodes,
which in turn helps the lower degree message nodes to converge
 Check node degree kept as uniform as possible and variable node
degree is non-uniform
– Code 14: Check node degree =14, Variable node degree =5, 6, 21, 23
 No attempt made to optimize the degree distribution for a given code
rate

Turbo and LDPC Codes

6/7/2006 95/133
 Density Evolution:
Richardson and Urbanke, 2001
 Given an irregular Tanner graph with a maximum d v and dc, what is the best degree
distribution?
– How many of the v-nodes should be degree dv, dv-1, dv-2,... nodes?
– How many of the c-nodes should be degree dc, dc-1,.. nodes?

 Question answered using Density Evolution

– Process of tracking the evolution of the message distribution during belief propagation

 For any LDPC code, there is a “worst case” channel parameter called the threshold such
that the message distribution during belief propagation evolves in such a way that the
probability of error converges to zero as the number of iterations tends to infinity

 Density evolution is used to find the degree distribution pair (λ, ρ) that maximizes this
threshold

Turbo and LDPC Codes

6/7/2006 96/133
 Density Evolution:
Richardson and Urbanke, 2001
 Step 1: Fix a maximum number of iterations

 Step 2: For an initial degree distribution, find the threshold

 Step 3: Apply a small change to the degree distribution

– If the new threshold is larger, fix this as the current distribution

 Repeat Steps 2-3

 Richardson and Urbanke identify a rate ½ code with degree distribution pair
which is 0.06 dB away from capacity
– “Design of capacity-approaching irregular low-density parity-check codes”, IEEE
Trans. Inf. Theory, Feb. 2001
 Chung et.al., use density evolution to design a rate ½ code which is 0.0045
dB away from capacity
– “On the design of low-density parity-check codes within 0.0045 dB of the Shannon
limit”, IEEE Comm. Letters, Feb. 2001

Turbo and LDPC Codes

6/7/2006 97/133
 More on Code Construction
 LDPC codes, especially irregular codes exhibit error floors at high SNRs
 The error floor is influenced by dmin
– Directly designing codes for large dmin is not computationally feasible
 Removing short cycles indirectly increases dmin (girth conditioning)
– Not all short cycles cause error floors
 Trapping sets and Stopping sets have a more direct influence on the error floor
 Error floors can be mitigated by increasing the size of minimum stopping sets
– Tian,et. al., “Construction of irregular LDPC codes with low error floors”, in Proc.
ICC, 2003
 Trapping sets can be mitigated using averaged belief propagation decoding
– Milenkovic, “Algorithmic and combinatorial analysis of trapping sets in structured
LDPC codes”, in Proc. Intl. Conf. on Wireless Ntw., Communications and Mobile
computing, 2005
 LDPC codes based on projective geometry reported to have very low error
floors
– Kou, “Low-density parity-check codes based on finite geometries: a rediscovery and
new results”, IEEE Tans. Inf. Theory, Nov.1998

Turbo and LDPC Codes

6/7/2006 98/133
 Encoding LDPC Codes
 A linear block code is encoded by performing the matrix multiplication c = uG

 A common method for finding G from H is to first make the code systematic
by adding rows and exchanging columns to get the H matrix in the form H =
[PT I]
– Then G = [I P]
– However, the result of the row reduction is a non-sparse P matrix
– The multiplication c =[u uP] is therefore very complex

 As an example, for a (10000, 5000) code, P is 5000 by 5000

– Assuming the density of 1’s in P is 0.5, then 0.5× (5000)2 additions are required per
codeword

 This is especially problematic since we are interested in large n (>105)

 An often used approach is to use the all-zero codeword in simulations

Turbo and LDPC Codes

6/7/2006 99/133
 Encoding LDPC Codes
 Richardson and Urbanke show that even for large n, the encoding
complexity can be (almost) linear function of n
– “Efficient encoding of low-density parity-check codes”, IEEE Trans. Inf.
Theory, Feb., 2001

 Using only row and column permutations, H is converted to an

approximately lower triangular matrix
– Since only permutations are used, H is still sparse
– The resulting encoding complexity in almost linear as a function of n

 An alternative involving a sparse-matrix multiply followed by

differential encoding has been proposed by Ryan, Yang, & Li….
– “Lowering the error-rate floors of moderate-length high-rate irregular
LDPC codes,” ISIT, 2003

Turbo and LDPC Codes

6/7/2006 100/133
 Encoding LDPC Codes
 Let H = [H1 H2] where H1 is sparse and

1  1 1 1 ... 1
   
1 1   1 1 ... 1
   
H2   1 1  and H 2T  1 ... 1
   
 1 ... 1   ... 1
   
 1 1  1
 Then a systematic code can be generated with G = [I H1TH2-T].
 It turns out that H2-T is the generator matrix for an accumulate-code (differential
encoder), and thus the encoder structure is simply:
u u

uH1TH -T
Multiply
2
by H1T
D
 Similar to Jin & McEliece’s Irregular Repeat Accumulate (IRA) codes.
– Thus termed “Extended IRA Codes”
Turbo and LDPC Codes
6/7/2006 101/133
 Performance Comparison
 We now compare the performance of the maximum-length UMTS
turbo code against four LDPC code designs.
 Code parameters
– All codes are rate ⅓
– The LDPC codes are length (n,k) = (15000, 5000)
• Up to 100 iterations of log-domain sum-product decoding
• Code parameters are given on next slide
– The turbo code has length (n,k) = (15354,5114)
• Up to 16 iterations of log-MAP decoding
 BPSK modulation
 AWGN and fully-interleaved Rayleigh fading
 Enough trials run to log 40 frame errors
– Sometimes fewer trials were run for the last point (highest SNR).

Turbo and LDPC Codes

6/7/2006 102/133
 LDPC Code Parameters
 Code 1: MacKay’s regular construction 2A
– See: D.J.C. MacKay, “Good error-correcting codes based on very sparse
matrices,” IEEE Trans. Inform. Theory, March 1999.
 Code 2: Richardson & Urbanke irregular construction
– See T. Richardson, M. Shokrollahi, and R. Urbanke, “Design of capacity-
approaching irregular low-density parity-check codes,” IEEE Trans.
Inform. Theory, Feb. 2001.
 Code 3: Improved irregular construction
– Designed by Chris Jones using principles from T. Tian, C. Jones, J.D.
Villasenor, and R.D. Wesel, “Construction of irregular LDPC codes with
low error floors,” in Proc. ICC 2003.
– Idea is to avoid small stopping sets
 Code 4: Extended IRA code
– Designed by Michael Yang & Bill Ryan using principles from M. Yang and
W.E. Ryan, “Lowering the error-rate floors of moderate-length high-rate
irregular LDPC codes,” ISIT, 2003.

Turbo and LDPC Codes

6/7/2006 103/133
 LDPC Degree Distributions
 The distribution of row-weights, or check-node degrees, is as follows:
i 1 2 3 4
3 1
Code number:
4 10000 4999 1 = MacKay construction 2A
5 13 5458 5000 2 = Richardson & Urbanke
6 9987 4542 3 = Jones, Wesel, & Tian
4 = Ryan’s Extended-IRA

 The distribution of column-weights, or variable-node degrees, is:

i 1 2 3 4
1 1
2 5000 8282 9045 9999
3 10000 2238 2267
4 2584 569 5000
5 206 1941
8 1
15 1689 1178

Turbo and LDPC Codes

6/7/2006 104/133
 -1
BER in AWGN
10
BPSK/AWGN Capacity:
-0.50 dB for r = 1/3
-2
10

-3
10
BER

-4
10
Code #1:
Code #3: Mackay 2A
-5
JWT Code #2:
10 R&U
Code #4:
IRA
-6
10

turbo
-7
10
0 0.2 0.4 0.6 0.8 1 1.2
Eb/No in dB
 DVB-S2 LDPC Code
 The digital video broadcasting (DVB) project was founded in 1993 by ETSI to
standardize digital television services

 The latest version of the standard DVB-S2 uses a concatenation of an outer

BCH code and inner LDPC code

 The codeword length can be either n =64800 (normal frames) or n =16200

(short frames)

 Normal frames support code rates 9/10, 8/9, 5/6, 4/5, 3/4, 2/3, 3/5, 1/2, 2/5,
1/3, 1/4
– Short frames do not support rate 9/10
 DVB-S2 uses an extended-IRA type LDPC code

 Valenti, et. al, “Turbo and LDPC codes for digital video broadcasting,” Chapter
12 of Turbo Code Application: A Journey from a Paper to Realizations,
Springer, 2005.

Turbo and LDPC Codes

6/7/2006 106/133
 FER for DVB-S2 LDPC Code
Normal Frames in BPSK/AWGN
0
10
r=9/10
r=8/9
r=5/6
r=4/5
-1
10 r=3/4
r=2/3
r=3/5
r=1/2
r=2/5
-2 r=1/3
FER

10 r=1/4

-3
10

-4
10
0 1 2 3 4 5
Eb/No in dB
 FER for DVB-S2 LDPC Code
Short Frames in BPSK/AWGN
0
10
r=8/9
r=5/6
r=4/5
r=3/4
-1
10 r=2/3
r=3/5
r=1/2
r=2/5
r=1/3
r=1/4
FER

-2
10

-3
10

-4
10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5
Eb/No in dB
 M-ary Complex Modulation
  = log2 M bits are mapped to the symbol xk, which is
chosen from the set S = {x1, x2, …, xM}
– The symbol is multidimensional.
– 2-D Examples: QPSK, M-PSK, QAM, APSK, HEX
– M-D Example: FSK, block space-time codes (BSTC)
 The signal y = hxk + n is received
– h is a complex fading coefficient.
– More generally (BSTC), Y = HX + N
 Modulation implementation in the ISCML
– The complex signal set S is created with the CreateConstellation
function.
– Modulation is performed using the Modulate function.

Turbo and LDPC Codes

6/7/2006 109/133
 Log-likelihood of Received Symbols
 Let p(xk|y) denote the probability that signal xk S was
transmitted given that y was received.
 Let f(xk|y) =  p(xk|y), where  is any multiplicative term
that is constant for all xk.
 When all symbols are equally likely, f(xk|y)  f(y|xk)
 For each signal in S, the receiver computes f(y|xk)
– This function depends on the modulation, channel, and receiver.
– Implemented by the Demod2D and DemodFSK functions, which
actually computes log f(y|xk).
 Assuming that all symbols are equally likely, the most
likely symbol xk is found by making a hard decision on f(y|
xk) or log f(y|xk).
Turbo and LDPC Codes
6/7/2006 110/133
 Example: QAM over AWGN.
 Let y = x + n, where n is complex i.i.d. N(0,N0/2 ) and the
average energy per symbol is E[|x|2] = Es
1   y  x k 2

p( y xk )  exp 
2 2  2
2

  y  x k 2

f ( y x k )  exp 
 2
2

2
 y  xk
log f ( y x k ) 
2 2
2
 E s y  xk

No

Turbo and LDPC Codes

6/7/2006 111/133
 Log-Likelihood of Symbol xk
 The log-likelihood of symbol xk is found by:
 k  log p (x k | y )
p(x k | y )
 log
 p(x k | y )
x m S

f (y | x k )
 log
 f (y | x m )
x m S

 log f (y | x k )  log  f (y | x
x m S
m )

 log f (y | x k )  log  exp log f (y | x )

x m S
m

 log f (y | x k )  max* log f (y | x m )

x m S

Turbo and LDPC Codes

6/7/2006 112/133
 The max* function
0.7

0.6 max* ( x, y )  log exp( x)  exp( y )

 max( x, y )  log1  exp  y  x  
0.5

 max( x, y )  f c  y  x 
0.4

0.3

f c ( z )  log1  exp  z )  
fc(|y-x|)

0.2

0.1

-0.1
0 1 2 3 4 5 6 7 8 9 10

|y-x|
 Capacity of Coded Modulation (CM)
 Suppose we want to compute capacity of M-ary
modulation
– In each case, the input distribution is constrained, so there is no
need to maximize over p(x)
– The capacity is merely the mutual information between channel
input and output.
 The mutual information can be measured as the following
expectation:
C  I ( X ; Y )  E x k ,n  log M  log p(x k | y ) nats

Turbo and LDPC Codes

6/7/2006 114/133
 Monte Carlo Calculation of the
Capacity of Coded Modulation (CM)
 The mutual information can be measured as the following
expectation:
C  I ( X ; Y )  E x k ,n  log M  log p (x k | y ) nats
 log M  E x k ,n   k  nats
E x k ,n   k 
 log 2 M  bits
log(2)
E x k ,n   k 
 bits
log(2)

 This expectation can be obtained through Monte Carlo

simulation.

Turbo and LDPC Codes

6/7/2006 115/133
 Simulation Block Diagram
This function is computed by This function is computed by
the CML function Demod2D the CML function Capacity

Modulator: xk Receiver: Calculate:

Pick xk at random Compute log f(y|xk)  k  log f ( y | x k )
from S for every xk  S  max* log f (y | x m )
x m S
nk

Noise Generator

After running many

Benefits of Monte Carlo approach: trials, calculate:
-Allows high dimensional signals to be studied. E  k 
-Can determine performance in fading. C
-Can study influence of receiver design. log(2)
 8
Capacity of 2-D modulation in AWGN 256QAM

6 c ity
Capacity (bits per symbol)

pa 64QAM
Ca
ed
in
5 ts ra
con
U n
D
2- 16QAM
4
16PSK
3 8PSK

2
QPSK

1
BPSK

0
-2 0 2 4 6 8 10 12 14 16 18 20
Eb/No in dB
 15
Capacity of M-ary Noncoherent FSK in AWGN
W. E. Stark, “Capacity and cutoff rate of noncoherent FSK
with nonselective Rician fading,” IEEE Trans. Commun., Nov. 1985.

M.C. Valenti and S. Cheng, “Iterative demodulation and decoding of turbo coded
M-ary noncoherent orthogonal modulation,” to appear in IEEE JSAC, 2005.

10 Noncoherent combining penalty

Minimum Eb/No (in dB)

M=2

5
M=4

M=16

M=64

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Rate R (symbol per channel use)
 15

Capacity of M-ary Noncoherent FSK in Rayleigh Fading

Ergodic Capacity (Fully interleaved)
Assumes perfect fading amplitude estimates available to receiver

10
Minimum Eb/No (in dB)

M=2

M=4
5

M=16

M=64

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Rate R (symbol per channel use)
 BICM
 Coded modulation (CM) is required to attain the
aforementioned capacity.
– Channel coding and modulation handled jointly.
– e.g. trellis coded modulation (Ungerboeck); coset codes (Forney)
 Most off-the-shelf capacity approaching codes are binary.
 A pragmatic system would use a binary code followed by
a bitwise interleaver and an M-ary modulator.
– Bit Interleaved Coded Modulation (BICM); Caire 1998.

u c' n cn Binary xk
Binary Bitwise to M-ary
Encoder Interleaver mapping

Turbo and LDPC Codes

6/7/2006 120/133
 Transforming Symbol Log-Likehoods
Into Bit LLRs
 Like the CM receiver, the BICM receiver calculates
log f (y|xk) for each signal in S.
 Furthermore, the BICM receiver needs to calculate the log-
likelihood ratio of each code bit:

p c n  1 | y 
 p x k | y  p y | x  p[ x
k k ]
x k S n(1) x k S n(1)
 n  log  log  log
p cn  0 | y   p x k | y   p y | x k  p[ xk ]
x k S n( 0 ) x k S n( 0 )

 max*
(1)
log f  y x k    max*
(0)
log f  y x k  
x k S n x k S n

(1 )
– where S n represents the set of symbols whose nth bit is a 1.

(0)
– and S n is the set of symbols whose nth bit is a 0.
Turbo and LDPC Codes
6/7/2006 121/133
 BICM Capacity
 BICM transforms the channel into  parallel binary
channels, and the capacity of the nth channel is:
C k  E c ,n  log( 2)  log p (c k | y ) nats
n

 p (c k | y ) 
 log( 2)  E cn ,n log  nats
 p ( c k  0 | y )  p (c k  1 | y ) 
 p ( c k  0 | y )  p (c k  1 | y ) 
 log( 2)  E cn ,n log  nats
 p (c k | y ) 
  p (c k  0 | y ) p (c k  1 | y ) 
 log( 2) - E cn ,n log exp log  exp log  nats
  p ( c k | y ) p ( c k | y ) 
  p (c k  0 | y ) p ( c k  1 | y ) 
 log( 2) - E cn ,n max* log , log  nats
  p (c k | y ) p ( c k | y ) 
  
 log( 2) - E cn ,n max* 0, (1) ck  k nats

Turbo and LDPC Codes

6/7/2006 122/133
 BICM Capacity (Continued)
 Since capacity over parallel channels adds, the capacity
of BICM is:

C   Ck
k 1

  nats

 
  log(2) - E cn ,n max* 0, (1) ck  k
k 1

  nats


  log(2)   E cn ,n max* 0, (1) ck  k
k 1

  bits


1

log(2) k 1

E cn ,n max* 0, (1) ck  k

Turbo and LDPC Codes

6/7/2006 123/133
 BICM Capacity
 As with CM, this can be computed using a Monte Carlo
integration. This function is computed by
the CML function Somap
For each bit, calculate:
Modulator: xk Receiver:
Pick xk at random Compute p(y|xk)  p y x 
xS n(1)
from S for every xk  S  n  log
nk  p y x 
xS n( 0 )

Noise Generator

For each symbol, calculate:

  bits


   E cn ,n max* 0, (1) ck k
k 1

Unlike CM, the capacity of BICM E 

depends on how bits are mapped to symbols
After running many C
trials, calculate: log(2)
Turbo and LDPC Codes
6/7/2006 124/133
 CM and BICM capacity for 16QAM in AWGN
1
CM M=16 QAM AWGN
0.9
BICM M=16 QAM gray
Code Rate R (4-bit symbols per channel use)

BICM M=16 QAM SP

0.8 BICM M=16 QAM MSP
BICM M=16 QAM Antigray
0.7 BICM M=16 QAM MSEW

0.6

0.5

0.4

0.3

0.2

0.1

0
-10 -5 0 5 10 15 20
Minimum Es/No (in dB)
 BICM-ID
 The conventional BICM receiver assumes that all bits in a
symbol are equally likely:
 p x | y   p y x 
xS n(1) xS n(1 )
 n  log  log
 p x | y   p y x 
xS n( 0 ) xS n( 0 )
 However, if the receiver has estimates of the bit
probabilities, it can use this to weight the symbol
likelihoods.  p y x  p(x | cn  1)
xS n(1)
 n  log
 p y x  p(x | c n  0)
xS n( 0 )
 This information is obtained from decoder feedback.
– Bit Interleaved Coded Modulation with Iterative Demodulation
– Li and Ritcey 1999.
Turbo and LDPC Codes
6/7/2006 126/133
 Mutual Information Transfer Chart
 Now consider a receiver that has a priori information
about the code bits (from a soft output decoder).
 Assume the following:
– The a priori information is in LLR form.
– The a priori LLR’s are Gaussian distributed.
– The LLR’s have mutual information Iv
 Then the mutual information Iz at the output of the receiver
can be measured through Monte Carlo Integration.
– Iz vs. Iv is the Mutual Information Transfer Characteristic.
– ten Brink 1999.

Turbo and LDPC Codes

6/7/2006 127/133
 Generating Random a Priori Input
1

0.9

0.8

0.7 There is a one-to-one correspondence

Mutual Information

between the mutual information

0.6 and the variance of the Gaussian
distributed a priori input
0.5

0.4

0.3

0.2

0.1

0
0 5 10 15 20 25 30 35 40 45 50
variance
 Mutual Information Characteristic
1

0.9 gray
SP
0.8 MSP
MSEW
0.7 Antigray

0.6

0.5
z
I

0.4 16-QAM
AWGN
0.3
6.8 dB
0.2

0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Iv
 EXIT Chart
1
16-QAM
0.9
AWGN
0.8 6.8 dB
adding curve for a FEC code
0.7 makes this an extrinsic information
transfer (EXIT) chart
0.6

0.5
z
I

0.4 gray
SP
0.3 MSP
MSEW
0.2 Antigray
K=3 Conv code
0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Iv
 EXIT Chart for
Space Time Block Code
1
16-QAM
0.9
8 dB
0.8
Rayleigh fading

0.7

0.6

0.5
z
I

0.4 1 by 1 MSP
2 by 1 Alamouti MSP
0.3 2 by 1 Alamouti huangNr1
2 by 2 Alamouti MSP
0.2 2 by 2 Alamouti huangNr2
K=3 Conv code
0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Iv
 EXIT Chart Analysis of Turbo Codes
 PCCC (turbo) codes can
be analyzed with an EXIT
chart by plotting the mutual
information transfer
characteristics of the two
decoders.
 Figure is from:
S. ten Brink, “Convergence
Behavior of Iteratively
Decoded Parallel
Concatenated Codes,”
IEEE Trans. Commun.,
Oct. 2001.

Turbo and LDPC Codes

6/7/2006 132/133
 Conclusions
 It is now possible to closely approach the Shannon limit by
using turbo and LDPC codes.
 Binary capacity approaching codes can be combined with
higher order modulation using the BICM principle.
 These code are making their way into standards
– Binary turbo: UMTS, cdma2000
– Duobinary turbo: DVB-RCS, 802.16
– LDPC: DVB-S2 standard.
 Software for simulating turbo and LDPC codes can be
found at www.iterativesolutions.com

Turbo and LDPC Codes

6/7/2006 133/133