Information and Coding Theory

Lecture-11: Linear Block Codes

Chadi Abou-Rjeily

Department of Electrical and Computer Engineering

Lebanese American University
[email protected]

March 31, 2020

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Mathematical Preliminary: Hamming distance (1)

Consider two sequences of length n given by

u = (u1 , . . . , un ) and v = (v1 , . . . , vn ). The
Hamming distance between u and v is given by:
n
X
dH (u, v ) = dH (ui , vi )
Richard Hamming (1915-1998)
i =1

where: 0, ui = vi ;
dH (ui , vi ) =
1, ui =
6 vi .

In other words, dH (u, v ) is the number of

components by which u and v differ.
Note that: dH (u, v ) = 0 ⇔ u = v

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Mathematical Preliminary: Hamming distance (2)

For binary sequences, the Hamming distance is related to the

Hamming weight by:

dH (u, v ) = wH (u + v )

In fact, if ui 6= vi , then ui + vi = 1.
Example: Let u = [1 1 0 0 1] and v = [1 0 1 0 0]:
dH (u, v ) is given by:

dH (u, v ) = 0 + 1 + 1 + 0 + 1 = 3

On the other hand, u + v = [0 1 1 0 1] resulting in:

wH (u + v ) = 3

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Introduction

In the previous lectures, we studied the channel coding

theorem which constitutes a theoretical bound on the
performance that can be achieved by a channel code.
In this lecture, we will study some explicit code constructions.
The purpose is to design and implement channel
encoders/decoders such that:
Information can be transmitted in a noisy environment as fast
as possible.
Information can be reconstituted efficiently at the output of
the decoder.
The complexity of the implementation of the encoder/decoder
must fall within acceptable limits.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Block Codes (1)

The input of a channel block encoder is composed of a

sequence of k bits.
The output of a channel block encoder is composed of a
sequence of n bits.
The rate of the code is given by:
k
Rc =
n
Rc corresponds to the number of information bits per
transmitted bit. Note that Rc < 1 since the channel encoder
introduces redundancy to ensure robust transmissions over
noisy channels.
Block encoders are memoryless. Successive encoded blocks
(of n symbols each) are independent.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Block Codes (2)
A code with parameters n and k will be called a (n, k) code.
The total number of messages is equal to 2k .
We say that the codebook is composed from 2k codewords.
Each codeword has a length n.
n − k redundant bits are introduced.
Examples:
A (5, 1) repetition code is given by:

A (3, 2) parity code is given by:

Note that the 22 = 4 codewords have even parity.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Linear Block Codes: Introduction (1)
Definition: A block code is linear if and only if the sum of any
two codewords is a codeword.
For example, the repetition code and the parity code are
linear.
Designate by Vn the set composed from all binary n-tuples:

Vn = {(v1 , . . . , vn ) ; vi ∈ {0, 1}}

Vn contains 2n elements.
Vn can be considered as a n-dimensional vector space over
{0, 1}.
Equivalent Definition: A (n, k) linear code is a k-dimensional
subspace of Vn .
For example, a plane is a 2-dimensional subspace of the
3-dimensional Euclidean space.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Linear Block Codes: Introduction (2)
In other words, it is possible to define a basis, and all the
codewords can be expressed as linear combinations of the
elements of this basis.
For example, in R3 , the xy-plane has degree 2 and a possible
choice of the basis corresponds to the two 3-dimensional
vectors u~ = (1, 0, 0) and ~
v = (0, 1, 0). Any vector of this plane
can be expressed in terms of this basis: (x, y , 0) = x u~ + y ~v
where x, y ∈ R.
Note that the basis is not unique.
For example, designate by {g1 , . . . , gk } the basis vectors
where each vector gi has a length n.
Each one of the 2k codewords can be expressed as a linear
combination of this basis.
In other words, the linear block encoding procedure associates
with every k-dimensional sequence u = (u1 , . . . , uk ) a
n-dimensional codeword v = (v1 , . . . , vn ) such that:
v = u1 g1 + u2 g2 + · · · + uk gk ; u1 , . . . , uk ∈ {0, 1}

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Generator matrix (1)
For a (n, k) linear block code we define the k × n generator
matrix G by:
   
g1 g1,1 g1,2 · · · g1,n
G =  ...  =  ... .. .. 
  
. ··· . 
gk gk,1 gk,2 · · · gk,n

In this case, the encoding procedure is equivalent to the simple

multiplication:
u → v : v 1×n = u 1×k G k×n
In other words:
 
g1
 
 g2 

v = uG = u1 u2 · · · uk  ..  = u1 g1 + u2 g2 + · · · + uk gk
 . 
gk

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Generator matrix (2)

For example, the generator matrix of the (5, 1) repetition code

is given by:  
G= 1 1 1 1 1
The encoding procedure is described by:
 
u = (u1 ) → v = uG = u1 u1 u1 u1 u1

In particular:
 
0 → 0G = 0 0 0 0 0
 
1 → 1G = 1 1 1 1 1

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Generator matrix (3)
For example, the generator matrix of the (3, 2) parity code is
given by:  
1 0 1
G=
0 1 1
The sequence u = (u1 , u2 ) is encoded to the codeword
v = (v1 , v2 , v3 ) such that:
 
    1 0 1
v = v1 v2 v3 = uG = u1 u2
0 1 1
 
= u1 u2 u1 + u2
In other words:
v1 = u1
v2 = u2
v3 = u1 + u2
where v3 is the parity-check redundant bit.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Generator matrix (4)
Consider the (5, 3) code whose generator matrix is given by:
 
1 1 1 0 0
G = 0 0 1 1 0 
1 1 1 1 1
The information sequence u = (u1 , u2 , u3 ) is encoded to the
codeword v = (v1 , v2 , v3 , v4 , v5 ) such that:
 
  1 1 1 0 0
v = uG = u1 u2 u3  0 0 1 1 0 
1 1 1 1 1
 
= u1 + u3 , u1 + u3 , u1 + u2 + u3 , u2 + u3 , u3
In other words, the codebook is given by:

{[0, 0, 0, 0, 0] , [1, 1, 1, 1, 1] , [0, 0, 1, 1, 0] , [1, 1, 0, 0, 1] ,

[1, 1, 1, 0, 0] , [0, 0, 0, 1, 1] , [1, 1, 0, 1, 0] , [0, 0, 1, 0, 1]}
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Systematic encoding (1)

It is often convenient to have the original data explicitly

evident in the codeword.
This is referred to as systematic encoding.
For systematic encoding, the generator matrix takes the
following form:
 
1 0 · · · 0 P1,1 · · · P1,n−k
.
  0 1 . . . .. P2,1 · · · P2,n−k
 
 
G = Ik P =  . . 
.. ..

 .. .. ... 0 .
..
. .


0 ··· 0 1 Pk,1 · · · Pk,n−k

where Ik is the k × k identity matrix and P is a k × (n − k)

matrix.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Systematic encoding (2)
For systematic encoders, the vector u = (u1 , . . . , uk ) is
encoded to the codeword v = (v1 , . . . , vn ) such that:

vi = ui , 1 ≤ i ≤ k;
vi +k = u1 P1,i + u2 P2,i + · · · + uk Pk,i , 1 ≤ i ≤ n − k.
For example, consider the (7, 4) Hamming code whose
generator matrix is given by:
 
1 0 0 0 0 1 1
 0 1 0 0 1 0 1 
G =  0 0 1

0 1 1 0 
0 0 0 1 1 1 1
then:
 
u1 u2 u3 u4 →
 
u1 , u2 , u3 , u4 , u2 + u3 + u4 , u1 + u3 + u4 , u1 + u2 + u4

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Systematic encoding (3)
Every generator matrix can be written in a systematic form by
performing:
row reductions.
column reordering.
Example:
Consider the matrix G given by:
   
g1 1 1 1 0 0
G =  g2  =  0 0 1 1 0 
g3 1 1 1 1 1
Replacing the second row by g1 + g2 results in:
 
1 1 1 0 0
 1 1 0 1 0 
1 1 1 1 1
Replacing the third row by g2 + g3 and permuting the columns:
   
1 1 1 0 0 1 0 0 1 1
 1 1 0 1 0  ⇔  0 1 0 1 1 
1 1 0 0 1 0 0 1 1 1
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Parity-check matrix (1)

Definition: The parity-check matrix of a linear (n, k) code C, is a

(n − k) × n matrix verifying:

cH T = 0n−k ; ∀c ∈ C

where 0n−k is the sequence composed of n − k zeros.

In other words, a codeword is orthogonal to each row of H.
The rows of H generate the null-space of the code C. A
codeword is orthogonal to every element of this null-space.
For example, let n = 3 and k = 2. Over R3 , we’ve seen that
u~ = (1, 0, 0) and ~
v = (0, 1, 0) are the basis of a subspace of
dimension k = 2 (xy-plane). The null space of this subspace
will have a dimension of n − k = 1. This is the subspace whose
basis is determined by w ~ = (0, 0, 1) (the z-axis). Note that any
vector of the z-axis is orthogonal to all vectors of the xy-plane.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Parity-check matrix (2)
Proposition: When G is written in a systematic form: G = [Ik P]
(with P being a k × (n − k) matrix), then H takes the form:
 
H = P T In−k
Proof:
GH T is given by:
  T
GH T = Ik P PT In−k
 
  P
= Ik P T
In−k
 
  P
= Ik P
In−k
= Ik P + PIn−k
= P + P = 0k×(n−k)
The last equality follows since the addition is a modulo-2
addition (0 + 0 = 0 and 1 + 1 = 0).
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Parity-check matrix (3)

Example, for the (7, 4) Hamming code:

 
1 0 0 0 0 1 1  
 0 1 0 1 1 1 1 0 0
0 0 1 0 1 
G =
 0 0
 ⇒ H= 1 0 1 1 0 1 0 
1 0 1 1 0 
1 1 0 1 0 0 1
0 0 0 1 1 1 1

For the (3, 2) parity-check code:

 
1 0 1  
G= ⇒ H= 1 1 1
0 1 1

In this case c = (c1 , c2 , c3 ) is a codeword if cH T = 0 implying

that:
c1 + c2 + c3 = 0 : even parity

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (1)

Assume that the codeword v was transmitted over the

channel. Over a BSC, the received vector r can be written as:

r =v +e

where e = (e1 , . . . , en ) is the error vector. Note that:

ei = 1 indicates a transmission error in the i-th component
ri 6= vi .
ei = 0 if and only if ri = vi .
The syndrome of the received vector r is defined by the
(n − k)-dimensional sequence s given by: s = rH T .
An important property of the syndrome is that it depends only
on the error pattern and not on the transmitted codeword:

s = rH T = (v + e)H T = vH T + eH T = eH T

since vH T = 0n−k because v is a codeword.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (2)
Syndrome Decoding: find the value of the transmitted codeword
based on the received vector r .
1 Compute the syndrome s = rH T .

2 Find the most probable error pattern e.

3 The output of the decoder will consist of:

v̂ = r − e = r + e

The error pattern e can not be determined in a unique way

from the syndrome s. In fact:

s = rH T = eH T ⇒
 
h1,1 · · · hn−k,1
   
 h1,2 · · · hn−k,2 

s1 · · · sn−k = e1 e2 · · · en  .. .. .. 
 . . . 
h1,n · · · hn−k,n

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (3)

Consequently:

 s1
 = h1,1 e1 + h1,2 e2 + · · · + h1,n en
..
 .

sn−k = hn−k,1 e1 + hn−k,2 e2 + · · · + hn−k,n en

Note that in the absence of error (e = (0, 0, . . . , 0)), all the

components of s are equal to zero.
Since the above system corresponds to a system of n − k
equations in n unknowns e1 , . . . , en , then it admits
2n−(n−k) = 2k possible solutions.
When decoding, we choose the most probable error pattern.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (4)
Proposition: The most probable error pattern is the one having
the smallest Hamming weight wH (e).
Proof:
The error vector is given by: e = (e1 , . . . , en ). Designating by
p ≤ 12 the parameter of the stationary and memoryless BSC
channel. Consequently:

p, ei = 1 (error) ;
p(ei ) = for i = 1, . . . , n
1 − p, ei = 0 (no error) .
Consequently:
n
Y
p(e) = p(ei )
i =1
= (1 − p)n−wH (e) p wH (e)
 wH (e)
n p
= (1 − p)
1−p
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (5)

Note that:
The first equality follows since the n components of e are
independent (since the channel is memoryless).
The second equality follows since:
The number of 1’s in e (of probability p each) is equal to
wH (e).
The number of 0’s in e (of probability 1 − p each) is equal to
n − wH (e).
p
Given that p ≤ 21 , then 0 ≤ 1−p ≤ 1 implying that p(e) is a
decreasing function of wH (e).
Therefore, the most frequent non-zero error pattern is the
vector e such that wH (e) has the smallest non-zero value.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (6)
Example: Consider the (7, 4) Hamming code whose parity-check
matrix is given by:
 
0 1 1 1 1 0 0
H= 1 0 1 1 0 1 0 
1 1 0 1 0 0 1

Determine the value of the transmitted codeword if the received

vector is r = (1, 0, 0, 0, 1, 1, 0).
The syndrome is given by s = rH T = [1 0 1] (note that r is
not a codeword implying that an error has occurred during the
transmission).
s = eH T implies that:

 1 = e2 + e3 + e4 + e5
0 = e1 + e3 + e4 + e6

1 = e1 + e2 + e4 + e7

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (7)
The above system of equations admits 2k = 24 = 16 possible
solutions.
These solutions can be determined, for example, by varying
the values of e1 , e2 , e3 and e4 in {0, 1} and solving for e5 , e6
and e7 .
The solutions are given by:
[e1 e2 e3 e4 ] = [0 0 0 0] ⇒ [e5 e6 e7 ] = [1 0 1] ; wH (e) = 2
[e1 e2 e3 e4 ] = [0 0 0 1] ⇒ [e5 e6 e7 ] = [0 1 0] ; wH (e) = 2
[e1 e2 e3 e4 ] = [0 0 1 0] ⇒ [e5 e6 e7 ] = [0 1 1] ; wH (e) = 3
[e1 e2 e3 e4 ] = [0 0 1 1] ⇒ [e5 e6 e7 ] = [1 0 0] ; wH (e) = 3
[e1 e2 e3 e4 ] = [0 1 0 0] ⇒ [e5 e6 e7 ] = [0 0 0] ; wH (e) = 1
[e1 e2 e3 e4 ] = [0 1 0 1] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [0 1 1 0] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [0 1 1 1] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Syndrome decoding (8)
[e1 e2 e3 e4 ] = [1 0 0 0] ⇒ [e5 e6 e7 ] = [1 1 0] ; wH (e) = 3
[e1 e2 e3 e4 ] = [1 0 0 1] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [1 0 1 0] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [1 0 1 1] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [1 1 0 0] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [1 1 0 1] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [1 1 1 0] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1
[e1 e2 e3 e4 ] = [1 1 1 1] ⇒ [e5 e6 e7 ] = [∗ ∗ ∗] ; wH (e) > 1

The error pattern having the smallest Hamming weight is

e = (0, 1, 0, 0, 0, 0, 0).
Consequently:
v = r +e = (1, 0, 0, 0, 1, 1, 0)+(0, 1, 0, 0, 0, 0, 0) = (1, 1, 0, 0, 1, 1, 0)
Note that, in this example, syndrome decoding necessitates
solving 6 systems of 3 unknowns each.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Maximum-Likelihood Detection (1)
Assume that the codeword c = (c1 , . . . , cn ) was transmitted
and that the vector r = (r1 , . . . , rn ) was received.
An error occurs when the decoder decides in the favor of the
codeword c ′ 6= c. The probability of error is given by:
X
P(e) = P(r )P(e|r )
r
X
= P(r )P(c ′ 6= c|r )
r
Given that P(r ) is independent from the decoding rule, then
minimizing the probability of error is equivalent to minimizing
P(e|r ) for all values of r .
Consequently:
min P(c ′ 6= c|r ) ⇔ max P(c ′ = c|r ) ∀ r
In other words, given the received vector r , the receiver
decides in the favor of the codeword c that maximizes:
P(c|r )
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Maximum-Likelihood Detection (2)
On the other hand:
P(r |c)P(c)
P(c|r ) =
P(r )
Since r is produced prior to decoding, then maximizing P(c|r )
is equivalent to maximizing:
P(r |c)P(c)

If we now assume that all the codewords are equiprobable,

then the optimal detection corresponds to deciding in the
favor of the codeword c that maximizes:
P(r |c)

A codeword that is selected on the basis of maximizing P(r |c)

is said to be selected according to the Maximum Likelihood
(ML) criterion .
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Maximum-Likelihood Detection (3)
Q
On the other hand: P(r |c) = ni=1 p(ri |ci ) where:

p, ri 6= ci (error);
p(ri |ci ) =
1 − p, ri = ci (no error).
Consequently:
Y n
P(r |c) = p(ri |ci ) = (1 − p)n−dH (r ,c) p dH (r ,c)
i =1
 dH (r ,c)
n p
= (1 − p)
1−p
p
Since 0 ≤ p ≤ 12 , then 0 ≤ 1−p ≤ 1. Consequently,
maximizing P(r |c) is equivalent to minimizing dH (r , c).
As a conclusion, a ML decoder decides in the favor of the
codeword that is the closest to r :
ĉ = arg min dH (r , c)
c∈C

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Maximum-Likelihood Detection (4)
Example: Consider the (7, 4) Hamming code whose parity-check
matrix is given by:
 
0 1 1 1 1 0 0
H= 1 0 1 1 0 1
 0 
1 1 0 1 0 0 1

By applying the ML criterion, determine the value of the

transmitted codeword if the received vector is
r = (1, 0, 0, 0, 1, 1, 0).
The generator matrix is given by:
 
1 0 0 0 0 1 1
 0 1 0 0 1 0 1 
G =  0 0 1 0

1 1 0 
0 0 0 1 1 1 1

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Maximum-Likelihood Detection (5)
The 2k = 16 codewords as well as the distances between these
codewords and r are shown below:
c1 = [0 0 0 0 0 0 0] , dH (r , c1 ) = 3 ; c9 = [1000011] , dH (r , c9 ) = 2
c2 = [0 0 0 1 1 1 1] , dH (r , c2 ) = 3 ; c10 = [1001100] , dH (r , c10 ) = 2
c3 = [0 0 1 0 1 1 0] , dH (r , c3 ) = 2 ; c11 = [1010101] , dH (r , c11 ) = 3
c4 = [0 0 1 1 0 0 1] , dH (r , c4 ) = 6 ; c12 = [1011010] , dH (r , c12 ) = 3
c5 = [0 1 0 0 1 0 1] , dH (r , c5 ) = 4 ; c13 = [1100110] , dH (r , c13 ) = 1
c6 = [0 1 0 1 0 1 0] , dH (r , c6 ) = 4 ; c14 = [1101001] , dH (r , c14 ) = 5
c7 = [0 1 1 0 0 1 1] , dH (r , c7 ) = 5 ; c15 = [1110000] , dH (r , c15 ) = 4
c8 = [0 1 1 1 1 0 0] , dH (r , c8 ) = 5 ; c16 = [1111111] , dH (r , c16 ) = 4

This implies that the most probable transmitted codeword is:

c13 = [1 1 0 0 1 1 0].
Note that the same result was obtained with the syndrome
decoding.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Maximum-Likelihood Detection (6)

Note that the complexity of the ML decoder grows

exponentially with k and this complexity becomes prohibitive
for large values of k.
In this context, the syndrome decoder requires a reduced
complexity.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Minimum Distance of a code (1)
The minimum distance of a code is defined as:

dmin = min{dH (x, x ′ ) ; x, x ′ ∈ C , x 6= x ′ }

where C corresponds to the codebook. x and x ′ are two

different codewords.
However, the Hamming distance between 2 binary vectors x
and x ′ verifies the relation:

dH (x, x ′ ) = wH (x + x ′ )

On the other hand, since the code is linear, then the sum of 2
codewords is also a codeword. In other words, x + x ′
corresponds to a codeword of C. Consequently:

dmin = min{wH (c) ; c ∈ C , c 6= 0}

The minimum distance of a linear block code is equal to the
minimum weight of its non-zero codewords.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Minimum Distance of a code (2)
The (5, 1) repetition codebook is given by:
{(0, 0, 0, 0, 0) , (1, 1, 1, 1, 1)}
and this code has a minimum distance of dmin = 5.
The (3, 2) parity-check codebook is given by:
{(0, 0, 0) , (0, 1, 1) , (1, 0, 1) , (1, 1, 0)}
and this code has a minimum distance of dmin = 2.
Consider the (5, 3) code whose generator matrix is:
 
1 1 1 0 0
G = 0 0 1 1 0 
1 1 1 1 1
the codebook is given by:
{[0, 0, 0, 0, 0] , [1, 1, 1, 1, 1] , [0, 0, 1, 1, 0] , [1, 1, 0, 0, 1] ,
[1, 1, 1, 0, 0] , [0, 0, 0, 1, 1] , [1, 1, 0, 1, 0] , [0, 0, 1, 0, 1]}
and this code has a minimum distance of dmin = 2.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Minimum Distance of a code (3)
Proposition: For a linear block code having a parity-check matrix
H, the minimum distance is equal to the smallest number of
columns of H whose sum is equal to zero.
Consider the (5, 1) repetition code:
 
1 1 0 0 0
   1 0 1 0 0 
G= 1 1 1 1 1 ; H=  1 0 0 1 0 


1 0 0 0 1
There is no zero column. There are no 2, 3 or 4 columns of H
whose sum is equal to zero. The sum of the 5 columns of H is
equal to zero and dmin = 5.
Consider the (3, 2) parity-check code:
 
H= 1 1 1
There is no column that is equal to zero, then dmin > 1. The
sum of any two columns of H is equal to zero and dmin = 2.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Minimum Distance of a code (4)

Consider the (7, 4) Hamming code:

 
0 1 1 1 1 0 0
H= 1 0 1 1 0 1 0 
1 1 0 1 0 0 1

There is no column that is equal to zero, then dmin > 1.

There are no two columns whose sum is equal to zero (no two
identical columns), then dmin > 2.
Consider the first, second and third columns:
       
0 1 1 0
 1 + 0 + 1  =  0 
1 1 0 0

Consequently, dmin = 3.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (1)
Note that if the error pattern e is identical to a non-zero
codeword, then:
s = rH T = (v + e)H T = vH T + eH T = 0n−k + 0n−k = 0n−k

In other words, the syndrome is zero even though r 6= v .

Errors of this kind are called undetectable errors.
The total number of undetectable errors is equal to the
number of non-zero codewords (2k − 1).
The total number of error patterns that a (n, k) linear code
can detect is:
n
|2 {z
− 1} − (2k − 1) = 2n − 2k
| {z }
possible non-zero error paterns undetectable error patterns

Note that in general, 2k is much smaller than 2n and only a

small fraction of error patterns passes through the decoder
without being detected.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (2)

When the codeword v is transmitted, the received vector r

can be expressed as:

r =v +e ⇒ dH (r , v ) = wH (e)

When wH (e) 6= 0 (an error was introduced by the channel):

If wH (e) < dmin , then r can not be a codeword since
dH (r , v ) = wH (e) < dmin and any two codewords are separated
by at least dmin . In this case, the syndrome of r is always
different from zero and the error will be detected.
If wH (e) ≥ dmin , then r can correspond to a codeword. In this
case, it is possible to have an undetectable error since the
syndrome of r will be equal to zero (since r is a codeword)
even if r 6= v .
A linear block code with minimum distance dmin is capable of
detecting all error patterns of dmin − 1 or fewer errors.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (3)

Note that even though such codes can not detect all error
patterns with dmin or more errors, they are capable of
detecting a large fraction of such errors.
In particular, we have previously proven that a (n, k) linear
code is capable of detecting a total number of error patterns
that is given by:

number of detectable error patterns = 2n − 2k

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (4)

For example, the (3, 2) parity code has a minimum distance of

dmin = 2. Consequently:
This code is capable of detecting all error patterns having
dmin − 1 = 1 error (one bit is inverted). In this case, the
inverted bit will break the parity of the transmitted codeword
and the receiver will detect the presence of an error over the
channel.
On the other hand, this code can not detect error patterns
with two errors since such patterns do not violate the parity of
the transmitted codeword.
Finally, this code is capable of detecting error patterns with
three errors since in this case the received codeword will have
an odd parity.
Overall, this code is capable of detecting the following
23 − 22 = 4 error patterns: (1, 0, 0), (0, 1, 0), (0, 0, 1) and
(1, 1, 1).

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (5)

The weight distribution of a code is given by the sequence

of numbers {Ai }ni=0 where Ai is the number of codewords
whose Hamming weight is equal to i .
The weight distribution of the (5, 1) repetition code
{(0, 0, 0, 0, 0), (1, 1, 1, 1, 1)} is A0 = 1, A5 = 1 and
A1 = A2 = A3 = A4 = 0.
The weight distribution of the (3, 2) parity code
{(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)} is A0 = 1, A1 = 0, A2 = 3
and A3 = 0.
Note that for any linear (n, k) code whose minimum distance
is dmin :
A0 = 1.
Pi n= 0 for 1 ≤ iP
A ≤ dmin − 1.
n k
i =0 Ai = 1 + i =dmin Ai = 2 (since the total number of
k
codewords is equal to 2 ).

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (6)

Since an undetected error occurs only when the error pattern

is identical to a non-zero codeword, then the probability of an
undetected error can be written as:
n
X
Pu = Ai p i (1 − p)n−i
i =1
Xn
= Ai p i (1 − p)n−i
i =dmin

Since in practice the probability p is small, then the dominant

term in the above summation corresponds to i = dmin :

Pu ≃ Admin p dmin (1 − p)n−dmin

≃ Admin p dmin

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Detection (7)

The weight distribution of the (7, 4) Hamming code (whose

minimum distance is equal to dmin = 3) is given by:

A0 = 1 ; A1 = A2 = 0 ; A3 = A4 = 7 ; A5 = A6 = 0 ; A7 = 1

Consequently:

Pu = 7p 3 (1 − p)4 + 7p 4 (1 − p)3 + p 7

For p = 10−2 :
Pu ≃ 7 × 10−6
In other words, if 1 million codewords are transmitted over a
BSC with parameter p = 10−2 , on the average there are only
7 erroneous bit sequences passing through the decoder
without being detected.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Correction (1)

Suppose that we want the linear code C to be capable of

correcting all errors of Hamming weight ≤ t.
In other words, if ci ∈ C was sent and r = ci + e was received
with wH (e) ≤ t, we want our decoder’s output to be ĉ = ci .
We have previously seen that the ML detection corresponds to
choosing the closest codeword to r .

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Correction (2)

The code will be capable of correcting all error patterns of

weight ≤ t if and only if the distance between each pair of
codewords is > 2t. In fact:
In fact, if dH (ci , cj ) > 2t for all pairs of codewords, the spheres
of radii t around ci and cj are disjoint.
Since dH (r , ci ) = wH (e) ≤ t, then r can not be closer to any
codeword cj 6= ci than it is to ci .
Note that since the Hamming distance is an integer, the
spheres will not overlap if the distance between each pair of
codewords is ≥ 2t + 1.
The last relation implies that dmin must verify:
dmin ≥ 2t + 1
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Correction (3)

In other words, a code with minimum distance dmin is capable

of correcting all error patterns with t errors where:
 
dmin − 1
t=
2

The (5, 1) repetition

 code with dmin = 5 is capable of
correcting 5−1 2 = 2 errors.
The (3,
 2−1  2) parity code with dmin = 2 is capable of correcting
2 = 0 errors. This code is applied since it is capable of
detecting all error patterns with dmin − 1 = 1 error.
The (7, 4) Hamming
 code with dmin = 3 is capable of
correcting 3−1 2 = 1 error.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Error Correction (4)

Designate byj dmin thek minimum distance of the block code

dmin −1
and let t = 2 .
Given that the block code is capable of correcting all error
patterns of t or less errors, then the probability of error (of
the entire codeword) can be upper bounded by:
n  
X n i
Pe ≤ p (1 − p)n−i
i
i =t+1

where the inequality follows from the fact that the block code
might correct some error patterns of t + 1 or more errors.
When p ≪ 1, the dominant term in the above summation is:
    j d −1 k   j d +1 k
n t+1 nk min +1 n k min
p = j
dmin −1 p 2
= j
dmin +1 p 2
t +1 2 +1 2

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Coding gain over AWGN channels (1)
Consider an AWGN channel where:
Y =X +Z
X is the transmitted signal having an energy E[X 2 ] = P.
Z stands for the AWGN noise having an energy E[Z 2 ] = N.
Y stands for the received signal.
The signal-to-noise ratio is defined as SNR = P/N.
The typical performance over AWGN channels is shown below:

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Coding gain over AWGN channels (2)

In the previous figure:

The curve “coded transmission without decoding” corresponds
to the curve “uncoded transmission” shifted by 10 log10 (n/k)
dB to the right.
This results from the fact that with linear block codes n bits
are used for the transmission of k information bits with n > k.
For a fair comparison between uncoded systems and coded
systems (that require the transmission of a larger number of
bits), an additional penalty of 10 log10 (n/k) dB is imposed on
coded systems.
This can be seen as normalizing the SNR.
Note that linear block codes result in higher performance gains
at higher SNRs.

Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes
Coding gain over AWGN channels (3)
It can be proven that at medium to high SNR, the coding
gain of a linear block code is given by:
10 log10 [Rc dmin ] dB
Note that Rc < 1: the simple redundancy results in a loss.
dmin > 1: the structure of the code results in a gain.
The product Rc dmin must be greater than 1 (to have a net
gain). In fact, the term Rc dmin must be maximized.
For example:
For the (7, 4) Hamming code, dmin = 3 and the coding gain is:
 
4
10 log10 [Rc dmin ] = 10 log10 3 ≈ 2.34 dB
7
For the (n, 1) repetition code, dmin = n and the coding gain is:
 
1
10 log10 [Rc dmin ] = 10 log10 n = 0 dB
n
In other words, there is no coding gain and repetition codes
are useless.
Chadi Abou-Rjeily Information and Coding Theory Lecture-11: Linear Block Codes