Information Theory and

Coding

Dr. Atri Mukhopadhyay

Assistant Professor
EEE Department
BITS Pilani Hyderabad Campus

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 About the Course
Course No. : ECE F344

Course Title : Information Theory and Coding

Lecture Class Timings : Tue, Thus, Sat 10:00 to 10:50 AM

Tutorial Timings : Fri 08:00 to 08:50 AM (Sec 1)

Tue 2:00 PM to 2:50 PM (Sec 2)

06/29/2023 2

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Evaluations Scheme
Sl. No. Evaluation Marks Time Date Component
Component Nature
1 Mid-Sem 90 90 mins As per Time Closed Book
Table
2 Quizzes 30 30 mins TBA Closed Book
3 Assignments 60 - TBA Open Book
5 Comprehensive Test 120 3 hours As per Time Closed Book
Table
Grand Total 300

1. No. of Announced Quizzes: 2

2. There will be –ve marking in all evaluation components.
3. No make-up for announced quizzes.
4. Make-up will be allowed for mid-term and end-term exam only for the
genuine cases with prior intimations.
06/29/2023 3

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Evaluations Scheme

1. All notices and related study materials will be shared via CMS only, and thus
you need to register in CMS.

06/29/2023 4

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Course description
• Information Theory basics; motivation and definitions

• Source Coding

• Channel Coding

• Cryptography

06/29/2023 5

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Text Books
Text Book:
 
• Information Theory, Coding and Cryptography, 3rd Ed., Dr. Ranjan Bose, Tata
McGrawHill, 2016

Reference Books:

• Elements of information theory, Thomas M.Cover and Joy A.Thomas, Wiley-

India
• Foundations of Coding, Jiri Adamek, John Wiley, 1991
• The Mathematics of Coding Theory, Paul Garrett, Pearson Education, 2005
• Information Theory, Inference and Learning Algorithms, David Mackay,
Cambridge University Press, 2003
• Coding Theory – A First course, Ling and Xing, Cambridge University press,
2004
06/29/2023 6

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Motivation to Study Information Theory

 Information theory applies to - The messages occurring in

any of the standard communications media, such as
telephone, radio, or television, and the signals involved in
electronic computers, electromechanical systems, and
other data-processing devices.
 Claude E. Shannon - The Mathematical Theory of
Communication” in the Bell System Technical Journal.
 The theory is even applicable to the signals appearing in
the nerve networks of humans and other animals.

06/29/2023 7

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Motivation to Study Information Theory

 The chief concern of information theory is to discover

mathematical laws governing systems designed to
communicate or manipulate information.
 It sets up quantitative measures of information and of the
capacity of various systems to transmit, store, and
otherwise process information.
 Some of the problems treated are related to finding the
best method for separating the wanted information, or
signal, from the extraneous information, or noise.
 Another problem is the setting of upper bounds on what it
is possible to achieve with a given information-carrying
medium.

06/29/2023 8

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Uncertainty and Information

 Let us consider the following three sentences:

 Tomorrow, the sun will rise in the east.
 The phone will ring in the next one hour.
 Tomorrow, it will snow in Hyderabad.

 Takeaways:
 The length of the sentence is not related to information content.
 The Uncertainty of an event (probability of occurrence)
enhances information content.
 If the output is exactly known, there is no need for
transmitting information.

06/29/202 9
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Uncertainty and Information

• The units of I(x ) are determined by the base of the logarithm,

i

which is usually selected as 2 or e.

• When the base is 2, the units are in bits and

• When the base is e, the units are in nats (natural units).

06/29/202 10
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Why Logarithm?

Example 1
Consider a binary source which tosses a fair coin and produces an
output equal to 1 if a head appears and a 0 if a tail appears.

I(x ) = -log2P(x ) = -log2 0.5 = 1

i i

Indeed, we have to use only one bit to represent the output from this
binary source (say, we use a 1 to represent a Head & 0 to represent
a Tail)

06/29/202 11
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Why Logarithm?
Example 1
• Now, let us say that the successive outputs from this binary
source is statistically independent (source is memoryless).
• If we consider m binary digits. We have 2m possible and
equiprobable m-bit blocks.
• Thus the probability of each block pattern is 2 -m
I(x ) = -log2P(x ) = -log2 2 -m = m
i i

• So, logarithm provides a desired additive property that

increases with the number of sources

06/29/202 12
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Why Logarithm?
Example 2
• Now, consider a source C which is a combination of two binary
sources A and B.
• The possible outputs are therefore equiprobable {00, 01, 10, 11}.
• P(C) = P(A)P(B) = 0.5 * 0.5 = 0.25
• Intuitively, the information content of the aggregate source C
should be the sum of the information content of independent
sources A and B. Recall from previous example.

I(x ) = -log2P(x ) = -log2 0.25 = 2

i i

06/29/202 13
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Why Logarithm?
Takeaway
Independent Event => Probabilities Multiply
Independent Sources => Information should add up

Logarithm does the job!!!

06/29/202 14
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Mutual Information

• Let us consider two discrete random variables X and Y

with possible outcomes xi , i = 1,2,3, …, n and yj , j =
1,2,3, …, m respectively
• Suppose, we want to determine the information of an
event X = xi after observing the event Y = yj

06/29/202 15
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Mutual Information
• Let us look at the two extreme situations
• X and Y are independent -> In this case, the occurrence of
provides no information about .
• X and Y are fully dependent -> In which case the occurrence of
determines the occurrence of the event
• A possible measure

• Simplifying

06/29/202 16
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Mutual Information
• We call the log of this measure as mutual information.
• Mutual information

• =
• When X and Y are statistically independent
=0
• When occurrence of Y = yj uniquely identifies X = xi ;
=1
• Therefore,

06/29/202 17
3
ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION
 Mutual Information

• Let X and Y be binary

random variables that
represent the input and
output of this BSC.

• Let the input symbols

be equally likely

06/29/2023 18

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Mutual Information

06/29/2023 19

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Mutual Information

Mutual Information about X = 0 given that Y = 0 is

Similarly,

06/29/2023 20

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Mutual Information

• If the channel is ideal (noiseless), [p = 0]

• If p = 0.5

• Channel is useless as the value of y cannot be determined

from the value of x

What if p = 1?

06/29/2023 21

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Conditional Self Information

• Conditional Self information of the event X = given Y = is

defined as

• Therefore, we may write

06/29/2023 22

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Conditional Self Information

• The conditional self-information can be interpreted as the self-

information about the event X = having observed the event Y
=.
• Recall that both ≥ 0 and ≥ 0.
• Therefore, < 0 when < and I(xi; yj) > 0 when > . Hence,
mutual information can be positive, negative or zero.

06/29/2023 23

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Mutual Information

Back to example

• If p = 0.1

• If p = 0.9

• If p = 1

06/29/2023 24

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Mutual Information

06/29/2023 25

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Mutual Information

• The Average Mutual Information between two random

variables X and Y is given by

06/29/2023 26

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Mutual Information

• If X and Y are statistically independent ( = 0), then there is no

average mutual information between X and Y

• An important property of the average mutual information is

that = 0 with equality if and only if X and Y are statistically
independent

06/29/2023 27

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Self Information

• When X represents the alphabet of possible output letters from a

source, H(X) represents the average information per source letter.
In this case, H(X) is called the entropy.

• The entropy of X can be interpreted as the expected value of

• We observe that since Hence H(X)

06/29/2023 28

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Theory and
Coding

Dr. Atri Mukhopadhyay

Assistant Professor
EEE Department
BITS Pilani Hyderabad Campus

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Self Information

• When X represents the alphabet of possible output letters from a

source, H(X) represents the average information per source letter.
In this case, H(X) is called the entropy.

• The entropy of X can be interpreted as the expected value of

• We observe that since Hence H(X)

06/29/2023 30

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Self Information

Example
• Consider a discrete binary source that emits a sequence of
statistically independent symbols. The output is either 0 with
probability p or 1 with probability ).

• Entropy can be calculated as

06/29/2023 31

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Self Information

Example
• The entropy of the source
is maximum when p = 0.5,
i.e., both 1 and zero are
equally likely.
• Find this by principles of
Maxima/minima.

06/29/2023 32

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Conditional Self Information

• The physical Interpretation is as follows:

• is the information (or uncertainty) in X after observing Y

• Based on the definitions:

06/29/2023 33

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Conditional Self Information

06/29/2023 34

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Proof that

Using Jensen Inequality

𝑡𝑓 ( 𝑥1 ) +(1− 𝑡 ) 𝑓 ( 𝑥 2 ) ≥ 𝑓 (𝑡 𝑥1 +(1 − 𝑡 ) 𝑥 2 )

Generalizing
𝑡 1 𝑓 ( 𝑥 1 )+𝑡 2 𝑓 ( 𝑥 2 )+ … 𝑡 𝑛 𝑓 ( 𝑥 𝑛 ) ≥ 𝑓 (𝑡 1 𝑥1 +𝑡 2 𝑥2 +… 𝑡 𝑛 𝑥 𝑛 )

𝑡𝑖 = 𝑃 ( 𝑋 = 𝑥 𝑖 )
𝐸[ 𝑓 ( 𝑋)]≥ 𝑓 (𝐸 [ 𝑋 ])

is convex

06/29/2023 35

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Proof that

( )
𝑛 𝑚
P(𝑥𝑖, 𝑦𝑗)
𝐼 (𝑋 ;𝑌 )=∑ ∑ P(𝑥𝑖, 𝑦 𝑗)𝑙𝑜𝑔
𝑖=1 𝑗=1 P(𝑥 𝑖) P( 𝑦 𝑗)

( )
𝑛 𝑚
P(𝑥𝑖)P( 𝑦 𝑗)
¿− ∑ ∑ P(𝑥𝑖, 𝑦 𝑗)𝑙𝑜𝑔
𝑖=1 𝑗=1 P(𝑥𝑖, 𝑦𝑗)

( ( )
𝑛 𝑚
P(𝑥𝑖)P(𝑦 𝑗)
≥−𝑙𝑜𝑔 ∑ ∑ P(𝑥𝑖, 𝑦 𝑗)
𝑖=1 𝑗=1 P(𝑥 𝑖, 𝑦𝑗)

(∑ )
𝑛 𝑚
¿−𝑙𝑜𝑔 P(𝑥𝑖)∑ P( 𝑦 𝑗)
𝑖=1 𝑗=1

¿ −𝑙𝑜𝑔 ( 1 )
¿0
06/29/2023 36

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Average Conditional Self Information

• Since
• Which means that X and Y are independent.
• Since is the average amount of uncertainty in X after we observe
Y and is the average amount of uncertainty of X, is the average
amount of uncertainty about X having observed Y.
• Since , the observation of Y does not increase the entropy. It can
only decrease entropy. Meaning, observing Y cannot reduce
information about X, it can only add information.

06/29/2023 37

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Joint Entropy

06/29/2023 38

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Theory and
Coding

Dr. Atri Mukhopadhyay

Assistant Professor
EEE Department
BITS Pilani Hyderabad Campus

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

• Compression of data due to efficient representation of the

symbols.
• Required in Text, Audio, Video, Data
• Efficient Transmission over channels
• Communication channels
• Storage

• The basic idea is to utilize the redundancy of the data

06/29/2023 40

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

• Suppose a discrete memoryless source (DMS) outputs a symbol

from a finite set of symbols . The number of binary digits
required for unique representation of each of the symbols is

• As a example we want to encode the set of English alphabets

• The minimum number of bits required is R = 5 (L = 26)

06/29/2023 41

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

• Each of the 5 bit sequences used to indicate an alphabet is called

as a codeword.
• The code words are of equal length. Hence, this is an example of
Fixed Length Code (FLC).
• FLC is efficient if each of the codeword is equiprobable.
• However, we know that some of the letters are less common (x,
q, z etc.) while some others are more frequently used (s, t, e etc.).

• What does our intuition say?

• When the source symbols are not equally likely, we should use a
Variable Length code.

06/29/2023 42

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

Example
• Suppose we have only eight alphabets of the English letter (A-H),
then the FLC would be

06/29/2023 43

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

Example
• An example variable length code could be

06/29/2023 44

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

Example
• Suppose we want to have code for the sequence A BAD CAB,
the fixed length and the variable length code representation
would be.

• The VLC uses lesser bits as more probable alphabets are

indicated by fewer bits.

06/29/2023 45

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

Example
• Let us look into another variable code sequence

06/29/2023 46

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

Example
• However there is a problem with VLC2.
• Consider the sequence of bits 0 1001 0001 which is used to
represent A BAD CAB in the second variable length coding
scheme.
• We could regroup the bits in a different manner to have [0] [10]
[0][1] [0][0][01] which A EAB AAD.
• We can also group the [0] [1][0][0][1] [0][0][0][1] which gives A
BAAB AAB.
• This problem is not present in VLC1, as no codeword form the
prefix of another codeword. (Prefix Condition)

06/29/2023 47

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

• In a code, if a symbol can be detected as soon as a sequence of

bits corresponding to any one of the possible codeword is
detected, we can that Instantaneous Code.
• End of the codeword can be recognized without looking into
any subsequent bits
• Uniquely decodable codes – Codes in which the encoded string
could be generated by only one possible input string
• However, we need to wait for the entire bit string before
decoding.

• VLC1 is uniquely decodable; VLC2 is not.

06/29/2023 48

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

Example

• Code 0, 01, 011, 111 for the symbols s1, s2, s3, s4, respectively is
not instantaneous (although it is uniquely decodable).
• To see this, consider the bit stream 011111 · · · 1.
• We can’t tell if the first symbol is s1, s2, or s3, although it’s clear
that after this first symbol we have a sequence of s4’s. Once we
see the last 1, we can then work backwards to eventually find out
what was the first symbol.

• Therefore, we need a systematic way of designing uniquely

decodable codes.

06/29/2023 49

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

06/29/2023 50

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

06/29/2023 51

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

• Average number of bits per source letter.

• Where, is the length of the codeword for symbol .

06/29/2023 52

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding (Morse Code)

• International Morse code encodes the 26 basic

Latin letters a through z, one accented Latin letter
(é), the Arabic numerals, and a small set of
punctuation and procedural signals (prosigns).
• There is no distinction between upper and lower
case letters.
• Each Morse code symbol is formed by a
sequence of dits and dahs.
• The dit duration is the basic unit of time
measurement in Morse code transmission. The
duration of a dah is three times the duration of a
dit. Each dit or dah within an encoded character
is followed by a period of signal absence, called a
space, equal to the dit duration.
• The letters of a word are separated by a space of
duration equal to three dits, and words are
separated by a space equal to seven dits.
06/29/2023 53

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding

• Suppose a DMS outputs a symbol every t seconds

• Each symbol is selected from a finite set of symbols
occurring with probabilities
• The entropy of this DMS in bits per symbol is given by

• Equality holds when the symbols are equally likely

06/29/2023 54

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Kraft Inequality

We observe that in the code tree of the order n = nL, the

number of terminal nodes eliminated from the total
number of 2n terminal nodes is

06/29/2023 55

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding Theorem

06/29/2023 56

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Source Coding Theorem

06/29/2023 57

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Efficiency of a code

06/29/2023 58

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 59

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 60

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 61

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 62

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 63

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 64

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 65

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

• Huffman Coding returns optimal code, only when the

probabilities of the symbols are negative powers of two.

06/29/2023 66

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 67

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 68

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 69

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Huffman Coding

06/29/2023 70

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon-Fano-Elias Coding

Shannon-Fano-Elias codeword lengths are

It uses the Cumulative Distribution Function to allocate the codewords. The Cumulative
distribution function is defined as

Let us define a modified cumulative distribution function as

06/29/2023 71

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon-Fano-Elias Coding

06/29/2023 72

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon-Fano-Elias Coding

The entropy for this distribution is 1.75 bits. However, the average codeword length
for the Shannon-Fano-Elias coding is 2.75 bits.

06/29/2023 73

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon-Fano-Elias Coding

06/29/2023 74

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Arithmetic Coding

Takeaways from Huffman Coding.

06/29/2023 75

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Arithmetic Coding

Example

Let our alphabet consists of only three symbols A, B and C with probabilities of
occurrence P(A) = 0.5, P(B) = 0.25, P(C) = 0.25. Suppose we are sending symbols
stream B A C A … Thus, the variable A corresponds [0, 0.5), B corresponds to [0.5,
0.75), C corresponds to [0.75,1).

06/29/2023 76

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 The Lempel Ziv Algorithm

• Huffman coding requires symbol probabilities. In real life

examples it is often not possible.
• Looking for a long time and estimating symbol probabilities is
not a feasible option in real life.
• Huffman coding works only with DMS source.
• Huffman coding does not work with sources with memory, e.g.
‘q-u’, ‘t-h’, ‘I-n-g’.
• Might be more efficient to use the statistical inter-dependence of
the letters in the alphabet along with their individual probabilities
of occurrence.

06/29/2023 77

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 The Lempel Ziv Algorithm

• A variable source coding scheme.

• Belongs to the class of Universal source coding algorithms –
performs compression without knowing the statistics.
• The variable length blocks are known as phrases.

06/29/2023 78

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 The Lempel Ziv Algorithm

06/29/2023 79

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 The Lempel Ziv Algorithm

06/29/2023 80

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 The Lempel Ziv Algorithm

• How long the table should be? As it is going to overflow

• A solution is using a predefined large vale for the dictionary. Overtime, the
encoder and decoder will update the table by substituting the less frequently
used entries by more frequently used ones.
• GIF and PNG formats use this compression.
06/29/2023 81

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Run Length Encoding

• An encoding technique used to reduce the size of a repeating string of

characters.
• The repeating string of characters is called a run.
• Typically RLE encodes a run of symbols into two bytes, a count and a symbol.
• Run-length encoding is supported by most bitmap file formats such as TIFF,
JPG, BMP, PCX and fax machines.

06/29/2023 82

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Rate Distortion Function

• Representation of an arbitrary real number requires infinite number of bits.

• Quantization into finite levels is a solution.
• If the levels are not equiprobable, then entropy encoding might help.
• Quantization of amplitude results in data compression at the cost of signal
distortion. Hence, quantization is a form of lossy compression.
• Distortion implies some measure of difference between the actual samples
and the corresponding quantized value .

06/29/2023 83

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Rate Distortion Function

06/29/2023 84

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Rate Distortion Function

06/29/2023 85

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Rate Distortion Function

06/29/2023 86

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Theory and
Coding

Dr. Atri Mukhopadhyay

Assistant Professor
EEE Department
BITS Pilani Hyderabad Campus

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding

• Until now we have discussed about compression of data by

utilizing the redundancy of the data.
• Now, we need to think about transmitting the data over a
communication channel.
• Channels are noisy.
• Received bit stream might be different from what was actually sent.

06/29/2023 88

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding

• For the case of passband communication, the demodulator

processes the channel-corrupted waveform and reduces each
waveform to a scalar or a vector that represents an estimate of the
transmitted data symbols.
• The detector, which follows the demodulator, may decide on
whether the transmitted bit is a 0 or a 1.
• This is called a hard decision decoding.
• This decision process at the decoder is like a binary quantization with two
levels.
• If there are more than 2 levels of quantization, the detector is said
to perform a soft decision decoding.
• In the extreme case, no quantization is performed for soft decision
decoding.

06/29/2023 89

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding

• A digital communication engineer has three dimensions to work

with:
• The transmitted signal power.
• The channel bandwidth
• The reliability of the communication system (in terms of the bit error
rate).
• Channel coding allows us to trade-off one of these commodities
(signal power, bandwidth or reliability) with respect to others.
• Channel coding adds redundant bits.

06/29/2023 90

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Models

• Binary Symmetric Channel

• A binary discrete-input, discrete-output channel
• If the modulator employs binary waveforms and the detector makes hard
decisions, then the channel may be viewed as one in which a binary bit stream
enters at the transmitting end and another bit stream comes out at the receiving
end.

06/29/2023 91

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding

06/29/2023 92

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Models

• BSC is a special case of composite discrete-input, discrete-output

channel (Discrete Memoryless Channel)
• Input to the channel consists of Q-ary
symbols,
• Output of the detector consists of Q-ary
symbols,

• Channel and the modulation is assumed to

be memoryless
• The input and output are related by

Where and

06/29/2023 93

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Models

06/29/2023 94

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Models

Some other Channel Models

• In terms of antenna count
• SISO
• SIMO
• MISO
• MIMO
• Relay Channels
• Multiple Access Channels
• Broadcast Channels
• Binary Erasure Channel (Some bits are lost)

06/29/2023 95

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Models

06/29/2023 96

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Capacity

• Consider a DMS with input to the channel consisting of Q-

ary symbols, and output of the detector consisting of Q-ary
symbols,
• Then the Average Mutual Information is defined as

06/29/2023 97

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Capacity

• The channel transition probabilities are determined by the

channel.
• Input Symbol probabilities are within the control of the
discrete channel encoder.
• Therefore, the average mutual information , maximized
over the set of input symbol probabilities depends only on
the channel transition probabilities .
• This quantity is called the capacity of the channel.

06/29/2023 98

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Capacity

The units of channel capacity is bits per channel use (provided the base of the
logarithm is 2). If the base of the logarithm is e, the units of capacity is in nats
per channel use (coming from the use of natural logarithm).
06/29/2023 99

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Capacity

06/29/2023 100

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Capacity

06/29/2023 101

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Symmetric Channel

06/29/2023 102

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Weakly Symmetric Channel

Each Column adds up to 2/3.

06/29/2023 103

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding

• Adding Redundancies in a controlled manner

• Encoding – Mapping of input data sequence to a channel
input sequence
• Done by Channel Encoder.
• Decoding – The channel output sequence in inverse mapped
to an output data sequence.
• Done by Channel Decoder.
• Channel Encoding is also known as Error Control Coding.

06/29/2023 104

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding

06/29/2023 105

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding – Block Codes

• Incoming message sequence is first sub-divided into

sequential blocks, each of length k bits.
• The cardinality of the message set is .
• Each k-bit long information block is mapped into an n-bit
block by the channel encoder where
• Hence for every k bit of information, (n – k) redundant bits
are added.

• Code Rate is given by the ratio

06/29/2023 106

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Channel Coding – Block Codes

• Code rate is always less than 1.

• Higher the number of redundant bits, lower is the code rate.
• More immune to noise, but lower communication rate

06/29/2023 107

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Noisy Channel Coding Theorem

06/29/2023 108

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Noisy Channel Coding Theorem

06/29/2023 109

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Noisy Channel Coding Theorem

06/29/2023 110

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Noisy Channel Coding Theorem

06/29/2023 111

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Noisy Channel Coding Theorem

06/29/2023 112

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Noisy Channel Coding Theorem

Observations
• As code rate decreases, there is a steep fall in the probability
of error (Pe).
• The decrease in Pe, is much more rapid than the decrease in
code rate.
• Hence, repetition code exchanges code rate for message
reliability.
• However, for repetition codes, code rate tends to zero, if we
want Pe to be tending to zero.
• Channel coding Theorem just states that code rate has to be
less than the channel capacity C.
• Finding a code is a challenge.
06/29/2023 113

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Maximum Entropy and Gaussian PDF

• Proving that Gaussian pdf gives the maximum entropy

given that the variable is unbounded.
• Calculate the Entropy of Gaussian Distribution

06/29/2023 114

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

Information Capacity Theorem is for band limited and power limited

Gaussian Channels

06/29/2023 115

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

06/29/2023 116

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

06/29/2023 117

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

06/29/2023 118

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

06/29/2023 119

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

If

06/29/2023 120

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Capacity Theorem

06/29/2023 121

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon Limit

06/29/2023 122

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon Limit

• Example Spread Spectrum Techniques (C is a linear function of W, and log

function of SNR)
06/29/2023 123

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon Limit

06/29/2023 124

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Shannon Limit

06/29/2023 125

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Information Theory and
Coding

Dr. Atri Mukhopadhyay

Assistant Professor
EEE Department
BITS Pilani Hyderabad Campus

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Error Correcting Codes

• Channels are not perfect

• Used for correcting error when messages are transmitted over a
noisy channel or retrieved from storage devices.
• Also known as Channel Coding as we are overcoming the
detrimental effects of the channel.
• Idea is to add redundancy in a controlled manner.
• The decoder can correct the errors if the number of errors is
within the limit for which the coding scheme is designed.

06/29/2023 127

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Error Correcting Codes

Block Diagram of a communication system (source

coding blocks are not shown here)

06/29/2023 128

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Error Correcting Codes

06/29/2023 129

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Basic Definitions

06/29/2023 130

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Basic Definitions

06/29/2023 131

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Basic Definitions

06/29/2023 132

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Basic Definitions

06/29/2023 133

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Basic Definitions

06/29/2023 134

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION

 Basic Definitions

06/29/2023 135

ELECTRICAL ELECTRONICS COMMUNICATION INSTRUMENTATION