Introduction to Information Theory:

Entropy (Avg Information) : The average

information per message of a source x is called its
entropy denoted by H(x) Hence,
The entropy of a source is a function of the message probabilities,
it is interesting to find the message probability distribution that
yields the maximum entropy, because the entropy is a measure of
uncertainty the probability distribution that generates the maximum
uncertainty will have the maximum entropy.
It is the probability of occurrence of the possible value of the
source symbol. thus equation gives the entropy in the units of bits
(per symbol)
Information rate (R): Average Number of bits of
information per second
the time rate at which source x emits symbols
is r
• Information rate, R = r H

• H = entropy or average information

• r = rate at which message are generated.
• Thus, information rate is represented in average
number of bits of information per second.
The average code word length L of the source encoder as
 Source Coding
• The primary function of a communication system is to transfer
a message signal from a sender to the receiver. In the process
of data transfer, the signal used to transfer the data should
have the necessary properties required for the transmission as
per the transmission channel. Hence, there are some
information coding techniques used to transform the message
signal into a transferrable signal through the transmission
channel. Two of such information coding schemes
are Huffman Coding and Shannon Fano Coding.
• The most basic difference between Huffman Coding and
Shannon Fano Coding is that the Huffman coding provides a
variable length encoding, while the Shannon Fano Coding
provides a limited length encoding.
 Huffman Coding
• Huffman coding is a compression technique that handles the data compression of
ASCII characters. Developed by David Huffman in 1952, the Huffman coding is
the compression algorithm that is based on the top−down approach. In this
algorithm, a binary tree is created in a top−down manner to produce a minimal
sequence. It translates the characters contained in a data file into a binary code. The
Huffman coding is a lossless text compression method because in this method, the
compressed data can be restored to its original format.
• Huffman coding uses a variable−length encoding scheme, where each symbol in
the input data is assigned a binary code based on its frequency of occurrence. This
allows for efficient compression of the data, as symbols that occur more frequently
in the input will be assigned shorter codes, reducing the overall size of the encoded
data.
 Shannon-Fano Coding
• Shannon Fano Coding is also an encoding algorithm used to
generate a uniquely decodable code. It was developed by Claude
Shannon and Robert Fano in 1949, earlier than the Huffman Coding
technique. Similar to the Huffman coding, the Shannon Fano Coding
also uses the probabilities of the data for encoding. However, this
algorithm does not ensure the optimal code generation. The Shannon
Fano coding uses cumulative distribution function.

• Instead of assigning binary codes to symbols based on their

frequency, it uses a hierarchical approach to divide the symbols into
two groups, with each group being assigned a prefix to form the
code for each symbol. This can also lead to efficient compression of
the data, but the encoding process is generally slower than Huffman
coding.
 Source Coding Techniques

1. Huffman Code.

2. Two-path Huffman Code.

3. Lemple-Ziv Code.

4. Fano Code.

5. Shannon Code .

6. Arithmetic Code.
 Huffman Code

1. Huffman Code.

With the Huffman code in the binary case the two

least probable source output symbols are joined
together, resulting in a new message alphabet with
one less symbol
1 take together smallest probabilites: P(i) +
P(j)
2 replace symbol i and j by new symbol
3 go to 1 - until end
Application examples: JPEG, MPEG, MP3
 1. Huffman Code.

ADVANTAGES:
• uniquely decodable code
• smallest average codeword
length

DISADVANTAGES:
• LARGE tables give complexity
• sensitive to channel errors
 1. Huffman Code.

Huffman is not universal!

it is only valid for one particular type of source!

For COMPUTER DATA data reduction

is
lossless no errors at reproduction
universal  effective for different types of
data
 Huffman Coding: Example
• Compute the Sourc Symbol
Huffman Code for e Probabilit
the source shown Symb y
ol sk pk
2   s0 0.1
HS    0.4 log  0. 
s1 0.2
1 4
2   s2 0.4
2 0.2log  0.  s3 0.2
1 2
2  0.1 2  1  s4 0.1
log  0. 

 2.12193
1
 Solution
A
Sourc Stage I
e
Symb
ol sk
s2 0.4

s1 0.2

s3 0.2

s0 0.1

s4 0.1
 Solution A
Source Stage I
Stage II Symbol
sk

s2 0. 0.
4 4

0. 0.
s1 2 2

0. 0.
2 2
s3
0. 0.
1 2
s0
 Solution A
•Source Stage IStage II Stage III
• s k

• s 2 0. 0. 0.
4 4 4
• S1
0. 0. 0.
• s3
2 2 4
• s0
0. 0. 0.
• s4 2 2 2

0. 0.
1 2
 Solution A
•SourceStage I Stage II Stage III S Stage
IV
• s k

• s 2 0. 0. 0. 0.
4 4 4 6
• s1
0. 0. 0. 0.
• s3
2 2 4 4
• s0
0. 0. 0.
• s4 2 2 2

0. 0.
1 2
 Solution A
•SourceStage I Stage II Stage III Stage
IV
• s k

• s 2 0. 0.4 0. 0.6
0

4 4
• s1 0.2
0
0. 1
0. 0. 4
• s3 0
2 0.2 4 1
0
• s0 0.
0. 1 0.
2 2 2
• s4 1

0.
1
 Solution A
•SourceStage I Stage II Stage III Stage Cod
IV e
• s k

• s 2 0. 0.4 0. 0.
0
00
4 4 6
• s1 0.2
0
10
1
0. 0. 0.
• s3 0 11
2 0.2 4 1 4
0
• s0 0. 01
0. 1 0.
2 2 2 0
• s4 1

0. 01
1 1
 Solution A
Sourc
e
Cont’d
Symbol Code
Probabilit word H S   2.12193
Symb y pk ck
ol sk
s0 0.1 010 L  0.4  2  0.2  2
s1 0.2 10 0.2  2  0.1  3 
s2 0.4 00 0.1  3
s3 0.2 11 2.2
s4 0.1 011
H S   L  H S 
THIS IS NOT THE ONLY 1
SOLUTION!
 Another Solution B
Source Stage I Stage II Stage III Cod
Stage IV Symbol e
sk

s2 0. 0. 0. 0.6
0
1
4 4 4 0
s1 0. 1 01
0. 0. 0 0. 4
s3 2 2 4 1 000
0
s0 0. 0. 1 0. 0010
2 2 2
s4 1
0011
0. 0.
1 2
 Another Solution B
Cont’d
Sourc Symbol Code
e Probabilit word H S   2.12193
Symb y pk ck
ol sk
s0 0.1 0010 L  0.4 1  0.2  2
s1 0.2 01 0.2  3  0.1  4 
s2 0.4 1 0.1  4
s3 0.2 000 2.2
s4 0.1 0011
H S   L  H S 
1
 What is the difference
between the two
solutions?
• They have the same average length
• They differ in the
K 1variance of 2the average
l L
code length 2
s  p  
k 0
k k

• Solution 
A
• s 2=0.16
• Solution
B
• s 2=1.36
 Shannon – Fano Encoding:

First the messages are ranked in a table in descending order of probability. The table is then progressively

divided into subsections of probability near equal as possible. Binary 0 is attributed to the upper subsection

and binary 1 to the lower subsection. This process continues until it is impossible to divide any further, the

following steps show the algorithmic procedure of Shannon – Fano encoding:

1-List the symbols in descending order of the probabilities.

2-Divide the table into as near as possible two equal values of probability.

3-Allocate binary 0 to the upper section and binary 1 to the lower section.

4-Divide both the upper section and the lower section into two.

5-Allocate binary 0 to the top half of each section and binary 1 to the lower half.

6-Repeat steps (4) and (5) until it is not possible to go any further.
For this example we can evaluate the efficiency of this system:
L = ? digits / symbol
H = ? bits / symbol
η = (H / L) *100% = ?
 Example 2:

Mutual Information

For this example we can evaluate the efficiency of this

system:
L = 2.72 digits / symbol
H = 2.67 bits / symbol
η = (H / L) *100% = ((2.67) / (2.72))*100% = 98.2%.
 Mutual Information
• Mutual information is a quantity that measures a relationship between two
random variables that are sampled simultaneously. In particular, it measures
how much information is communicated, on average, in one random
variable about another.
• It is closely linked to the concept of entropy. This is because it can also be
known as the reduction of uncertainty of a random variable if another is
known. Therefore, a high mutual information value indicates a large
reduction of uncertainty whereas a low value indicates a small reduction.
• An important theorem from information theory says that the mutual
information between two variables is 0 if and only if the two variables are
statistically independent.
• The formal definition of the mutual information of two Discrete random
variables X and Y , whose joint distribution is defined by P(X, Y ) is given
by

• P(X) and P(Y ) are the marginal distributions of X and Y

 Mutual Information
• Whereas to compute the mutual information
for continuous random variables the summations have to be
replaced by the integrals.
Mutual Information
Mutual Information
 Discrete Channel Capacity
Channel Capacity
 The communication between A and B is a consequence of a physical act performed

by A to induce the desired state in B. This transfer of information is subject to noise.

 The communication is successful if the transmitter A and the receiver B agree on

what was sent.

 In this part we define the channel capacity as the logarithm of the number of

distinguishable signals that can be sent through the channel.

 Source symbols from some finite alphabet are mapped into some sequence of channel

symbols, which then produces the output sequence of the channel. The output

sequence is random but has a distribution that depends on the input sequence.
 Discrete Channel Capacity
Channel Capacity
• Each of the possible input sequences induces a probability

distribution on the output sequences.

• We show that we can choose a “non confusable” subset of input

sequences so that with high probability there is only one highly

likely input that could have caused the particular output.

• We can transmit a message with very low probability of error and

reconstruct the source message at the output. The maximum rate

at which this can be done is called the capacity of the channel.

 Discrete Channel Capacity
• Definition We define a discrete channel to be a system consisting of an input
alphabet X and output alphabet Y and a probability transition matrix p(y|x)
that expresses the probability of observing the output symbol y given that
we send the symbol x.
• The channel is said to be memoryless if the probability distribution of
the output depends only on the input at that time and is conditionally
independent of previous channel inputs or outputs.
 Shannon-Hartley Law

• The Shannon-Hartley Capacity

Theorem, more commonly known as
the Shannon-Hartley theorem or
Shannon's Law, relates the system
capacity of a channel with the
averaged received signal power, the
average noise power and the
bandwidth.
 Shannon-Hartley Law
 The Shannon-Hartley Capacity Theorem, more commonly
known as the Shannon-Hartley theorem or Shannon's Law,
relates the system capacity of a channel with the averaged
received signal power, the average noise power and the
bandwidth.
 It is an application of the noisy channel coding theorem to the
archetypal case of a continuous-time analog communications
channel subject to Gaussian noise.
 The theorem establishes Shannon's channel capacity for such
a communication link, a bound on the maximum amount of
error-free digital data (that is, information) that can be
transmitted with a specified bandwidth in the presence of the
noise interference, assuming that the signal power is
bounded, and that the Gaussian noise process is characterized
by a known power or power spectral density.
 Shannon-Hartley Law
• This capacity relationship can be stated as:
where:
• C is the capacity of the channel (bits/s)
• S is the average received signal power
• N is the average noise power
• W is the bandwidth (Hertz)

Shannon–Hartley
Theorem
R=Information Rate
C=Channel Capacity
 Trade-off between bandwidth and SNR
To increase the information rate the trade between bandwith and signal noise ratio
are held.
Definition of Bandwith;
• The maximum number of data is transferred over the internet. It is a part
of information technology.
Definition of signal noise ratio;
• Signal noise ratio compared the desired signal to the level of background noise.
Trade between bandwith and signal noise ratio;
• To occupy the smallest system the bandwith and signal noise ratio work against
each other.
• Bandwith has the fixed quantity and niether can be changed. So the channel
capacity is directly proportional to the signal noise ratio.
• While the channel capacity is changed by the signal noise ratio to occupy the
smallest system or the allocated bandwith.
• The noise of the bandwith increases and we can observe the signal noise relation
also increased in the background.
• Hence, in relation to bandwith and signal noise relation they work
simultaneously in reference to the channel capacity directly proportional to the
signal to noise ratio.
 Trade-off between bandwidth and SNR
The message rate is given as
 Trade-off between bandwidth and SNR

**This gives the maximum information transmission rate possible for a system of
given power but no bandwidth limitations