Digital Comm Notes
Digital Comm Notes
x(t) = X( f )e
j 2ft
df
Fourier Transform
Time domain Frequency Domain
Inverse Fourier Transform
Frequency domain Time Domain
Spectrum
The spectrum of a signal is a plot which shows
how the signal amplitude or power is
distributed as a function of frequency.
X( f ) = x(t)e
j 2ft
dt = e
j 2ft
dt =
1
j2f
e
j 0.5ft
e
j 0.5ft
[ ]
0.5
0.5
=
sin(f )
f
Time Domain Frequency Domain
Amp. Amp.
Time(s) Frequency (Hz)
Band limited signals
A band limited signal is a signal who has a finite
spectrum.
Most of signal energy in the spectrum is contained in a
finite range of frequencies. finite range of frequencies.
After that range, the signal power is almost zero or
negligible value.
X(f)
+ f
H
- f
H
Freq.
Symmetrical Signal
Positive = Negative
Converting an Analog Signal to a Discrete
Signal (A/D)
Can be done through three basic steps:
1- Sampling 1- Sampling
2- Quantization
3- Coding
Sampling
Process of converting the continuous time
signal to a discrete time signal.
Sampling is done by taking Samples at
specific times spaced regularly. specific times spaced regularly.
V(t) is an analog signal
V(nT
s
) is the sampled signal
T
s
= positive real number that represent the spacing of
the sampling time
n = sample number integer
Sampling
Original Analog Signal
Before Sampling
Sampled Analog Signal
After Sampling
Sampling
The closer the Ts value, the closer the sampled
signal resemble the original signal.
Note that we have lost some values of the
original signal, the parts between each original signal, the parts between each
successive samples.
Can we recover these values? And How?
Can we go back from the discrete signal to
the original continuous signal?
Sampling Theorem
A bandlimited signal having no spectral components
above f
max
(Hz), can be determined uniquely by values
sampled at uniformintervals of Ts seconds, where
An analog signal can be reconstructed from a sampled An analog signal can be reconstructed from a sampled
signal without any loss of information if and only if it is:
Band limited signal
The sampling frequency is at least twice the signal
bandwidth
Ts
1
2 f
max
Quantization
Quantization is a process of approximating a
continuous range of values, very large set of
possible discrete values, by a relatively small
range of values, small set of discrete values. range of values, small set of discrete values.
Continuous range infinte set of values
Discrete range finite set of values
Quantization
Dynamic range of a signal
The difference between the highest to lowest
value the signal can takes.
Quantization
In the Quantization process, the dynamic range of a
signal is divided into L amplitude levels denoted by m
k,
where k = 1, 2, 3, .. L
L is an integer power of 2
L = 2
k
L = 2
K is the number of bits needed to represent the amplitude
level.
For example:
If we divide the dynamic range into 8 levels,
L = 8 = 2
3
We need 3 bits to represent each level.
Quantization
Example:
Suppose we have an analog signal with the values
between [0, 10]. If we divide the signal into four
levels. We have
m1 [ 0, 2.5 ]
m2 [ 2.5, 5 ]
m3 [ 5 , 7.5]
m4 [ 7.5, 10]
Quantization
For every level, we assign a value for the signal
if it falls within the same level.
M1 = 1.25 if the signal in m1
M2 = 3.75 if the signal in m2
Q [ v(t) ] =
M3 = 6.25 if the signal in m3
M4 = 8.75 if the signal in m4
Quantization
Original Analog Signal
Before Quantization
Quantized Analog Signal
After Quantization
Quantization
Original Discrete Signal
Before Quantization
Quantized Discrete Signal
After Quantization
Quantization
The more quantization levels we take the
smaller the error between the original and
quantized signal.
Quantization step Quantization step
The smaller the the smaller the error.
=
Dynamic Range
No. of Quantization levels
=
S
max
S
min
L
Coding
Assigning a binary code to each quantization
level.
For example, if we have quantized a signal into
16 levels, the coding process is done as the 16 levels, the coding process is done as the
following:
Step Code Step Code Step Code Step Code
0 0000 4 0100 8 1000 12 1100
1 0001 5 0101 9 1001 13 1101
2 0010 6 0110 10 1010 14 1110
3 0011 7 0111 11 1011 15 1111
Coding
The binary codes are represented as pulses
Pulse means 1
No pulse means 0 No pulse means 0
After coding process, the signal is ready to be
transmitted through the channel. And
Therefore, completing the A/D conversion of
an analog signal.
Chapter 3: Source Coding
12
th
November 2008
Abdullah Al-Meshal
3.1 Measure of Information
What is the definition of Information ?
News, text data, images, videos, sound etc..
In Information Theory
Information is linked with the element of surprise or
uncertainty uncertainty
In terms of probability
Information
The more probable some event to occur the less information
related to its occurrence.
The less probable some event to occur the more information
we get when it occurs.
Example1:
The rush hour in Kuwait is between 7.00 am
8.00 am
A person leaving his home to work at 7.30 will
NOT be surprised about the traffic jam almost
no information is gained here no information is gained here
A person leaving his home to work at 7.30 will BE
surprised if THERE IS NO traffic jam:
He will start asking people / family / friends
Unusual experience
Gaining more information
Example 2
The weather temperature in Kuwait at summer
season is usually above 30
o
It is known that from the historical data of the
weather, the chance that it rains in summer is weather, the chance that it rains in summer is
very rare chance.
A person who lives in Kuwait will not be surprised by
this fact about the weather
A person who lived in Kuwait will BE SURPRISED if it
rains during summer, therefore asking about the
phenomena. Therefore gaining more knowledge
information
How can we measure information?
Measure of Information
Given a digital source with N possible outcomes
messages, the information sent from the digital
source when the j
th
message is transmitted is source when the j message is transmitted is
given by the following equation:
[ Bits ]
I
j
= log
2
(
1
p
j
)
Example 1
Find the information content of a message
that takes on one of four possible outcomes
equally likely
Solution Solution
The probability of each outcome = P =
Therefore,
1
0.25
I = log
2
(
1
0.25
) =
log(
1
0.25
)
log(2)
= 2 bits
Example 2
Suppose we have a digital source that
generates binary bits. The probability that it
generates 0 is 0.25, while the probability generates 0 is 0.25, while the probability
that it generates 1 is 0.75. Calculate the
amount of information conveyed by every bit.
Example 2 (Solution)
For the binary 0 :
For the binary 1:
I = log
2
(
1
0.25
) = 2 bits
bits I 42 . 0 )
1
( log = =
For the binary 1:
Information conveyed by the 0 is more than the
information conveyed by the 1
bits I 42 . 0 )
75 . 0
1
( log
2
= =
Example 3:
A discrete source generates a sequence of ( n )
bits. How many possible messages can we
receive fromthis source?
Assuming all the messages are equally likely to Assuming all the messages are equally likely to
occur, how much information is conveyed by
each message?
Example 3 (solution):
The source generates a sequence of n bits,
each bit takes one of two possible values
a discrete source generates either 0 or 1
Therefore: Therefore:
We have 2
N
possible outcomes
The Information Conveyed by each outcome
I = log
2
(
1
2
n
) =
log(2
n
)
log(2)
=
nlog(2)
log(2)
= n bits
3.3 Entropy
The entropy of a discrete source S is the
average amount of information ( or
uncertainty ) associated with that source.
m= number of possible outcomes
Pj = probability of the j
th
message
H(s) = p
j
log
2
(
1
p
j
) [bits]
j=1
m
Importance of Entropy
Entropy is considered one of the most
important quantities in information theory.
There are two types of source coding: There are two types of source coding:
Lossless coding lossless data compression
Lossy coding lossy data compression
Entropy is the threshold quantity that
separates lossy from lossless data
compression.
Example 4
Consider an experiment of selecting a card at
random from a cards deck of 52 cards. Suppose
were interested in the following events:
Getting a picture, with probability of :
Getting a number less than 3, with probability of:
Getting a number between 3 and 10, with a probability of:
Calculate the Entropy of this randomexperiment.
52
12
8
52
32
52
Example 4 (solution) :
The entropy is given by :
Therefore,
H(s) = p
j
log
2
(
1
p
j
) [bits]
j=1
3
H(s) =
12
52
log
2
(
52
12
) +
8
52
log
2
(
52
8
) +
32
52
log
2
(
52
32
) =1.335 bits
Source Coding Theorem
First discovered by Claude Shannon.
Source coding theorem
A discrete source with entropy rate H can be
encoded with arbitrarily small error probability at
any rate L bits per source output as long as L > H
encoded with arbitrarily small error probability at
any rate L bits per source output as long as L > H
Where
H = Entropy rate
L = codeword length
If we encode the source with L > H Trivial Amount of errors
If we encode the source with L < H were certain that an error will
occur
3.4 Lossless data compression
Data compression
Encoding information in a relatively smaller size than their original size
Like ZIP files (WinZIP), RAR files (WinRAR),TAR files etc..
Data compression:
Lossless: the compressed data are an exact copy of the original data
Lossy: the compressed data may be different than the original data
Loseless data compression techniques:
Huffman coding algorithm
Lempel-Ziv Source coding algorithm
3.4.1 Huffman Coding Algorithm
A digital source generates five symbols with the
following probabilities:
S , P(s)=0.27
T, P(t)=0.25
U, P(t)=0.22
V,P(t)=0.17
W,P(t)=0.09
Use Huffman Coding algorithm to compress this
source
Step1: Arrange the symbols in a
descending order according to their
probabilities
S
0.27
S
0.27
T
0.25
T
0.25
U
0.22
U
0.22
V
0.17
V
0.17
W
0.09
W
0.09
Step 2: take the symbols with the
lowest probabilities and form a leaf
LIST
V
0.17
V
0.17
U
0.22
U
0.22
T
0.25
T
0.25
S
0.27
S
0.27
W
0.09
W
0.09
V,W(x1)
0.26
V,W(x1)
0.26
Step 3: Insert the parent node to
the list
LIST
V
0.17
V
0.17
U
0.22
U
0.22
T
0.25
T
0.25
S
0.27
S
0.27
W
0.09
W
0.09
V,W(x1)
0.26
V,W(x1)
0.26
Step 3: Insert the parent node to
the list
LIST
S
0.27
S
0.27
V
0.17
V
0.17
U
0.22
U
0.22
T
0.25
T
0.25
0.27 0.27
W
0.09
W
0.09
V,W(x1)
0.26
V,W(x1)
0.26
X1
0.26
X1
0.26
Step 4: Repeat the same procedure
on the updated list till we have only
one node
LIST
S
0.27
S
0.27
U
0.22
U
0.22
T
0.25
T
0.25
0.27 0.27
V
0.17
V
0.17
W
0.09
W
0.09
V,W(x1)
0.26
V,W(x1)
0.26
X1
0.26
X1
0.26
T
0.25
T
0.25
U
0.22
U
0.22
X2
0.47
X2
0.47
LIST
S
0.27
S
0.27 0.27 0.27
V
0.17
V
0.17
W
0.09
W
0.09
X1
0.26
X1
0.26
X1
0.26
X1
0.26
T
0.25
T
0.25
U
0.22
U
0.22
X2
0.47
X2
0.47
X2
0.47
S
0.27
S
0.27
X3
0.53
X3
0.53
LIST
X3
0.53
X3
0.53
X2 X2
X3 X3
X4
1
X4
1
0.53 0.53
X2
0.47
X2
0.47
V
0.17
V
0.17
W
0.09
W
0.09
X1
0.26
X1
0.26
T
0.25
T
0.25
U
0.22
U
0.22
0.47 0.47
S
0.27
S
0.27
X3
0.53
X3
0.53
X2 X2
X3 X3
X4
1
X4
1
Step 5: Label each branch of the tree with 0 and
1
0 1
V
0.17
V
0.17
W
0.09
W
0.09
X1
0.26
X1
0.26
T
0.25
T
0.25
U
0.22
U
0.22
0.47 0.47
S
0.27
S
0.27
X3
0.53
X3
0.53
Huffman Code Tree
0
0
0
1
1
1
X2 X2
X3 X3
X4
1
X4
1
Codeword of w = 010
0 1
V
0.17
V
0.17
W
0.09
W
0.09
X1
0.26
X1
0.26
T
0.25
T
0.25
U
0.22
U
0.22
0.47 0.47
S
0.27
S
0.27
X3
0.53
X3
0.53
Huffman Code Tree
0
0
0
1
1
1
X2 X2
X3 X3
X4
1
X4
1
Codeword of u=10
0 1
V
0.17
V
0.17
W
0.09
W
0.09
X1
0.26
X1
0.26
T
0.25
T
0.25
U
0.22
U
0.22
0.47 0.47
S
0.27
S
0.27
X3
0.53
X3
0.53
Huffman Code Tree
0
0
0
1
1
1
As a result:
Symbol Probability Codeword
S 0.27 00
T 0.25 11
U 0.22 10
V 0.17 011
W 0.09 010
Symbols with higher probability of occurrence have a shorter
codeword length, while symbols with lower probability of
occurrence have longer codeword length.
Average codeword length
The Average codeword length can be
calculated by:
L = P l
m