Scribd
Upload
37 views

Lec2 - Data Compression PDF

1) Data compression algorithms address problems with efficiently representing digital data by reducing storage and transmission bandwidth requirements. 2) Discrete sources can be compressed based on their information content and statistical correlations between symbols. Analog sources can be compressed based on fidelity criteria and temporal waveform correlations. 3) The goal of source coding is to represent a source at a given data rate while maintaining a specified level of fidelity. This allows lower bit rates to be used for a given source fidelity.

Uploaded by

rayanmat
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
37 views

Lec2 - Data Compression PDF

1) Data compression algorithms address problems with efficiently representing digital data by reducing storage and transmission bandwidth requirements. 2) Discrete sources can be compressed based on their information content and statistical correlations between symbols. Analog sources can be compressed based on fidelity criteria and temporal waveform correlations. 3) The goal of source coding is to represent a source at a given data rate while maintaining a specified level of fidelity. This allows lower bit rates to be used for a given source fidelity.

Uploaded by

rayanmat
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

Data Compression

Skalar “Digital Commun icatiins” Chapter 13


THOMAS M. COVER JOY A. THOMAS”ELEMENTS OF
INFORMATION THEORY” 2nd edition,chapter 5
Data`compression
❖ Data compression is a general term for the various algorithm and program
developed to address the problems associated with efficient description of
information sources.
❖ Usually digital data are bot represented in compact form.
❖ They need more storage than they require and high transmission bandwidth.
❖ The efficient description of information sources permit:
▪ reduction in the memory required to store data
▪ reduction bandwidth required the transport source of
data
Discrete sources versus Analog
sources
❖ Discrete sources
▪ The ability to form reduced date rate description is related to the:
▪ information content
▪ statistical correlation among source symbole.
❖ Anlog sources
The ability to form reduced data rate description is related to:
▪ Stated fiedility criterion and hence amplitude distribution
▪ temporal correlation of the source waveforms
❖ In Conclusion the goal of the source coding is to form:
▪ a good fidelity description of the source for a given available data rat
▪ To permit low bit rates to obtain a specific fidelity description of the source.
Discrete Sources
❖a. A Finite discrete source: The output of the
source contain s finite number of symbols ie output
of 10 bits A/D converter or output of computer
keyboard.
The information in each symbol is given by:
𝐼 𝑋𝑖 = −𝑙𝑜𝑔2 𝑃𝑖 bits
Where 𝑃𝑖 𝑖𝑠 𝑡ℎ𝑒 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑜𝑐𝑐𝑢𝑟𝑎𝑛𝑐𝑒 𝑜𝑓 𝑠𝑦𝑚𝑏𝑜𝑙 𝑋𝑖
The average self information for the symbols in alphabet set is called
source entropy and is given by:

𝐻 𝑥 = 𝐸 𝐼 𝑋𝑖
𝑀 𝑏𝑖𝑡𝑠
= −෍ 𝑃𝑖 𝑙𝑜𝑔2 𝑃𝑖
𝑖=1 𝑠𝑦𝑚𝑏𝑜𝑙
And E{} is the expected value of X
Example: Entropy of a binary
Source
• Assume that a binary source generate ones and zeros
with probability of 0,1 and 0.9 repectively
• 𝐻 𝑥 = − σ2𝑖=1 𝑃𝑖 𝑙𝑜𝑔2 (𝑃𝑖 ) = 0.1𝑙𝑜𝑔2 0.1 + 0.9𝑙𝑜𝑔2 0.9
• =0.47 bis/symbol
• The result obtained because information depends on
the probability , small probability contains information
higher than large probability.
• However , the upper bound is 𝑙𝑜𝑔2 M and this always
sttitafy the equality below
• 0 ≤ 𝐻 𝑥 ≤ 𝑙𝑜𝑔2 (𝑀) where M is number of Symbols
Discrete Sources (continued
• Memoryless

A discrete source is said to be memoryless if the symbol emitted by the source are statistically independent .
Example if two symbol is chosen at a time, the joint probability of the two elements is simlpy the product of
their probabilities

𝑃 𝑋𝑖 , 𝑋𝑘 = 𝑃(𝑋𝑖 ‫) 𝑘𝑋(𝑃 𝑖𝑋 𝑃 = ) 𝑘𝑋ہ‬


Discrete Sources (continued
• A result of statistical independence is that the
information required to transmit a sequence of M
symbol> (called an ,M-tuple) from a given alphabet is
prectsely M times the average informant required to
transmit a single symbol.
• ThlS happens the probability of a statistically
independent M-·tuple is given by:
𝑀

𝑃 𝑋1 , 𝑋2, 𝑋3…………, 𝑋𝑀 = ෑ 𝑃(𝑋𝑚 )


𝑚=1
• The information (Entropy) in each symbol is given by:
𝐼 = −𝑙𝑜𝑔2 𝑃(𝑋𝑖 ) 𝑏𝑖𝑡𝑠
The Average information (entropy) per symbol of a
statistically independent M-Tuple is given by:
Discrete Sources (continued
1 1 𝑚=𝑀
𝐻𝑀 𝑋 = 𝐸{ −𝑙𝑜𝑔2 𝑃 𝑋1 , 𝑋2 , 𝑋3 , … … . . 𝑋𝑀 = ෍ {−𝑝(𝑋𝑚 )𝑙𝑜𝑔2 𝑃( 𝑋𝑚 )}
𝑀 𝑀 𝑚=1
= H(X) bits/symbol

This is also called source Entropy and E(X) is the expected value of X.

With Memory

The discrete is said to have memory if the source elements composing the sequence are not independents
𝐻 𝑥 ≤ 𝑙𝑖𝑚𝑀→∞ 𝐻𝑀 𝑋 𝑎𝑛𝑑 𝐻𝑀 (𝑋)𝑚𝑒𝑚𝑜𝑟𝑦 < 𝐻𝑀 (𝑋)𝑛𝑜𝑚𝑒𝑚𝑜𝑟𝑦

In conclusion the average entropy per symbol of the M-tuple from a` source with memory decreases
as the length M increases.
It is more efficient to encode symols from a source with memory in groups of several symbols rather
than to encode them one at a time.
Example 2

• Consider the binary first-order Markov source


described by the the state transition diagram
shown

You might also like