Multimedia Systems and Applications

Sudipta Mahapatra

1
Lossy Image Compression

Theory of Quantization

2
 Outline

• Quantization
• Scalar and Vector Quantization
• Uniform and non-uniform quantization
• Rate-Distortion Function and the Source
Coding Theorem

3
 Why to go for Lossy Compression

• Limited compression performance of lossless

schemes though exact reconstruction is
guaranteed;
• Achievable compression ratio is restricted by
the source entropy;
• Lossless predictive coding schemes perform
better as the error entropy is less due to the
spatial redundancy in images; still, comp. ratio is
no better than 0.5 for most practical images.
4
 Lossy Image compression

• Exploit the available psycho-visual

redundancy; the loss in quality is not visually
perceptible;
• Apply a quantization process to transform
domain images.
• Transform used to de-correlate an image.

5
 Quantization
• The process of representing a set of values with
a much smaller set.
• Example: Set of numbers between -10 to 10 .
• Represent using the nearest integer.
• [-10 to 10] is mapped to [-10, …, 0, …, 10].
• Result in loss of information.

6
 Quantization
– Encoder mapping – involves generating a definite
codeword for all the symbols that map into a
particular interval.
– Decoder – Generates a reconstructed value for every
codeword generated by the encoder.
Input Code Output
Codes 000 -3.5
001 -2.5
010 -1.5
011 -0.5
100 0.5
000 001 010 011 100 101 110 111 101 1.5
110 2.5
111 3.5
7
 Quantizer Input Output mapping
Uniform Quantisation: All the intervals are of the same size; equal to
the spacing of decision boundaries, ) .
1. Midrise Quantizer (# of intervals, L is odd; does not include zero).
2. Midtread quantizer (L is even; includes zero as an o/p level, useful in control
systems and audio coding schemes-need to represent silence intervals )

(1) (2)
8
9
10
 Uniform quantization of a
Uniformly distributed Source
# levels=M; Input uniformly distributed in [-Xmax,Xmin].
X max

M
Distortion (mean squared quantization error or msqe),
2
M / 2 i
 2i  1  1
 2    x 
2
 dx 2 / 12.
1
( i  1) 
 2  2 X max

(2)
11
Quantization error
 Uniform quantization of a
Uniformly distributed Source
Input lies in the interval [-Xmax, Xmax];
Quantization error les in the interval [-  /2, /2]; As the input is
uniform, the error is also uniform.
Thus, msqe=the second moment
of a random variable uniformly
distributed in the interval and is
given by
2 /2
 q (1 / )  q 2 dq 2 / 12. Quantization error
 /2
2
Signal power,  q (2 X max ) 2 / 12 ? As =2Xmax/M,
2 2
SNR(dB) 10 log10 ( s /  q ) 6.02 log 2 M dB 6.02ndB.
12
 Quantizer design – Rate & Distortion
• Although quantization is a simple process, the
design of the quantizer greatly influences on
the amount of compression and the loss
incurred.
• Encoder design involves construction of
intervals.
• Decoder design involves selection of
reconstruction values.
• The encoder-decoder pair is called a quantizer.
13
 Rate & Distortion
● Assume that the input x is modelled by a random
variable X with a pdf fx(x).
● # of intervals=M; the end-point of the intervals are
known as the decision boundaries.
● Decision boundaries: bi i 0
M

● Reconstruction levels:  yi i 1
M

● The quantization operation is defined as

Q( x)  yi iff bi  1  x  bi
● Mean Square Quantization Error
 M bi
  [ x  Q( x)] f X ( x)dx   ( x  yi ) 2 f X ( x)dx
2
q
2
 bi  1
i 1
14
 Rate (Fixed length codes)
• If fixed length codeword is used, size of the
output alphabet gives the rate.
• If M is the # of quantizer outputs, rate is given by
R log 2 M 
• The quantizer design problem is:
Given an input pdf fx (x) and the number
of levels M in the quantizer, find the decision
boundaries {bi} and the reconstruction levels
{yi} so as to minimize the mean squared
quantization error. 15
 Rate (Variable length codes)
• If variable length codewords are used such
as Huffman or arithmetic coding, selection
of the decision boundaries also effects the
rate of the quantizer along with the alphabet
size.
y1 1110 y5 01
y2 1100 y6 101
y3 100 y7 1101
y4 00 y8 1111
• For variable length codewords, if Ii is the
length of codeword yi , rate
M M bi
R  I i P( yi )  I i  f X ( x)dx 16
bi  1
i 1 i 1
 Quantizer Design Problem
Given a distortion constraint q 2
 D *
,
find the decision boundaries,
reconstruction levels and binary codes
that minimize the rate subject to the
distortion constraint;
Or, given a rate constraint R R , find
*

the decision boundaries, reconstruction

levels and binary codes that minimize
the distortion subject to the rate
constraint. 17
 Scalar Quantization
• Quantize each sample independently
• A scalar quantizer Q(.) is a function that maps a
continuous-valued variable s having a probability
density function p(s) into a discrete set of
reconstruction levels
• Applies a set of the decision levels di (i=1, 2, …, L)
on the continuous-valued samples s, such that
if ,],
where, L represents number of output levels.
18
 Vector Quantization
• Each of the samples is not quantized.
• A set of continuous-valued samples, expressed
collectively as a vector is represented by a limited
number of vector states.
• Encoding sequences of outputs is better than
encoding individual samples (Shanon).

19
 Quantizer Performance
• Determined by its distortion measure.
• Let be a continuous-valued variable and sˆ Q ( s )
be the quantized variable.
• Quantization error, ŝ
• Distortion, D = E[] ŝ
• We should design and so that the distortion D
should be minimized.

20
• Two different approaches for optimal quantizer
design namely
● Lloyd-Max Quantizers

● Entropy-constrained Quantizers

• Minimize D = E[] w.r.t and , subjected to the

constraint: L, the number of output states in the
quantizer is fixed.

• The above quantizers perform non-uniform

quantization in general and are known as Lloyd-
Maxquantizers

21
• Minimize D = E[] w.r.t and , subjected to the
constraint that the source entropy, is a constant; L
may vary.

• The above quantizers are called Entropy-

Constrainedquantizers.

22
• In fixed-length coding, the rate R for quantizers
with L states is given by 

• In variable-length coding the rate R for

quantizersis given by

• Lloyd-Maxquantizersare more suited for use with

fixed-length coding.

• Entropy-Constrainedquantizers are more

suitable for use with variable-length coding.

23
Design of Lloyd-Max Quantizers

• Design of Lloyd-Max quantizers requires the

minimization of
L di
D = E[] =  d p(s) ds
i 1
i 1

• Setting the partial derivatives of D w.r.t and to

zero, we obtain the necessary conditions for
minimization.

24
Design of Lloyd-Max Quantizers
di

s P(s) r  r
ri  di  1
, 1 i L, d i 
i i  1

di , 1 i L
2
P ( s )
di  1

• The decision and the reconstruction levels are

solutions to the above set of nonlinear equations.

• Closed form solutions to above equations do not

exist and they need to be solved by numerical
techniques. 25
• Using numerical techniques, the above equations
could be solved in an iterative way.

• Solutions to above equations are iteratively

repeated until a convergence in the decision and
reconstruction levels are achieved.

• The initial values for the decision variables can be

assumed to be those corresponding to uniform
quantization.

• In most of the cases, the convergence is achieved

quite fast for a wide range of initial values.
26
Uniform and Non-Uniform Quantization
• Lloyd-Max quantizers perform non-uniform
quantization if the pdf of the input variable is not
uniform.

• Lloyd-Max quantizers perform finer quantization

wherever the pdf is large.

• In finer quantization, decision levels more closely

packed and consequently more number of
reconstruction levels.

27
• Lloyd-Max quantizers perform coarser quantization
wherever the pdf is low.

• In coarser quantization, decision levels more

widely packed and hence less number of
reconstruction levels.

• In uniform quantization, the reconstruction levels

are equally spaced.

- =θ

where quantization step-size, θ is a constant.

28
• If the pdf of the input variable s is uniform in the
interval [A, B], i.e.,
1
A s  B
• = B A

0 otherwise

• The design of Lloyd-Max quantizer leads to a

uniform quantizer, where
B A

L
di  A  i 0 i L

ri di  1  1 i L 29
2
• If the pdf exhibits even symmetric properties about
its mean e.g., Gaussian and Laplacian distributions.

• Then the decision and the reconstruction levels

have some symmetry relations for both uniform and
non-uniform quantizers.

• When the pdfis even symmetric about its mean, the

quantizer is to be designed for only levels or
levels, depending upon whether L is even or odd,
respectively.

30
Fig: Typical Quantizer Characteristics for L even (=8)
31
Fig: Typical Quantizer Characteristics for L odd (=7)

32
 Rate-Distortion Function
• Shannon’s Coding Theorem on noiseless channels
considers the channel, as well as the encoding
process to be lossless.

• With the introduction of quantizers, the encoding

process becomes lossy, even if the channel
remains as lossless.

• In most cases of lossy compressions, a limit is

generally specified on the maximum tolerable
distortion D from fidelity consideration.
33
 R-D Function
• Given a distortion measure D, how to obtain
the smallest possible rate ?

• The answer is provided by a branch of information

theory that is known as the rate-distortion theory.

• The corresponding function that relates the

smallest possible rate to the distortion, is called
the rate-distortion function R(D).

34
 R-D Function

Fig: Typical Rate Distortion Function

35
 R-D Function
• At no distortion (D=0), the corresponding rate
R(0) is equal to the entropy, as per Shannon’s
coding theorem for noiseless channels.

• Rate-distortion functions can be computed

analytically for simple sources and distortion
measures.

• Computer algorithms exist to compute R(D) when

analytical methods fail or are impractical.

36
 Source Coding Theorem
• There exists a mapping from the source symbols
to code words such that for a given distortion D,
R(D) bits/symbol are sufficient to enable source
reconstruction with an average distortion
arbitrarily close to D.

• The actual bits R is given by

37
 Example
• Lena Colour Image Quantized image (2-bit)
Original image Scalar Quantized Image

38
What you should be able to do at this point:

1. Define quantization.
2. Distinguish between scalar and vector quantization.
3. Distinguish between uniform and non-uniform quantization.
4. Define quantization error and optimum scalar quantizer design
criteria.
5. Design a Lloyd-Max quantizer.
6. Define rate-distortion function.
7. State the source coding theorem.
8. Determine the minimum possible rate for a given SNR to encode
a quantized Gaussian signal.

39
 Assignments

1. Pick up any monochrome image from the archive.

(a) Obtain the bit-plane mapped images for all the eight bit
planes.
(b) Represent the image by a 8-bit gray code defined as follows:
g 7 b7
g i bi  bi 1 , 0 i 6
where, b7 b6 b1b0 represents the binary values. Obtain the
gray-coded bit plane images for all the eight planes.
(c) Compare the two sets of bit-plane images in (a) and (b). In
what sense should gray-coded bit-planes be better? Justify
your answer.
40
 Assignments (Contd.)
2. (a) On the above bit-plane images, perform 1-D run-length
coding. If each run is to be represented by a 6-bit value,
calculate the compression ratio (comp. bits/ uncomp. bits) for:
(i) binary-mapped bit-planes and (ii) gray-coded bit-planes.
(b) From the statistics of gray-coded bit-planes, obtain the
probabilities of the run-lengths. Assign Huffman codes to the
run-lengths and calculate the compression ratio for the
resulting encoded bit stream. Compare this result with that of
(a)-(ii).
3. (a) Using the same monochrome image, obtain the predicted
image and the error image using a1 a 2 a3 a 4 0.25.
(b) Compute the histograms and the entropies of the original
image and the error image.
41