Quantization

Input-output characteristic of a scalar quantizer

x
Output

Sometimes, this
convention is used:

xq 2

M representative levels

xq 1
xq
tq

-1

t q+1

t q+2

Q x

input signal x

M-1 decision
thresholds
Bernd Girod: EE398A Image and Video Compression

Quantization no. 1

Example of a quantized waveform

Original and Quantized Signal

Quantization Error

Bernd Girod: EE398A Image and Video Compression

Quantization no. 2

Lloyd-Max scalar quantizer

Problem : For a signal x with given PDF f X ( x) find a quantizer with

M representative levels such that

d MSE E X X

min.

Solution : Lloyd-Max quantizer

[Lloyd,1957] [Max,1960]

M-1 decision thresholds exactly

half-way between representative
levels.

1
tq xq 1 xq q 1, 2,, M -1
2

M representative levels in the

centroid of the PDF between two xq

successive decision thresholds.
Necessary (but not sufficient)
conditions

Bernd Girod: EE398A Image and Video Compression

tq1

x f

( x)dx

tq

tq 1

q 0,1, , M -1

f X ( x)dx

tq

Quantization no. 3

Iterative Lloyd-Max quantizer design

1.
2.

Guess initial set of representative levels xq q 0,1, 2,, M -1

Calculate decision thresholds
1
tq xq 1 xq q 1, 2,, M -1
2

3.

Calculate new representative levels

tq1

xq

x f

( x)dx

tq

tq1

q 0,1,, M -1

f X ( x)dx

tq

4.

Repeat 2. and 3. until no further distortion reduction

Bernd Girod: EE398A Image and Video Compression

Quantization no. 4

Example of use of the Lloyd algorithm (I)

X zero-mean, unit-variance Gaussian r.v.

Design scalar quantizer with 4 quantization indices with
minimum expected distortion D*
Optimum quantizer, obtained with the Lloyd algorithm

Decision thresholds -0.98, 0, 0.98

Representative levels 1.51, -0.45, 0.45, 1.51
D*=0.12=9.30 dB
Boundary
Reconstruction

Bernd Girod: EE398A Image and Video Compression

Quantization no. 5

Example of use of the Lloyd algorithm (II)

Convergence

Quantization Function

Initial quantizer B:
decision thresholds , 0,
Quantization Function

Initial quantizer A:
decision thresholds 3, 0 3

6
4
2
0
-2
-4
-6

10

15

6
4
2
0
-2
-4
-6

0.2

0.2

0.15

0.4

10

10

SNROUT final = 9.2978

10

0.1

15

5
0

15

0
10

10

15

SNROUT10final = 9.298

15

8
6

Iteration Number

Iteration Number

SNROUT [dB]

Iteration Number

SNROUT [dB]

10

15

Iteration Number

After 6 iterations, in both cases, (D-D*)/D*<1%

Bernd Girod: EE398A Image and Video Compression

Quantization no. 6

Example of use of the Lloyd algorithm (III)

X zero-mean, unit-variance Laplacian r.v.

Design scalar quantizer with 4 quantization indices with
minimum expected distortion D*
Optimum quantizer, obtained with the Lloyd algorithm

Decision thresholds -1.13, 0, 1.13

Representative levels -1.83, -0.42, 0.42, 1.83
D*=0.18=7.54 dB
Threshold
Representative

Bernd Girod: EE398A Image and Video Compression

Quantization no. 7

Example of use of the Lloyd algorithm (IV)

Convergence
Initial quantizer A,
decision thresholds 3, 0 3
10
5
0
-5
-10

Initial quantizer B,
decision thresholds , 0,
Quantization Function

10

Iteration Number

12

14

10
5
0
-5
-10

0.2

0.2

6
4

10

15

SNRfinal = 7.5415
0

10

Iteration Number

10

12

14

16

Iteration Number
0.4

0
0
8

16

0.4

SNR [dB]

Quantization Function

15

SNR [dB]

0
0
8

6
4

10

15

SNRfinal = 7.5415
0

Iteration Number10

15

After 6 iterations, in both cases, (D-D*)/D*<1%

Bernd Girod: EE398A Image and Video Compression

Quantization no. 8

Lloyd algorithm with training data

xq ; q 0,1, 2,, M -1

1.

Guess initial set of representative levels

2.

Assign each sample xi in training set T to closest representative xq

Bq x T : Q x q
3.

Calculate new representative levels

xq

4.

q 0,1,2,, M -1

1
Bq

q 0,1,, M -1

xBq

Repeat 2. and 3. until no further distortion reduction

Bernd Girod: EE398A Image and Video Compression

Quantization no. 9

Lloyd-Max quantizer properties

Zero-mean quantization error

E X X 0

Quantization error and reconstruction decorrelated

E X X X 0

Variance subtraction property

E X X

2
X

2
X

Bernd Girod: EE398A Image and Video Compression

Quantization no. 10

High rate approximation

Approximate solution of the "Max quantization problem,"

assuming high rate and smooth PDF [Panter, Dite, 1951]

x( x) const

1
3

f X ( x)

Distance between two

successive quantizer
representative levels

Probability density
function of x

Approximation for the quantization error variance:

d E X X

1 3 f ( x)dx

12M 2 X
x

Number of representative levels

Bernd Girod: EE398A Image and Video Compression

Quantization no. 11

High rate approximation (cont.)

High-rate distortion-rate function for scalar Lloyd-Max quantizer

d R 2 X2 22 R

1 3
with f X ( x)dx
12 x

2
X

Some example values for

uniform
Laplacian
Gaussian

9
2

1
4.5

3
2.721
2

Bernd Girod: EE398A Image and Video Compression

Quantization no. 12

High rate approximation (cont.)

Partial distortion theorem: each interval makes an (approximately)

equal contribution to overall mean-squared error

Pr ti X ti 1 E X X ti X ti 1

Pr t j X t j 1 E X X t j X t j 1

for all i, j

[Panter, Dite, 1951], [Fejes Toth, 1959], [Gersho, 1979]

Bernd Girod: EE398A Image and Video Compression

Quantization no. 13

Entropy-constrained scalar quantizer

Lloyd-Max quantizer optimum for fixed-rate encoding, how can we

do better for variable-length encoding of quantizer index?

Problem : For a signal x with given pdf

rate
M 1

f X ( x) find a quantizer with

R H X pq log 2 pq

such that

q 0

d MSE E X X

min.
2

Solution: Lagrangian cost function

J d R E X X

H X min.
2

Bernd Girod: EE398A Image and Video Compression

Quantization no. 14

Iterative entropy-constrained scalar quantizer design

1.

2.

Guess initial set of representative levels xq ; q 0,1, 2,, M -1

and corresponding probabilities pq

Calculate M-1 decision thresholds

tq =
3.

xq -1 xq
2

log 2 pq 1 log 2 pq
2 xq -1 xq

q 1, 2,, M -1

Calculate M new representative levels and probabilities pq

tq1

xq

xf

( x)dx

tq

tq1

q 0,1,, M -1

f X ( x)dx

tq

4.

Repeat 2. and 3. until no further reduction in Lagrangian cost

Bernd Girod: EE398A Image and Video Compression

Quantization no. 15

Lloyd algorithm for entropy-constrained

quantizer design based on training set
1.

2.

Guess initial set of representative levels xq ; q 0,1, 2,, M -1

and corresponding probabilities pq

Assign each sample xi in training set T to representative

2

J
q

x
minimizing Lagrangian cost x i q log 2 pq

xq

Bq x T : Q x q
3.

Calculate new representative levels and probabilities pq

xq
4.

q 0,1, 2,, M -1

1
Bq

q 0,1,, M -1

xBq

Repeat 2. and 3. until no further reduction in overall Lagrangian

2
cost
J J
x Q x
log p

xi

xi

q xi

xi

Bernd Girod: EE398A Image and Video Compression

Quantization no. 16

Example of the EC Lloyd algorithm (I)

X zero-mean, unit-variance Gaussian r.v.

Design entropy-constrained scalar quantizer with rate R2 bits, and
minimum distortion D*
Optimum quantizer, obtained with the entropy-constrained Lloyd
algorithm

11 intervals (in [-6,6]), almost uniform

D*=0.09=10.53 dB, R=2.0035 bits (compare to fixed-length example)

Threshold
Representative level

Bernd Girod: EE398A Image and Video Compression

Quantization no. 17

Example of the EC Lloyd algorithm (II)

Same Lagrangian multiplier used in all experiments

Quantization Function

R [bit/symbol] SNR [dB]

4
2
0

4
2
0

-4

-4

-6

-6

10

15

20

25

Iteration Number

30

35

40

SNRfinal = 10.5363

8
6
5

10

15

20

25

30

35

40

Rfinal = 2.0044

1.5
0

-2

-2

10

Initial quantizer B, only 4 intervals

(<11) in [-6,6], with the same length

12

0
2.5

Quantization Function

Initial quantizer A, 15 intervals

(>11) in [-6,6], with the same length

10

15

20

25

30

35

40

R [bit/symbol] SNR [dB]

10

15

20

Iteration Number

12

SNRfinal = 8.9452

10
8
6
0
2.5

10

15

20

Rfinal = 1.7723

2
1.5
1

Iteration Number

Bernd Girod: EE398A Image and Video Compression

10

15

20

Iteration Number

Quantization no. 18

Example of the EC Lloyd algorithm (III)

X zero-mean, unit-variance Laplacian r.v.

Design entropy-constrained scalar quantizer with rate R2 bits
and minimum distortion D*
Optimum quantizer, obtained with the entropy-constrained Lloyd
algorithm

21 intervals (in [-10,10]), almost uniform

D*= 0.07=11.38 dB, R= 2.0023 bits (compare to fixed-length example)

Decision threshold
Representative level

Bernd Girod: EE398A Image and Video Compression

Quantization no. 19

Example of the EC Lloyd algorithm (IV)

Same Lagrangian multiplier used in all experiments
Initial quantizer A, 25 intervals
(>21 & odd) in [-10,10], with the
same length

10
5
0
-5
-10

10

15

20

25

30

35

Initial quantizer B, only 4 intervals

(<21) in [-10,10], with the same length
Quantization Function

Quantization Function

40

10
5
0
-5
-10

10

10

SNRfinal = 11.407

10

15

20

25

30

35

40

Rfinal = 2.0063
2
1

10

15

20

25

30

35

40

R [bit/symbol] SNR [dB]

15

20

15
10

SNRfinal = 7.4111

5
0

10

15

20

Rfinal = 1.6186

2
1

Iteration Number

15

Iteration Number

Iteration Number

R [bit/symbol] SNR [dB]

10

Iteration Number

15

20

Convergence in cost faster than convergence of decision thresholds

Bernd Girod: EE398A Image and Video Compression

Quantization no. 20

High-rate results for EC scalar quantizers

For MSE distortion and high rates, uniform quantizers (followed by

entropy coding) are optimum [Gish, Pierce, 1968]

Distortion and entropy for smooth PDF and fine quantizer interval

2
d
12
2

H X h X log 2

Distortion-rate function

1 2 h X 2 R
d R 2
2
12

e
is factor
or 1.53 dB from Shannon Lower Bound
6
1 2 h X 2 R
D R
2
2
2 e
Bernd Girod: EE398A Image and Video Compression

Quantization no. 21

Comparison of high-rate performance of

scalar quantizers

High-rate distortion-rate function

d R 2 X2 22 R

Scaling factor

Shannon LowBd

Laplacian

6
0.703
e
e
0.865

Gaussian

Uniform

Lloyd-Max

Entropy-coded

9
4.5
2
3
2.721
2

Bernd Girod: EE398A Image and Video Compression

e2
1.232
6
e
1.423
6
Quantization no. 22

Deadzone uniform quantizer

Quantizer
output x

Bernd Girod: EE398A Image and Video Compression

Quantizer
input x

Quantization no. 23

Embedded quantizers

Motivation: scalability decoding of compressed

bitstreams at different rates (with different qualities)
Nested quantization intervals
Q0

Q1
Q2

In general, only one quantizer can be optimum

(exception: uniform quantizers)

Bernd Girod: EE398A Image and Video Compression

Quantization no. 24

Example: Lloyd-Max quantizers for Gaussian PDF

2-bit and 3-bit optimal

quantizers not embeddable
Performance loss for
embedded quantizers

-.98

.98

0 0 0 01 1 1 1
0 0 1 10 0 1 1
0 1 0 10 1 0 1
-1.75 -1.05 -.50

Bernd Girod: EE398A Image and Video Compression

.50 1.05 1.75

Quantization no. 25

Information theoretic analysis

Successive refinement Embedded coding at multiple

rates w/o loss relative to R-D function
E d ( X , X 1 ) D1

I ( X ; X 1 ) R( D1 )
I ( X ; X ) R( D )

E d ( X , X 2 ) D2

Successive refinement with distortions D1 and D2 D1

can be achieved iff there exists a conditional distribution

f X , X
1

( x1 , x2 , x) f X

( x2 , x) f X

X 2

( x1 , x2 )

Markov chain condition

X X 2 X 1
Bernd Girod: EE398A Image and Video Compression

[Equitz,Cover, 1991]
Quantization no. 26

Embedded deadzone uniform quantizers

x=0
8

Q0

Q1
Q2

Supported in JPEG-2000 with general for

quantization of wavelet coefficients.
Bernd Girod: EE398A Image and Video Compression

Quantization no. 27

Vector quantization

Bernd Girod: EE398A Image and Video Compression

Quantization no. 28

LBG algorithm

Lloyd algorithm generalized for VQ [Linde, Buzo, Gray, 1980]

Best representative
vectors
for given partitioning
of training set

Best partitioning
of training set
for given
representative
vectors

Assumption: fixed code word length

Code book unstructured: full search

Bernd Girod: EE398A Image and Video Compression

Quantization no. 29

Design of entropy-coded vector quantizers

Extended LBG algorithm for entropy-coded VQ

[Chou, Lookabaugh, Gray, 1989]

Lagrangian cost function: solve unconstrained problem rather than

constrained problem

J d R E X X H X min.

Unstructured code book: full search for

J xi q xi xq log 2 pq
2

The most general coder structure:

Any source coder can be interpreted as VQ with VLC!

Bernd Girod: EE398A Image and Video Compression

Quantization no. 30

Lattice vector quantization

Amplitude 2
cell

pdf

representative
vector

Amplitude 1

Bernd Girod: EE398A Image and Video Compression

Quantization no. 31

8D VQ of a Gauss-Markov source

SNR [dB]

18

LBG fixed
CWL

12

E8-lattice

LBG variable
CWL

6
scalar

r 0.95

0
0

0.5

1.0

Rate [bit/sample]
Bernd Girod: EE398A Image and Video Compression

Quantization no. 32

8D VQ of memoryless Laplacian source

9
E8-lattice

SNR [dB]

LBG variable
CWL

scalar

3
LBG fixed
CWL

0.5

1.0

Rate [bit/sample]
Bernd Girod: EE398A Image and Video Compression

Quantization no. 33

Reading

Taubman, Marcellin, Sections 3.2, 3.4

J. Max, Quantizing for Minimum Distortion, IEEE Trans.
Information Theory, vol. 6, no. 1, pp. 7-12, March 1960.
S. P. Lloyd, Least Squares Quantization in PCM, IEEE
Trans. Information Theory, vol. 28, no. 2, pp. 129-137,
March 1982.
P. A. Chou, T. Lookabaugh, R. M. Gray, Entropyconstrained vector quantization, IEEE Trans. Signal
Processing, vol. 37, no. 1, pp. 31-42, January 1989.

Bernd Girod: EE398A Image and Video Compression

Quantization no. 34