Lossless Image

Compression
Yao Wang
Polytechnic Institute of NYU, Brooklyn, NY 11201
With contribution from Zhu Liu
Partly based on A. K. Jain, Fundamentals of Digital Image
Processing
Yao Wang, NYU-Poly EL5123: Lossless Compression
2
Lecture Outline
Introduction: need for compression
Review of basic probability theory
Binary encoding
Fixed length coding
Variable length coding
Huffman coding
Other variable length code (LZW, arithmetic)
Runlength coding of bilevel images
Fax coding standard
Lossless Predictive coding
Coding block diagram
Design of linear predictors
Yao Wang, NYU-Poly EL5123: Lossless Compression
3
Necessity for Signal Compression
Size
One Page of Text 2 KB
One 640x480 24-bit color still image 900 KB
Voice ( 8 Khz, 8-bit) 8 KB /second
Audio CD DA (44.1 Khz, 16-bit) 176 KB/second
Animation ( 320x640 pixels, 16-bit color, 16 frame/s) 6.25 MB/second
Video (720x480 pixels, 24-bit color, 30 frame/s) 29.7 MB/second
Storage requirement for various uncompressed
data types
Goal of compression
Given a bit rate, achieve the best quality
Given an allowed distortion, minimize the data amount
(Rate-Distortion or RD tradeoff)
Yao Wang, NYU-Poly EL5123: Lossless Compression
4
Image Coding Standards
by ITU and ISO
G3,G4: facsimile standard
JBIG: The next generation facsimile standard
ISO Joint Bi-level Image experts Group
JPEG: For coding still images or video frames.
ISO Joint Photographic Experts Group
JPEG2000: For coding still images, more efficient than JPEG
Lossless JPEG: for medical and archiving applications.
MPEGx: audio and video coding standards of ISo
H.26x: video coding standard of ITU-T
ITU: International telecommunications union
ISO: International standards organization
Yao Wang, NYU-Poly EL5123: Lossless Compression
5
A Typical Compression System
Transfor-
mation
Quanti-
zation
Binary
Encoding
Prediction
Transforms
Model fitting
...
Scalar Q
Vector Q
Fixed length
Variable length
(Huffman,
arithmetic, LZW)
Input
Samples
Transformed
parameters
Quantized
parameters
Binary
bitstreams
Motivation for transformation ---
To yield a more efficient representation of the original samples.
Yao Wang, NYU-Poly EL5123: Lossless Compression
6
Review of Basic Probability Theory
A. Papoulis and S. Pillai, Probability, Random
Variables, and Stochastic Processes, McGraw-
Hill, Inc., 2002.
http://www.mhhe.com/engcs/electrical/papoulis/
Random Variable (RV)*
The set, S, of all possible outcomes of a particular
experiment is called the sample space for the
experiment.
An event is any collection of possible outcomes of an
experiment, that is, any subset of S.
Random variable is a function from a sample space S
into the real numbers.
*From G. Casella and R. Berger, Statistical Inference, Duxbury Press, 1990.
Yao Wang, NYU-Poly EL5123: Lossless Compression
7
Examples of Random Variables
Tossing two coins, X is the number of
heads, and Y is the number of tails
X and Y take on values {0, 1, 2}
Discrete type
X is the lifetime of a certain brand of light
bulbs
X take on values [0, +)
Continuous type
Yao Wang, NYU-Poly EL5123: Lossless Compression
8
Distribution, Density, and Mass
Functions
The cumulative distribution function (cdf) of a random
variable X, is defined by
If X is a continuous random variable (taking value over a
continuous range)
F
X
(x) is continuous function.
The probability density function (pdf) of X is given by
If X is a discrete random variable (taking a finite number
of possible values)
F
X
(x) is step function.
The probability mass function (pmf) of X is given by
x. all for ), .( Pr ) ( x X x F
X
=
) .( Pr ) ( x X x p
X
= =
) ( ) ( x F
dx
d
x f
X X
=
The percentage of time that X=x.
Yao Wang, NYU-Poly EL5123: Lossless Compression
9
Special Cases
Binomial (discrete)
Poisson distribution
Normal (or Gaussian) N(,
2
)
Uniform over (x
1
, x
2
),
Laplacian L(, b)
. ,..., 1 , 0 , ) 1 ( } { n k p p
k
n
k X P
k n k
=
|
|

\
|
= =

) 2 /( ) (
2 2
2
1
) (



=
x
e x f

=
otherwise 0
1
) (
2 1
1 2
x x x
x x
x f
,... 1 , 0 ,
!
} { = = =

k
k
a
e k X P
k
a
b x
e
b
x f
/ | |
2
1
) (

=
Figures are from
http://mathworld.wolfram.com
Yao Wang, NYU-Poly EL5123: Lossless Compression
10
Expected Values
The expected (or mean) value of a random variable X:
The variance of a random variable X:
Mean and variance of common distributions:
Uniform over range (x
1
, x
2
): E{x} = (x
1
+x
2
)/2, VarX = (x
2
-x
1
)
2
/12
Gaussian N(,
2
): Ex = , VarX =
2
Laplace L(, b): Ex = , VarX = 2b
2

=
= =

discrete is X if ) (
continuous is X if ) (
} {
X x
X
X
x X xP
dx x xf
X E
( )
( )

= =

discrete is X if ) (
continuous is X if ) (
} {
X
2
2
2
x
X
X X
X
x X P x
dx x f x
X Var

Yao Wang, NYU-Poly EL5123: Lossless Compression

11
Functions of Random Variable
Y=g(X)
Following the example of the lifetime of the bulb, let Y
represents the cost of a bulb, which depends on its lifetime X
with relation
Expectation of Y
Variance of Y
X Y =

=
= =

discrete is X if ) ( ) (
continuous is X if ) ( ) (
} {
X x
X
Y
x X P x g
dx x f x g
Y E
( )
( )

= =

discrete is X if ) ( ) (
continuous is X if ) ( ) (
} {
X
2
2
2
x
X
X Y
Y
x X P x g
dx x f x g
Y Var

Yao Wang, NYU-Poly EL5123: Lossless Compression

12
Two RVs
We only discuss discrete RVs (i.e. X and Y for both
discrete RVs)
The joint probability mass function (pmf) of X and Y is
given by
The conditional probability mass function of X given Y is
Important relations
) , .( Pr ) , ( y Y x X y x p
XY
= = =
) | .( Pr ) / (
/
y Y x X y x p
Y X
= = =
) ( ) / ( ) , (
/
y p y x p y x p
Y Y X XY
=

= = =
Y y
XY X
y Y x X x p ) , ( . Pr ) (
Yao Wang, NYU-Poly EL5123: Lossless Compression
13
Conditional Mean and Covariance
Conditional mean
Correlation
Correlation matrix
Covariance
Covariance matrix

= = = =
X
|
) | ( } | {
x
y X
y Y x X xP y X E

= = = =
Y y X,
,
) , ( } {
x
Y X
y Y x X xyP XY E R
( )( )
Y X Y X Y X Y X
R Y X E C = =
, ,
} {
[ ]
(

=
)
`

=
2
2
Y XY
XY X
Y X
Y
X
C
C
Y X
Y
X
E

C
[ ]
{ }
{ }
{ }
2 2 2
2
2
,
X X
XY
XY
X E
Y E R
R X E
Y X
Y
X
E + =
(

=
)
`

= R
Yao Wang, NYU-Poly EL5123: Lossless Compression
14
Multiple RVs
The definitions for two RVs can be easily extended to
multiple (N>2) RVs, X
1
,X
2
, , X
N
The joint probability mass function (pmf) is given by
Covariance matrix is
[ ]
(
(
(
(
(

(
(
(
(

=
2
2 1
2
2
2 21
1 12
2
1
2 2 1 1
2 2
1 1
...
... ... ... ...
...
...
...
N N N
N
N
N N
N N
C C
C C
C C
X X X
X
X
X
E

C
) ,..., , .( Pr ) ,..., , (
2 2 1 1 2 1 N N N
x X x X x X x x x p = = = =
Yao Wang, NYU-Poly EL5123: Lossless Compression
15
Binary Encoding
Binary encoding
To represent a finite set of symbols using binary
codewords.
Fixed length coding
N symbols represented by (int) log
2
(N) bits.
Variable length coding
more frequently appearing symbols represented by
shorter codewords (Huffman, arithmetic, LZW=zip).
The minimum number of bits required to
represent a source is bounded by its entropy.
Yao Wang, NYU-Poly EL5123: Lossless Compression
16
Entropy of a Source
Consider a source of N symbols, r
n
, n=1,2,,N. This
source can be considered as a discrete RV with N
possible outcomes.
Suppose the probability of symbol r
n
is p
n
. The self
information of symbol r
n
is defined as,
The entropy of this source, which represents the average
information, is defined as:
The self information indicates the number of bits required
to convey a particular outcome (r
n
) of the underlying RV.
The entropy indicates the average number of bits
required to specify any possible outcomes.
) ( log
2
bits p H
n n
=

=
=
N
n
n n
bits p p H
1
2
) ( log
Yao Wang, NYU-Poly EL5123: Lossless Compression
17
Shannon Source Coding Theory
For a source with p
n
=2
-l
n
, we can design a code
such that the length of the codeword for r
n
is
l
n
=-log
2
p
n
, and the average bit rate is

= = = H p p l p l
n n n n 2
log
Yao Wang, NYU-Poly EL5123: Lossless Compression
18
Shannon Source Coding Theory
For an arbitrary source, a code can be designed
so that -log
2
p
n
<= l
n
<= -log
2
p
n
+1
This is Shannon Source Coding Theorem, which
states the lower and upper bound for variable
length coding.
The Shannon theorem only gives the bound but
not the actual way of constructing the code to
achieve the bound.
1 ) 1 (log log
, 1 log log
2 2
2 2
+ = + = =
+

H p p l p l p p H
have we p l p Since
n n n n n n
n n n
Yao Wang, NYU-Poly EL5123: Lossless Compression
19
Huffman Coding
A method to realize the Shannon bound
for any given source
Procedure of Huffman coding
Step 1: Arrange the symbol probabilities p
n
in
a decreasing order and consider them as leaf
nodes of a tree.
Step 2: While there is more than one node:
Find the two nodes with the smallest probability
and arbitrarily assign 1 and 0 to these two nodes
Merge the two nodes to form a new node whose
probability is the sum of the two merged nodes.
Yao Wang, NYU-Poly EL5123: Lossless Compression
20
Example of Huffman Coding
Symbol Prob Codeword Length
1
2 36/49 1 1


1 1
3 8/49 01 2

0
1
1 4/49 13/49 001 3
0



0 5/49
0 1/49 000 3
. 16 . 1 log ; 4 . 1
49
67
3
49
1
3
49
4
2
49
8
1
49
36
2
= = = = + + + =
n n
p p H I
Yao Wang, NYU-Poly EL5123: Lossless Compression
21
Example of Huffman Coding
1.00
Section
VIII
Section
VII
Section
VI
Section
V
Section
IV
Section
III
Section
II
Section
I
.57
.43
.30
.27
.43
.27
.30 .30 .30 .30 .30
.23
.20
.23 .23 .23 .23
.15
.12
.15 .15 .15
.12 .12
.12
.08
.08 .08
.06 .06
.06 .06
.06
.06
b(2):00
b(1):10
b(3):010
b(4):111
b(0): 1100
b(5):1101
b(6): 0110
b(7): 0111
.20
0
1
0
1
0
1
0
1
0
1
1
1
0
0
bits. 68 . 2 log Entropy bits; 71 . 2 Rate Bit Average
2
= = = =

n
n n
n
n n
p p l p
Yao Wang, NYU-Poly EL5123: Lossless Compression
22
Disadvantage of Huffman Coding
At least one bit has to be used for each symbol.
Solution
Vector Huffman coding: To obtain higher compression,
we can treat each group of M symbols as one entity
and give each group a codeword based on the group
probability. (M-th order Huffman coding)
Consider the m symbols as the outcome of m RVs
Joint Entropy of a pair of variables (X,Y) with a joint
distribution p(x,y)

=
x y
y x p y x p Y X H ) , ( log ) , ( ) , (
2

=
1
) ,..., ( log ) ,..., ( ... ) ,..., (
1 2 1 1
x x
M M M
M
x x p x x p X X H
Yao Wang, NYU-Poly EL5123: Lossless Compression
23
Example of Vector Huffman Coding
(1)
Consider an alphabetic source of three
symbols: a, b, c.
Assume the symbols are independent.
6
1
) ( ,
6
1
) ( ,
3
2
) ( = = = c p b p a p
) ( ) ( ) ( y p x p xy p =
36
1
6
1
6
1
) (
36
1
6
1
6
1
) (
9
1
3
2
6
1
) (
36
1
6
1
6
1
) (
36
1
6
1
6
1
) (
9
1
3
2
6
1
) (
9
1
6
1
3
2
) (
9
1
6
1
3
2
) (
9
4
3
2
3
2
) (
= = = = = =
= = = = = =
= = = = = =
cc p cb p ca p
bc p bb p ba p
ac p ab p aa p
Yao Wang, NYU-Poly EL5123: Lossless Compression
24
Example of Vector Huffman Coding
(2)
Design a Huffman code which uses a separate
codeword for each symbol
33 . 1
3
4
2
6
1
2
6
1
1
3
2
= = + + = l
a
b
c
2/3
1/6
1/6
0
1
0
1
1/3
1
01
00
Symbol Probability
Yao Wang, NYU-Poly EL5123: Lossless Compression
25
Example of Vector Huffman Coding
(3)
Design a Huffman code which use a separate
codeword for each group of two symbols.
aa 4/9
ab 1/9
ac 1/9
ba 1/9
bb 1/36
bc 1/36
ca 1/9
cb 1/36
cc 1/36
0
1
1/18
0
1
1/18
0
1
1/9
2/9
0
1
2/9
0
1
3/9
5/9
1
011
0101
0100
00111
00110
000
00101
00100
0
1
0
1
0
1
27 . 1
36
46
5
36
1
5
36
1
3
9
1
5
36
1
5
36
1
4
9
1
4
9
1
3
9
1
1
9
4
2
1
= =
(

+ + + + + + + + = l
Yao Wang, NYU-Poly EL5123: Lossless Compression
26
Explain in entropy relation H(X,Y) and
H(X), H(Y|X)
Yao Wang, NYU-Poly EL5123: Lossless Compression
27
Conditional Entropy
If (X,Y) ~ p(x,y), then the conditional entropy
H(Y|X) is defined as
1
st
order conditional Huffman coding
Assume there are N symbols {a
i
}, I =1,,N, we have
to build N different Huffman coder C
i
based on p(a
j
|a
i
)

=
=
=
= =
x y
x y
x y
x
x y p y x p
x y p x y p x p
x y p x y p x p
x X Y H x p X Y H
) | ( log ) , (
) | ( log ) | ( ) (
) | ( log ) | ( ) (
) | ( ) ( ) | (
2
2
2
Yao Wang, NYU-Poly EL5123: Lossless Compression
28
Example of Conditional Huffman Coding
{ }

=
=
otherwise
j i if
a a p
j i
4 / 1
2 / 1
/
5 . 1 5 . 1
3
1
3
5 . 1
4
1
log
4
1
2
2
1
log
2
1
) | ( log ) | ( ) | (
) | ( ) (
2 2
3
1
2
3
1
= =
= = =
=

=
=
H
a a p a a p a X H
a X H a p H
j
i j i j i
i
i i
A source with three symbols A
= {a
1
, a
2
, a
3
} has the following
probability distributions
a
1
|a
1
1/2
1/2
0
1
0
1
1
01
00
a
2
|a
1
1/4
a
3
|a
1
1/4
a
2
|a
2
1/2
1/2
0
1
0
1
1
01
00
a
1
|a
2
1/4
a
3
|a
2
1/4
a
3
|a
3
1/2
1/2
0
1
0
1
1
01
00
a
1
|a
3
1/4
a
2
|a
3
1/4
5 . 1 2
4
1
2 1
2
1
= + = l
What is
the previous
Symbol?
Input
a
1
a
2
a
3
Yao Wang, NYU-Poly EL5123: Lossless Compression
29
Other Variable Length Coding Methods
Huffman coding
Assign variable length codewords to a fixed length sequence of
symbols
LZW coding (Lempel, Ziv, and Welsh)
Assign fixed-length codewords to variable length sequences of
source symbols
Does not require priori knowledge of the symbol probabilities.
(universal code)
Not as efficient as Huffman for a given distribution
Arithmetic coding
Assign variable-length codewords to variable length sequences
of source symbols
More efficient than Huffman coding (bit rate closer to the source
entropy)
Requires significantly more computation
Yao Wang, NYU-Poly EL5123: Lossless Compression
30
Significance of Compression for
Facsimile
For a size A4 document (digitized to
1728x2376 pixels)
Without compression
Group 1 Fax: 6 min/page
Group 2 Fax: 3 min/page
With compression
Group 3 Fax: 1 min/page
Fax becomes popular only after the
transmission time is cut down to below 1
min/page.
Yao Wang, NYU-Poly EL5123: Lossless Compression
31
Runlength Coding of Bi-Level Images
1D Runlength Coding:
Count length of white and length of black alternatingly
Represent the last runlength using EOL
Code the white and black runlength using different
codebook (Huffman Coding)
2D Runlength Coding:
Use relative address from the last transition in the line
above
Used in Facsimile Coding (G3,G4)
Yao Wang, NYU-Poly EL5123: Lossless Compression
32
Example of 1-D Runlength Coding
X
X
2
4
2
1
6
4
2
1
2
4
X
2
1
5
1
6
1
5
1
5
1
X
2 5 6 5 5
X

X
X
Yao Wang, NYU-Poly EL5123: Lossless Compression
33
Example Continued
Design the Huffman coding table for all
possible runlength symbols
Do on the board
Yao Wang, NYU-Poly EL5123: Lossless Compression
34
Example of 2-D Runlength Coding
Relative address coding (RAC) is based on the principle of tracking
the binary transitions that begin and end each black and white run.
c is current transition, e is the last transition in the same line, c is the
first similar transition past e in the previous line.
If ec <= cc, d = ec; if cc < ec, d = cc.
Yao Wang, NYU-Poly EL5123: Lossless Compression
35
CCITT Group 3 and Group 4 Facsimile
Coding Standard the READ Code
Relative Element Address Designate
The first line in every K lines is coded using
1D runlength coding, and the following K-1
lines are coded using 2D runlength coding
The reason for 1D RLC is used for every K line is
to suppress propagation of transmission errors.
Group 4 method is designed for more
secure transmission, such as leased data
line where the bit error rate is very low.
Yao Wang, NYU-Poly EL5123: Lossless Compression
36
Predictive Coding
Motivation
The value of a current pixel usually does not
change rapidly from those of adjacent pixels.
Thus it can be predicted quite accurately from
the previous samples.
The prediction error will have a non-uniform
distribution, centered mainly near zero, and
can be specified with less bits than that
required for specifying the original sample
values, which usually have a uniform
distribution.
Yao Wang, NYU-Poly EL5123: Lossless Compression
37
Lossless Predictive Coding System
+
Predictor
Delay
(frame store)
Binary
Encoding
+
-
f
e
f

f
+
+
+
p
f
Encoder
Decoder
Binary
Decoding
+
B
e
B
e
e
Predictor
Delay
(frame store)
p
f
f

f
Yao Wang, NYU-Poly EL5123: Lossless Compression
38
Linear Predictor
Let f
0
represent the current pixel, and f
k
, k =
1,2,,K the previous pixels that are used to
predict f
0
. For example, if f
0
=f(m,n), then f
k
=f(m-
i,n-j) for certain i, j 0. A linear predictor is
a
k
are called linear prediction coefficients or
simply prediction coefficients.
The key problem is how to determine a
k
so that
a certain criterion is satisfied.

=
=
K
k
k k
f a f
1
0

Yao Wang, NYU-Poly EL5123: Lossless Compression

39
LMMSE Predictor (1)
The designing criterion is to minimize the mean
square error (MSE) of the predictor.
The optimal a
k
should minimize the error

= =

=
2
1
0
2
0 0
2
} |

{|
K
k
k k p
f a f E f f E
. ,..., 2 , 1 , 0
1
0
2
K l f f a f E
a
l
K
k
k k
l
p
= =
)
`

\
|
=

Let R(k,l)=E{f
k
f
l
}

=
= =
K
k
k
K l l R l k R a
1
,..., 2 , 1 ), , 0 ( ) , (
Yao Wang, NYU-Poly EL5123: Lossless Compression
40
LMMSE Predictor (2)
In matrix format
r R a r Ra
K R
R
R
a
a
a
K K R K R K R
K R R R
K R R R
K
1 1
1
) , 0 (
) 2 , 0 (
) 1 , 0 (
) , ( ) , 2 ( ) , 1 (
) 2 , ( ... ) 2 , 2 ( ) 2 , 1 (
) 1 , ( ... ) 1 , 2 ( ) 1 , 1 (

= =
(
(
(
(

=
(
(
(
(

(
(
(
(

M M
L
M O M M
The MSE of this predictor
r R r R a r R k R a R f f f E
T T
K
k
k p
1
1
0 0 0
2
) 0 , 0 ( ) 0 , 0 ( ) 0 , ( ) 0 , 0 ( } )

{(

=
= = = =

=
= =
K
k
k
K l l R l k R a
1
,..., 2 , 1 ), , 0 ( ) , (
Yao Wang, NYU-Poly EL5123: Lossless Compression
41
An Example of Predictive Coder
Assume the current pixel f
0
= f(m, n) is predicted from two pixels, one on the
left, f
1
= f(m, n-1), and one on the top, f
2
= f(m-1,n). Assume the pixel values
have zero means. The correlations are: R(0,0) = R(1,1) = R(2,2) =
f
2
, R(0,1)
=
h

f
2
, R(0,2) =
v

f
2
, R(1,2) = R(2,1) =
d

f
2
.
[ ]
|
|

\
|
+
=
(

+
=
(

= =
|
|

\
|

+
=
(

=
(

=
(

=
(

=
(

=
(

d
f p
d
v h
d
h d v v h
f p
h d v
v d h
d v
h
d
d
d
v
h
d
d
a
a
isotropic is n correlatio the if
a
a
R R R
a
a
a
a
R
R
a
a
R R
R R

1
2
1
1
1
1
, ,
1
2
1 ) 2 , 0 ( ) 1 , 0 ( ) 0 , 0 (
1
1
1
1
1
1
1
1
) 2 , 0 (
) 1 , 0 (
) 2 , 2 ( ) 1 , 2 (
) 1 , 2 ( ) 1 , 1 (
2
2 2
2
1
2
2 2
2
2
1 2
2 2
2
1
2
1
2
1
) , 1 ( ) 1 , ( ) , (

2 1
n m f a n m f a n m f + =
)} 1 , ( ) , 1 ( { ) 1 , 2 (
)} , 1 ( ) 1 , ( { ) 2 , 1 (
)} , 1 ( ) , ( { ) 2 , 0 (
)} 1 , ( ) , ( { ) 1 , 0 (
)} , 1 ( ) , 1 ( { ) 2 , 2 (
)} 1 , ( ) 1 , ( { ) 1 , 1 (
)} , ( ) , ( { ) 0 , 0 (
) , 1 (
) 1 , (
) , (
2
1
0
=
=
=
=
=
=
=
=
=
=
n m f n m f E R
n m f n m f E R
n m f n m f E R
n m f n m f E R
n m f n m f E R
n m f n m f E R
n m f n m f E R
n m f f
n m f f
n m f f
f(m,n)
f(m-1,n)
f(m,n-1)
Note: when the pixel value has non-zero mean, the above predictor can be applied to mean-shifted values.
Yao Wang, NYU-Poly EL5123: Lossless Compression
42
Homework (1)
1. Consider a discrete source with an alphabet A = {a
1
, a
2
, , a
L
}.
Compute the entropy of the source for the following two cases:
(a) the source is uniformly distributed, with p(a
l
) = 1/L, l = 1, 2, , L.
(b) For a particular k, p(a
k
) = 1 and p(a
l
) = 0; l k.
2. A source with three symbols A = {a
1
, a
2
, a
3
} has the following
probability distributions:
(a) Calculate the 1st order entropy, 2nd order entropy, 1st order
conditional entropy.
Hint: the probability for having a pair of symbols a
i
a
j
is P(a
i
)P(a
j
|a
i
).
(b) Design the 1st order, 2nd order, and 1st order conditional
Huffman codes for this source. Calculate the resulting bit rate for
each case. Compare it to the corresponding lower and upper
bounds defined by the entropy. Which method has the lowest bit
rate per symbol? How do they compare in complexity?
{ } { }

=
= =
otherwise
j i if
a a p i a p
j i i
6 / 1
3 / 2
/ ; all for , 3 / 1
Yao Wang, NYU-Poly EL5123: Lossless Compression
43
Homework (2)
3. For the following image which consists of 8 symbols, (a) Determine the probability of each
symbol based on its occurrence frequency; (b) Find its entropy; (c) Design a codebook for
these symbols using Huffman coding method. Calculate the average bit rate and compare it to
the entropy.
4. For the following bi-level image: (a) Give its run-length representation in one dimensional RLC.
Assume that each line start with W and mark the end with EOL. (b) Suppose we want to use
the same codeword for the black and white run-lengths, determine the probability of each
possible run-length (including the symbol EOL) and calculate the entropy of this source; (c)
Determine the Huffman code for the source containing all possible runlengths, calculate the
average bit rate of this code, and compare it to the entropy.
0 1 2 3
2 3 5 5
2 5 5 6
4 4 6 7
White pixel
Black pixel
Yao Wang, NYU-Poly EL5123: Lossless Compression
44
Homework (3)
5. Assume the current pixel f
0
= f(m, n) is predicted from three pixels, f
1
= f(m, n-1), f
2
=
f(m-1,n), f
3
= f(m-1,n-1). Assume the pixel values have zero means. The correlations
are: R(0,0) = R(1,1) = R(2,2) = R(,3,3)=
f
2
, R(0,1) = R(2,3)=
h

f
2
, R(0,2) =R(1,3)=

f
2
, R(1,2) = R(0,3) =
d

f
2
. We want to predict f0 from f1,f2,f3 using a linear
predictor f0=a1 f1 +a2 f2 + a3 f3. Determine the predictor coefficients a1,a2,a3 that
will minimize the prediction MSE. For arbitrary values of
h,

v
,
d,
you can just write
down the equation for a1,a2,a3. For the specific case with
h
=
v
=
d
= , write down
the solution explicitly.
f0
f2
f1
f3
Yao Wang, NYU-Poly EL5123: Lossless Compression
45
Homework (4)
6. Suppose we want to code an image using lossless predictive coding, where
a new pixel is predicted by the average of the top and left pixel values. For
the following image:
Determine the predicted image, the prediction error image, and the
reconstructed image. Also determine the entropy of the prediction error
image, which provides a lower bound on the required bit rate. Only do for
the lower 3x3 region, assuming the top row and leftmost column are know.
(
(
(
(

9 7 6 8
10 11 9 7
9 7 8 6
7 4 6 5
Yao Wang, NYU-Poly EL5123: Lossless Compression
46
Reading
R. Gonzalez, Digital Image Processing,
Section 8.1 - 8.4
A.K. Jain, Fundamentals of Digital Image
Processing, Section 11.1 11.3