Adaptive Huffman Coding

• One pass
• During the pass calculate the frequencies
CSEP 590
• Update the Huffman tree accordingly
Data Compression – Coder – new Huffman tree computed after transmitting the
Autumn 2007 symbol
– Decoder – new Huffman tree computed after receiving the
symbol
Adaptive Huffman Coding • Symbol set and their initial codes must be known
ahead of time.
• Need NYT (not yet transmitted symbol) to indicate a
new leaf is needed in the tree.

CSEP 590 - Lecture 2 - Autumn 2007 2

Optimal Tree Numbering Weight the Nodes

• a : 5, b: 2, c : 1, d : 3 • a : 5, b: 2, c : 1, d : 3

11

5 6
a a
3 3
d d
1 2
c b c b

CSEP 590 - Lecture 2 - Autumn 2007 3 CSEP 590 - Lecture 2 - Autumn 2007 4

Number the Nodes Adaptive Huffman Principle

• a : 5, b: 2, c : 1, d : 3 • In an optimal tree for n symbols there is a
numbering of the nodes y1<y2<... <y2n-1 such
11 7 that their corresponding weights x1,x2, ... , x2n-1
6 satisfy:
5 5 6
a – x1 < x2 < ... < x2n-1
3 3 3 4 – siblings are numbered consecutively
d
• And vice versa
1 1 2 2
c b – That is, if there is such a numbering then the tree is
optimal. We call this the node number invariant.
Number the nodes as they are removed from the
priority queue.
CSEP 590 - Lecture 2 - Autumn 2007 5 CSEP 590 - Lecture 2 - Autumn 2007 6

1
 Initialization Initialization
• Symbols a1, a2, ... ,am have a basic prefix code, • The tree will encode up to m + 1 symbols
used when symbols are first encountered. including NYT.
• Example: a, b ,c, d, e, f, g, h, i, j • We reserve numbers 1 to 2m + 1 for node
numbering.
0 1
• The initial Huffman tree consists of a single
0 1 0 1 node weight
0 1 0 1 0 1 0 1
0 2m + 1
1 0 1 NYT node number
a b c d e f 0

g h i j
CSEP 590 - Lecture 2 - Autumn 2007 7 CSEP 590 - Lecture 2 - Autumn 2007 8

Coding Algorithm Decoding Algorithm

1. If a new symbol is encountered then output the 1. Decode the symbol using the current tree.
code for NYT followed by the fixed code for the 2. If NYT is encountered then use the fixed code to
symbol. Add the new symbol to the tree. decode the symbol. Add the new symbol to the
2. If an old symbol is encountered then output its tree.
code. 3. Update the tree to preserve the node number
3. Update the tree to preserve the node number invariant.
invariant.

CSEP 590 - Lecture 2 - Autumn 2007 9 CSEP 590 - Lecture 2 - Autumn 2007 10

Updating the Tree Example

1. Let y be leaf (symbol) with current weight x.* • aabcdad in alphabet {a,b,..., j}
2. If y the root update x by 1, otherwise,
3. Exchange y with the largest numbered node with
the same weight (unless it is the parent).** 0 21
4. Update x by 1 NYT
5. Let y be the parent with its weight x and go to 2.

*We never update the weight of NYT

** This exchange will preserve the node number invariant fixed code for a
output = 000
fixed code
CSEP 590 - Lecture 2 - Autumn 2007 11 CSEP 590 - Lecture 2 - Autumn 2007 12

2
 Example Example
• aabcdad • aabcdad
1 21 1 21
0 1 0 1

0 19 1 20 0 19 1 20
NYT a NYT a

output = 000 output = 0001

CSEP 590 - Lecture 2 - Autumn 2007 13 CSEP 590 - Lecture 2 - Autumn 2007 14

Example Example
• aabcdad • aabcdad
1 21 2 21
0 1 0 1

0 19 2 20 0 19 2 20
NYT a NYT a

NYT
fixed code for b

output = 0001 output = 00010001

CSEP 590 - Lecture 2 - Autumn 2007 15 CSEP 590 - Lecture 2 - Autumn 2007 16

Example Example
• aabcdad • aabcdad
2 21 2 21
0 1 0 1

0 19 2 20 0 19 2 20
0 1 a 0 1 a
0 17 0
a 18 0 17 1
a 18
NYT b NYT b

output = 00010001 output = 00010001

CSEP 590 - Lecture 2 - Autumn 2007 17 CSEP 590 - Lecture 2 - Autumn 2007 18

3
 Example Example
• aabcdad • aabcdad
2 21 3 21
0 1 0 1

1 19 2 20 1 19 2 20
0 1 a 0 1 a
0 17 1
a 18 0 17 1
a 18
NYT b NYT b
fixed code for c
NYT
output = 00010001 output = 0001000100010

CSEP 590 - Lecture 2 - Autumn 2007 19 CSEP 590 - Lecture 2 - Autumn 2007 20

Example Example
• aabcdad 3 21 • aabcdad 3 21
0 1 0 1

1 19 2 20 1 19 2 20
0 1 a 0 1 a

0 17 a
1 18 0 17 a
1 18
0 1 b 0 1 b
0 15 0a 16 0 15 1a 16
NYT c NYT c

output = 0001000100010 output = 0001000100010

CSEP 590 - Lecture 2 - Autumn 2007 21 CSEP 590 - Lecture 2 - Autumn 2007 22

Example Example
• aabcdad 3 21 • aabcdad 3 21
0 1 0 1

1 19 2 20 2 19 2 20
0 1 a 0 1 a

1 17 a
1 18 1 17 a
1 18
0 1 b 0 1 b
0 15 a1 16 0 15 a1 16
NYT c NYT c

output = 0001000100010 output = 0001000100010

CSEP 590 - Lecture 2 - Autumn 2007 23 CSEP 590 - Lecture 2 - Autumn 2007 24

4
 Example Example
4 21
• aabcdad 4 21 • aabcdad 0 1
0 1
2 19 2 20
2 19 2 20 0 1 a
0 1 a
1 17 a
1 18
1 17 a
1 18 0 1 b
0 1 b
0 15 a1 16
0 15 a1 16 0 1 c
NYT c fixed code for d 0 13 0
a 14
NYT
NYT d
output = 0001000100010000011
output = 0001000100010000011
CSEP 590 - Lecture 2 - Autumn 2007 25 CSEP 590 - Lecture 2 - Autumn 2007 26

Example Example
4 21 4 21
• aabcdad 0 1 • aabcdad 0 1

2 19 2 20 2 19 2 20
0 1 a 0 1 a

1 17 a
1 18 1 17 a
1 18 exchange!
0 1 b 0 1 b
0 15 a1 16 1 15 a1 16
0 1 c 0 1 c
0 13 1
a 14 0 13 1
a 14
NYT d NYT d

output = 0001000100010000011 output = 0001000100010000011

CSEP 590 - Lecture 2 - Autumn 2007 27 CSEP 590 - Lecture 2 - Autumn 2007 28

Example Example
4 21 4 21
• aabcdad 0 1 • aabcdad 0 1

2 19 2 20 2 19 2 20
0 1 a 0 1 a

1 17 1 18 1 17 2 18 exchange!
b 0 1 b 0 1

1 15 1a 16 1 15 1a 16
0 1 c 0 1 c
0 13 1
a 14 0 13 1
a 14
NYT d NYT d

output = 0001000100010000011 output = 0001000100010000011

CSEP 590 - Lecture 2 - Autumn 2007 29 CSEP 590 - Lecture 2 - Autumn 2007 30

5
 Example Example
4 21 4 21
• aabcdad 0 1 • aabcdad 0 1

2 19 2 20 2 19 3 20
a 0 1 a 0 1

1 17 2 18 1 17 2 18
b 0 1 b 0 1

1 15 1
a 16 1 15 a1 16
0 1 c 0 1 c
0 13 1
a 14 0 13 1
a 14
NYT d NYT d

output = 0001000100010000011 output = 0001000100010000011

CSEP 590 - Lecture 2 - Autumn 2007 31 CSEP 590 - Lecture 2 - Autumn 2007 32

Example Example
5 21 Note: the first a is coded 5 21
• aabcdad 0 1 as 000, the second as 1, • aabcdad 0 1
and the third as 0
2 19 3 20 3 19 3 20
a 0 1 a 0 1

1 17 2 18 1 17 2 18
b 0 1 b 0 1

1 15 1
a 16 1 15 a1 16
0 1 c 0 1 c
0 13 1
a 14 0 13 1
a 14
NYT d NYT d

output = 00010001000100000110 output = 00010001000100000110

CSEP 590 - Lecture 2 - Autumn 2007 33 CSEP 590 - Lecture 2 - Autumn 2007 34

Example Example
6 21 6 21
• aabcdad 0 1 • aabcdad 0 1

3 19 3 20 3 19 3 20
a 0 1
exchange! a 0 1

1 17 2 18 1 17 2 18
b 0 1 d 0 1

1 15 1a 16 1 15 1a 16
0 1 c 0 1 c
0 13 1
a 14 0 13 1
a 14
NYT d NYT b

output = 000100010001000001101101 output = 000100010001000001101101

CSEP 590 - Lecture 2 - Autumn 2007 35 CSEP 590 - Lecture 2 - Autumn 2007 36

6
 Example Example
6 21 6 21
• aabcdad 0 1 • aabcdad 0 1

3 19 3 20 3 19 4 20
a 0 1 a 0 1

2 17 2 18 2 17 2 18
d 0 1 d 0 1

1 15 1
a 16 1 15 a1 16
0 1 c 0 1 c
0 13 1
a 14 0 13 1
a 14
NYT b NYT b

output = 000100010001000001101101 output = 000100010001000001101101

CSEP 590 - Lecture 2 - Autumn 2007 37 CSEP 590 - Lecture 2 - Autumn 2007 38

Example Data Structure for Adaptive Huffman

7 21 7 1. Fixed code table
• aabcdad 0 1 a 0 1 2. Binary tree with
b parent pointers
3 19 4 20 3 4 3. Table of pointers
a 0 1 c
a 0 1 nodes into tree
d 4. Doubly linked list
2 17 2 18 . 2 2 to rank the nodes
d 0 1 0 1
. d
1 15 1
a 16 . 1 1
a
0 1 c j 0 1 c
0 13 1
a 14 0 a
1
NYT b NYT b
NYT
output = 000100010001000001101101
CSEP 590 - Lecture 2 - Autumn 2007 39 CSEP 590 - Lecture 2 - Autumn 2007 40

In Class Exercise Huffman Summary

• Decode using adaptive Huffman coding • Statistical compression algorithm

assuming the following fixed code • Prefix code
0 1 • Fixed-to-variable length code
• Optimization to create a best code
0 1 0 1
• Symbol merging
0 1 0 1 0 1 0 1 • Context
a b c d e f g h • Adaptive coding
• Decoder and encoder behave almost the same
• 00110000 • Need for data structures and algorithms

CSEP 590 - Lecture 2 - Autumn 2007 41 CSEP 590 - Lecture 2 - Autumn 2007 42