CDI15

4. Arithmetic coding / decoding

303 SXD

4.1. Segment coding

4.2. Binary representation of segments
4.3. Arithmetic coding
4.4. Arithmetic decoding
4.5. A note on the bit encoding of a segment
4.6. Arithmetic coding with conditional probabilities
4.7. Incremental arithmetic coding based on segment rescaling
 2

4.1. Segment coding

Consider a source alphabet 𝒜𝒜 = {𝑎𝑎1 , … , 𝑎𝑎𝑛𝑛 } , with probabilities
{𝑝𝑝1 , … , 𝑝𝑝𝑛𝑛 }. Let 𝑀𝑀 = 𝑎𝑎𝑗𝑗1 ⋯ 𝑎𝑎𝑗𝑗𝑁𝑁 be a source message of length 𝑁𝑁. In seg-
ment coding (the first step in what will be arithmetic coding) a segment (or
interval)
𝑆𝑆𝑀𝑀 = [𝑙𝑙𝑀𝑀 , ℎ𝑀𝑀 ) ⊂ [0,1)
is assigned to any such 𝑀𝑀. Moreover, this assignment has the following
properties:
1. ℎ𝑀𝑀 − 𝑙𝑙𝑀𝑀 = 𝑃𝑃(𝑀𝑀)
2. 𝑆𝑆𝑀𝑀 ∩ 𝑆𝑆𝑀𝑀′ = ∅ for any pair two distinct messages 𝑀𝑀, 𝑀𝑀′ of length 𝑁𝑁.
3. ⋃𝑀𝑀 𝑆𝑆𝑀𝑀 = [0,1), the union is with respect to all messages of length 𝑁𝑁.
4. If 𝑀𝑀 is a prefix of 𝑀𝑀′, then 𝑆𝑆𝑀𝑀′ ⊂ 𝑆𝑆𝑀𝑀 .
So the length of 𝑆𝑆𝑀𝑀 is 𝑃𝑃(𝑀𝑀); the different 𝑆𝑆𝑀𝑀 , for fixed 𝑁𝑁, form a partition
of [0,1), and the partition for 𝑁𝑁 ′ > 𝑁𝑁 is a refinement of that for 𝑁𝑁.
 3

The case 𝑁𝑁 = 1
For messages of length 1 (any one of the symbols 𝑎𝑎𝑗𝑗 of 𝒜𝒜), we set
𝑆𝑆𝑎𝑎𝑗𝑗 = �𝑝𝑝1 + ⋯ + 𝑝𝑝𝑗𝑗−1 , 𝑝𝑝1 + ⋯ + 𝑝𝑝𝑗𝑗−1 + 𝑝𝑝𝑗𝑗 � = [𝜎𝜎𝑗𝑗−1 , 𝜎𝜎𝑗𝑗 ),
where we define 𝜎𝜎𝑗𝑗 = 𝑝𝑝1 + ⋯ + 𝑝𝑝𝑗𝑗−1 + 𝑝𝑝𝑗𝑗 , 𝑗𝑗 = 1, … , 𝑛𝑛 (we say that
𝜎𝜎1 , … , 𝜎𝜎𝑛𝑛 ) is the cumulative probability distribution. Thus we have
𝑙𝑙𝑎𝑎𝑗𝑗 = 𝜎𝜎𝑗𝑗−1 , ℎ𝑎𝑎𝑗𝑗 = 𝜎𝜎𝑗𝑗 , ℎ𝑎𝑎𝑗𝑗 − 𝑙𝑙𝑎𝑎𝑗𝑗 = 𝑝𝑝𝑗𝑗
From the definitions it follows that 0 < 𝑝𝑝1 = 𝜎𝜎1 < ⋯ < 𝜎𝜎𝑛𝑛 = 1 and hence
that the segments 𝑆𝑆𝑎𝑎𝑗𝑗 cover [0,1) and are pairwise disjoint. See the left
column of the illustration on next page for an example in which 𝑛𝑛 = 3.
Example. Before considering the general construction, we first describe it
in a simple example. We will see how to obtain the segment 𝑆𝑆𝑀𝑀 for the
message 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 produced by the source {′𝑎𝑎′ : 0.2, ′𝑏𝑏 ′ : 0.5, ′𝑐𝑐 ′ : 0.3}.
 4

By our stipulation of the case 𝑁𝑁 =

1.0 0.7 0.3 0.27
c c c c 1, we know that
0.7 0.55 0.27 0.255 𝑆𝑆𝑏𝑏 = [0.2, 0.7).
Let us proceed now to assign an in-
b b b b
terval to 𝑏𝑏𝑏𝑏 (second colum of the
0.2 0.3 0.22 0.23 illustration), then to 𝑏𝑏𝑏𝑏𝑏𝑏 (third col-
a
a
0.0
a a
0.2 0.22 umn) and finally to 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 (fourth
column). The interval 𝑆𝑆𝑏𝑏𝑏𝑏 is ob-
Example tained by subdividing
𝒜𝒜 = {𝑎𝑎, 𝑏𝑏, 𝑐𝑐}, 𝑝𝑝 = (0.2, 0.5, 0.3), 𝑀𝑀 = 𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏.
𝑆𝑆𝑏𝑏 = [𝑙𝑙𝑏𝑏 = 0.2, ℎ𝑏𝑏 = 0.7)
into segments of relative length 𝑝𝑝(𝑎𝑎) = 0.2, 𝑝𝑝(𝑏𝑏) = 0.5 and 𝑝𝑝(𝑐𝑐 ) = 0.3
and choosing the 𝑎𝑎-segment as 𝑆𝑆𝑏𝑏𝑏𝑏 . So the division points are
𝑙𝑙𝑏𝑏𝑏𝑏 = 0.2, ℎ𝑏𝑏𝑏𝑏 = 𝑙𝑙𝑏𝑏𝑏𝑏 = 0.2 + 0.5 × 𝜎𝜎(𝑎𝑎) = 0.30,
ℎ𝑏𝑏𝑏𝑏 = 𝑙𝑙𝑏𝑏𝑏𝑏 = 0.2 + 0.5 × 𝜎𝜎(𝑏𝑏) = 0.55, ℎ𝑏𝑏𝑏𝑏 = 0.2 + 0.5 × 𝜎𝜎(𝑐𝑐 ) = 0.70 , so
𝑆𝑆𝑏𝑏𝑏𝑏 = [0.2,0.3), 𝑆𝑆𝑏𝑏𝑏𝑏 = [0.30, 0.55), 𝑆𝑆𝑏𝑏𝑏𝑏 = [0.55,0.70).
 5

Now the interval 𝑆𝑆𝑏𝑏𝑏𝑏𝑏𝑏 is obtained in a similar way: subdivide 𝑆𝑆𝑏𝑏𝑏𝑏 into in-
tervals that are proportional to 𝑝𝑝(𝑎𝑎), 𝑝𝑝(𝑏𝑏) and 𝑝𝑝(𝑐𝑐 ) and choose as 𝑆𝑆𝑏𝑏𝑏𝑏𝑏𝑏
the segment corresponding to 𝑏𝑏. Actually we have
ℎ𝑏𝑏𝑏𝑏𝑏𝑏 = 𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏 = 0.2 + 0.1 × 𝜎𝜎(𝑎𝑎) = 0.22,
ℎ𝑏𝑏𝑏𝑏𝑏𝑏 = 𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏 = 0.2 + 0.1 × 𝜎𝜎(𝑏𝑏) = 0.27,
ℎ𝑏𝑏𝑏𝑏𝑏𝑏 = 0.2 + 0.1 × 𝜎𝜎(𝑐𝑐 ) = 0.30,
and hence
𝑆𝑆𝑏𝑏𝑏𝑏𝑏𝑏 = [𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏 , ℎ𝑏𝑏𝑏𝑏𝑏𝑏 ) = [0.22,0.27).
Finally we get, following the same procedure with 𝑆𝑆𝑏𝑏𝑏𝑏𝑏𝑏 ,
𝑆𝑆𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 = �0.22 + 0.05 × 𝜎𝜎(𝑏𝑏), 0.22 + 0.05 × 𝜎𝜎(𝑐𝑐 )� = [0.255,0.270).
 6

The general case

Suppose that we already know the interval 𝑆𝑆𝑀𝑀 = [𝑙𝑙𝑀𝑀 , ℎ𝑀𝑀 ) of a message of
length 𝑁𝑁 and that ℎ𝑀𝑀 − 𝑙𝑙𝑀𝑀 = 𝑃𝑃(𝑀𝑀). Then the interval of 𝑀𝑀′ = 𝑀𝑀𝑎𝑎𝑗𝑗 is de-
fined as follows:
𝑆𝑆𝑀𝑀′ = [𝑙𝑙𝑀𝑀′ , ℎ𝑀𝑀′ ) = �𝑙𝑙𝑀𝑀 + 𝑃𝑃(𝑀𝑀) × 𝜎𝜎𝑗𝑗−1 , 𝑙𝑙𝑀𝑀 + 𝑃𝑃(𝑀𝑀) × 𝜎𝜎𝑗𝑗 �.
We note that
ℎ𝑀𝑀′ − 𝑙𝑙𝑀𝑀′ = 𝑃𝑃(𝑀𝑀) × �𝜎𝜎𝑗𝑗 − 𝜎𝜎𝑗𝑗−1 � = 𝑃𝑃(𝑀𝑀)𝑝𝑝𝑗𝑗 = 𝑃𝑃(𝑀𝑀′ ).
Remark. The conditions 1-4 at the beginning are a direct consequence of
the definitions.
 7

4.2. Binary representation of segments

Binary representation of numbers in the unit segment

0.270
0 0.255 1
0 1
0 1 0 1
0 1 0 1 0 1 0 1
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
0 101 01 01 01 01 0 1 010 1 010 1 010 1 01 01 01 01 01 0 1 01 0 10101 0 1 01 0 1 01 01 0 1 0 10 1 01

0.010001 0.011001
0.010000
The binary unit segment

Examples. 0.5→0.1, 0.25→0.01, 0.75→0.11, 0.125→0.001;

0.255→0.01000, 0.270→0.01001;
0.011001→1/4+1/8+1/64=25/64=0.390625.
 8

4.3. Arithmetic coding

The basic idea of arithmetic coding is to select an element in the seg-
ment 𝑺𝑺𝑴𝑴 = [𝒍𝒍𝑴𝑴 , 𝒉𝒉𝑴𝑴 ) that requires the minimal number of bits. Then we
encode 𝑴𝑴 using the binary word formed with those bits.
This can be accomplished as follows. Suppose that the first bit that is dif-
ferent in the binary representations of 𝒍𝒍𝑴𝑴 and 𝒉𝒉𝑴𝑴 is the 𝒓𝒓-th, so that we
will have
𝑙𝑙𝑀𝑀 = 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 0 ···, ℎ𝑀𝑀 = 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1 ···
If 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1 < ℎ𝑀𝑀 , then the number 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1, or the word
𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1, satisfies the requirements, for 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1 ∈ 𝑆𝑆𝑀𝑀 and
any other number in the interval 𝑆𝑆𝑀𝑀 will require more bits. Thus we en-
code 𝑀𝑀 as 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1. If 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 1 = ℎ𝑀𝑀 , then the shortest bi-
nary word will be 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 if 𝑙𝑙𝑀𝑀 = 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 0. Otherwise 𝑙𝑙𝑀𝑀 =
0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 0𝑥𝑥, with 𝑥𝑥 ≠ 0. If 𝑥𝑥 = 0 ⋯, we can take 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 01,
else if 𝑥𝑥 = 1 ⋯ 10 (𝑠𝑠 ones), we may take 0. 𝑏𝑏1 𝑏𝑏2 ··· 𝑏𝑏𝑟𝑟−1 01 ⋯ 11 (𝑠𝑠 + 1
ones).
 9

Remark. For the decoding, it will be convenient to include in the encoding

the length of the original message. Thus it can be represented as a pair (𝐿𝐿,
𝑐𝑐), where 𝐿𝐿 is the length of the of the message and 𝑐𝑐 is the binary string
given by the arithmetic encoding.
For example, if S = [‘a’ : 0.2, ‘b’ : 0.5: ‘c’ : 0.3] and M=”babc”, then the arithmetic
encoding is [4,”010001”], for the binimals of 0.255 and 0.270 are, respectively:
01000001010 ⋯
01000101000 ⋯
 10

4.4. Arithmetic decoding and effective computations

Suppose we have the arithmetically encoded message
𝑪𝑪 = [𝑵𝑵, 𝒙𝒙] (𝑵𝑵 the number of symbols, 𝒙𝒙 the binary string encoding).
Let 𝑷𝑷 be a representation of the source as a list of pairs (symbol, probabil-
ity). Let 𝑨𝑨 denote the list of pairs (symbol, cumulative-probability). The
decoding algorithm is implemented in the function AD(C,P) which will be
studied, together with the encoding function AE(M,P).

Example. If P=[(’a’, 0.25), (’b’, 0.4}, (’c’, 0.15), (’d’, 0.1), (’e’, 0.1)] and
M= “badbbdcbabea”,
then the arithmetic encoding AE(M,P) is
C=[12,”0101010110111011011100101”]
and AD(C,P) is “badbbdcbabea”.
Remark. This description of arithmetic encoding and decoding would work if
floats had unlimited precision, but in pyzo it only works for short messages (of
the order of 54 encoded bits, due to the fact of that floats have 16 significant
digits, which amounts to 16*log(10)/log(2)≃ 53.15 bits.
 11

4.5. A note on the bit encoding of a segment

𝑛𝑛 𝑛𝑛+1
The question is how to select a binary interval [ 𝑘𝑘 , 𝑘𝑘 ) of highest length
2 2
1/2𝑘𝑘 (least 𝑘𝑘) contained in a segment [𝑎𝑎, 𝑏𝑏) ⊆ [0,1).
1
In particular, ≤ 𝑏𝑏 − 𝑎𝑎, hence −𝑘𝑘 ≤ log 2 (𝑏𝑏 − 𝑎𝑎), or 𝑘𝑘 ≥ −log 2 (𝑏𝑏 − 𝑎𝑎).
2𝑘𝑘
It follows that the least possible 𝑘𝑘 is 𝑘𝑘 = �−log2 (𝑏𝑏 − 𝑎𝑎)�.
𝑘𝑘
Now we must have 𝑎𝑎 ≤ 𝑛𝑛⁄2 , or 2𝑘𝑘 𝑎𝑎 ≤ 𝑛𝑛, and the least possible integer 𝑛𝑛
is 𝑛𝑛𝑘𝑘 = �2𝑘𝑘 𝑎𝑎�.
𝑛𝑛𝑘𝑘 𝑛𝑛𝑘𝑘 +1
With these choices, if the binary segment 𝐼𝐼𝑘𝑘 = [ , 𝑘𝑘 ) is contained in
2𝑘𝑘 2
𝑛𝑛 +1
[𝑎𝑎, 𝑏𝑏), which is equivalent to say that 𝑘𝑘 𝑘𝑘 ≤ 𝑏𝑏, then we are done: we can
2
take the binary representation of 𝑛𝑛𝑘𝑘 as the bit encoding of [𝑎𝑎, 𝑏𝑏).
𝑛𝑛𝑘𝑘 +1
If > 𝑏𝑏, then the interval 𝐼𝐼𝑘𝑘+1 is contained in [𝑎𝑎, 𝑏𝑏), for
2𝑘𝑘
𝑛𝑛𝑘𝑘+1 1
𝑛𝑛𝑘𝑘+1 − 1 < 2𝑘𝑘+1 𝑎𝑎, or < 𝑎𝑎 + ,
2𝑘𝑘+1 2𝑘𝑘+1
 12

and therefore
𝑛𝑛𝑘𝑘+1 +1 1 1 1
< 𝑎𝑎 + + = 𝑎𝑎 + 𝑘𝑘 ≤ 𝑎𝑎 + (𝑏𝑏 − 𝑎𝑎) = 𝑏𝑏.
2𝑘𝑘+1 2𝑘𝑘+1 2𝑘𝑘+1 2
So in this case we can take the binary representation of 𝑛𝑛𝑘𝑘+1 as the bit
encoding of [𝑎𝑎, 𝑏𝑏).
This can be easily programmed into an alternative bit encoder of intervals.
Note. If 𝑥𝑥 is the binary string representing 𝑛𝑛/2𝑘𝑘 , the function segment(x)
in cdi.py returns the endpoints of 𝐼𝐼𝑘𝑘 .
From the analysis above, it is easy to show that arithmetic coding becomes
practically optimal for long messages. Indeed, for messages 𝑀𝑀 of length 𝑁𝑁,
the average length is ℓ�𝑁𝑁 = ∑𝑀𝑀 𝑃𝑃(𝑀𝑀)ℓ(𝑀𝑀), where ℓ(𝑀𝑀) is the number of
bits of the arithmetic encoding of 𝑀𝑀. Since we just have seen that
1 1
ℓ(𝑀𝑀) ≤ �log 2 ( )� +1≤ log 2 ( ) +2
𝑃𝑃 𝑀𝑀 𝑃𝑃 𝑀𝑀

we get
𝐻𝐻𝑁𝑁 ≤ ℓ�𝑁𝑁 ≤ 𝐻𝐻𝑁𝑁 + 2
 13

where 𝐻𝐻𝑁𝑁 stands for the entropy of messages of length 𝑁𝑁 (the first ine-
quality by Shannon’s theorem). Now 𝐻𝐻𝑁𝑁 = 𝑁𝑁𝑁𝑁, where 𝐻𝐻 is the entropy of
the symbol source (an easy calculation using that 𝑃𝑃�𝑎𝑎𝑗𝑗1 ··· 𝑎𝑎𝑗𝑗𝑁𝑁 � =
∏𝑘𝑘 𝑝𝑝𝑗𝑗𝑘𝑘 )). Therefore
2
𝐻𝐻 ≤ ℓ� ≤ 𝐻𝐻 + ,
𝑁𝑁

where ℓ� is the average length per symbol, and this means that ℓ� ≈ 𝐻𝐻 for
large 𝑁𝑁.
Note that if sending the encoding of messages 𝑀𝑀 of length 𝑁𝑁 includes a
fixed overhead (like sending the number 𝑁𝑁 as part of the encoding), then
the conclusion still holds (the 2 above is replaced by some constant 𝐾𝐾𝑁𝑁
𝐾𝐾𝑁𝑁
such that → 0 as 𝑁𝑁 grows).
𝑁𝑁
 14

4.6. Arithmetic coding with conditional probabilities

When the probabilities are conditional (so they depend on the current
symbol and of the preceding symbols in the message), the (high) cumula-
tive probabilities for each symbol 𝑎𝑎𝑗𝑗 ∈ 𝒜𝒜 when we are scanning symbol
the symbol 𝑥𝑥𝑘𝑘 in the message 𝑥𝑥1 ⋯ 𝑥𝑥𝑘𝑘 ⋯ 𝑥𝑥𝑁𝑁 , are given by
𝜎𝜎(𝑎𝑎𝑘𝑘 |𝑥𝑥1 , … , 𝑥𝑥𝑘𝑘−1 ) = ∑𝑘𝑘𝑖𝑖=1 𝑃𝑃(𝑎𝑎𝑖𝑖 |𝑥𝑥1 , … , 𝑥𝑥𝑘𝑘−1 ).
For examples, see L1016-1.
 15

4.7. Incremental arithmetic coding based on segment rescaling

If the current segment [𝑙𝑙, ℎ) in AE is contained in [0,1/2), then we know
that next bit is 0. So we can send 0 to the decoder and rescale [𝑙𝑙, ℎ) to
[2𝑙𝑙, 2ℎ). This rescaling, which in binary is a shift to the left, forgets the in-
formation about that bit, but this is not a loss because it is already available
to the decoder. Similarly, we can send a 1 to the decoder if [𝑙𝑙, ℎ) is con-
tained in [1/2,1) and rescale [𝑙𝑙, ℎ) to [2𝑙𝑙 − 1,2ℎ − 1).
The decoding has to perform the same rescaling each time a bit is received.
The implementation of these rules is illustrated by the functions SAE and
SAD in cdi_SACD.py.
 16

4.8. Arithmetic coding with integers

We will work with blocks of 𝑚𝑚 bits (we may take 𝑚𝑚 = 16, giving a total of
216 = 65536 numbers). This can be done as follows:
Let 𝑁𝑁 be the length of the message, 𝑀𝑀.
If 𝑓𝑓[𝑥𝑥] is the frequency of symbol 𝑥𝑥, let 𝐹𝐹 [𝑥𝑥] = ∑𝑦𝑦≤𝑥𝑥 𝑓𝑓(𝑦𝑦).
Encoding algorithm (of a message 𝑀𝑀):
𝑙𝑙 = 0; ℎ = 𝑁𝑁
for 𝑥𝑥 in 𝑀𝑀:
𝑢𝑢 = ℎ − 𝑙𝑙 + 1
𝑢𝑢·𝐹𝐹 [𝑥𝑥]
ℎ = 𝑙𝑙 + � �
𝑁𝑁
𝑢𝑢·𝑓𝑓[𝑥𝑥]
𝑙𝑙 = 𝑙𝑙 + � � −1
𝑁𝑁

𝑢𝑢 = ℎ − 𝑙𝑙 + 1
 17

ℎ = 𝑙𝑙 + (𝑢𝑢 · 𝜎𝜎′[𝑥𝑥])/𝜎𝜎 ′ − 1
𝑙𝑙 = ℎ − (𝑢𝑢 · 𝐹𝐹 ′ [𝑥𝑥])/𝜎𝜎′)
Decoding algorithm
𝑥𝑥 = encoding value (string)
𝑙𝑙 = 0; ℎ = 𝜎𝜎 ′

𝑢𝑢 = ℎ − 𝑙𝑙 + 1
ℎ = 𝑙𝑙 + (𝑢𝑢 · 𝜎𝜎′[𝑥𝑥])/𝜎𝜎 ′ − 1
𝑙𝑙 = ℎ − (𝑢𝑢 · 𝐹𝐹 ′ [𝑥𝑥])/𝜎𝜎′)