12. Video Compression: Video compression is based on two principles. The first is the spatial redundancy that exists in each frame, The second is the fact that most of the time, a video frame is very similar to its immediate neighbors, ‘This is called temporal redundancy. A typical technique for video compression should therefore start by encoding the first frame using a still image compression method. It should then encode each successive frame by identifying the differences between the frame and its predecessor, and encoding these differences. If a frame is very different from its predecessor (as happens with the first frame of a shot), it should be coded independently of any other frame, In the video compression literature, a frame that is coded using its predecessor is called inter frame (or just inter), while a frame that is coded independently is called intra frame (or just intra), Video compression is normally lossy. Encoding a frame F; in terms of its predecessor Fi1 introduces some distortions. As a result, encoding the next frame Fi: in terms of (the already distorted) F; increases the distortion. Even in lossless video compression, a frame may lose some bits. This may happen during transmission or after a long shelf stay. Ifa frame F; has lost some bits, then all the frames following it, up to the next intra frame, are decoded improperly, perhaps even leading to accumulated errors. This is why intra frames should be used from time to time inside a sequence, not just at its beginning. An intra frame is labeled /, and an inter frame is labeled P (for predictive) With this in mind itis easy to imagine a situation where the encoder encodes frame 2 based on both frames 1 and 3, and writes the frames on the compressed stream in the order 1, 3, 2. The decoder reads them in this order, decodes frames 1 and 3 in parallel, outputs frame 1, then decodes frame 2 based on frames 1 and 3. Naturally, the frames should be clearly tagged (or time stamped). ‘A frame that is encoded based on both past and future frames is labeled B (for bidirectional). Predicting a frame based on its successor makes sense in cases where the movement of an object in the picture gradually uncovers a background area, Such an area may be only partly known in the current frame but may be better known in the next frame, Thus, the next frame is a natural candidate for predicting this area in the current frame. ‘The idea of a B frame is so useful that most frames in a compressed video presentation may be of this type. We therefore end up with a sequence of compressed frames of the three types 1, P, and B. An I frame is decoded independently of any other frame. A P frame is decoded using the preceding I or P frame, A B frame is decoded using the preceding and following / or P frames, Figure ‘12a shows a sequence of such frames in the order in which they are generated by the encoder (andinput by the decoder). Figure 12b shows the same sequence in the order in which the frames are output by the decoder and displayed. The frame labeled 2 should be displayed after frame 5, so ‘each frame should have two time stamps, its coding time and its display time, Be re ee re ee 1°23 4 5 6 7 8 9 WHR a) Time BoP BOB BOP BOB BOP 1 u (a) Coding Order. (b) Display Order. (b) Figure 12: We start with a few intuitive video compression methods. ‘Subsampling: The encoder selects every other frame and writes it on the compressed stream. This, yields a compression factor of 2, The decoder inputs a frame and duplicates it to create two frames. Differencing: A frame is compared to its predecessor. If the difference between them is small (just a few pixels), the encoder encodes the pixels that are different by writing three numbers on the compressed stream for each pixel: its image coordinates, and the difference between the values of the pixel in the two frames. If the difference between the frames is large, the current frame is written on the output in raw format, A lossy version of differencing looks at the amount of change in a pixel. If the difference between the intensities of a pixel in the preceding frame and in thecurrent frame is smaller than a certain (user controlled) threshold, the pixel is not considered different, Block Differencing: This is a further improvement of differencing. The image is divided into blocks of pixels, and each block B in the current frame is compared with the corresponding block P in the preceding frame, If the blocks differ by more than a certain amount, then B is compressed by writing its image coordinates, followed by the values of all its pixels (expressed as differences) on the compressed stream. The advantage is that the block coordinates are small numbers (smaller than a pixel’s coordinates), and these coordinates have to be written just once for the entire block On the downside, the values of all the pixels in the block, even those that haven't changed, have to be written on the output, However, since these values are expressed as differences, they are small numbers. Consequently, this method is sensitive to the block size. 12.4 Motion Compensation ‘The difference between consecutive frames is small because it is the result of moving the scene, the camera, or both between frames. This feature can therefore be exploited to achieve better compression. If the encoder discovers that a part P of the preceding frame has been rigidly moved to a different location in the current frame, then P can be compressed by writing the following three items on the compressed stream: its previous location, its current location, and information identifying the boundaries of P. In principle, such a part can have any shape. In practice, we are limited to equalize blocks {normally square but can also be rectangular). The encoder scans the current frame block by block. For each block B it searches the preceding frame for an identical block C (if compression is to be lossless) or for a similar one (if it can be lossy). Finding such a block, the encoder writes the difference between its past and present locations on the output. This difference is of the form (Cx - Bu Gy - By) = (Ax, Ay), s0 it is called a motion vector. Figure 13 a,b shows a simple example where the sun and trees are moved rigidly to the right (because of camera movement) while the child moves a different distance to the let (this is scene movement) Motion compensation is effective if objects are just translated, not scaled or rotated. Drastic changes in illumination from frame to frame also reduce the effectiveness of this method. In general, motion compensation is lossy. The following paragraphs discuss the main aspects of ‘motion compensation in detail. Figure 14 shows the flow of information through the motion compensation process.le RB (a) (b) Figure 13: Motion Compensation. Figure 14: Flow of information in motion compensation processFrame Segmentation: The current frame is divided into equal-size nonoverlapping blocks. The blocks may be squares or rectangles. The latter choice assumes that motion in video is mostly horizontal, so horizontal blocks reduce the number of motion vectors without degrading the compression ratio, The block size is important, because large blocks reduce the chance of finding a match, and small blocks result in many motion vectors. In practice, block sizes that are integer powers of 2, such as 8 or 16, are used, since this simplifies the software. Search Threshold: Each block B in the current frame is first compared to its counterpart C in the preceding frame. If they are identical, or if the difference between them is less than a preset ‘threshold, the encoder assumes that the block hasn't been moved, Block Search: This is a time-consuming process, and so has to be carefully designed, If B is the current block in the current frame, then the previous frame has to be searched for a block identical to or very close to B. The search is normally restricted to a small area (called the search area) around B, defined by the maximum displacement parameters dx and dy. These parameters specify the maximum horizontal and vertical distances, in pixels, between B and any matching block in the previous frame, If B is a square with side b, the search area will contain (b + 2dx)(b + 2dy) pixels (Figure 15) and will consist of (2dx+1)(2dy +1) distinct, overlapping bxb squares. The number of candidate blocks in this area is therefore proportional to dx « dy. Figure 15: Search Area. Distortion Measure: This is the most sensitive part of the encoder. The distortion measure selects the best match for block B. It has to be simple and fast, but also reliable. The mean absolute difference (or mean absolute error) calculates the average of the absolute differences between a pixel By in B and its counterpart Cy in a candidate block C:bo 1 DD Bu Sul a This involves b? subtractions and absolute value operations, b? additions, and one division. This, measure is calculated for each of the (2dx+1)(2dy +1) distinct, overlapping bxb candidate blocks, and the smallest distortion (say, for block ,) is examined. If it is smaller than the search threshold, then Cj is selected as the match for B. Otherwise, there is no match for B, and B has to be encoded without motion compensation, Suboptimal Search Methods: These methods search some, instead of all, the candidate blocks in the (b+2dx)(b2dy) area. They speed up the search for a matching block, at the expense of compression efficiency. Motion Vector Correction: Once a block C has been selected as the best match for B, a motion vector is computed as the difference between the upper-left corner of C and the upper-left corner of B, Regardless of how the matching was determined, the motion vector may be wrong because of noise, local minima in the frame, or because the matching algorithm is not perfect. It is possible to apply smoothing techniques to the motion vectors after they have been calculated, in an attempt to improve the matching, Spatial correlations in the image suggest that the motion vectors should also be correlated. If certain vectors are found to violate this, they can be corrected, This step is costly and may even backfire. A video presentation may involve slow, smooth motion of most objects, but also swift, jerky motion of some small objects. Correcting motion vectors may interfere with the motion vectors of such objects and cause distortions in the compressed frames. Coding Motion Vectors: A large part of the current frame (perhaps close to half of it) may be converted to motion vectors, which is why the way these vectors are encoded is crucial; it must also be lossless. Two properties of motion vectors help in encoding them: (1) They are correlated and (2) their distribution is nonuniform. As we scan the frame block by block, adjacent blocks normally have motion vectors that don’t differ by much; they are correlated, The vectors also don’t point in all directions. There are often one or two preferred directions in which all or most motion vectors point; the vectors are nonuniformly distributed. No single method has proved ideal for encoding the motion vectors. Arithmetic coding, adaptive Huffman coding, and various prefix codes have been tried, and all seem to perform well. Here are two different methods that may perform better 1. Predict a motion vector based on its predecessors in the same row and its predecessors in the same column of the current frame. Calculate the difference between the prediction and the actualvector, and Huffman encode it. This algorithm is important. It is used in MPEG and other compression methods. 2. Group the motion vectors in blocks. Ifall the vectors in a block are identical, the block is encoded by encoding this vector. Other blocks are encoded as in 1 above. Each encoded block starts with a code identifying its type. Coding the Prediction Error: Motion compensation is lossy, since a block B is normally matched to a somewhat different block C. Compression can be improved by coding the difference between the current uncompressed and compressed frames on a block by block basis and only for blocks that differ much, This is usually done by transform coding, The difference is written on the output, following each frame, and is used by the decoder to improve the frame after it has been decoded, 14 MPEG The name MPEG is an acronym for Moving Pictures Experts Group. MPEG is a method for video compression, which involves the compression of digital images and sound, as well as synchronization of the two. There currently are several MPEG standards, MPEG-1 is intended for intermediate data rates, on the order of 1.5 Mbit/sec MPEG-2 is intended for high data rates of at least 10 Mbit/ sec. MPEG-3 was intended for HDTV compression but was found to be redundant and was merged with MPEG-2, MPEG-4 is intended for very low data rates of less than 64 Kbit/sec. A third international body, the ITU-T, has been involved in the design of both MPEG-2 and MPEG-4 ‘The formal name of MPEG-1 is the international standard for moving picture video compression, 1S11172-2. Like other standards developed by the ITU and 1S0, the document describing MPEG-1 has normative and informative sections. A normative section is part of the standard specification. It is intended for implementers, is written in a precise language, and should be strictly adhered to when implementing the standard on actual computer platforms. An informative section, on the other hand, illustrates concepts discussed elsewhere, explains the reasons that led to certain choices and decisions, and contains background material. An example of a normative section is the various tables of variable codes used in MPEG. An example of an informative section is the algorithm used by MPEG to estimate motion and match blocks, MPEG does not require any particular algorithm, and an MPEG encoder can use any method to match blocks. ‘The importance of a widely accepted standard for video compression is apparent from the fact that many manufacturers (of computer games, DVD movies, digital television, and digital recorders, among others) implemented MPEG-1 and started using it even before it was finallyapproved by the MPEG committee. This also was one reason why MPEG-1 had to be frozen at an high data early stage and MPEG-2 had to be developed to accommodate video applications rates. ‘To understand the meaning of the words “intermediate data rate" we consider a typical example of video with a resolution of 360%288, a depth of 24 bits per pixel, and a refresh rate of 24 frames per second, The image part of this video requires 360«288*24x24 = 59,719,680 bits/s. For the audio part, we assume two sound tracks (stereo sound), each sampled at 44 kHz with 16-bit samples. The data rate is 2x 44,000x16 = 1,408,000 bits/s. The total is about 61.1 Mbit/s and this is supposed to be compressed by MPEG-1 to an intermediate data rate of about 1.5 Mbit/s (the size of the sound track alone), a compression factor of more than 40! Another aspect is the decoding speed. ‘An MPEG-compressed movie may end up being stored on a CD-ROM or DVD and has to be decoded and played in real time. MPEG uses its own vocabulary. An entire movie is considered a video sequence. It consists of pictures, each having three components, one luminance (¥ } and two chrominance (Cb and Cr). ‘The luminance component contains the black-and white picture, and the chrominance components provide the color hue and saturation, Each component is a rectangular array of samples, and each row of the array is called a raster line. A pel is the set of three samples. The eye is sensitive to small spatial variations of luminance, but is less sensitive to similar changes in chrominance. As a result, MPEG-1 samples the chrominance components at half the resolution of the luminance component. ‘The term intra is used, but inter and nonintra are used interchangeably. ‘The input to an MPEG encoder is called the source data, and the output of an MPEG decoder is the reconstructed data, The source data is organized in packs (Figure 16b), where each pack starts with a start code (32 bits) followed by a header, ends with a 32-bit end code, and contains a number of packets in between. A packet contains compressed data, either audio or video. The size of a packet is determined by the MPEG encoder according to the requirements of the storage or transmission medium, which is why a packet is not necessarily a complete video picture. It can be any part of a video picture or any part of the audio. ‘The MPEG decoder has three main parts, called layers, to decode the audio, the video, and the system data. The system layer reads and interprets the various codes and headers in the source data, and routes the packets to either the audio or the video layers (Figure 16a) to be buffered and later decoded, Each of these two layers consists of several decoders that work simultaneously.Audio >| decoder Audio layer Audio j-+| decoder System decoder || Clock Source data }— t-+| Video decoder Video layer |-[ Video |__| decoder (a) start code pack header] packed] packet packetfend code (b) Figure 16: (a) MPEG Decoder Organization. (b) Source Format. 14.1 MPEG 1 MPEG uses 1, P, and B pictures. They are arranged in groups, where a group can be open or closed, The pictures are arranged in a certain order, called the coding order, but (after being decoded) they are output and displayed in a different order, called the display order. In a closed group, P and B pictures are decoded only from other pictures in the group. In an open group, they can be decoded from pictures outside the group. Different regions of a B picture may use different