EC8002-Multimedia Compression and Communication Department of ECE

5.1 Multimedia Transmission

A generic picture of multimedia transmission is shown in Fig 5.1. Node N1 is
transmitting multimedia information through a communication link L(N1, N2) to node N2.
Each node contains a buffer that acts as temporary storage for the multimedia objects either
retrieved from secondary storage (eg. Disk, tape) or received from a live source (e.g. video
camera). Multimedia information is formatted into packets to make it suitable for
transmission over a communication link. This information is then transmitted as a stream of
packets. The details of these packets depend on the communications protocols

used.
Figure 5.1 Transmission of multimedia streams on a networked multimedia system
Transmitted information is viewed as a stream of multimedia objects. Two object
streams are shown in Figure 5.1. Stream (A) comprises type A objects : A1, A2, ….Am, Am+1, ….
Stream (B) comprises type B objects B1, B2, ….Bn, Bn+1, ….
The two multimedia streams A and B shown in Fig. can be written as follows.
Stream (A) := A1, A2, ….Am, Am+1, …. := Ψ(Ap)
Stream (B) := B1, B2, ….Bn, Bn+1, …. := Ψ(Bq)
Where Ap refers to any object in the Stream (A), and Ψ(Ap) refers to a sequence of type A
objects.
Similarly, Bq refers to any object in the Stream (B), and Ψ(Bq) refers to a sequence of
type – B objects. Temporal relationships arise out of the various types of interactions and
possible relationships between the objects in the Ψ(Ap) and Ψ(Bq) streams. In representing
temporal activities, t is used to represent time instants, ∆t is used to represent the difference
between two time instants, and T is used to represent time intervals.
5.2 CLASSIFICATION OF TEMPORAL RELATIONSHIPS

A multimedia presentation consists of one or more multimedia documents. Each

document may contain one or more multimedia objects. Temporal relationships come into
play when these multimedia documents are treated as streams of multimedia objects.
Temporal relationships are important on standalone systems as well as on networked

St. Joseph’s College of Engineering 1

EC8002-Multimedia Compression and Communication Department of ECE

multimedia systems. On a networked multimedia system, continuous multimedia traffic is

transmitted as a stream of multimedia objects.
The temporal (timing) relationships between related objects is important in a
multimedia presentation. For example, a speaker’s lip movement and the words she utters
must match in a video clip. A multimedia presentation includes multiple media types with
well-defined temporal relationships between them. It is essential to maintain the required
temporal relationships within specified bounds to achieve a good-quality presentation. The
type of temporal relationship and accuracy to which it must be maintained depend on the
application and the type of media used in it.
Temporal relationship is classified based on three important aspects of temporal
relationships. They are

• Object stream inter-actions

• Media levity
• Media synchronization
Each of these aspects has three options, which are listed and briefly described in Table 5.1.
Table 5.1 Classification of Temporal Relationships

Object Stream Interactions

Intramedia : Temporal relationship between the objects of one stream.

Example: Animation without sound.
Intermedia : Temporal relationship between the objects of two parallel streams.
Example: Video clip with sound.
User Interaction: The temporal constraints in terms of response time.
Example: Entertainment system
Media Levity

Live: Live presentation demanding minimum end - to -end delay.

Example : Video conference
Stored : Stored presentations retrieved from secondary storage.
Example : Video on demand.
Mixed : Combination of live and stored presentations.
Example : Collaborative conference
Media Synchronization

Asynchronous: No well-defined timing relationships between objects.

Example : Text entry on keyboard.
Synchronous : Well-defined temporal relationship between the objects of different streams.
Example : Video with sound
Isochronous: Well-defined time relationship between the objects of the same stream.
Example: Sound

St. Joseph’s College of Engineering 2

EC8002-Multimedia Compression and Communication Department of ECE

5.2.1 Object Stream Interactions

It is based on the method by which multimedia object streams interact with one another.
This leads to three types of temporal relationships
(a) Intramedia temporal relationships
(b) Intermedia temporal relationships
(c) User interaction-based temporal relationship
(a) Intramedia Temporal Relationships
The temporal relationships that arise out of the interaction between the objects of a single
stream are called intramedia temporal relationships. For example, if an animation clip is
stored at the rate of 14 frames per second (fps), then this clip must be played back at 14 fps.
This is an intramedia temporal relationship constraint. This type of constraint is also called
intrastream synchronization or serial synchronization. In Figure 5.1, intra media temporal
relationships relate to the relationship between the objects belonging either to Stream (A) or
Stream (B).
(b) Intermedia Temporal Relationships
The temporal relationships that arise out of the interaction between the objects of two or
more streams are called Intermedia temporal relationships. A video clip is a good example of
a multimedia presentation requiring intermedia temporal relationships. A precise temporal
relationship must be maintained between the image frames and the digitized voice to
maintain lip synchronization. Such temporal constraints are also called interstream
synchronization or parallel synchronization. In Fig 5.1, intermedia temporal relationships
arise from the interaction between Ap and Bq objects.
(c) User Interaction-Based Temporal Relationships.
An interactive multimedia system adds another dimension to the temporal demands put on
the system. The interactivity relates to the interaction between the user and the system. The
time taken by the system to respond to a user interaction is called the response time. The
response time of a standalone or networked interactive multimedia system is one of the main
issues considered under this category. For example, in a good interactive multimedia system,
the response time should be consistent and as small as possible. This type of constraint is
also called event synchronization.

5.2.2 Media Levity

The temporal relationships required for Multimedia systems can also be classified based on
the levity of the media presentation. Based on their levity, three types of presentations are
considered.
(a) Live

St. Joseph’s College of Engineering 3

EC8002-Multimedia Compression and Communication Department of ECE

(b) Stored
(c) Mixed
a) Live Presentations
Presentations in which the information is captured, transmitted, and presented in real time
are called live presentations. A video conference is an example of a live presentation. Live
presentations demand minimum end-to-end delay.
b) Stored Presentations
A stored presentation is captured and stored on secondary storage systems such as
magnetic tape, disk, or CD-ROM. To display this information, it is retrieved from the
secondary storage system. There is no real-time constraint during the capture and storage
processes. During the retrieval process, however, the system demands strict temporal
relationships. Video on demand is an example of a stored and retrieved, nonlive
presentation.
c) Mixed Presentation
A mixed presentation uses a combination of live and stored information. A collaborative
conference is an example of a mixed presentation. One window on the computer screen
shows live pictures, while other windows are used to show stored information and user
interactions.

5.2.3 Media synchronization

Any multimedia presentation requires composition of object streams and two types of
synchronization. The composition has two aspects.
a) Spatial composition- relates to the placement of multimedia objects in space at any
point in time.
b) Temporal composition- takes care of synchronization between object steams
Types of synchronization
a) Point synchronization is used when a single object in one stream must be
synchronized with the object in another stream at one point in time.
b) Stream synchronization is done continuously over a period of time, as in the case of a
video presentation. Stream synchronization is also called continuous synchronization.
Multimedia systems can be classified based on the level of synchronization between the
various object streams in the system. Three types of systems are considered.
(a) Asynchronous Media
(b) Synchronous Media
(c) Isochronous Media

St. Joseph’s College of Engineering 4

EC8002-Multimedia Compression and Communication Department of ECE

a) Asynchronous Media
In an Asynchronous system, there is no well-defined timing relationship between the
objects of one or more object stream (s). This synchrony can either be intrastream or
interstream. A well-known example of intrastream asynchronous communication is a user
typing text on a keyboard. Since no user can type at a constant rate the time gaps between
consecutive character vary.
Figure 5.2 shows two object streams : Stream (A) := Ψ(An) and Stream (B) := Ψ(Bn).
Three objects are shown for Stream (A), namely An-1, An, and An+1. The leading edges of these
objects occur at time instants t(An-1), t(An), and t(An+1) respectively. Similarly, the leading
edges of Stream (B) objects, Bn-1, Bn, and Bn+1 occur at t(Bn-1), t(Bn), and t(Bn+1) respectively.

Figure 5.2 Temporal relationships between objects in two streams

i)Intrastream Asynchrony For intrastream asynchrony, only one object steam, say Stream (A),
must be considered. The time gap between the leading edges of two consecutive objects
such as An-1, An is represented by ∆t(An, An-1) and is given by
∆t(An, An-1) = t(An) - t(An-1) (5.3)
Similarly, the time gap between An and An+1 is given by
∆t(An+1, An) = t(An+1) - t(An) (5.4)
The general form of intrastream asynchrony implies that the consecutive time gaps
between objects are not equal; that is
∆t(An+1, An) ≠ ∆t(An, An-1) (5.5)
ii)Interstream Asynchrony For interstream asynchrony, two (or more ) object streams, say
Stream (A) and Stream (B), must be considered. The time gap between the leading edges of

St. Joseph’s College of Engineering 5

EC8002-Multimedia Compression and Communication Department of ECE

two related objects from the two streams, such as An-1 and Bn-1 is represented by ∆t(An-1, Bn-1)
and is given by
∆t(An-1, Bn-1) = t(An-1) – t(Bn-1) (5.6)
The time gap between the leading edges of the next two related objects, such as A n
and Bn, is given by
∆t(An, Bn) = t (An) – t(Bn) (5.7)
When two streams are said to be asynchronous, it implies that the time gaps between
the objects in the two streams are non zero and not equal to each other, such as
∆t(An-1, Bn-1) ≠ ∆t(An, Bn) ≠ 0 (5.8)
For example, in a collaborative conference, one window may be used for live video,
and another window may be used for browsing through some text stored on a server. In this
case, the video stream and the text stream need not have any synchronization.
b) Synchronous Streams:
In synchronous multimedia, well-defined temporal relationships exist between the
objects. The term synchronous is derived from the Greek, meaning means same time.
The term synchronous is applicable mainly to the temporal relationship between two
different media stream; it is therefore an intermedia concept. Synchronous temporal
relationships are specified for multimedia object streams as they are captured, stored,
retrieved, transmitted, received and played back.
For interstream synchrony two (or more) object streams, say Stream (A) and Stream (B) in
Figure 5.2 - must be considered. Interstream synchrony implies that time gaps between
related objects are equal to a constant called Tconstant; that is
∆t(Bn-1, An-1) = ∆t(Bn, An) = ∆t(Bn+1, An+1) = Tconstant (5.10)
The most often-used meaning of synchrony is that the corresponding objects of the
two streams occur at the same time (Tconstant = 0), that is
∆t(Bn-1, An-1) = ∆t(Bn, An) = ∆t(Bn+1, An+1) = 0 (5.11)
A video clip, when played along with audio, is asynchronous media. Not only must the
sound and video streams play isochronously that is, at the correct rate, they must also be
synchronized to each other.
c) Isochronous Streams The term isochronous is also derived from the Greek, meaning
equal time. For isochronous streams – streams involving only intrastream synchrony –
only one object stream, say Stream (A), needs to be considered. An isochronous
stream has equal time gaps between consecutive object. This time is equal to the
period , Tperiod, of the stream; that is
∆t(An+1, An) = ∆t(An, An-1) = ∆t(An-1, An-2) = …. = Tperiod, (5.9)

St. Joseph’s College of Engineering 6

 EC8002-Multimedia Compression and Communication Department of ECE

A video clip played without sound is an isochronous media. If it was stored at 24f[s,
then it must be played back at 24 fps. It is not acceptable to play the video at 20 fps in one
second and the compensate by playing it at 28 fps in the next.

Synchronous and isochronous behaviors need not occur at the same time; it is possible
to have one without the other. It is possible for two media streams to work in lock-step
manner (i.e., synchronously) even if they are not isochronous (i.e., there is no fixed time
period between consecutive objects). For example, in a multimedia slide show each slide is
synchronized to some voice or music. This time allocated to each slide need not be the same.
It is also possible for the change over from one slide to the next to be controlled by the user,
making the slide presentation asynchronous. The two media in this slide show – still images
and sound – are synchronized, but neither of them is isochronous.
It is also possible to have two isochronous steams without having any synchronization
between them. For example, in a collaborative conference the users may be watching the live
video in one window and an animation in another. The live vide is likely to be running
between 20 and 30 fps. The animation may run between 15 and 20 fps. Both streams are
isochronous, but they are not synchronized to each other.

5.3 Models for Temporal Specifications

The timing relationship between the various objects in a presentation must be clearly
specified to achieve the desired results. Some systems require precise specification, while
other systems require relative, or even imprecise, specification.
Systems such as video have their inherent, or natural temporal constraints. Systems
such as entertainment packages have temporal constraints imposed artificially. Interactive
systems have response time specifications. Most stored presentation allow various Temporal
Access Control (TAC) operations, such as play, fast forward, rewind, pause, looping, and
browsing. All these activities require methods for specifying temporal constraints and
relationships.
Different models for specifying temporal relationships are used for different
applications.
i) Relative Timing constraints Model.
ii) Absolute timing constraints Model
In the absolute temporal specification model there are two options.
(a) Specification of instants
(b) Specification of duration
(a) Relative Time Specification
St. Joseph’s College of Engineering 7
EC8002-Multimedia Compression and Communication Department of ECE

Many multimedia presentations do not need exact specification of when the various
media objects should appear; relative position of the interacting objects is more important.
Consider the example of an educational courseware called “Introduction to Multimedia”. It is
likely that in such a system a welcome screen appears first, followed by a voice message,
called the introduction message . A transition from the welcome screen to a menu screen
may be designed to overlap with the introduction message.
Such relative specifications are more important to this package than absolute timing
information. It is not even possible to specify exact timing information, because the time
taken for presenting various objects will vary from one computer to another. On the other
hand, the voice message must play isochronously regardless of the computer speed.
There are seven fundamental Relative Temporal Relationships (RTRs) between two
objects, as shown in Figure 5.3 (Allen, 1983). Each of the first six RTRs have inverse relations,
giving rise to six more RTRs. The inverse of the ‘equal’ relation is the same as the original
relation. This gives a total of thirteen unique RTRs.

Figure 5.3 Relative temporal relationships – RTRs

Table 5.2 Inverse Transforms of Relative Temporal Relationships

(A before B)-1 = (B before A) = (A before -1 B) = (A after B) (5.12)

(A meets B)-1 = (B meets A) = (A meets -1 B) = (A met_by B) (5.13)

(A during B)-1 = (B during A) = (A during -1 B) = (A envelops B) (5.14)

St. Joseph’s College of Engineering 8

 EC8002-Multimedia Compression and Communication Department of ECE

(A overlaps B)-1 = (B overlaps A) = (A overlaps -1 B) = (A overlapped_by B) (5.15)

(A starts B)-1 = (B starts A) = (A starts -1 B) = (A started_by B) (5.16)

(A ends B)-1 = (B ends A) = (A ends -1 B) = (A ended_by B) (5.17)

(A equals B)-1 = (B equals A) = (A equals -1 B) = (A equals B) (5.18)

The inverse of (A before B) is represented as (A before B)-1 , and is equal to (B before

A). It is also equal to (A before -1 B), which can be written as (A after B). This and other
inverse transformations are given in Table 5.2.
Temporal Transformations The inverse transformation is only one type of temporal
transformation. Other types of temporal transformations include scaling and shifting (Buford,
1994, 185). Temporal Transformations can be used to map stored temporal relationships to
a real-time play-out. Graphical representation of these transformations are shown in Figure
5.4. Scaling is used for fast forward and slow motion operations. Inverting can be used for
rewind operation.

Figure 5.4 Temporal Transformations

Spatial transformations Transformations that take place in space (e.g, on the screen
or in audio) are called spatial transformations. These include transitions such as fading of
images or audio, mixing of channels, colour changes, etc. Temporal specifications are required
to specify the instants or Intervals over which spatial transformations take place.
(b) Absolute Time Specification
A time line is used to represent absolute temporal specifications. Absolute time
specifications can be described in terms of instants or in terms of intervals.

St. Joseph’s College of Engineering 9

EC8002-Multimedia Compression and Communication Department of ECE

Figure 5.5 Absolute time specifications

In some situations, exact time specification is not required; i.e., inexact timing
specification is either adequate or necessary. The three types of specifications are shown in
Figure 5.5
Instant-Based Specification : Instant-Based Specification is appropriate when there is
a need to state that a play-out begins exactly at time t1 and finishes at time t3, as shown in
Figure 5.5(a)
The value of these instants may be specified either with respect to the reference time
t0 (e.g.., t0 = 0.00, t1 = 10 sec, t2 = 20 sec., etc.) or in terms of local time (e.g.., t1 = 05:45Hr., t2
= 17:54Hr.) Complete date and time stamping is also used, especially in applications in which
information from two different nodes is combined synchronously.
Duration – Based Specification : A duration – Based Specification gives the interval for
the play-out. The starting time t1 and the play-out interval Tplay are specified, as shown in
Figure 5.5 (b). It is easy to map a fixed interval specification to a starting and ending instant-
based specification. An interval-based specification becomes more important when an
inexact specification is required.
In exact timing specification : In some applications, an In exact temporal specification
must be used to specify the play-out period. For example, it may be appropriate to specify
that the music should start “soon” after the still image, and as soon as the music finishes the
image should be faded out. A lower bound and an upper bound on the play-out period is
specified, as shown in Figure 5.5 (c).

St. Joseph’s College of Engineering 10