Igli Tafa

Lectures
Multimedia Concepts

Lecture *Multimedia-Igli Tafa*

1
 Outcome
• What is multimedia
• History of Multimedia
• Reasons to Use Multimedia
• Introduction to Multimedia
• Interactive Multimedia
• Categories of Multimedia
• Multimedia Elements
• Multimedia Files
• Multimedia Application
• Audio Encoding
• Video Encoding
– The JPEG Standard
– The MPEG Standard
• Multimedia Process Scheduling
• Rate Monotonic Scheduling
• Earliest Deadline First Scheduling
Lecture *Multimedia-Igli Tafa*
2
 Outcome
• Multimedia File System Paradigms
• VCR Control Functions
• Near Video on Demand
• Near Video on Demand with VCR Functions
• File Placement
• Two Alternative File Organization Strategies
• Placing Files for Near Video on Demand
• Placing Multiple files on a Single Disk
• Placing Files on Multiple Disks
• Caching
• File Caching
• Disk Scheduling for Multimedia
• Dynamic Disk Scheduling
• QoS Problem
• An End-to-End Problem
• OS Problems in Supporting Multimedia
• Memory Management in Multimedia Supporting
• Network Interface in Multimedia Support

Lecture *Multimedia-Igli Tafa*

3
 What is multimedia

• Using more than one media element.

– Sound, Text, Video, Graphics, Animation
• More applications used more of our senses
are activated
• Give Life to Information
• Encourages users to embrace and
internalize

Lecture *Multimedia-Igli Tafa*

4
 History of Multimedia

• Began with silent movies

• Sound occurred about 50 years ago
• As technology improves, multimedia
techniques improve
• More uses has created a larger demand
• Digitized world
• Growth of Internet

Lecture *Multimedia-Igli Tafa*

5
 Reasons to Use Multimedia

• Inspiration from different sources

• Accommodation of learning styles
• People remember only
– 20% sight
– 30% hear

Lecture *Multimedia-Igli Tafa*

6
 Introduction to Multimedia

Video On Demand: (a) ADSL

Lecture *Multimedia-Igli Tafa* vs. (b) cable 7
 Introduction to Multimedia

• Some data rates

– multimedia, high performance I/O devices
• Note: 1 Mbps = 10 bits/sec but 1 GB = 230 bytes
Lecture
6 *Multimedia-Igli Tafa*
8
 Interactive Multimedia

• To respond directly and to control any or all

of these media elements
• Active participants vs Passive recipients

Lecture *Multimedia-Igli Tafa*

9
 Categories of Multimedia

• Linear media: Start at the beginning and

progress through a set of sequence of events
until they reach the end.
• Non-Linear Media: user has control over
the order of events

Lecture *Multimedia-Igli Tafa*

10
 Multimedia Elements

• Text:
– Hypertext, Hotword, True Type fonts, Printer
fonts, Kerning, Leading, Screen fonts
• Graphics:
– Clip Art, Charts, Stock photography, Digital
cameras, scanners
• Animation:
– Convey information that is important
– Change or movement
Lecture *Multimedia-Igli Tafa*
11
 Multimedia Elements

• Sound: Any sound digitized is called audio

– Midi, WAV, AIF, AU
• Video: Used to illustrate a proper technique
or advertise
– AVI, MOV
– Large files, 5 sec. = 5mb
• Bandwidth: Amount of data a channel can
carry
Lecture *Multimedia-Igli Tafa*
12
 Multimedia Files

A movieLecture
may*Multimedia-Igli
consist of Tafa*
several files 13
 Multimedia Applications

• World Wide Web

• HTML editors
• CBT: Computer Based Training
• CAT: Computer Assisted Training
• MCIA: Multimedia Computer-assisted
instruction

Lecture *Multimedia-Igli Tafa*

14
 Multimedia Application

• International Access
• Standard Use
• Ensuring a level or mastery
• Update skills of current employees
• Immediate feedback
• Simulated experiences

Lecture *Multimedia-Igli Tafa*

15
 Multimedia Application

• Electronic References
• Push Technology
• Kiosk
• Advertise and marketing
• Electronic catalogs and malls
• Games

Lecture *Multimedia-Igli Tafa*

16
 Multimedia Application

• Up-to-date articles
• Software on demand
• Free standing information stations
• Grab Attention and online slide shows
• Marketing Goods and purchases
• Edutainment

Lecture *Multimedia-Igli Tafa*

17
 Audio Encoding

• Audio Waves Converted to Digital

– electrical voltage input
– binary number as output
Lecture *Multimedia-Igli Tafa*
18
 Audio Encoding

• Error induced by finite sampling

– called quantization noise

• Examples of sampled sound

– telephone – pulse code modulation
– audio compact disks

Lecture *Multimedia-Igli Tafa*

19
 Video Encoding

Scanning PatternLecture
for*Multimedia-Igli
NTSC Video Tafa* and Television 20
 The JPEG Standard

RGB input data and block preparation

Lecture *Multimedia-Igli Tafa*
21
 The JPEG Standard

One block of the Y matrix and the DCT coefficients

Lecture *Multimedia-Igli Tafa*
22
 The JPEG Standard

Computation of the quantized DCT coefficients

Lecture *Multimedia-Igli Tafa*

23
 The MPEG Standard

Order of quantized values when

Lecture *Multimedia-Igli Tafa*
transmitted
24
 The MPEG Standard
MPEG-2 has three kinds of frame: I, P, B
1. Intracoded frames
- Self-contained JPEG-encoded pictures
2. Predictive frames
- Block-by-block difference with last frame
3. Bi-directional frames
- Differences with last and next frame

Lecture *Multimedia-Igli Tafa*

25
The MPEG Standard

Consecutive Video Frames

Lecture *Multimedia-Igli Tafa*

26
 Multimedia Process Scheduling

• Periodic processes displaying a movie

• Frame rates and processing requirements
may be different for each movie
Lecture *Multimedia-Igli Tafa*
27
 Rate Monotonic Scheduling

Used for processes which meet these conditions

1. Each periodic process must complete within its period
2. No process dependent on any other process
3. Each process needs same CPU time each burst
4. Any nonperiodic processes have no deadlines
5. Process preemption occurs instantaneously, no
overhead

Lecture *Multimedia-Igli Tafa*

28
Earliest Deadline First Scheduling

• Real Time Scheduling algorithms

– RMS
– EDF Lecture *Multimedia-Igli Tafa*
29
 Earliest Deadline First Scheduling

Another example of real-time scheduling with RMS and EDF

Lecture *Multimedia-Igli Tafa*
30
Multimedia File System Paradigms

Pull and Push Servers

Lecture *Multimedia-Igli Tafa*
31
 VCR Control Functions

• Rewind is simple
– set next frame to zero
• Fast forward/backward are trickier
– compression makes rapid motion complicated
– special file containg e.g. every 10th frame

Lecture *Multimedia-Igli Tafa*

32
Near Video on Demand

New streamLecture
starting at regular intervals
*Multimedia-Igli Tafa*
33
Near Video on Demand with VCR Functions

Buffering for Rewind

Lecture *Multimedia-Igli Tafa*
34
 File Placement

Frame 1 Frame 2 Frame 3

Audio Text
Frame Frame

Placing a File on a Single Disk

• Interleaving
– Video, audio, text in*Multimedia-Igli
Lecture single contiguous
Tafa* file per movie
35
Two Alternative File Organization Strategies

• Noncontiguous Movie Storage

(a) small disk blocks
(b) large disk blocks
Lecture *Multimedia-Igli Tafa*
36
Two Alternative File Organization Strategies

Trade-offs between small, large blocks

1. Frame index
- heavier RAM usage during movie play
- little disk wastage
• Block index (no splitting frames over blocks)
- low RAM usage
- major disk wastage
• Block index (splitting frames over blocks allowed)
- low RAM usage
- no disk wastage
- extra seeks

Lecture *Multimedia-Igli Tafa*

37
Placing Files for Near Video on Demand

Optimal frame placement for near video on demand

Lecture *Multimedia-Igli Tafa*
38
Placing Multiple files on a Single Disk

• Zipf's law for N=20

• Squares for 20 largest cities in US
– sorted on rank order
Lecture *Multimedia-Igli Tafa*
39
Placing Multiple files on a Single Disk

• Organ-pipe distribution of files on server

– most popular movie in middle of disk
– next most popular either on either side, etc.
Lecture *Multimedia-Igli Tafa*
40
Placing Files on Multiple Disks

Organize multimedia files on multiple disks

(a) No striping
(b) Same striping pattern for all files
(c) Staggered striping
(d) Random striping Lecture *Multimedia-Igli Tafa*
41
 Caching

Block Caching
(a) Two users, same movie 10 sec out of sync
(b) Merging two streams into one
Lecture *Multimedia-Igli Tafa*
42
 File Caching

• Most movies stored on DVD or tape

– copy to disk when needed
– results in large startup time
– keep most popular movies on disk

• Can keep first few min. of all movies on disk

– start movie from this while remainder is fetched

Lecture *Multimedia-Igli Tafa*

43
Disk Scheduling for Multimedia
Stream

Order in which disk requests are processed 

Static Disk Scheduling

• In one round, each movie asks for one frame
Lecture *Multimedia-Igli Tafa*
44
 Dynamic Disk Scheduling

• Scan-EDF algorithm
– uses deadlines & cylinder numbers for scheduling
Lecture *Multimedia-Igli Tafa*
45
 QoS Problem

• Selection of proper VC
• Problems with Bandwidth: FDM and TDM
• Problems with Delay: Line Transmission
and Buffer Delay
• Problems with Jitter: Dynamic Packets
during transmission
• Problems with Reliabilities: OS must solve
it?
Lecture *Multimedia-Igli Tafa*
46
 An End-To-End Problem
160 ---

Client
160 160
160 148
160 190
... ...

• Server Application • Client Application

• Operating System Network •
Operating System
Routers
• Network Protocol • Network Protocol

Lecture *Multimedia-Igli Tafa*

47
 OS Problems in Supporting
Multimedia
• Process Scheduling
• Memory Management
• Storage Scheduling
• Network Interface

Lecture *Multimedia-Igli Tafa*

48
 Memory Management in
Multimedia Supporting
 Paging:
– page faults cause jitter
– allocation causes jitter
 global vs. local
– solution: lock in pages

Lecture *Multimedia-Igli Tafa*

49
Network Interface in Multimedia
Support
• TCP
– guarantees delivery
– stream semantics
– fixed flow control
– unicast
– … big bleah!

Lecture *Multimedia-Igli Tafa*

50
Network Interface in Multimedia
Support
 UDP
– multicast add-on
– checksum cannot be turned off
– no notion of priority
– no flow control
– … little bleah!

Lecture *Multimedia-Igli Tafa*

51
Network Interface in Multimedia
Support
• RTP
– multicast add-on
– packet sequence
– flow control

Lecture *Multimedia-Igli Tafa*

52
 Multimedia: audio
 analog audio signal
sampled at constant rate
quantization quantized
 telephone: 8,000 error value of
samples/sec analog value

audio signal amplitude

 CD music: 44,100 analog
signal
samples/sec
 each sample quantized,
i.e., rounded
 e.g., 28=256 possible time
quantized values sampling rate
 each quantized value (N sample/sec)

represented by bits,
e.g., 8 bits for 256
values
Multmedia Networking
7-53
 Multimedia: audio
 example: 8,000 samples/sec,
256 quantized values:
quantization
64,000 bps error
quantized
value of
 receiver converts bits back to analog value

audio signal amplitude

analog signal: analog

 some quality reduction signal

example rates
 CD: 1.411 Mbps time

 MP3: 96, 128, 160 kbps sampling rate

(N sample/sec)
 Internet telephony: 5.3 kbps
and up

Multmedia Networking
7-54
 spatial coding example: instead

Multimedia: video of sending N values of same

color (all purple), send only two
values: color value (purple) and
number of repeated values (N)
 video: sequence of images
……………………...…
……………………...…
displayed at constant rate
 e.g. 24 images/sec
 digital image: array of pixels
 each pixel represented by
bits
 coding: use redundancy
frame i
within and between images to
decrease # bits used to encode
image temporal coding example:
instead of sending
 spatial (within image) complete frame at i+1,
send only differences from
 temporal (from one image frame i
to next)
Multmedia Networking frame i+1
7-55
 spatial coding example: instead

Multimedia: video of sending N values of same

color (all purple), send only two
values: color value (purple) and
number of repeated values (N)
 CBR: (constant bit rate):
video encoding rate fixed ……………………...…
……………………...…
 VBR: (variable bit rate):
video encoding rate changes
as amount of spatial,
temporal coding changes
 examples:
 MPEG 1 (CD-ROM) 1.5
Mbps frame i
 MPEG2 (DVD) 3-6 Mbps
 MPEG4 (often used in temporal coding example:
Internet, < 1 Mbps) instead of sending
complete frame at i+1,
send only differences from
frame i

Multmedia Networking frame i+1

7-56
 Multimedia networking: 3 application types

 streaming, stored audio, video

 streaming: can begin playout before downloading entire file
 stored (at server): can transmit faster than audio/video will
be rendered (implies storing/buffering at client)
 e.g., YouTube, Netflix, Hulu
 conversational voice/video over IP
 interactive nature of human-to-human conversation limits
delay tolerance
 e.g., Skype
 streaming live audio, video
 e.g., live sporting event (futbol)
Multmedia Networking
7-57
Cumulative
data
Streaming stored video:

2. video
1. video sent 3. video received,
recorded network played out at
(e.g., 30 delay client time
frames/ (fixed in (30 frames/sec)
streaming:
this at this time, client
sec)
playing
example) out early part of video,
while server still sending later
part of video
Multmedia Networking
7-58
Streaming stored video: challenges
 continuous playout constraint: once client playout begins, playback must match
original timing
 … but network delays are variable (jitter), so will need client-side buffer to
match playout requirements
 other challenges:
 client interactivity: pause, fast-forward, rewind, jump through video
 video packets may be lost, retransmitted

Multmedia Networking
7-59
 Streaming stored video: revisted
constant bit
rate video client video
Cumulative

constant bit
transmission reception rate video
data

variable playout at client

buffered
network

video
delay

client time
playout
delay
client-side buffering and playout delay: compensate
for network-added delay, delay jitter
Multmedia Networking
7-60
 Client-side buffering, playout
buffer fill level,
Q(t)
variable fill playout rate,
rate, x(t) e.g., CBR r

client application
video server buffer, size B

client

Multmedia Networking
7-61
 Client-side buffering, playout
buffer fill level,
Q(t)
variable fill playout rate,
rate, x(t) e.g., CBR r

client application
video server buffer, size B

client

1. Initial fill of buffer until playout begins at t p

2. playout begins at tp,
3. buffer fill level varies over time as fill rate x(t)
varies andMultmedia
playout rate r is constant
Networking
7-62
 Client-side buffering, playout
buffer fill level,
Q(t)
variable fill playout rate,
rate, x(t) e.g., CBR r

client application
video server buffer, size B

playout buffering: average fill rate (x), playout rate (r):

x < r: buffer eventually empties (causing freezing of video playout until
buffer again fills)
x > r: buffer will not empty, provided initial playout delay is large enough
to absorb variability in x(t)
 initial playout delay tradeoff: buffer starvation less likely
with larger delay, but larger delay until user begins watching
Multmedia Networking
7-63
 Streaming multimedia: UDP
server sends at rate appropriate for client
 often: send rate = encoding rate = constant rate
 transmission rate can be oblivious to congestion
levels
short playout delay (2-5 seconds) to remove
network jitter
error recovery: application-level,
timeipermitting
RTP [RFC 2326]: multimedia payload types
UDP may not go through firewalls
Multmedia Networking
7-64
 Streaming multimedia: HTTP
multimedia file retrieved via HTTP GET
send at maximum possible rate under TCP

variable
rate, x(t)

video TCP send TCP receive application

file buffer buffer playout buffer
server client

fill rate fluctuates due to TCP congestion control,

retransmissions (in-order delivery)
larger playout delay: smooth TCP delivery rate
HTTP/TCP passes more easily
Multmedia through firewalls
Networking
7-65
 Streaming multimedia: DASH
DASH: Dynamic, Adaptive Streaming over HTTP
server:
 divides video file into multiple chunks
 each chunk stored, encoded at different rates
 manifest file: provides URLs for different chunks
client:
 periodically measures server-to-client bandwidth
 consulting manifest, requests one chunk at a time
• chooses maximum coding rate sustainable given current
bandwidth
• can choose different coding rates at different points in time
(depending on available bandwidth at time)
Multmedia Networking
7-66
 Streaming multimedia: DASH
• DASH: Dynamic, Adaptive Streaming over
HTTP
• “intelligence” at client: client determines
– when to request chunk (so that buffer
starvation, or overflow does not occur)
– what encoding rate to request (higher quality
when more bandwidth available)
– where to request chunk (can request from URL
server that is “close” to client or has high
available bandwidth)
Multmedia Networking
7-67
 Content distribution networks
• challenge: how to stream content (selected
from millions of videos) to hundreds of
thousands of simultaneous users?

• option 1: single, large “mega-server”

– single point of failure
– point of network congestion
– long path to distant clients
– multiple copies of video sent over outgoing link
….quite simply: thisMultmedia
solution
Networkingdoesn’t scale
7-68
 Content distribution networks
• challenge: how to stream content (selected from
millions of videos) to hundreds of thousands of
simultaneous users?

• option 2: store/serve multiple copies of videos at

multiple geographically distributed sites (CDN)
– enter deep: push CDN servers deep into many access
networks
• close to users
• used by Akamai, 1700 locations
– bring home: smaller number (10’s) of larger clusters in
POPs near (but not within) access networks
• used by Limelight Multmedia Networking
7-69
 CDN: “simple” content access scenario
Bob (client) requests video http://netcinema.com/6Y7B23V
video stored in CDN at http://KingCDN.com/NetC6y&B23V

1. Bob gets URL for for video

http://netcinema.com/6Y7B23V
from netcinema.com 2. resolve http://netcinema.com/6Y7B23V
web page 2 via Bob’s local DNS
1
6. request video from 5
KINGCDN server, 4&5. Resolve
streamed via HTTP http://KingCDN.com/NetC6y&B23
via KingCDN’s authoritative DNS,
netcinema.com 3. netcinema’s DNS returns URL
4 which returns IP address of KIingCDN
http://KingCDN.com/NetC6y&B23V server with video
3

netcinema’s
authorative DNS KingCDN.com KingCDN
Multmedia Networking
authoritative DNS 7-70
 CDN cluster selection strategy
• challenge: how does CDN DNS select “good” CDN
node to stream to client
– pick CDN node geographically closest to client
– pick CDN node with shortest delay (or min # hops) to
client (CDN nodes periodically ping access ISPs,
reporting results to CDN DNS)
– IP anycast

• alternative: let client decide - give client a list of

several CDN servers
– client pings servers, picks “best”
– Netflix approach Multmedia Networking
7-71
 Case study: Netflix
30% downstream US traffic in 2011
owns very little infrastructure, uses 3rd party
services:
 own registration, payment servers
 Amazon (3rd party) cloud services:
• Netflix uploads studio master to Amazon cloud
• create multiple version of movie (different endodings) in
cloud
• upload versions from cloud to CDNs
• Cloud hosts Netflix web pages for user browsing
 three 3rd party CDNs host/stream Netflix content:
Akamai, Limelight, Level-3
Multmedia Networking
7-72
 Case study: Netflix
upload copies of
Amazon cloud
multiple versions of
video to CDNs Akamai CDN

Netflix registration,
accounting servers
3. Manifest file
2. Bob browses returned for
requested video Limelight CDN
Netflix video
2
3
1

1. Bob manages
Netflix account
Level-3 CDN
4. DASH streaming

Multmedia Networking
7-73
 Voice-over-IP (VoIP)
• VoIP end-end-delay requirement: needed to maintain
“conversational” aspect
– higher delays noticeable, impair interactivity
– < 150 msec: good
– > 400 msec bad
– includes application-level (packetization,playout), network
delays
• session initialization: how does callee advertise IP address,
port number, encoding algorithms?
• value-added services: call forwarding, screening,
recording
• emergency services: 911
Multmedia Networking
7-74
 VoIP characteristics
• speaker’s audio: alternating talk spurts, silent periods.
– 64 kbps during talk spurt
– pkts generated only during talk spurts
– 20 msec chunks at 8 Kbytes/sec: 160 bytes of data
• application-layer header added to each chunk
• chunk+header encapsulated into UDP or TCP segment
• application sends segment into socket every 20 msec
during talkspurt

Multmedia Networking
7-75
 VoIP: packet loss, delay
network loss: IP datagram lost due to
network congestion (router buffer overflow)
delay loss: IP datagram arrives too late for
playout at receiver
 delays: processing, queueing in network; end-
system (sender, receiver) delays
 typical maximum tolerable delay: 400 ms
loss tolerance: depending on voice encoding,
loss concealment, packet loss rates between
1% and 10% can be tolerated
Multmedia Networking
7-76
 Delay jitter
constant bit
rate client
Cumulative

constant bit
transmission reception rate playout
data

variable at client

buffered
network

data
delay
(jitter)
client time
playout
end-to-end delays ofdelay
two consecutive packets: difference can
be more or less than 20 msec (transmission time difference)
Multmedia Networking
7-77
 VoIP: fixed playout delay
• receiver attempts to playout each chunk
exactly q msecs after chunk was generated.
– chunk has time stamp t: play out chunk at t+q
– chunk arrives after t+q: data arrives too late for
playout: data “lost”
• tradeoff in choosing q:
– large q: less packet loss
– small q: better interactive experience

Multmedia Networking
7-78
 VoIP: fixed playout delay
 sender generates packets every 20 msec during talk spurt.
 first packet received at time r
 first playout schedule: begins at p
 second playout schedule: begins at p’
packets

packets loss
generated
packets
playout schedule
received
p' - r

playout schedule
p-r

tim e
Multmedia
r
Networking
5-79
p p'
 Adaptive playout delay (1)
goal: low playout delay, low late loss rate
approach: adaptive playout delay adjustment:
 estimate network delay, adjust playout delay at
beginning of each talk spurt
 silent periods compressed and elongated
 chunks still played out every 20 msec during talk
spurt
i = (1)d
adaptively destimate i-1 + delay:
packet (ri – ti)( EWMA -
exponentially weighted moving average, recall TCP RTT
estimate):
delay estimate small constant, time received - time sent
after ith packet e.g. 0.1 (timestamp)
measured delay of ith packet
Multmedia Networking
7-80
 Adaptive playout delay (2)
 also useful to estimate average deviation of delay, v i :
vi = (1)vi-1 + |ri – ti – di|
 estimates di, vi calculated for every received
packet, but used only at start of talk spurt

 for first packet in talk spurt, playout time is:

playout-timei = ti + di + Kvi

remaining packets in talkspurt are played out

periodically Multmedia Networking
5-81
 Adaptive playout delay (3)
Q: How does receiver determine whether packet is
first in a talkspurt?
if no loss, receiver looks at successive timestamps
 difference of successive stamps > 20 msec -->talk spurt
begins.
with loss possible, receiver must look at both time
stamps and sequence numbers
 difference of successive stamps > 20 msec and
sequence numbers without gaps --> talk spurt begins.

Multmedia Networking
7-82
 VoiP: recovery from packet loss (1)
Challenge: recover from packet loss given small tolerable delay
between original transmission and playout
 each ACK/NAK takes ~ one RTT
 alternative: Forward Error Correction (FEC)
 send enough bits to allow recovery without retransmission (recall two-
dimensional parity in Ch. 5)

simple FEC
 for every group of n chunks, create redundant chunk by exclusive OR-ing n
original chunks
 send n+1 chunks, increasing bandwidth by factor 1/n
 can reconstruct original n chunks if at most one lost chunk from n+1 chunks,
with playout delay

Multmedia Networking
7-83
VoiP: recovery from packet loss (2)
another FEC scheme:
 “ piggyback lower
quality stream”
 send lower resolution
audio stream as
redundant information
 e.g., nominal
stream PCM at 64 kbps
and redundant stream
GSM at 13 kbps
 non-consecutive loss: receiver can conceal loss
 generalization: can also append (n-1)st and (n-2)nd low-bit rate
chunk
Multmedia Networking
7-84
 VoiP: recovery from packet loss (3)

interleaving to conceal loss:

 audio chunks divided into smaller  if packet lost, still have most
units, e.g. four 5 msec units per 20 of every original chunk
msec audio chunk
 no redundancy overhead,
 packet contains small units from
different chunks but increases playout delay
Multmedia Networking
7-85
 Voice-over-IP: Skype
Skype clients (SC)
 proprietary application-layer
protocol (inferred via
reverse engineering)
 encrypted msgs Skype

login server supernode (SN)
 P2P components:
clients: skype peers connect directly to each other
for VoIP call
supernode
overlay
network
 super nodes (SN): skype peers with special
functions

 overlay network: among SNs to locate SCs

 login server
Application Layer
2-86
 P2P voice-over-IP: skype
skype client operation:
1. joins skype network by
contacting SN (IP address
cached) using TCP Skype
login server
2. logs-in (usename,
password) to centralized
skype login server
3. obtains IP address for
callee from SN, SN overlay
 or client buddy list

4. initiate call directly to callee

Application Layer
2-87
 Skype: peers as relays
• problem: both Alice, Bob
are behind “NATs”
– NAT prevents outside peer
from initiating connection to
insider peer
– inside peer can initiate
connection to outside
 relay solution: Alice, Bob
maintain open connection
to their SNs
 Alice signals her SN to connect
to Bob
 Alice’s SN connects to Bob’s
SN
 Bob’s SN connects to Bob over
open connection Bob initially
initiated to his SN
Application Layer
2-88
 Real-Time Protocol (RTP)

 RTP specifies packet  RTP runs in end

structure for packets systems
carrying audio, video  RTP packets
data encapsulated in UDP
 RFC 3550 segments
 RTP packet provides  interoperability: if two
 payload type VoIP applications run
identification RTP, they may be able
 packet sequence to work together
numbering
 time stamping
Multmedia Networking
7-89
 RTP runs on top of UDP
RTP libraries provide transport-layer interface
that extends UDP:
• port numbers, IP addresses
• payload type identification
• packet sequence numbering
• time-stamping

Multmedia Networking
5-90
 RTP example
example: sending 64 kbps  RTP header indicates
PCM-encoded voice over type of audio encoding
RTP in each packet
application collects  sender can change
encoded data in chunks, encoding during
conference
e.g., every 20 msec = 160
bytes in a chunk  RTP header also
audio chunk + RTP contains sequence
header form RTP packet, numbers, timestamps
which is encapsulated in
UDP segment
Multmedia Networking
7-91
 RTP and QoS
RTP does not provide any mechanism to
ensure timely data delivery or other QoS
guarantees
RTP encapsulation only seen at end systems
(not by intermediate routers)
 routers provide best-effort service, making no
special effort to ensure that RTP packets arrive
at destination in timely matter

Multmedia Networking
7-92
 RTP header
payload sequence Synchronization Miscellaneou
time stamp
type number type Source ID s fields

payload type (7 bits): indicates type of encoding currently being

used. If sender changes encoding during call, sender
informs receiver via payload type field
Payload type 0: PCM mu-law, 64 kbps
Payload type 3: GSM, 13 kbps
Payload type 7: LPC, 2.4 kbps
Payload type 26: Motion JPEG
Payload type 31: H.261
Payload type 33: MPEG2 video

sequence # (16 bits): increment by one for each RTP packet sent
detect packet loss, restore packet sequence
Multmedia Networking
5-93
 RTP header
payload sequence Synchronization Miscellaneou
time stamp
type number type Source ID s fields

timestamp field (32 bits long): sampling instant of first

byte in this RTP data packet
 for audio, timestamp clock increments by one for each sampling
period (e.g., each 125 usecs for 8 KHz sampling clock)
 if application generates chunks of 160 encoded samples,
timestamp increases by 160 for each RTP packet when source
is active. Timestamp clock continues to increase at constant rate
when source is inactive.

SSRC field (32 bits long): identifies source of RTP stream. Each
stream in RTP session has distinct SSRC
Multmedia Networking
7-94
 RTSP/RTP programming
assignment
build a server that encapsulates stored video
frames into RTP packets
 grab video frame, add RTP headers, create
UDP segments, send segments to UDP socket
 include seq numbers and time stamps
 client RTP provided for you
also write client side of RTSP
 issue play/pause commands
 server RTSP provided for you
Multmedia Networking
7-95
Real-Time Control Protocol (RTCP)
 works in conjunction  each RTCP packet
with RTP contains sender and/or
 each participant in RTP receiver reports
session periodically  report statistics useful to
application: # packets
sends RTCP control
sent, # packets lost,
packets to all other interarrival jitter
participants
 feedback used to control
performance
 sender may modify its
transmissions based on
feedback
Multmedia Networking
7-96
 RTCP: multiple multicast senders
sender RTP
RTCP

RTCP
RTCP

receivers

 each RTP session: typically a single multicast address; all RTP

/RTCP packets belonging to session use multicast address
 RTP, RTCP packets distinguished from each other via distinct
port numbers
 to limit traffic, each participant reduces RTCP traffic as number of
conference participants increases
Multmedia Networking
5-97
 RTCP: packet types
receiver report packets: source description
 fraction of packets lost, last packets:
sequence number, average  e-mail address of sender,
interarrival jitter sender's name, SSRC of
sender report packets: associated RTP stream
 SSRC of RTP stream,  provide mapping between
current time, number of the SSRC and the
packets sent, number of user/host name
bytes sent

Multmedia Networking
7-98
 RTCP: stream synchronization

 RTCP can synchronize  each RTCP sender-report

different media streams packet contains (for most
within a RTP session recently generated packet
 e.g., videoconferencing in associated RTP stream):
app: each sender generates  timestamp of RTP
one RTP stream for video, packet
one for audio.  wall-clock time for
 timestamps in RTP when packet was
packets tied to the video, created
audio sampling clocks  receivers uses association
 not tied to wall-clock to synchronize playout of
time audio, video
Multmedia Networking
7-99
 RTCP: bandwidth scaling
RTCP attempts to limit  75 kbps is equally shared
its traffic to 5% of among receivers:
session bandwidth  with R receivers, each receiver
gets to send RTCP traffic at
75/R kbps.
example : one sender,
 sender gets to send RTCP
sending video at 2 Mbps
traffic at 25 kbps.
RTCP attempts to limit
RTCP traffic to 100 Kbps  participant determines RTCP
packet transmission period by
RTCP gives 75% of rate
calculating avg RTCP packet
to receivers; remaining 25%
size (across entire session)
to sender
and dividing by allocated
rate
Multmedia Networking
7-100
SIP: Session Initiation Protocol [RFC 3261]
long-term vision:
all telephone calls, video conference calls
take place over Internet
people identified by names or e-mail
addresses, rather than by phone numbers
can reach callee (if callee so desires), no
matter where callee roams, no matter what
IP device callee is currently using

Multmedia Networking
7-101
 SIP services
 SIP provides  determine current IP
mechanisms for call address of callee:
setup:  maps mnemonic
identifier to current IP
 for caller to let callee address
know she wants to  call management:
establish a call  add new media streams
 so caller, callee can during call
agree on media type,  change encoding during
encoding call
 invite others
 to end call  transfer, hold calls
Multmedia Networking
7-102
 Example: setting up call to known IP address
Bob
Alice

 Alice’s SIP invite message

167.180.112.24 193.64.210.89 indicates her port number, IP
address, encoding she prefers
to receive (PCM law)
Bob's
terminal rings
 Bob’s 200 OK message
indicates his port number, IP
address, preferred encoding
(GSM)
 Law audio
port 38060  SIP messages can be sent
over TCP or UDP; here sent
G SM
port 48753
over RTP/UDP

 default SIP port number is

time Multmedia time
Networking 5060
5-103
 Setting up a call (more)
• codec negotiation: • rejecting a call
– suppose Bob doesn’t – Bob can reject with
have PCM law encoder replies “busy,” “gone,”
– Bob will instead reply “payment required,”
with 606 Not Acceptable “forbidden”
Reply, listing his • media can be sent over
encoders. Alice can then RTP or some other
send new INVITE
protocol
message, advertising
different encoder

Multmedia Networking
7-104
 Example of SIP message
INVITE sip:[email protected] SIP/2.0
Via: SIP/2.0/UDP 167.180.112.24  Here we don’t know
From: sip:[email protected] Bob’s IP address
To: sip:[email protected]
 intermediate SIP
Call-ID: [email protected]
Content-Type: application/sdp servers needed
Content-Length: 885  Alice sends,
receives SIP messages
c=IN IP4 167.180.112.24
using SIP default port
m=audio 38060 RTP/AVP 0
506

Notes:  Alice specifies in

 HTTP message syntax header that SIP client
 sdp = session description protocol
sends, receives SIP
 Call-ID is unique for every call
Multmedia Networking
messages over UDP
7-105
 Name translation, user location
• caller wants to call • result can be based on:
callee, but only has – time of day (work,
callee’s name or e-mail home)
address. – caller (don’t want boss to
call you at home)
• need to get IP address of
– status of callee (calls sent
callee’s current host:
to voicemail when callee
– user moves around is already talking to
– DHCP protocol someone)
– user has different IP
devices (PC, smartphone,
car device)
Multmedia Networking
7-106
 SIP registrar
one function of SIP server: registrar
 when Bob starts SIP client, client sends SIP
REGISTER message to Bob’s registrar server

register message:
REGISTER sip:domain.com SIP/2.0
Via: SIP/2.0/UDP 193.64.210.89
From: sip:[email protected]
To: sip:[email protected]
Expires: 3600

Multmedia Networking
7-107
 SIP proxy
• another function of SIP server: proxy
• Alice sends invite message to her proxy server
– contains address sip:[email protected]
– proxy responsible for routing SIP messages to callee, possibly
through multiple proxies
• Bob sends response back through same set of SIP
proxies
• proxy returns Bob’s SIP response message to Alice
– contains Bob’s IP address
• SIP proxy analogous to local DNS server plus TCP setup

Multmedia Networking
7-108
 SIP example: [email protected] calls [email protected]
Poly SIP
2. UMass proxy forwards request registrar
to Poly registrar server
2 3 3. Poly server returns redirect response,
indicating that it should try [email protected]

UMass 4. Umass proxy forwards request Eurecom SIP

SIP proxy to Eurecom registrar server 4 registrar
7
1. Jim sends INVITE 6-8. SIP response returned to Jim 5. eurecom
message to UMass 8 5 registrar
6 forwards INVITE
SIP proxy. 1
to 197.87.54.21,
which is running
keith’s SIP client
9
128.119.40.186
9. Data flows between clients
197.87.54.21
Multmedia Networking
7-109
 Comparison with H.323

 H.323: another signaling  H.323 comes from the

protocol for real-time, ITU (telephony)
interactive multimedia  SIP comes from IETF:
 H.323: complete, vertically borrows much of its
integrated suite of protocols
concepts from HTTP
for multimedia conferencing:
signaling, registration,  SIP has Web flavor;
admission control, transport, H.323 has telephony
codecs flavor
 SIP: single component.  SIP uses KISS principle:
Works with RTP, but does Keep It Simple Stupid
not mandate it. Can be
combined with other
protocols, services Multmedia Networking
7-110
Network support for multimedia

Multmedia Networking
7-111
 Dimensioning best effort networks

• approach: deploy enough link capacity so that

congestion doesn’t occur, multimedia traffic flows
without delay or loss
– low complexity of network mechanisms (use current “best
effort” network)
– high bandwidth costs
• challenges:
– network dimensioning: how much bandwidth is “enough?”
– estimating network traffic demand: needed to determine
how much bandwidth is “enough” (for that much traffic)
Multmedia Networking
7-112
Providing multiple classes of service
thus far: making the best of best effort service
 one-size fits all service model
alternative: multiple classes of service
 partition traffic into classes
 network treats different classes of traffic differently
 granularity: differential
(analogy: VIP service versus regular service)
service among multiple
classes, not among 0111

individual connections
 history: ToS bits

Multmedia Networking
7-113
Multiple classes of service: scenario

H3
H1
R1 R2

H4
H2 R1 output 1.5 Mbps link
interface
queue

Multmedia Networking
7-114
 Scenario 1: mixed HTTP and VoIP
example: 1Mbps VoIP, HTTP share 1.5 Mbps
link.
 HTTP bursts can congest router, cause audio loss
 want to give priority
R1 to audio over
R2
HTTP

Principle 1
packet marking needed for router to distinguish
between different classes; and new router policy to
treat packets accordingly
Multmedia Networking
7-115
 Principles for QOS guarantees (more)
what if applications misbehave (VoIP sends higher than
declared rate)
 policing: force source adherence to bandwidth allocations
marking, policing at network edge
1 Mbps
phone R1 R2

1.5 Mbps link

packet marking and policing

Principle 2
provide protection (isolation) for one class from others
Multmedia Networking
7-116
 Principles for QOS guarantees (more)
• allocating fixed (non-sharable) bandwidth to flow:
inefficient use of bandwidth if flows doesn’t use its
allocation

1 Mbps 1 Mbps logical link

R1
phone R2

1.5 Mbps link

0.5 Mbps logical link

Principle 3
while providing isolation, it is desirable to use
resources Multmedia
as efficiently
Networking as possible
7-117
Scheduling and policing mechanisms
scheduling: choose next packet to send on link
FIFO (first in first out) scheduling: send in order of
arrival to queue
 real-world example?
 discard policy: if packet arrives to full queue: who to
discard?
• tail drop: drop arriving packet
• priority: drop/remove on priority basis
• random: drop/remove randomly

packet packet
arrivals queue link departures
(waiting area) (server)
Multmedia Networking
7-118
 Scheduling policies: priority
high priority queue
priority scheduling: (waiting area)

send highest priority arrivals departures

queued packet
classify link
multiple classes, with low priority queue
(server)
(waiting area)
different priorities
2
 class may depend on 1 3 4 5
arrivals
marking or other
header info, e.g. IP packet
in 1 3 2 4 5
service
source/dest, port
numbers, etc. departures
1 3 2 4 5
 real world example?
Multmedia Networking
7-119
 Scheduling policies: still more
Round Robin (RR) scheduling:
multiple classes
cyclically scan class queues, sending one complete packet from each class (if available)
real world example?

2
1 3 4 5
arrivals

packet
in 1 3 2 4 5
service

departures
1 3 3 4 5

Multmedia Networking
7-120
 Scheduling policies: still more
Weighted Fair Queuing (WFQ):
generalized Round Robin
each class gets weighted amount of service
in each cycle
real-world example?

Multmedia Networking
7-121
 Policing mechanisms
goal: limit traffic to not exceed declared parameters
Three common-used criteria:
(long term) average rate: how many pkts can be sent per unit time (in the long
run)
 crucial question: what is the interval length: 100 packets per sec or 6000 packets per min
have same average!
peak rate: e.g., 6000 pkts per min (ppm) avg.; 1500 ppm peak rate
(max.) burst size: max number of pkts sent consecutively (with no intervening
idle)

Multmedia Networking
7-122
 Policing mechanisms:
implementation
token bucket: limit input to specified burst size and
average rate

bucket can hold b tokens

tokens generated at rate r token/sec unless bucket full
over interval of length t: number of packets admitted
less than or equal to Multmedia
(r t +Networking
b) 7-123
 Policing and QoS guarantees
token bucket, WFQ combine to provide
guaranteed upper bound on delay, i.e., QoS
guarantee!
arriving token rate, r
traffic
bucket size, b
per-flow
rate, R
WFQ

D = b/R
arriving max
traffic
Multmedia Networking
7-124
 Differentiated services
• want “qualitative” service classes
– “behaves like a wire”
– relative service distinction: Platinum, Gold, Silver
• scalability: simple functions in network core, relatively complex
functions at edge routers (or hosts)
– signaling, maintaining per-flow router state difficult with large number of
flows
• don’t define define service classes, provide functional components
to build service classes

Multmedia Networking
7-125
 Diffserv architecture
marking
r
edge router:
b scheduling
 per-flow traffic management
 marks packets as in-profile and
out-profile ..
.

core router:
 per class traffic management
 buffering and scheduling based
on marking at edge
 preference given to in-profile
packets over out-of-profile
packets Multmedia Networking
7-126
 Edge-router packet marking
 profile: pre-negotiated rate r, bucket size b
 packet marking at edge based on per-flow profile

rate r

user packets

possible use of marking:

 class-based marking: packets of different classes marked
differently
 intra-class marking: conforming portion of flow marked
differently than non-conforming one
Multmedia Networking
5-127
 Diffserv packet marking: details
packet is marked in the Type of Service (TOS) in
IPv4, and Traffic Class in IPv6
6 bits used for Differentiated Service Code Point
(DSCP)
 determine PHB that the packet will receive
 2 bits currently unused
DSCP unused

Multmedia Networking
7-128
 Classification, conditioning
may be desirable to limit traffic injection rate
of some class:
user declares traffic profile (e.g., rate, burst
size)
traffic metered, shaped if non-conforming

Multmedia Networking
7-129
Forwarding Per-hop Behavior (PHB)
PHB result in a different observable
(measurable) forwarding performance behavior
PHB does not specify what mechanisms to use
to ensure required PHB performance behavior
examples:
 class A gets x% of outgoing link bandwidth over
time intervals of a specified length
 class A packets leave first before packets from class
B

Multmedia Networking
7-130
 Forwarding PHB

PHBs proposed:
expedited forwarding: pkt departure rate of
a class equals or exceeds specified rate
 logical link with a minimum guaranteed rate
assured forwarding: 4 classes of traffic
 each guaranteed minimum amount of
bandwidth
 each with three drop preference partitions

Multmedia Networking
7-131
Per-connection QOS guarantees
basic fact of life: can not support traffic demands
beyond link capacity

1 Mbps R1
phone R2

1 Mbps 1.5 Mbps link

phone

Principle 4
call admission: flow declares its needs, network may
block call (e.g., busy signal) if it cannot meet needs
Multmedia Networking
7-132
 QoS guarantee scenario
 resource reservation
 call setup, signaling (RSVP)
 traffic, QoS declaration
 per-element admission control

request/
reply

 QoS-sensitive scheduling
(e.g., WFQ)

Multmedia Networking
7-133