Multicast Communication

Introduction to multicast
Multicast communication requires coordination and agreement. The aim is for members of a group to receive copies of messages sent to the group Many different delivery guarantees are possible
e.g. agree on the set of messages received or on delivery ordering

A process can multicast by the use of a single operation instead of a send to each member
For example in IP multicast aSocket.send(aMessage) The single operation allows for:
efficiency delivery guarantees

System model
The system consists of a collection of processes which can communicate reliably over 1-1 channels Processes fail only by crashing (no arbitrary failures) Processes are members of groups - which are the destinations of multicast messages In general process p can belong to more than one group Operations
multicast(g, m) sends message m to all members of process group g deliver (m) is called to get a multicast message delivered. It is different from receive as it may be delayed to allow for ordering or reliability.

Multicast message m carries the id of the sending process sender(m) and the id of the destination group group(m) We assume there is no falsification of the origin and destination of messages

Open and closed groups

Closed groups only members can send to group, a member delivers to itself they are useful for coordination of groups of cooperating servers Open they are useful for notification of events to groups of interested processes

Clos ed group

Open group

Reliability of one-to-one communication

The term reliable 1-1 communication is defined in terms of validity and integrity as follows: validity:
any message in the outgoing message buffer is eventually delivered to the incoming message buffer integrity: the message received is identical to one sent, and no messages are delivered twice validity - by use of acknowledgements and retries integrity by use checksums, reject duplicates (e.g. due to retries) If allowing for malicious users, use security techniques

Basic multicast
A correct process will eventually deliver the message provided, the multicaster does not crash The primitives are called B-multicast and B-deliver A straightforward but ineffective method of implementation:
use a reliable 1-1 send (i.e. with integrity and validity as above) To B-multicast(g,m): for each process p g, send(p, m); On receive (m) at p: B-deliver (m) at p

Problem
if the number of processes is large, the protocol will suffer from the ack-implosion

Implementation of basic multicast over IP

Each process p maintains:
a sequence number, Spg for each group it belongs to and Rqg, the sequence number of the latest message it has delivered from process q

For process p to B-multicast a message m to group g

it piggybacks Spg on the message m, using IP multicast to send it the piggybacked sequence numbers allow recipients to learn about messages they have not received

On receive (g,m, S) at p:
if S = Rqg +1 B-deliver (m) and increment Rqg by 1 if S < Rqg +1 reject the message because it has been delivered before if S > Rqg +1 note that a message is missing, request missing message from sender. (will use a hold-back queue)

If the sender crashes, then a message may be delivered to some members of the group but not all.

Reliable multicast
The protocol is correct even if the multicaster crashes it satisfies criteria for validity, integrity and agreement it provides operations R-multicast and R-deliver Integrity - a correct process, p delivers m at most once. Also p group(m) and m was supplied to a multicast operation by sender(m) Validity - if a correct process multicasts m, it will eventually deliver m Agreement - if a correct process delivers m then all correct processes in group(m) will eventually deliver m
integrity as for 1-1 communication

validity - simplify by choosing sender as the one process

agreement agree all or nothing - atomicity, even if multicaster crashes

Reliable multicast algorithm the message to Agreement - every correct process B-multicastsover basic the others. If pIntegrity--a correct process will 1-1 channels used for B-multicast does not R-deliver Validity because then this is because to itself multicast the reliable B-deliver it didnt B-deliver - because no guarantee integrity others did either.
processes can belong to several closed groups to R-multicast a message, a process B-multicasts it to
processes in the group including itself

when a message is B-delivered, the recipient B-multicasts it to the group, then R-delivers it. Duplicates are detected.

primitives R-multicast and R-deliver

Reliable multicast can be implemented efficiently over IP multicast by holding back messages until every member can receive them.

Reliable multicast over IP multicast

This protocol assumes groups are closed. It uses:
piggybacked acknowledgement messages negative acknowledgements when messages are missed

Process p maintains: For process p to R-multicast message m to group g

IP multicasts the message to g, increments Spg by 1

the piggybacked values in a messagefor each group it to learn about Spg a message sequence number allow recipients belongs to and messages they have not yet received Rqg sequence number of latest message received from process q to g
piggyback Spg and +ve acks for messages received in the form

<q, Rqg >

A process on receipt by of a message to g with S from p

Iff S=Rpg+1 R-deliver the message and increment Rpg by 1 If S Rpg discard the message If S> Rpg + 1 or if R<Rqg (for enclosed ack <q,R>)
then it has missed messages and requests them with negative acknowledgements puts new message in hold-back queue for later delivery

The hold-back queue for arriving multicast messages

Simplifies the protocol, allowing sequence numbers to represent sets of messages. Hold-back queues are also used for ordering protocols.

Message proc es sing deliv er Hold-bac k queue Deliv ery queue When deliv ery guarantees are met

Incoming mes sages

Reliability properties of reliable multicast over IP

Integrity - duplicate messages detected and rejected. IP multicast uses checksums to reject corrupt messages Validity - due to IP multicast in which sender delivers to itself Agreement - processes can detect missing messages. They must keep copies of messages they have delivered so that they can re-transmit them to others. discarding of copies of messages that are no longer needed :
when piggybacked acknowledgements arrive, note which processes have received messages. When all processes in g have the message, discard it.

Ordered multicast
The basic multicast algorithm delivers messages to processes in an arbitrary order. A variety of orderings may be implemented FIFO ordering
If a correct process issues multicast(g, m) and then multicast(g,m ), then every correct process that delivers m will deliver m before m .

Causal ordering
If multicast(g, m) multicast(g,m ), where is the happened-before relation between messages in group g, then any correct process that delivers m will deliver m before m .

Total ordering
If a correct process delivers message m before it delivers m, then any other correct process that delivers m will deliver m before m.

Ordering is expensive in delivery latency and bandwidth consumption

Display from a bulletin board program

Users run bulletin board applications which multicast messages One multicast group per topic (e.g. os.interesting) Require reliable multicast - so that all members receive messages Ordering:

total (makes the numbers the same at all sites)

Bulletin board:os.interesting Item 23 24 25 26 27 end From A.Hanlon G.Joseph A.Hanlon T.LHeureux M.Walker Subject Mach Microkernels Re: Microkernels RPC performance Re: Mach

causal (makes replies come after original message)

FIFO (gives sender order

Implementation of FIFO ordering over basic multicast

FIFO ordered multicast with the operations FO-multicast and FO-deliver. It can be implemented on top of any basic multicast Each process p holds:
Spg a count of messages sent by p to g and Rqg the sequence number of the latest message to g that p delivered from q

For p to FO-multicast a message to g, it piggybacks Spg on the message, B-multicasts it and increments Spg by 1 On receipt of a message from q with sequence number S, p checks whether S = Rqg + 1. If so, it FO-delivers it. if S > Rqg + 1 then p places message in hold-back queue until intervening messages have been delivered. (note that Bmulticast does eventually deliver messages unless the sender crashes)

Implementation of totally ordered multicast

The general approach is to attach totally ordered identifiers to multicast messages
each receiving process makes ordering decisions based on the identifiers similar to the FIFO algorithm, but processes keep group specific sequence numbers operations TO-multicast and TO-deliver

we present two approaches to implementing total ordered multicast over basic multicast
1. using a sequencer (only for non-overlapping groups) 2. the processes in a group collectively agree on a sequence number for each message

Total ordering using a sequencer

A process wishing to TO-multicast m to g attaches a unique id, id(m) and sends it to the sequencer and the members.

Other processes: B-deliver <m,i>

put <m,i> in hold-back queue

B-deliver order message, get g and S and i from order message wait till <m,i> in queue and S = rg, TO-deliver m and set rg to S+1 The sequencer keeps sequence number sg for group g When it B-delivers the message it multicasts an order message to members of g and increments sg.

The ISIS algorithm for total ordering

this protocol is for open or closed groups
P2 1 Message
1. the process P1 B-multicasts a message to members of the group

3 2 2 1 P4

2. the receiving processes propose numbers and return them to the sender

3 Agreed Seq 1
3. the sender uses the proposed numbers to generate an agreed number

2 3 P3

P1

ISIS total ordering - agreement of sequence numbers

Each process, q keeps:
Aqg the largest agreed sequence number it has seen and Pqg its own largest proposed sequence number

1. Process p B-multicasts <m, i> to g, where i is a unique identifier for m. 2. Each process q replies to the sender p with a proposal for the messages agreed sequence number of
Pqg := Max(Aqg, Pqg ) + 1. assigns the proposed sequence number to the message and places it in its hold-back queue

3. p collects all the proposed sequence numbers and selects the largest as the next agreed sequence number, a. It B-multicasts <i, a> to g. Recipients set Aqg := Max(Aqg, a ), attach a to the message and re-order hold-back queue.

Causally ordered multicast

We present an algorithm of Birman 1991 for causally ordered multicast in non-overlapping, closed groups. It uses the happened before relation (on multicast messages only)
that is, ordering imposed by one-to-one messages is not taken into account

It uses vector timestamps - that count the number of multicast messages from each process that happened before the next message to be multicast

Causal ordering using vector timestamps

each process has its own vector timestamp

To CO-multicast m to g, a process adds 1 to its entry in the vector timestamp and B-multicasts m and the vector timestamp When a process B-delivers m, it places it in a hold-back queue until messages earlier in the causal ordering have been delivered:a) earlier messages from same sender have been delivered b) any messages that the sender had delivered when it sent the multicast message have been delivered

Note: a process can immediately CO-deliver to itself its own messages (not shown)

then it CO-delivers the message and updates its timestamp

Summary
Multicast communication can specify requirements for reliability and ordering, in terms of integrity, validity and agreement B-multicast a correct process will eventually deliver a message provided the multicaster does not crash reliable multicast in which the correct processes agree on the set of messages to be delivered; we showed two implementations: over B-multicast and IP multicast delivery ordering FIFO, total and causal delivery ordering. FIFO ordering by means of senders sequence numbers total ordering by means of a sequencer or by agreement of sequence numbers between processes in a group causal ordering by means of vector timestamps the hold-back queue is a useful component in implementing multicast protocols