UNIT II -MESSAGE ORDERING & SNAPSHOTS

Chapter 6
Message ordering and group communication: Message ordering paradigms –
Asynchronous execution with synchronous communication –Synchronous program
order on an asynchronous system –Group communication – Causal order (CO) - Total
order.
 Message ordering paradigms
• Notations
• ) Network (N, L); event set (E, ≺)
• ) message mi : send and receive events si and r i
• ) send and receive events: s and r .
• ) M, send (M), and receive(M)
• ) Corresponding events: a ∼ b denotes a and b occur at the same process
• ) send-receive pairs T = {(s, r ) ∈ Ei × Ej | s corresponds to r }
 Message ordering paradigms

• The order of delivery of messages in a distributed system is an important

aspect of system executions

• because it determines the messaging behavior that can be expected by the

distributed program.

• Distributed program logic greatly depends on this order of delivery.

 Message ordering paradigms

• The programmer can then code the program logic with respect to this

behaviour

• Several orderings on messages have been defined: (i) non-FIFO, (ii)

FIFO,(iii) causal order, and (iv) synchronous order.

 Message ordering paradigms

Asynchronous executions

• Definition : (A-execution) An asynchronous execution (or A-execution) is an

execution E≺ for which the causality relation is a partial order

Message ordering paradigms
 Message ordering paradigms
• FIFO executions

• On any logical link in the system, messages are necessarily delivered in the

order in which they are sent.

• Although the logical link is inherently non- FIFO, most network protocols

provide a connection-oriented service at the transport layer.

 Message ordering paradigms

• Therefore, FIFO logical channels can be realistically assumed

• when designing distributed algorithms. A simple algorithm to
implement a
• FIFO logical channel over a non-FIFO channel would use a separate
numbering
 Message ordering paradigms

• scheme to sequence the messages on each logical channel. The

sender

• assigns and appends a sequence_num, connection_id tuple to each

message.

• The receiver uses a buffer to order the incoming messages as per the

• sender’s sequence numbers, and accepts only the “next” message in

sequence.
 Message ordering paradigms
Causally ordered (CO) executions
Message ordering paradigms
Message ordering paradigms
Synchronous execution (SYNC)or S exec
• When all the communication between pairs of processes uses
synchronous send and receive primitives, the resulting order is the
synchronous order.
• As each synchronous communication involves a handshake between the
receiver and the sender, the corresponding send and receive events can be
viewed as occuring instantaneously and atomically.
• In a timing diagram, the “instantaneous” message communication can be
shown by bidirectional vertical message lines.
• Figure 6.3(a) shows a synchronous execution on an asynchronous system.
• Figure 6.3(b) shows the equivalent timing diagram with the
corresponding instantaneous message communication.
Asynchronous execution with synchronous communication
• Executions realizable with synchronous communication (RSC)
• Hierarchy of ordering paradigms
• Simulations
 Asynchronous execution with synchronous communication

• When all the communication between pairs of processes is by using

synchronous send and receive primitives, then resulting order is synchronous
order.
• The send and receive events of a message appear instantaneous(immediately),
see the example in Figure 6.3. We now address the following question:
• If a program is written for an asynchronous system, say a FIFO system,
will it still execute correctly if the communication is done by
synchronous primitives instead?
There is a possibility that the program may lead to deadlock

• Charron-Bost et al. [7] observed that a distributed algorithm designed

to run correctly on asynchronous systems (called A-executions) may
not run correctly on synchronous systems (called S-executions).
• An algorithm that runs on an asynchronous system may be a deadlock
on a synchronous system.
Examples
• Figure 6.4 shows the asynchronous execution
• illustrated this code(fig 6.4) in Figure 6.5(a) using a timing diagram,
will lead to deadlock if run with synchronous primitives.
• The executions in Figure 6.5(b)–(c) will also be a deadlock when run
on a synchronous system.
 Executions realizable with synchronous communication (RSC)
• In an A-execution, the messages can be made to appear instantaneous if
there exists a linear extension of the execution, such that each send event is
immediately followed by its corresponding receive event in this linear
extension. (send immediately followed by receive event)
• Such an A-execution can be realized under synchronous communication and
is called a Realizable with Synchronous Communication (RSC) execution.
• Definition 6.11 (RSC execution): An A-execution E≺ is an RSC
execution if and only if there exists a non-separated linear extension
of the partial order E≺.
• In the non-separated linear extension, if the adjacent send event and
its corresponding receive event are viewed atomically(indivisible),
then that pair of events shares a common past and a common future
with each other.
• Crown: study the characterization of the execution in terms of a
graph structure called a crown; the crown leads to a feasible test for a
RSC execution
Crown: Definition
Crown
Let E be an execution. A crown of size k in E is a sequence ( (si ,r i ), i ∈ { 0, . . ., k-1 }
) of pairs of corresponding send and receive events such that:
s0 ≺ r 1, s1 ≺ r 2, ... ... sk−2 ≺ r k−1 , sk−1 ≺ r 0.

s1 r2 r3 r2 s3 r2
P1
m3
m1 m3
m2 s1s3 m2 r3 m2
P2 2
s r1 m1 s1
m1
P3
s2 r1 s2 r1
(a) (b) (c)
Figure 6.5: Illustration of non-RSC A-executions and crowns.
Fig 6.5(a): crown is ((s1, r 1), (s2, r 2 )) as we have s1 ≺ r 2 and s2 ≺ r1
Fig 6.5(b) (b) crown is ((s1, r 1), (s2, r 2 )) as we have s1 ≺ r 2 and s2 ≺ r 1
Fig 6.5(c): crown is ((s1, r 1), (s3, r 3), (s2, r 2 )) as we have s1 ≺ r 3 and s3 ≺ r 2 and s2 ≺ r 1
Fig 6.2(a): crown is ((s1, r 1), (s2, r 2), (s3, r 3 )) as we have s1 ≺ r 2 and s2 ≺ r 3 and s3 ≺ r 1.

Message Ordering and Group Commnication 30 /

Crown: Characterization of RSC Executions
s1 r2 r3 r2 s3 r2
P1
m3
m1 m3
m2 s1s3 m2 r3 m 2
P
2 s2 r1 m1 s1
m1
P3
s2 r1 s2 r1
(a) (b) (c)
Figure 6.5: Illustration of non-RSC A-executions and crowns.
Fig 6.5(a): crown is ((s1, r 1), (s2, r 2 )) as we have s1 ≺ r 2 and s2 ≺ r1
Fig 6.5(b) (b) crown is ((s1, r 1), (s2, r 2 )) as we have s1 ≺ r 2 and s2 ≺ r 1
Fig 6.5(c): crown is ((s1, r 1), (s3, r 3), (s2, r 2 )) as we have s1 ≺ r 3 and s3 ≺ r 2 and s2 ≺ r 1
Fig 6.2(a): crown is ((s1, r 1), (s2, r 2), (s3, r 3 )) as we have s1 ≺ r 2 and s2 ≺ r 3 and s3 ≺ r 1.
Some observations
In a crown, si and r i +1
may or may not be on same process
Non-CO execution must have a crown
CO executions (that are not synchronous) have a crown (see Fig 6.2(b)) Cyclic
dependencies of crown ⇒ cannot schedule messages serially ⇒ not RSC

Message Ordering and Group Commnication 31 /

Crown Test for RSC executions
1 Define the ‹→: T × T relation on messages in the execution (E, ≺) as follows. Let
‹→ ([s, r ], [sj, r j ]) iff s ≺ r j . Observe that the condition s ≺ r j (which has the form used
in the definition of a crown) is implied by all the four conditions: (i) s ≺ s j, or (ii)
s ≺ r j , or (iii) r ≺ s j, and (iv) r ≺ rj .
2 Now define a directed graph G‹→ = (T , ‹→), where the vertex set is the set of messages
T and the edge set is defined by ‹→.
Observe that ‹→: T × T is a partial order iff G‹→ has no cycle, i.e., there must not be a
cycle with respect to ‹→ on the set of corresponding (s, r ) events.
3 Observe from the defn. of a crown that G‹→ has a directed cycle iff (E, ≺) has a crown.

Crown criterion
An A-computation is RSC, i.e., it can be realized on a system with synchronous communication, iff
it contains no crown.
Crown test complexity: O(|E |) (actually, # communication events)

Timestamps for a RSC execution

Execution (E, ≺) is RSC iff there exists a mapping from E to T (scalar timestamps) such that
for any message M, T (s(M)) = T (r(M))
for each (a, b) in (E × E ) \ T , a ≺ b =⇒ T (a) < T (b)
Message Ordering and Group Commnication 15 / 52
Hierarchy of Message Ordering Paradigms
FIFO CO A

SYNC

CO

A
FIFO
SYNC

(a) (b)

Figure 6.7: Hierarchy of message ordering paradigms. (a) Venn diagram (b) Example executions.

An A-execution is RSC iff A is an S-execution.

RSC ⊂ CO ⊂ F I F O ⊂ A.
More restrictions on the possible message orderings in the smaller classes. The
degree of concurrency is most in A, least in S Y N C.
A program using synchronous communication easiest to develop and verify.
A program using non-FIFO communication, resulting in an A-execution, hardest to
design and verify.

Message Ordering and Group Commnication 33 /

Simulations: Async Programs on Sync Systems
Pi
m
RSC execution: schedule events as per a Pi,j
non-separated linear extension m’
adjacent (s, r ) events sequentially
m
Pj,i
partial order of original A-execution
unchanged m’
P
If A-execution is not RSC: j
partial order has to be changed; or Figure 6.8: Modeling channels as processes to
model each Ci,j by control process Pi,j simulate an execution using asynchronous
and use sync communication (see Fig primitives on an synchronous system.
6.8)
Enables decoupling of sender from
receiver.
This implementation is expensive.

Message Ordering and Group Commnication 34 /

Simulations: Synch Programs on Async Systems

Schedule msgs in the order in which they appear in S-program

partial order of S-execution unchanged
Communication on async system with async primitives
When sync send is scheduled:
) wait for ack before completion

Message Ordering and Group Commnication 35 /

Synchronous program order on an asynchronous system

• Rendezvous
• Algorithm for binary rendezvous
Synchronous program order on an asynchronous system

• Implementation of synchronous communication is not realized

naturally, so implemented through asynchronous.
• Distributed programs are deterministic and Non deteministic
• Deterministic, i.e., repeated runs of the same program will produce the
same partial order.
• Deterministic receive ⇒ deterministic execution ⇒ (E, ≺) is fixed
• Nondeterminism (besides due to unpredictable message delays):
→Receive call does not specify sender
→Multiple sends and receives enabled at a process; can be
executed in interchangeable order
• If i sends to j, and j sends to i concurrently using blocking synchronous calls,
there results a deadlock, similar to the one in Figure 6.4.
• If the receive call at one of the processes can be scheduled before the send
call, then there is no deadlock.
• In this section, we consider scheduling synchronous communication events
(over an asynchronous system).
• How to schedule (nondeterministic) sync communication calls over async
system?
• Match send or receive with corresponding event
 Rendezvous(meet at an agreed time and place)
• One form of group communication is called multiway rendezvous, which is a
synchronous communication among an arbitrary number of asynchronous
processes.
• All the processes involved “meet with each other,” i.e., communicate
“synchronously” with each other at one time.
• Here,rendezvous between a pair of processes at a time, which is called binary
rendezvous as opposed to the multiway rendezvous.
• Binary rendezvous communication was supported by such as CSP and Ada.
• In these languages, the repetitive command (the ∗ operator) over the
alternative command (the ||operator) on multiple guarded commands (each
having the form Gi → CLi) is used, as follows:
∗[G1 → CL1 || G2 −→ CL2 || · · · || Gk → CLk ]
• Each communication command may be a part of a guard Gi, and may also appear
within the statement block CLi.
• A guard Gi is a boolean expression.
• If a guard Gi evaluates to true then CLi is said to be enabled, otherwise CLi is said to
be disabled.
• A send command of local variable x to process Pk is denoted as “x !Pk.”
• A receive from process Pk into local variable x is denoted as “Pk ? x.”
• Some typical observations about synchronous communication under binary
rendezvous are as follows:
• For the receive command, the sender must be specified
• Send and received commands may be individually disabled or enabled.
• A command is disabled if it is guarded and the guard evaluates to false.
• Synchronous communication is implemented by scheduling messages
(asynchronous communication).
• Scheduling involves pairing of matching send and receive commands that are both
enabled.
Assumptions:
• Binary rendezvous explicitly assumes that multiple send and receives are
enabled.
• Any send or receive event that can be “matched” with the corresponding
receive or send event can be scheduled.
• This is dynamically scheduling the ordering of events and the partial order
of the execution.
 Algorithm for binary rendezvous
Binary rendezvous (implementation using tokens)
• Token for each enabled interaction
• multiple interactions are enabled, a process chooses one of them and tries to “synchronize”
with the partner process.
Common features:
• Schedule online, atomically, in a distributed manner
• the scheduling code at any process does not know the application code of other processes.
• Schedule in a deadlock-free manner(ie Crown-free scheduling) (safety);
• sender and receiver are enabled for a message when it is scheduled
• Schedule to satisfy the progress property (i.e., find a schedule within a bounded
number of steps) in addition to the safety (i.e., correctness) property
Additional features of a good algorithm are:
(i) symmetry or some form of fairness, i.e., not favoring particular
processes over others during scheduling,
(ii) efficiency, i.e., using as few messages as possible, and involving
as low a time overhead as possible.
 Bagrodia’s Algorithm for Binary Rendezvous
Assumptions:
• Receives are always enabled
• Send, once enabled, remains enabled
• To break deadlock, PIDs used to introduce asymmetry
• Each process schedules one send at a time
• The algorithm illustrates how crown-free message scheduling is achieved on-line.

• Message types: M, ack(M), request(M), permission(M)

• Process blocks when it knows it can successfully synchronize the current
message with the partner process
• Each process maintains a queue that is processed in FIFO order, only
when the process is unblocked.
• When a process is blocked waiting for a particular message that it is
currently synchronizing, any other message that arrives is queued up.
The algorithm to enforce synchronous order is given in Algorithm 6.1.
• The key rules to prevent cycles among the messages:
• 1. To send to a lower priority process, messages M and ack(M) are involved in
that order. The sender issues send(M) and blocks until ack(M) arrives.
• Thus, when sending to a lower priority process, the sender blocks waiting for
the partner process to synchronize and send an acknowledgement.
• To send to a higher priority process, messages request(M), permission(M),
and M are involved, in that order.
• The sender issues send(request(M)), does not block, and awaits permission.
When permission(M) arrives, the sender issues send(M).
Bagrodia’s Algorithm for Binary Rendezvous: Code
(message types)
M, ack(M), request(M), permission(M)

1 P i wants to execute SEND(M) to a lower priority process Pj :

P i executes send(M) and blocks until it receives ack(M) from Pj . The send event SEND(M) now completes.
Any M’ message (from a higher priority processes) and request(M’) request for synchronization (from a lower priority processes) received
during the blocking period are queued.

2 P i wants to execute SEND(M) to a higher priority process Pj :

1 Pi seeks permission from Pj by executing send(request(M)).

/ / to avoid deadlock in which cyclically blocked processes queue messages.

2 While Pi is waiting for permission, it remains unblocked.

1 If a message M r arrives from a higher priority process Pk , Pi accepts M r by scheduling a RECEIVE(M’) event and then
executes send(ack(M’)) to Pk .

2 If a request(M’) arrives from a lower priority process Pk , Pi executes send(permission(M’)) to Pk and blocks waiting for the
message M r . When M r arrives, the RECEIVE(M’) event is executed.

3 When the permission(M) arrives, Pi knows partner Pj is synchronized and Pi executes send(M). The SEND(M) now completes.

3 Request(M) arrival at Pi from a lower priority process Pj :

At the time a request(M) is processed by P i , process P i executes send(permission(M)) to Pj and blocks waiting for the message M. When
M arrives, the RECEIVE(M) event is executed and the processunblocks.
4 Message M arrival at P i from a higher priority process Pj :
At the time a message M is processed by P i , process P i executes RECEIVE(M) (which is assumed to be always enabled) and then
send(ack(M)) to Pj .

5 Processing when Pi is unblocked:

When P i is unblocked, it dequeues the next (if any) message from the queue and processes it as a message arrival (as per Rules 3 or 4).

Message Ordering and Group Commnication 47 /

Group communication
 Causal order (CO)
• The Raynal–Schiper–Toueg algorithm
 Total order
• Centralized algorithm for total order
• Three-phase distributed algorithm