UNITIV CONSENSUS AND RECOVERY

Consensus and Agreement Algorithms: Problem Definition–Overview of Results–

Agreementina Failure-Free System(Synchronous and Asynchronous) – Agreement in
Synchronous Systems with Failures; Checkpointing and Rollback Recovery: Introduction–
Background and Definitions–Issues in Failure Recovery– Checkpoint-based Recovery–
Coordinated Checkpointing Algorithm-Algorithm for Asynchronous Checkpointing and
Recovery.

TEXTBOOKS

1.Kshemkalyani Ajay D, Mukesh Singhal, “Distributed Computing: Principles, Algorithms and

Systems”, Cambridge Press,2011.
 Consensus and agreement algorithms

I. Problem definition

Consensus in Distributed computing

Consensus

Agreeing that something is truth

 Distributed computing

 A network of nodes operating together.

Failure Modes

 Fail-Stop= a node dies

 Fail-recover= a node dies and come back later

 Byzantine= a node misbehaves

Consensus Overview

Achieving Consensus= distributed system acting as one entity.

Consensus Problem= getting nodes in a distributed system to agree on something(value,
operation, etc.)

 Agreement among the processes in a distributed system is a fundamental requirement for a

wide range of applications.

 Many forms of coordination require the processes to exchange information to negotiate with
one another and eventually reach a common understanding or agreement, before taking
application-specific actions.
Some assumptions underlying study of agreement algorithms:

1.Failure models

2. Synchronous/ Asynchronous communication

3. Network connectivity.

4.Sender identification.

5.Channel reliability.

6.Authenticated vs. non-authenticated messages.

7.Agreement variable.
The Byzantine agreement and other problems

Byzantine Agreement (single source has an initial value)

Agreement: All non-faulty processes must agree on the same value.

Validity: If the source process is non-faulty, then the agreed upon value by all the non-faulty

processes must be the same as the initial value of the source.

Termination: Each non-faulty process must eventually decide on a value.

Consensus Problem (all processes have an initial value)

Agreement: All non-faulty processes must agree on the same (single) value.

Validity: If all the non-faulty processes have the same initial value, then the agreed upon value by
all the non-faulty processes must be that same value.

Termination: Each non-faulty process must eventually decide on a value.

Interactive Consistency (all processes have an initial value)

Agreement: All non-faulty processes must agree on the same array of values A[v1 : : : vn].

Validity: If process i is non-faulty and its initial value is vi , then all non-faulty processes agree on
vi as the ith element of the array A. If process j is faulty, then the non-faulty processes can agree
on any value for A[j ].

Termination: Each non-faulty process must eventually decide on the array A.

II. Overview of Results
Solvable Variants of the Consensus Problem in Asynchronous Systems.
G
 III. Agreement in a failure-free system (synchronous or asynchronous)

 In a failure-free system, consensus can be reached by collecting information from the different
processes, arriving at a “decision,” and distributing this decision in the system.

 A distributed mechanism would have each process broadcast its values to others, and each process
computes the same function on the values received.

 The decision can be reached by using an application specific function – some simple examples being
the majority, max, and min functions.

 Algorithms to collect the initial values and then distribute the decision may be based on the token
circulation on a logical ring, or the three-phase tree-based broadcast–convergecast–broadcast, or direct
communication with all nodes.

 In a synchronous system, this can be done simply in a constant number of rounds (depending on the
specific logical topology and algorithm used).

 In an asynchronous system, consensus can similarly be reached in a constant number of message hops.
IV. Agreement in (message-passing) synchronous systems with failures
Consensus algorithms for Byzantine failures (synchronous system)

Upper bound on Byzantine processes

In a system of n processes, the Byzantine agreement problem can be solved in a synchronous

system only if the number of Byzantine processes f is such that f ≤ [n−1/3].
We informally justify this result using two steps:

 With n = 3 processes, the Byzantine agreement problem cannot be solved if the number of
Byzantine processes f = 1.

 The argument uses the illustration in Figure 14.3, which shows a commander Pc and two
lieutenant

 processes Pa and Pb.

 The malicious process is the lieutenant Pb in the first scenario (Figure 14.3(a)) and hence Pa
should agree on the value of the loyal commander Pc, which is 0.

 But note the second scenario (Figure 14.3(b)) in which Pa receives identical values from Pb and
Pc, but now Pc is the disloyal commander whereas Pb is a loyal lieutenant.
 In this case, Pa needs to agree with Pb. However, Pa cannot distinguish between the two
scenarios and any further message exchange does not help because each process has already
conveyed what it knows from the third process.

 In both scenarios, Pa gets different values from the other two processes.

 In the first scenario, it needs to agree on a 0, and if that is the default value, the decision is
correct, but then if it is in the second indistinguishable scenario, it agrees on an incorrect value.

 A similar argument shows that if 1 is the default value, then in the first scenario, Pa makes an
incorrect decision. This shows the impossibility of agreement when n=3 and f =1.

Byzantine agreement tree algorithm: exponential (synchronous system)

Recursive formulation
We begin with an informal description of how agreement can be achieved with n = 4 and f = 1
processes [20, 25], as depicted in Figure 14.4.
 In the first round, the commander Pc sends its value to the other three lieutenants, as shown by dotted
arrows.

 In the second round, each lieutenant relays to the other two lieutenants, the value it received from the
commander in the first round.

 At the end of the second round, a lieutenant takes the majority of the values it received (i) directly from
the commander in the first round, and (ii) from the other two lieutenants in the second round.

 The majority gives a correct estimate of the “commander’s” value. Consider Figure 14.4(a) where the
commander is a traitor.

 The values that get transmitted in the two rounds are as shown. All three lieutenants take the majority of
(1, 0, 0) which is “0,” the agreement value.

 In Figure 14.4(b), lieutenant Pd is malicious. Despite its behavior as shown, lieutenants Pa and Pb agree
on “0,” the value of the commander.
F
Bzantine Generals (iterative formulation), Sync, Msg-passing
Phase-king algorithm for consensus: polynomial (synchronous system)

The phase-king algorithm proposed by Berman and Garay solves the consensus problem under the
same model, requiring f +1 phases, and a polynomial number of messages.

Operation

Each phase has a unique ”phase king” derived, say, from PID.

Each phase has two rounds:

1. In 1st round, each process sends its estimate to all other processes.

2. In 2nd round, the ”Phase king” process arrives at an estimate based on the values it received in
1st round, and broadcasts its new estimate to all others.
 T

Message pattern for the phase-king algorithm.

The Phase King Algorithm: Code
V. Checkpointing and Rollback Recovery

Introduction

• Rollback recovery protocols

– restore the system back to a consistent state after a failure
– achieve fault tolerance by periodically saving the state of a process during the failure-free execution
– treats a distributed system application as a collection of processes that communicate over a network

• Checkpoints - the saved states of a process

• Why is rollback recovery of distributed systems complicated?

– messages induce inter-process dependencies during failure-free operation

• Rollback propagation
– the dependencies may force some of the processes that did not fail to roll back
– This phenomenon is called “domino effect” 22
If each process takes its checkpoints independently, then the system can not avoid the domino
effect this scheme is called independent or uncoordinated checkpointing

Techniques that avoid domino effect

Coordinated checkpointing rollback recovery

processes coordinate their checkpoints to form a system-wide consistent state

Communication-induced checkpointing rollback recovery

forces each process to take checkpoints based on information piggybacked on the

application

Log-based rollback recovery

combines checkpointing with logging of non-deterministic events

relies on piecewise deterministic (PWD) assumption 23

 VI. Background and Definitions

A local checkpoint

 All processes save their local states at certain instants of time

 A local check point is a snapshot of the state of the process at a given instance

 Assumption

 A process stores all local checkpoints on the stable storage

 A process is able to roll back to any of its existing local checkpoints

 𝑪𝒊,𝒌

 The kth local checkpoint at process 𝑃𝑖

 𝑪𝒊,𝟎
24
 A process 𝑃𝑖 takes a checkpoint 𝐶𝑖,0 before it starts execution
Consistent states

 A global state of a distributed system

 a collection of the individual states of all participating processes and the states of the
communication channels

 Consistent global state

 a global state that may occur during a failure-free execution of distribution of distributed
computation

 if a process’s state reflects a message receipt, then the state of the corresponding sender
must reflect the sending of the message

 A global checkpoint

 a set of local checkpoints, one from each process 25

•A consistent global checkpoint

– a global checkpoint such that no message is sent by a process after taking its local point that
is received by another process before taking its checkpoint.

Consistent states - examples

Interactions with outside world

 A distributed system often interacts with the outside world to receive input data or deliver
the outcome of a computation

 Outside World Process (OWP)

 a special process that interacts with the rest of the system through message passing

 A common approach

 save each input message on the stable storage before allowing the application program to
process it

 Symbol “||”

 An interaction with the outside world to deliver the outcome of a computation

27
Messages

 In-transit message -messages that have been sent but not yet received

 Lost messages - messages whose ‘send’ is done but ‘receive’ is undone due to rollback

 Delayed messages

 messages whose ‘receive’ is not recorded because the receiving process was either down or
the message arrived after rollback

 Orphan messages

 messages with ‘receive’ recorded but message ‘send’ not recorded

 do not arise if processes roll back to a consistent global state

 Duplicate messages

 arise due to message logging and replaying during process recovery 28

Messages – example

• In-transit
– 𝑚1 , 𝑚2
• Lost
– 𝑚1
• Delayed
– 𝑚1 , 𝑚5
• Orphan
– none
• Duplicated
– 𝑚4 , 𝑚5

29
 VII.Issues in failure recovery

 Checkpoints : {𝐶𝑖,0 , 𝐶𝑖,1 }, {𝐶𝑗,0 , 𝐶𝑗,1 , 𝐶𝑗,2 }, and {𝐶𝑘,0 , 𝐶𝑘,1 , 𝐶𝑘,2 }

 Messages : A - J

 The restored global consistent state : {𝐶𝑖,1 , 𝐶𝑗,1 , 𝐶𝑘,1 }

30
• The rollback of process 𝑃𝑖 to checkpoint 𝐶𝑖,1 created an orphan message H

• Orphan message I is created due to the roll back of process 𝑃𝑗 to checkpoint 𝐶𝑗,1

• Messages C, D, E, and F are potentially problematic

– Message C: a delayed message

– Message D: a lost message since the send event for D is recorded in the restored state for
𝑃𝑗 , but the receive event has been undone at process 𝑃𝑖 .

– Lost messages can be handled by having processes keep a message log of all the sent
messages

– Messages E, F: delayed orphan messages. After resuming execution from their checkpoints,
processes will generate both of these messages

31
 VIII. Checkpoint-based recovery

 In the checkpoint-based recovery approach, the state of each process and the communication
channel is checkpointed frequently so that, upon a failure, the system can be restored to a
globally consistent set of checkpoints.

 It does not rely on the PWD assumption, and so does not need to detect, log, or replay non-
deterministic events.

 Checkpoint-based protocols are therefore less restrictive and simpler to implement than log-
based rollback recovery.

 However, checkpoint-based rollback recovery does not guarantee that prefailure execution can
be deterministically regenerated after a rollback.
Therefore, checkpoint-based rollback recovery may not be suitable for applications that require
frequent interactions with the outside world.

Checkpoint-based rollback-recovery techniques can be classified into three categories:

uncoordinated checkpointing, coordinated checkpointing, and communication-induced
checkpointing

Uncoordinated Checkpointing

•Each process has autonomy in deciding when to take checkpoints

•Advantages

– The lower runtime overhead during normal execution

•Disadvantages

– Domino effect during a recovery

– Recovery from a failure is slow because processes need to iterate to find a consistent set of
checkpoints

– Each process maintains multiple checkpoints and periodically invoke a garbage collection
algorithm

– Not suitable for application with frequent output commits

•The processes record the dependencies among their checkpoints caused by message exchange
during failure-free operation
Direct dependency tracking technique

• Assume each process 𝑃𝑖 starts its execution with an initial heckpoint 𝐶𝑖,0

• 𝐼𝑖,𝑥 : checkpoint interval, interval between 𝐶𝑖,𝑥−1 and 𝐶𝑖,𝑥

• When 𝑃𝑗 receives a message m during 𝐼𝑗,𝑦 , it records the dependency from 𝐼𝑖,𝑥 to 𝐼𝑗,𝑦 , which
is later saved onto stable storage when 𝑃𝑗 takes 𝐶𝑗,𝑦

35
Coordinated Checkpointing

• Blocking Checkpointing

– After a process takes a local checkpoint, to prevent orphan messages, it remains blocked
until the entire checkpointing activity is complete

– Disadvantages

• the computation is blocked during the checkpointing

• Non-blocking Checkpointing

– The processes need not stop their execution while taking checkpoints

– A fundamental problem in coordinated checkpointing is to prevent a process from

receiving application messages that could make the checkpoint inconsistent.

36
Coordinated Checkpointing

• Example (a) : checkpoint inconsistency

– message m is sent by 𝑃0 after receiving a checkpoint request from the checkpoint

coordinator

– Assume m reaches 𝑃1 before the checkpoint request

– This situation results in an inconsistent checkpoint since checkpoint 𝐶1,𝑥 shows the
receipt of message m from 𝑃0 , while checkpoint 𝐶0,𝑥 does not show m being sent from 𝑃0

• Example (b) : a solution with FIFO channels

– If channels are FIFO, this problem can be avoided by preceding the first post-checkpoint
message on each channel by a checkpoint request, forcing each process to take a
checkpoint before receiving the first post-checkpoint message.
37
Coordinated Checkpointing

38
Communication-induced Checkpointing
• Two types of checkpoints -autonomous and forced checkpoints

• Communication-induced checkpointing piggybacks protocol-related information on each

application message

• The receiver of each application message uses the piggybacked information to determine if it
has to take a forced checkpoint to advance the global recovery line

• The forced checkpoint must be taken before the application may process the contents of the
message

• In contrast with coordinated checkpointing, no special coordination messages are exchanged

• Two types of communication-induced checkpointing

– model-based checkpointing and index-based checkpointing.

39
 IX. Koo-Toueg coordinated checkpointing algorithm

• A coordinated checkpointing and recovery technique that takes a consistent set of

checkpointing and avoids domino effect and livelock problems during the recovery

• Includes 2 parts: the checkpointing algorithm and the recovery algorithm.

Checkpointing algorithm

• Assumptions: FIFO channel, end-to-end protocols, communication failures do not partition the
network, single process initiation, no process fails during the execution of the algorithm.

• Two kinds of checkpoints: permanent and tentative.

• Permanent checkpoint: local checkpoint, part of a consistent global checkpoint.

• Tentative checkpoint: temporary checkpoint, become permanent checkpoint when the algorithm
terminates successfully.
40
First Phase

 An initiating process Pi takes a tentative checkpoint and requests all other processes to take

tentative checkpoints.

 Each process informs Pi whether it succeeded in taking a tentative checkpoint.

 A process says “no” to a request if it fails to take a tentative checkpoint, which could be due to

several reasons, depending upon the basic application.

 If Pi learns that all the processes have successfully taken tentative checkpoints, Pi decides that

all tentative checkpoints should be made permanent; otherwise, Pi decides that all the tentative

checkpoints should be thrown-away.

Second Phase
Pi informs all the processes of the decision it reached at the end of the first phase. A process, on
receiving the message from Pi will act accordingly.
Either all or none of the processes advance the checkpoint by taking permanent checkpoints.
The algorithm requires that after a process has taken a tentative checkpoint, it cannot send
messages related to the basic computation until it is informed of Pi’s decision.
Correctness: for 2 reasons
 Either all or none of the processes take permanent checkpoint
 No process sends message after taking permanent checkpoint
Optimization: maybe not all of the processes need to take checkpoints (if not change since the last
checkpoint)
An Optimization

The above protocol may cause a process to take a checkpoint even when it is not necessary for
consistency. Since taking a checkpoint is an expensive operation, we avoid taking checkpoints.
Consider the example shown in Fig.1. The set {x1, y1, z1} is a consistent set of checkpoints.

Figure 1: Example of checkpoints taken unnecessarily

 Suppose process X decides to initiate the checkpointing algorithm after receiving message m.

 It takes a tentative checkpoint x2 and sends “take tentative checkpoint" messages to processes

Y and Z, causing Y and Z to take checkpoints y2 and z2, respectively.

 {x2, y2, z2} forms a consistent set of checkpoints.

 {x2, y2, z1} forms a consistent set of checkpoints.

 There is no need for process Z to take checkpoint z2 because Z has not sent any message since

its last checkpoint.

 Process Y must take a checkpoint since has sent messages since its last checkpoint.
B.The Rollback Recovery Algorithm

 The rollback recovery algorithm restores the system state to a consistent state after a failure.

 The rollback recovery algorithm assumes that a single process invokes the algorithm.

 It assumes that the checkpoint and the rollback recovery algorithms are not invoked
concurrently. The rollback recovery algorithm has two phases.

First Phase

 An initiating process Pi sends a message to all other processes to check if they all are willing to
restart from their previous checkpoints.

 A process may reply “no” to a restart request due to any reason (e.g., it is already participating
in a checkpointing or a recovery process initiated by some other process).

 If Pi learns that all processes are willing to restart from their previous checkpoints,
 Pi decides that all processes should roll back to their previous checkpoints. Otherwise, Pi aborts
the roll back attempt and it may attempt a recovery at a later time.

Second Phase

 Pi propagates its decision to all the processes.

 On receiving Pi’s decision, a process acts accordingly.

 During the execution of the recovery algorithm, a process cannot send messages related to the
underlying computation while it is waiting for Pi’s decision.

Correctness

 All processes restart from an appropriate state because if processes decide to restart, then they
resume execution from a consistent state (the checkpointing algorithm takes a consistent set of
checkpoints).
An Optimization

The above recovery protocol causes all processes to roll back irrespective of whether a process
needs to roll back or not. Consider the example shown in below figure.

The above protocol, in the event of failure of process X, the above protocol will require processes
X, Y, and Z to restart from checkpoints x2, y2, and z2, respectively.

Process Z need not roll back because there has been no interaction between process Z and the
other two processes since the last checkpoint at Z.
 X.Juang-Venkatesan algorithm for asynchronous checkpointing and recovery

 System Model and Assumptions

 The algorithm makes the following assumptions about the underlying system:

 The communication channels are reliable, deliver the messages in FIFO order and have infinite
buffers.

 The message transmission delay is arbitrary, but finite.

 The processors directly connected to a processor via communication channels are called its
neighbors.

 The underlying computation or application is assumed to be event-driven: a processor P waits

until a message m is received, it processes the message m, changes its state from s to s′, and
sends zero or more messages to some of its neighbors.
 Then the processor remains idle until the receipt of the next message.

 The new state s′ and the contents of messages sent to its neighbors depend on state s and the
contents of message m.

 The events at a processor are identified by unique monotonically increasing numbers, ex0, ex1,
ex2,...
 To help recovery after a process failure and restore the system to a consistent state, two types of
log storage are maintained, volatile log and stable log.

 Accessing the volatile log takes less time than accessing the stable log, but the contents of the
volatile log are lost if the corresponding processor fails.

 The contents of the volatile log are periodically flushed to the stable storage.

A. Asynchronous Checkpointing

 After executing an event, a processor records a triplet {s,m,msgs_sent} in its volatile storage.

 Where s is the state of the processor before the event, m is the message ,and msqs_sent is the
set of messages that were sent by the processor during the event.
 A local checkpoint at a processor consists of the record of an event occurring at the processor
and it is taken without any synchronization with other processors.

 Periodically, a processor independently saves the contents of the volatile log in the stable
storage and clears the volatile log. This operation amounts to taking a local checkpoint.

The recovery algorithm

Notation and data structure

The following notation and data structure are used by the algorithm:
Basic idea

 Since the algorithm is based on asynchronous checkpointing, the main issue in the recovery is
to find a consistent set of checkpoints to which the system can be restored.

 The recovery algorithm achieves this by making each processor keep track of both the number
of messages it has sent to other processors as well as the number of messages it has received
from other processors.

 Recovery may involve several iterations of roll backs by processors.

 Whenever a processor rolls back, it is necessary for all other processors to find out if any
message sent by the rolled back processor has become an orphan message.

 Orphan messages are discovered by comparing the number of messages sent to and received
from neighboring processors.
 For example, if RCVDi←j(CkPti) > SENTj→i(CkPtj) (that is, the number of messages received
by processor pi from processor pj is greater than the number of messages sent by processor pj to
processor pi, according to the current states the processors), then one or more messages at
processor pj are orphan messages.

 In this case, processor pj must roll back to a state where the number of messages received
agrees with the number of messages sent.

 Consider an example shown in Fig.13.3. Suppose processor Y crashes at the point indicated
and rolls back to a state corresponding to checkpoint ey1.

 According to this state, Y has sent one message to X; according to X’s current state (ex2), X
has received two messages from Y.
 X must roll back to a state preceding ex2 to be consistent with Y’s state.

 If X rolls back to checkpoint ex1, then it will be consistent with Y’s state, ey1. Processor Z
must roll back to checkpoint ez2 to be consistent with Y’s state, ey1.

 Processors X and Z will have to resolve any such mutual inconsistencies.

The Algorithm

 When a processor restarts after a failure, it broadcasts a ROLLBACK message that it had
failed1.

 The recovery algorithm at a processor is initiated when it restarts after a failure or when it
learns of a failure at another processor.

 Because of the broadcast of ROLLBACK messages, the recovery algorithm is initiated at all
processors.
H
 The rollback starts at the failed processor and slowly diffuses into the entire system through
ROLLBACK messages.

 The procedure has |N| iterations. During the kth iteration (k ≠ 1), a processor pi does the
following:

 (i) based on the state CkPti it was rolled back in the (k - 1)th iteration, it computes
SENTi→j(CkPti)) for each neighbor pj and sends this value in a ROLLBACK message to that
neighbor.

 (ii) pi waits for and processes ROLLBACK messages that it receives from its neighbors in kth
iteration and determines a new recovery point CkPti for pi based on information in these
messages.

 At the end of each iteration, at least one processor will roll back to its final recovery point,
unless the current recovery points are already consistent.
An Example

 Consider an example shown in below figure consisting of three processors.

 Suppose processor Y fails and restarts.

 If event ey2 is the latest checkpointed event at Y, then Y will restart from the state
corresponding to ey2.
 Because of the broadcast nature of ROLLBACK messages, the recovery algorithm is initiated
at processors X and Z.

 Initially, X, Y, and Z set CkPtX ← ex3, CkPtY ← ey2 and CkPtZ ← ez2, respectively, and X, Y,
and Z send the following messages during the first iteration:

 Y sends ROLLBACK(Y,2) to X and ROLLBACK(Y,1) to Z;

 X sends ROLLBACK(X,2) to Y and ROLLBACK(X,0) to Z;

 Z sends ROLLBACK(Z,0) to X and ROLLBACK(Z,1) to Y.

 Since RCVDX←Y (CkPtX) = 3 > 2 (2 is the value received in the ROLLBACK(Y,2) message
from Y), X will set CkPtX to ex2 satisfying RCVDX←Y (ex2) = 1≤ 2.

 Since RCVDZ←Y (CkPtZ) = 2 > 1, Z will set CkPtZ to ez1 satisfying RCV DZ←Y (ez1) = 1 ≤ 1.
 At Y, RCVDY←X(CkPtY ) = 1 < 2 and RCVDY←Z(CkPtY ) = 1 = SENTZ←Y (CkPtZ).

 Y need not roll back further.

 In the second iteration,

 Y sends ROLLBACK(Y,2) to X and ROLLBACK(Y,1) to Z;
 Z sends ROLLBACK(Z,1) to Y and ROLLBACK(Z,0) to X;
 X sends ROLLBACK(X,0) to Z and ROLLBACK(X, 1) to Y.

 If Y rolls back beyond ey3 and loses the message from X that caused ey3, X can resend this
message to Y because ex2 is logged at X and this message available in the log.

 The second and third iteration will progress in the same manner.

 The set of recovery points chosen at the end of the first iteration, {ex2, ey2, ez1}, is consistent,
and no further rollback occurs.