Topics of Distributed System Section

• Distributed deadlock detection

– Global state and distributed snapshot
– Logical clock and vector clock, …
• Distributed mutual exclusion
• Distributed transaction execution
• Fault tolerance
– Quorum-based approach
– Leader-based approach: Paxos and PBFT
• Consistency
 Absence of Global Clock
• Problem: synchronizing the activities of
different part of the system
• What about using a single shared clock?
– two different processes can see the clock at
different times due to unpredictable transmission
delays
• What about using radio synchronized clocks?
– Propagation delays are unpredictable
• Software approaches
– Clock synchronization algorithms
– Logical clocks
 Cristian's Algorithm
• Basic idea: get the current time from a time server.
• Issues:
– Error due to communication delay - can be estimated as (T1-T0-I)/2
– Time correction on client must be gradual
 The Berkeley Algorithm

a) The time daemon asks all the other machines for their clock values
b) The machines answer
c) The time daemon tells everyone how to adjust their clock
 Limitation
• They all require network delay to be stable to
some extent

• Whether this assumption always holds is

questionable in some environments
 Logical clock - motivation
• Review
– Happened-before relation
– Consistent global state
– Chandy Lamport protocol
 New problem
• Monitor node sometimes needs to observe
other nodes’ events continuously

• Distributed snapshot is a heavy protocol

– Performing a snapshot for each event is
unacceptable
 New problem
• Still use deadlock detection as an example
– When a node receives a request or decides to
grant a resource to another node, it sends this
event to the monitor
– The monitor needs to decide whether the system
is in a deadlock based on the incoming sequence
of events
 Message delivery
• Receive: a node gets a message from the
network
• Delivery: the node actually processes the
message
– In deadlock detection, the monitor’s “deliver” will
add the event to graph and check for cycle
• Can the monitor deliver a message when
receiving it, assuming each channel is FIFO?
 Counter example

p0
Ask Grant
p1
Ask
Ask
p2
 Counter example
No deadlock: good
Monitor
p0->p1 p1->p2 p2->p0
remove
p0->p1
p0
Ask Grant
p1
Ask
Ask
p2
 Counter example
Deadlock: false alarm
Monitor

p0
Ask Grant
p1
Ask
Ask
p2
 Solution: Causal delivery
• Definition: sendi(m) → sendj(m’) deliverk(m)
→ deliverk(m’)

• If we can achieve causal delivery, then the

monitor will always observe a consistent
global state.

• TCP can guarantee casual delivery when i=j,

but cannot guarantee it when i≠j.
 Start with stronger assumptions
• Assume all nodes have access to a global clock

• Assume synchronous network:

– Network delays are bounded (∆).

• Can you think about a solution?

 Start with stronger assumptions
• Attach a timestamp ts to each message

• Deliver messages in the order of ts

• Is this complete?
 Start with stronger assumptions
• Attach a timestamp ts to each message

• Deliver messages in the order of ts

• When a node’s clock reaches t, it can deliver

all messages with ts <= t – ∆.
 An ideal clock
• Clock condi on (for correctness): e → e’
clock(e) < clock(e’)
– Is the opposite required?
• Gap detection (for liveness): given two events e
and e’ with clock(e) < clock(e’), determine
whether e’’ exists such that clock(e) < clock(e’’) <
clock(e’)
– Even real time clock cannot achieve this. Need
additional help (bounded latency in previous slide).
 Logic clock
• Also called Lamport clock
• Each node maintains a local integer called LC
Logic clock
 Logic clock
• Also called Lamport clock
• Each node maintains a local integer called LC

• Does it satisfy clock condition?

• Can it provide gap detection?
 Logic clock
• Also called Lamport clock
• Each node maintains a local integer called LC

• Does it satisfy clock condition? Yes.

• Can it provide gap detection? No.
– Could you think about a solution?
 Logic clock
• Also called Lamport clock
• Each node maintains a local integer called LC

• Does it satisfy clock condition? Yes.

• Can it provide gap detection? No.
– A node can deliver e with ts if it receives a
message with at least ts-1 from each node.
 Limitation of Logic clock
• Cannot differentiate concurrent events and
related events
– Clock condition: e→e’ clock(e) < clock(e’).
Implementation uses it in the opposite direction:
clock(e) < clock(e’) deliver e before e’
– Can cause unnecessary delays in message delivery
Example of limitation

To deliver 6 from p2, the

monitor must wait for p1
and p3 to report at least 5:
the latter is unnecessary in
this example.
 Limitation of Logic clock
• Cannot differentiate concurrent events and
related events
– Clock condition: e→e’ clock(e) < clock(e’).
Implementation uses it in the opposite direction:
clock(e) < clock(e’) deliver e before e’
– Can cause unnecessary delays in message delivery
• It’s better to have a stronger property:
– Strong clock condition: e→e’ ≡ clock(e) < clock(e’).
– Note again that even real time clock cannot
achieve this property.
 Vector clock
• The clock of an event e is the minimal
consistent cut θ(e) that includes e
– A (consistent) cut is equivalent to a (consistent)
global state
 Vector clock
• The clock of an event e is the minimal
consistent cut θ(e) that includes e
• Define clock comparison as set operation:
– e→e’ ≡ θ(e) θ(e’)
– e and e’ are concurrent if θ(e) θ(e’) and θ(e’)
θ(e)
 Vector clock
• A consistent cut can be uniquely identified by
its frontier

• We can use a vector VC(e) to represent the

frontier of the minimal consistent cut of e
– VC(e)[i] = number of events on node i
 Vector clock
• VC(e41) = [4, 1 ,3]

How to compare two vector clocks?

Vector clock

For all events from

process i, their VC[i]s
are incremental and
have no gaps.
 Go back to the problem
• Assume processes send a notification message
to the monitor for each of their events.
• When the monitor receives m from pj, when
can the monitor deliver m?
 Go back to the problem
• Assume processes send a notification message
to the monitor for each of their events.
• When the monitor receives m from pj, when
can the monitor deliver m?
– When the monitor has received and delivered all
messages happened before m. But how to achieve
this with vector clock?
 Go back to the problem
• Note that m’s vector clock identifies the minimal
consistent cut that contains m.

• Example: if m is from p1 and its VC is [5,1,3],

– m is the 5th event on p1
– On p2, there is 1 event happened before m
– On p3, there are 3 events happened before m.
– Therefore, if the monitor wants to deliver m, it must
first deliver 4 events from p1, 1 event from p2, and 3
events from p3.
 Solution
• Monitor maintains an array D[1-n]
– When it delivers m’ from pk, set D[k] = VC(m’)[k]

• Causal Delivery: Deliver message m from

process pj as soon as both of the following
conditions are satisfied:
– D[j] = VC(m)[j]-1
– D[k] ≥ VC(m)[k], k≠j
 Example

(1,1) (1,0)
Monitor

(1,0)
p0

p1
(1,1)
 Properties of vector clock
• Property 1 (Strong Clock Condition)
– e→e’ ≡ VC(e) < VC(e’)
• Property 2 (Simple Strong Clock Condition)
– ei →ej ≡ VC(ei)[i]≤VC(ej)[i]
– Can you prove it?
• Property 3 (Concurrent)
– ei||ej ≡(VC(ei)[i] > VC(ej)[i]) (VC(ej)[j] > VC(ei)[j])
 Properties of vector clock
• Properties 4 - 6: Read the paper. 4 and 5 are
used to determine whether a cut is consistent.

• Property 7 (Weak Gap-Detection)

– VC(ei)[k] < VC(ej)[k] ek, s.t. ¬(ek →ei) (ek →ej)
– When i=k, ek, s.t. ei → ek → ej
 Data race detection by vc (Cont’d)
• Usually, shared variable are supposed to be
protected by locks

Lock() Lock()
Write x Write x
this is perfectly safe
Y=x+5 Unlock()
Unlock()
• Accessing shared variable without locks is
called a data race
– It is usually a bug.
– How to detect it?
 Open Questions
• How to use Vector Clocks
– What are messages here?
• How to be efficient about this
– There can be too many memory accesses
 Basic idea
• If there is a happened-before relationship
between conflicting operations, then no
problem. Otherwise, data race.

• Happened-before relationship is established

by unlock -> lock

• Treat unlock as send and lock as receive.

 Basic Approach (DJIT+ Algo)
• Each thread t maintains a vector clock Ct
– Ct[u] denotes last operation at thread u
• For each lock m, maintain a vector clock Lm
• When thread t releases lock m, update Lm to
be Ct
• When thread i acquires lock m
– Ci = max(Ci, Lm), Ci[i]++ (same as on receiving a
message)
 Basic Approach (Contd)
• For every variable x, maintain Rx and Wx also
• Rx(t) records clock of last read on x by t
• Wx(t) records clock of last write on x by t
 Checking Race Conditions
• Three types of conflicts: write-write, read-write,
write-read.

• When Thread u reads x

– Check if Wx is less than Cu
• When Thread u writes x
– Check if Wx is less than Cu
• When Thread u writes x
– Check if Rx is less than Cu