0% found this document useful (0 votes)

34 views

Disc

Uploaded by

swathi sp

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

34 views

Disc

Uploaded by

swathi sp

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 299

8/9/2024

SSWT ZG526
Distributed Computing
BITS Pilani
Pilani | Dubai | Goa | Hyderabad

BITS Pilani
Pilani | Dubai | Goa | Hyderabad

LECTURE 1

1
8/9/2024

Text and References

T1 Ajay D. Kshemkalyani, and Mukesh Singhal “Distributed Computing: Principles,

Algorithms, and Systems”, Cambridge University Press, 2008 (Reprint 2013).

R1 Kai Hwang, Geoffrey C. Fox, and Jack J. Dongarra, “Distributed and Cloud
Computing: From Parallel processing to the Internet of Things”, Morgan
Kaufmann, 2012 Elsevier Inc.

R2 John F. Buford, Heather Yu, and Eng K. Lua, “P2P Networking and Applications”,
Morgan Kaufmann, 2009 Elsevier Inc.

SSWT ZG526 - Distributed Computing 3 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

index
• DISTRIBUTED COMPUTING
• Contact Session – 1

• 1Introduction - introduction to Distributed computing in terms of various hardware and software models T1 (Chap.1)

• 1.2 Multiprocessing and Multi computing System, Distributed System Design Issues 1.3 Distributed Communication
Model (RPC) 1.4 Review of different communication models Review of Design issues and Challenges for building
distributed systems
• Contact Session – 2
• M2: Logical Clocks & Vector Clocks

• 2.1 Distributed Computational Model and Logical Clocks. T1 (Chap.3)

• 2.2 Lamport Logical Clocks 2.3 Vector Clocks 2 Review of logical clocks. Review Lamport logical and vector clocks
examples

4 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

2
8/9/2024

index

• Contact Session – 3

• M3: Global state and snapshot recording algorithms

• 3.1 Global States, Principles to use to record the global states T1 (Chap.4 & 5)

• 3 Review of recording global state

• Contact Session – 4
• M3: Global state and snapshot recording algorithms

• 3.2 Chandy Lamport global state recording Algorithm for FIFO channels and Lai yang Algorithm for non-FIFO
channels

• 4 Review of algorithms Chandy Lamport global state recording Algorithm and Lai yang Algorithm for FIFO and non-
FIFO channels

5 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

index

• Contact Session – 5

• M4: Message ordering and Termination Pre-CH/CS

• 4.1 Casual Ordering of messages; Birman Schipher Stephenson (BSS) Algorithm with Example T1 (Chap.6)

• 4.2 Schipher Eggli Sandoz (SES) Protocol for casual ordering with example

• 5 Review of Birman Schipher Stephenson (BSS) Algorithm Schipher Eggli Sandoz (SES) Algorithm with examples
detection

• Contact Session – 6

• M5: Distributed Mutual Exclusion

• 5.1 Distributed Mutual Exclusion; Centralized Algorithm T1 (Chap.9, 10)

• 5.2 Lamport DME Algorithm with Examples
• 5.3 Ricart Agrawala DME Algorithm with Example
• 6 Review of DME algorithms like, Lamport. Ricart Agrawala algorithms with example.
6 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

3
8/9/2024

index

• Contact Session – 7

• M5: Distributed Mutual Exclusion

• 5.4 Maekawa’s DME Algorithm with Example T1 (Chap.9, 10)

• 5.5 Token Based DME, Broadcast Based Algorithm; Suzuki Kasami Algorithm

• 5.6 Raymond’s Tree Based Algorithm

• 7 Review of DME algorithms like Maekawa’s,Broadcast Based, Suzuki Kasam and Raymond Tree based algorithms
with example .

• Contact Session – 8

• Review 8 Review of previous Modules M1 to M5 for Mid Term test preperation

7 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

index

• Contact Session – 9

• M6: Deadlock Detection

• 6.1 Deadlocks in distribution system
• 6.2 Chandy Misra Haas(CMH) Algorithm for AND Model (Edge Chasing)

• 6.3 Chandy Misra Haas(CMH) Algorithm for OR Model (Diffusion Computation)

• 9 Review of Chandy Misra Haas algorithms for deadlock.

• Contact Session – 10
• M7: Consensus and Agreement Algorithm

• 7.1 Agreement Algorithm T1 (Chap.14)

• 7.2 Oral Message Algorithm

• 7.3 Applications of Byzantine Algorithm

• 10 Review of Agreement and OM algorithms 8 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

4
8/9/2024

index

• Contact Session – 11

• M8: Peer to Peer Computing and Overlay graphs

• 8.1 Introduction, P2P Architecture T1 (Chap.18)

• 8.2 Design of Unstructured peer to peer networks

• 11 Review of deign of structured P2P network.

• Contact Session – 12

• M8: Peer to Peer Computing and Overlay graphs

• 8.3 Design of structured peer too peer networks T1 (Chap.18)
• 8.4 Security Solutions for threats in P2P networks 12 Review of deign of unstructured P2P network

9 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

index

• Contact Session – 13

• M9: Cluster computing, Grid Computing

• 9.1 Cluster computing Introduction R2 (Chap.2, 7)

• 9.2 Design Components of cluster computers

• 13 Review of Cluster computing.

• Contact Session – 14

• M9: Cluster computing, Grid Computing

• 9.3 Grid Computing Introduction R2 (Chap.2, 7)

• 14 Review of grid computing.

10 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

5
8/9/2024

index

• Contact Session – 15

• M10: Internet of Things

• 9.4 IoT ; IoT Architecture

• 15 Review of IoT Architecture and Technologies

• Contact Session – 16
• Review CS 16 Review of previous Modules M6 to M10 for Comprehensive exam preparation

11 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Massively Multiplayer Online Games

• Player's login from different parts of the world into a single virtual world
• Different physical times
– Logical clocks (S2)
• Consistent view of the virtual world
– Global state (S3 & S4)
• Player communication
– Causal ordering in group communication (S5)
• Send broadcasts on the network, placement of game servers
– Distributed graph algos (S6)

SSWT ZG526 - Distributed Computing 12 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

6
8/9/2024

Massively Multiplayer Online Games

• Completion of a stage or set of steps

– Termination detection (S5)
• Fight for resources
– Distributed Mutual Exclusion and Deadlocks (S7 & S9)
• A team of players making a decision
– Consensus protocols (S10)
• Game servers' architecture
– Client/Server, Peer-to-Peer, Clusters, Grid (S11 - S14)
• Large scale networks - millions of users, devices
– IoT (S15)

SSWT ZG526 - Distributed Computing 13 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Module Details

SSWT ZG526 - Distributed Computing 14 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

7
8/9/2024

Module Details

M1 - Introduction to Distributed Computing

• Introduction to Distributed computing

• Motivation, Multiprocessor Vs Multicomputer Systems
• Distributed Communication Model; RPC
• Design issues and challenge

References : T1 (Chap.1)

SSWT ZG526 - Distributed Computing 15 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

• Introduction to Distributed Computing

• Motivation
• Classification and introduction to Parallel Systems
• Distributed Communication Models
• Design issues and challenges
• Summary

SSWT ZG526 - Distributed Computing 16 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

8
8/9/2024

DISTRIBUTED COMPUTING

• Distributed computing is a field of computer science that studies

distributed systems.
• A distributed system is a software system in which components located on
networked computers communicate and coordinate their actions by passing
messages.
• A distributed system is a collection of independent entities that cooperate to solve a
problem that cannot be individually solved.

SSWT ZG526 - Distributed Computing 17 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

DISTRIBUTED COMPUTING

• Multiple independent machines with no physical

common clock
• No shared memory but somehow a global state
needs to get created when required
• May be physically distant machines and needs
good network infrastructure and protocols /
control algorithms to communicate
• Machines communicate via messages and
somehow the system figures out the message
order
• The machines doing pieces of the work can jointly
figure out when a task terminates
SSWT ZG526 - Distributed Computing 18 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

9
8/9/2024

DISTRIBUTED SYSTEM CHARACTERISTICS

A distributed system can be characterized as a collection of mostly autonomous

processors communicating over a communication network and having the following
features:

– No common physical clock

– No shared memory
– Geographical separation
– Autonomy and heterogeneity
(Different processors with different speed and OS)

SSWT ZG526 - Distributed Computing 19 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

RELATION TO COMPUTER SYSTEM COMPONENTS

• The distributed software is also termed as middleware.

• A distributed execution is the execution of processes across the distributed system
to collaboratively achieve a common goal.
• An execution is also sometimes termed a computation or a run.
• The machines can access and use shared resources without conflicts, e.g. writing a
file on a network storage
• If the program is stuck with deadlock, it can be detected and resolved.
• Independent machines can have joint decisions made via agreement protocol - e.g.
majority voting by just passing messages
• All this logic can then be built into the distributed system using various architectural
patterns - client/server, peer to peer etc.

SSWT ZG526 - Distributed Computing 20 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

10
8/9/2024

RELATION TO COMPUTER SYSTEM COMPONENTS

A Middleware Service for Distributed Applications

SSWT ZG526 - Distributed Computing 21 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

RELATION TO COMPUTER SYSTEM COMPONENTS

Relation between software components on a machine

SSWT ZG526 - Distributed Computing 22 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

11
8/9/2024

RELATION TO COMPUTER SYSTEM COMPONENTS

Complex application directly on network stack has Easier programming where complexity is handled in
to solve distributed computing problems distributed middleware / os services

Networking OS Distributed OS

SSWT ZG526 - Distributed Computing 23 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Examples of Distributed Systems

• Cluster computing systems

• Banking applications
• Internet caching systems, e.g. Akamai
• Distributed Databases
• Peer-to-peer systems for content sharing
• Media streaming systems
• Real-time process control, e.g. aircraft control systems
• IoT or sensor networks

SSWT ZG526 - Distributed Computing 24 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

12
8/9/2024

Topics for today

• Introduction to Distributed Computing

• Motivation
• Classification and introduction to Parallel Systems
• Distributed Communication Models
• Design issues and challenges
• Summary

SSWT ZG526 - Distributed Computing 25 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MOTIVATION

The motivation for using a distributed system is some or all of the following
requirements:

1. Inherently distributed computations

• In many applications such as money transfer in banking, or reaching consensus
among parties that are geographically distant, the computation is inherently
distributed.

SSWT ZG526 - Distributed Computing 26 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

13
8/9/2024

MOTIVATION

2. Resource sharing
• Resources such as peripherals, complete data sets in databases, special libraries, as
well as data (variable/files) cannot be fully replicated at all the sites because it is
often neither practical nor cost-effective.
• Further, they cannot be placed at a single site because access to that site might
prove to be a bottleneck. Therefore, such resources are typically distributed across
the system.
• For example, distributed databases such as DB2 partition the data sets across
several servers, in addition to replicating them at a few sites for rapid access as well
as reliability.

SSWT ZG526 - Distributed Computing 27 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MOTIVATION

3. Access to geographically remote data and resources

• In many scenarios, the data cannot be replicated at every site participating in the
distributed execution because it may be too large or too sensitive to be replicated.
• For example, payroll data within a multinational corporation is both too large and
too sensitive to be replicated at every branch office/site. It is therefore stored at a
central server which can be queried by branch offices.
• Similarly, special resources such as supercomputers exist only in certain locations.
• Advances in the design of resource-constrained mobile devices as well as in the
wireless technology with which these devices communicate have given further
impetus to the importance of distributed protocols and middleware.

SSWT ZG526 - Distributed Computing 28 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

14
8/9/2024

MOTIVATION

4. Enhanced reliability
• A distributed system has the inherent potential to provide increased reliability
because of the possibility of replicating resources and executions, as well as the
reality that geographically distributed resources are not likely to crash or
malfunction at the same time under normal circumstances.

Reliability entails several aspects:

1. Availability, i.e., the resource should be accessible at all times
2. Integrity, i.e., the value/state of the resource should be correct
3. Fault-tolerance, i.e., the ability to recover from system failures

SSWT ZG526 - Distributed Computing 29 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MOTIVATION

5. Increased performance/cost ratio

• By resource sharing and accessing geographically remote data and resources, the
performance/cost ratio is increased. Although higher throughput has not necessarily
been the main objective behind using a distributed system, nevertheless, any task
can be partitioned across the various computers in the distributed system.
• Such a configuration provides a better performance/cost ratio than using special
parallel machines. This is particularly true of the NOW configuration.

SSWT ZG526 - Distributed Computing 30 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

15
8/9/2024

MOTIVATION

6. Scalability
• As the processors are usually connected by a wide-area network, adding more
processors does not pose a direct bottleneck for the communication network.

7. Modularity and incremental expandability

• Heterogeneous processors may be easily added into the system without affecting
the performance, as long as those processors are running the same middleware
algorithms. Similarly, existing processors may be easily replaced by other
processors.

SSWT ZG526 - Distributed Computing 31 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Example Distributed System : Netflix

SSWT ZG526 - Distributed Computing 32 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

16
8/9/2024

Example Distributed System : Netflix

Widely distributed network of content caching servers

SSWT ZG526 - Distributed Computing 33 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

• Introduction to Distributed Computing

• Motivation
• Classification and introduction to Parallel Systems
• Distributed Communication Models
• Design issues and challenges
• Summary

SSWT ZG526 - Distributed Computing 34 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

17
8/9/2024

PARALLEL SYSTEMS

A parallel system may be broadly classified as belonging to one of three types:

1. A multiprocessor system
2. A multicomputer parallel system
3. Array processors

SSWT ZG526 - Distributed Computing 35 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MULTIPROCESSOR SYSTEMS

• A multiprocessor system is a parallel system in which the multiple processors have

direct access to shared memory which forms a common address space.
• Such processors usually do not have a common clock.
• A multiprocessor system usually corresponds to a uniform memory access (UMA)
architecture in which the access latency, i.e., waiting time, to complete an access to any
memory location from any processor is the same.
• The processors are in very close physical proximity and are connected by an
interconnection network.
• All the processors usually run the same operating system, and both the hardware and
software are very tightly coupled.

SSWT ZG526 - Distributed Computing 36 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

18
8/9/2024

MULTIPROCESSOR SYSTEMS

SSWT ZG526 - Distributed Computing 37 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MULTIPROCESSOR SYSTEMS

• The processors are usually of the same type, and are housed within the same
box/container with a shared memory.
• The interconnection network to access the memory may be a bus, although for
greater efficiency, it is usually a multistage switch with a symmetric and regular
design.
• There are two popular interconnection networks
1. The Omega network and
2. The Butterfly network

SSWT ZG526 - Distributed Computing 38 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

19
8/9/2024

MULTIPROCESSOR SYSTEMS

• Two popular interconnection networks – the Omega network and the Butterfly
network , each of which is a multi-stage network formed of 2×2 switching elements.
• Each 2×2 switch allows data on either of the two input wires to be switched to the
upper or the lower output wire.
• In a single step, however, only one data unit can be sent on an output wire. So if the
data from both the input wires is to be routed to the same output wire in a single
step, there is a collision.
• Various techniques such as buffering or more elaborate interconnection designs can
address collisions.

SSWT ZG526 - Distributed Computing 39 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MULTIPROCESSOR SYSTEMS

INTERCONNECTION NETWORKS FOR SHARED MEMORY MULTIPROCESSOR SYSTEMS

Omega network and Butterfly network

for n = 8 processors P0–P7 and memory banks M0–M7

SSWT ZG526 - Distributed Computing 40 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

20
8/9/2024

MULTIPROCESSOR SYSTEMS

Interconnection Networks

a) A crossbar switch - faster

b) An omega switching network - cheaper

SSWT ZG526 - Distributed Computing 41 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MULTICOMPUTER PARALLEL SYSTEM

• A multicomputer parallel system is a parallel system in which the multiple

processors do not have direct access to shared memory.
• The memory of the multiple processors may or may not form a common address
space. Such computers usually do not have a common clock.
• The processors are in close physical proximity and are usually very tightly coupled
(homogenous hardware and software), and connected by an interconnection
network.
• The processors communicate either via a common address space or via message-
passing usually corresponds to a non-uniform memory access (NUMA) architecture
in which the latency to access various shared memory locations from the different
processors varies.

SSWT ZG526 - Distributed Computing 42 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

21
8/9/2024

MULTICOMPUTER PARALLEL SYSTEM

Non-uniform memory access (NUMA) multiprocessor

SSWT ZG526 - Distributed Computing 43 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MULTICOMPUTER PARALLEL SYSTEM

A wrap-around 4×4 mesh.

For a k×k mesh which will contain k2
processors, the maximum path length
between any two processors is
2(k/2−1).

SSWT ZG526 - Distributed Computing 44 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

22
8/9/2024

MULTICOMPUTER PARALLEL SYSTEM

• A k-dimensional hypercube has 2k processor-and-memory units. Each such unit is a

node in the hypercube, and has a unique k-bit label. Each of the k dimensions is
associated with a bit position in the label

Four-dimensional hypercube.

SSWT ZG526 - Distributed Computing 45 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

ARRAY PROCESSORS

• Array processors belong to a class of parallel computers that are physically co-
located, are very tightly coupled, and have a common system clock (but may not
share memory and communicate by passing data using messages).
• Array processors and systolic arrays that perform tightly synchronized processing
and data exchange in lock-step for applications such as DSP and image processing
belong to this category.
• These applications usually involve a large number of iterations on the data.

SSWT ZG526 - Distributed Computing 46 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

23
8/9/2024

ARRAY PROCESSORS

SSWT ZG526 - Distributed Computing 47 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Flynn’s Taxonomy

SSWT ZG526 - Distributed Computing 48 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

24
8/9/2024

Some basic concepts

• Coupling
– Tight - SIMD, MISD shared memory systems
– Loose - NOW, distributed systems, no shared memory
• Speedup
– how much faster can a program run when given N processors as opposed to 1 processor — T(N) / T(1)
– Optional reading: Amdahl’s Law, Gustafson’s Law
• Parallelism / Concurrency of program
– Compare time spent in computations to time spent for communication via shared memory or message
passing
• Granularity
– Average number of compute instructions before communication is needed across processors
• Note:
– Coarse granularity —> Distributed systems else use tightly coupled multi-processors/computers
– High concurrency doesn’t lead to high speedup if granularity is too small leading to high overheads

SSWT ZG526 - Distributed Computing 49 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

• Introduction to Distributed Computing

• Motivation
• Classification and introduction to Parallel Systems
• Distributed Communication Models
• Design issues and challenges
• Summary

SSWT ZG526 - Distributed Computing 50 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

25
8/9/2024

REMOTE PROCEDURE CALL (RPC)

• Remote Procedure Call (RPC) is a powerful technique for constructing distributed,

client-server based applications.
• It is based on extending the conventional local procedure calling, so that the called
procedure need not exist in the same address space as the calling procedure.
• The two processes may be on the same system, or they may be on different systems
with a network connecting them.

SSWT ZG526 - Distributed Computing 51 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

REMOTE PROCEDURE CALL (RPC)

SSWT ZG526 - Distributed Computing 52 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

26
8/9/2024

REMOTE PROCEDURE CALL (RPC)

SSWT ZG526 - Distributed Computing 53 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

MESSAGE QUEUEING

• Message queueing is a method by which process (or program instances) can

exchange or pass data using an interface to a system-managed queue of messages.
• Messages can vary in length and be assigned different types or usages.
• A message queue can be created by one process and used by multiple processes
that read and/or write messages to the queue.
• For example, a server process can read and write messages from and to a message
queue created for client processes.
• The message type can be used to associate a message with a particular client
process even though all messages are on the same queue.

SSWT ZG526 - Distributed Computing 54 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

27
8/9/2024

PUBLISH–SUBSCRIBE PATTERN

• Publish–subscribe is a messaging pattern where senders of messages, called

publishers, do not program the messages to be sent directly to specific receivers,
called subscribers, but instead categorize published messages into classes without
knowledge of which subscribers, if any, there may be.
• Similarly, subscribers express interest in one or more classes and only receive
messages that are of interest, without knowledge of which publishers, if any, there
are.

SSWT ZG526 - Distributed Computing 55 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

PUBLISH–SUBSCRIBE PATTERN

• Publish–subscribe is a sibling of the message queue paradigm, and is typically one

part of a larger message-oriented middleware system.
• Most messaging systems support both the pub/sub and message queue models in
their API, e.g. Java Message Service (JMS).
• This pattern provides greater network scalability and a more dynamic network
topology, with a resulting decreased flexibility to modify the publisher and the
structure of the published data.

SSWT ZG526 - Distributed Computing 56 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

28
8/9/2024

PUBLISH–SUBSCRIBE PATTERN

• The sender (also called a publisher) uses a topic-based approach to publish

messages to topic A and to topic B.
• Three receivers (also called subscribers) subscribe to these topics;
– one receiver subscribes to topic A,
– one receiver subscribes to topic B, and
– one receiver subscribes to both topic A and to topic B.

SSWT ZG526 - Distributed Computing 57 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

PUBLISH–SUBSCRIBE PATTERN

SSWT ZG526 - Distributed Computing 58 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

29
8/9/2024

Topics for today

Fault tolerance
• Fault tolerance requires maintaining correct and efficient operation in spite of any failures of links,
nodes, and processes

Security
• Distributed systems security involves various aspects of cryptography, secure channels, access
control, key management – generation and distribution, authorization, and secure group
management.

SSWT ZG526 - Distributed Computing 62 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

31
8/9/2024

DESIGN ISSUES AND CHALLENGES

Applications Programming Interface (API) and transparency

• The API for communication and other specialized services is important for the ease of use and wider
adoption of the distributed systems services by non-technical users.

Scalability and modularity

• The algorithms, data (objects), and services must be as distributed as possible. Various techniques
such as replication, caching and cache management, and asynchronous processing help to achieve
scalability.

SSWT ZG526 - Distributed Computing 63 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

NEW APPLICATIONS POSE NEW CHALLENGES

• Mobile networks
– distributed graph algorithms
• Sensor networks collecting and transmitting physical parameters
– large volume streaming, algorithms for location estimation
• Ubiquitous or Pervasive computing where processors are embedded everywhere to perform
application functions e.g. smart homes
– groups of wireless sensors/actuators connected to Cloud backend
• Peer-to-peer e.g. gaming, content distribution
– object storage/lookup/retrieval/replication, self-organising networks, privacy

SSWT ZG526 - Distributed Computing 64 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

32
8/9/2024

NEW APPLICATIONS POSE NEW CHALLENGES

• Publish-subscribe e.g. movie streaming, online trading markets - receive only data of interest
– data streaming systems and filtering / matching algorithms
• Distributed agents or robots e.g. in mobile computing
– swarm algorithms, coordination among agents (like an ant colony)
• Distributed data mining
– e.g. user profiling - Data is spread in many repositories

SSWT ZG526 - Distributed Computing 65 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

33
8/9/2024

BITS Pilani
Pilani | Dubai | Goa | Hyderabad

Contact Session – 2
M2 -Logical clocks

References : T1 (Chap.3)

Topics for today

» A model of distributed computing
» Lamport’s logical clock
» Vector clock
» Efficient implementation of vector clock
» Other techniques, e.g. Matrix and Fowler-Zwaenepoel’s
» Physical clock sync using Network Time Protocol
» Physical or logical clocks ?
» Summary

SSWT ZG526 - Distributed Computing 2 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

1
8/9/2024

A distributed program

• A distributed program is composed of a set of n asynchronous processes, p1,

p2, ..., pi , ..., pn.
• The processes do not share a global memory and communicate solely
by message passing.
• The processes do not share a global clock that is instantaneously
accessible to these processes.
• Process executions and message transfers are asynchronous.

SSWT ZG526 - Distributed Computing 3 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Causality and clocks

• The concept of causality between events is fundamental to the design and analysis of parallel and distributed computing and operating systems.

• Usually causality is tracked using physical time.

• In distributed systems, it is not possible to have global physical time; it is possible to realize only an approximation of it.

• Causality among events in a distributed system is a powerful concept in reasoning, analyzing, and drawing inferences about a computation.

SSWT ZG526 - Distributed Computing 4 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

2
8/9/2024

Why is logical clock essential ?

• Physical time synchronisation across many nodes and high message
density with high accuracy is nearly impossible
z
• We will study a measurement based technique in NTP
• Need a way to order events across processes to understand how
computation across processes is progressing
• Need to ensure that
x y
• incoming messages are delivered before they are expected by a
process
• no message meant to arrive at a process after an event is delivered
y = f(x)
before the event
• unrelated events can proceed concurrently
time

SSWT ZG526 - Distributed Computing 5 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Causality

• Causality is influence by which one event, process or state contributes to the

production of another event, process or state where the cause is partly responsible

for the effect, and the effect is partly dependent on the cause.

SSWT ZG526 - Distributed Computing 6 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

3
8/9/2024

Uses of causality and logical clocks (1)

The knowledge of the causal precedence relation among the events of processes helps solve a variety of problems in distributed systems some of these problems is as follows.

Distributed algorithms design

The knowledge of the causal precedence relation among events helps ensure liveness and fairness in mutual exclusion algorithms, helps maintain consistency in replicated

databases, and helps design correct deadlock detection algorithms to avoid phantom and undetected deadlocks.

Concurrency measure

The knowledge of how many events are causally dependent is useful in measuring the amount of concurrency in a computation. All events that are not causally related can be

executed concurrently. Thus, an analysis of the causality in a computation gives an idea of the concurrency in the program.

SSWT ZG526 - Distributed Computing 7 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Uses of causality and logical clocks (2)

Tracking of dependent events

In distributed debugging, the knowledge of the causal dependency among events helps construct a consistent state for resuming

reexecution; in failure recovery, it helps build a checkpoint; in replicated databases, it aids in the detection of file inconsistencies in

case of a network partitioning.

Knowledge about the progress

The knowledge of the causal dependency among events helps measure the progress of processes in the distributed computation.

This is useful in discarding obsolete information, garbage collection, and termination detection, e.g. how much of a distributed file

write has finished.

SSWT ZG526 - Distributed Computing 8 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

4
8/9/2024

A model of distributed executions

“happens before” “happens before” A space-time diagram

space

“happens before”

» Result of logically ordered computation is same and independent of absolute time

SSWT ZG526 - Distributed Computing 9 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

A model of distributed executions (2)

“happens before” “happens before” logically concurrent events

space

“happens before”

» Logically concurrent events may happen in different physical times - depends on message delays
» Assume they happened at the same time … computation result will not change

SSWT ZG526 - Distributed Computing 10 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

5
8/9/2024

Models of Communication Networks

• There are several models of services provided by communication networks,
namely, FIFO, Non-FIFO
• Distributed middleware can provide additional semantics - e.g. causal order

C1
P1 P2
m4 | m3 | m1 | m2 |

C3
C2 FIFO channel e.g. TCP
non-FIFO channel C4
e.g. UDP
m3 | m4 | P3
m1 | m2 |

SSWT ZG526 - Distributed Computing 11 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Models of Communication Networks

• non-FIFO: No restrictions in message order on link
• FIFO: Messages are delivered in same sequence as sent on a link
• Causal ordering (CO) aka ‘happened before’ relationship among messages across links
• If send(mij) —> send(mkj) then receive(mij) —> receive(mkj)
• Ensures that causally related messages for same destination are delivered preserving the same order.
• CO is most useful in building distributed algorithms, e.g. DB replica updates (case study later)

Send 1 Send 2
P1
CO FIFO

P2
Receive 1 Receive 2 non-FIFO

SSWT ZG526 - Distributed Computing 12 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

6
8/9/2024

Logical Clocks: Scalar time and Vector time

• What is a logical clock

• In the absence of perfect clock synchronisation, assign logical numbers
to events in each processor so that across processor events get ordered.
• 2 main techniques - scalar and vector
• The goal is to assign a causal order or “happened before” when assigning
numbers instead of actually finding physical time
• Causality among events in a distributed system is a powerful concept in
reasoning, analysing, and drawing inferences about a computation.

SSWT ZG526 - Distributed Computing 13 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Implementing logical clocks

Requires addressing two issues

• Data structures local to every process to represent logical time and

• A protocol (set of rules) to update the data structures to ensure the consistency condition.

Each process pi maintains data structures that allow it the following two capabilities:

• A local logical clock , denoted by lc , that helps process p measure its own progress.
i i

• A logical global clock , denoted by gci , that is a representation of process pi’s local view of the logical global time. It allows this process to assign consistent timestamps to its local events.

Typically, lci is a part of gci .

SSWT ZG526 - Distributed Computing 14 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

7
8/9/2024

Lamport clock with scalar time

• Proposed by Lamport as an attempt to totally order events in a distributed

system.
• It however only creates a partial order as shown later
• Time domain is the set of non-negative integers (scalar).
• The logical local clock of a process pi and its local view of the global time are
squashed into one integer variable Ci.
• 2 rules R1 and R2 to update the clocks

SSWT ZG526 - Distributed Computing 15 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Rules for Lamport clock (1)

• R1: Before executing an event (send, receive, or internal),

process pi executes the following:
Ci := Ci + d (d > 0)
In general, every time R1 is executed, d can have a different value; however,
typically d is kept at 1 (e.g. for event counting - more later).

R1: 1 R1: 2 <rule applied>: <clock value>

e1 e2
P1
0

SSWT ZG526 - Distributed Computing 16 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

8
8/9/2024

Rules for Lamport clock (2)

• R2: Each message piggybacks the clock value of its sender at

sending time. When a process pi receives a message with
timestamp Cmsg , it executes the following actions:
➢ Ci := max(Ci , Cmsg )
➢ Execute R1.
➢ Deliver the message. 2

e2
P1

R2: 2

P2
e3
0
R2: 3

SSWT ZG526 - Distributed Computing 17 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Lamport clock example

Assume d=1
R1: 1 R1: 2 <rule applied>: <clock value>
e1 e2
P1
0 R2: 2
R1: 1
e3
P2
0
R2: 1

P3
0 e4 e5 e6

R1: 1 R2: 2 R2: 3

SSWT ZG526 - Distributed Computing 18 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

9
8/9/2024

Another example
• Distributed algorithms such as resource synchronization often depend on some method of ordering events to function.

• For example, consider a system with two processes and a disk.

• The processes send messages to each other, and also send messages to the disk requesting access.

• The disk grants access in the order the messages were sent.

• Now, imagine process 1 sends a message to the disk asking for access to write, and then sends a message to process 2 asking it to read.

• Process 2 receives the message, and as a result sends its own message to the disk.

• Now, due to some timing delay, the disk receives both messages at the same time: how does it determine which message happened-before the other?

• A logical clock algorithm provides a mechanism to determine facts about the order of such events.

SSWT ZG526 - Distributed Computing 19 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Basic Properties

Consistency Property
• Scalar clocks satisfy the monotonicity and hence the consistency property:
for two events ei and ej,

* However, converse is not true (more about this later)

Total Ordering
• A partial order is created because events may have same time stamp
• Can be used to also totally order events in a distributed system.
• But, the main problem in totally ordering events is that two or more events at different processes
may have identical timestamp.

SSWT ZG526 - Distributed Computing 20 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

10
8/9/2024

Consistency
Assume d=1
1 2
e1 e2
P1
1
e3
P2

P3
e4 e5 e6
1 2 3

SSWT ZG526 - Distributed Computing 21 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Total Ordering

• A tie-breaking mechanism is needed to order such events. A tie is broken as follows:

➢ Process identifiers are linearly ordered and tie among events with identical scalar timestamp is
broken on the basis of their process identifiers.
➢ The lower the process identifier in the ranking, the higher the priority.
➢ The timestamp of an event is denoted by a tuple (t, i) where t is its time of occurrence and i is the
identity of the process where it occurred.
➢ The total order relation ≺ on two events x and y with timestamps (h,i) and (k,j), respectively, is
defined as follows:

total order relation

SSWT ZG526 - Distributed Computing 22 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

11
8/9/2024

Total ordering (example)

Assume d=1
1 2
e1 e2
P1
1
e3
P2

P3
e4 e5 e6
1 2 3

Use process index (P1<P2) to make e1 precede e3 and force a total order
SSWT ZG526 - Distributed Computing 23 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Properties (1)

No Strong Consistency
1 2 3

• The system of scalar clocks is not strongly consistent; that e1 e2 e3

P1
is, for two events ei and ej,

P2
• The reason that scalar clocks are not strongly consistent e4

is that the logical local clock and logical global clock of a 3

process are squashed into one, resulting in the loss of P3

e5 e6 e7
causal dependency information among events at
different processes. 1 2 3

SSWT ZG526 - Distributed Computing 24 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

12
8/9/2024

Properties (2)

Event counting
1 2 3
• If the increment value d is always 1, the scalar time
e1 e2 e3
has the following interesting property: if event e has a
P1
timestamp h, then h-1 represents the minimum
logical duration, counted in units of events, required height(e4) = 2
before producing the event e; P2
e4

• We call it the height of the event e. 3

• In other words, h-1 events have been produced P3

e5 e6 e7
sequentially before the event e regardless of the
processes that produced these events. 1 2 3

SSWT ZG526 - Distributed Computing 25 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Vector time
P2
• The system of vector clocks was developed
independently by Fidge, Mattern and Schmuck.
P1 P3
• In the system of vector clocks, the time domain is
represented by a set of n-dimensional non-negative
integer vectors.

• So views of local clock and non-local clocks from

clock(P3, t) = [2, 4, 6]
perspective of a process are represented in a
vector. view of P3’s local clock
view of P1, P2’s clocks

SSWT ZG526 - Distributed Computing 26 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

13
8/9/2024

Vector clock rules (1)

Process pi uses the following two rules R1 and R2 to update its clock:

• R1: Before executing an event, process pi updates its local logical time as
follows:
vti[i] := vti[i] + d (d > 0)

(1,0,0) R1: (2,0,0)

e1 e2
P1

SSWT ZG526 - Distributed Computing 27 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Vector clock rules (2)

•R2: Each message m is piggybacked with the vector clock vt of

the sender process at sending time. On the receipt of such a
message (m,vt), process pi executes the following sequence of
actions:

➢ Update its global logical time as follows:

1≤ k≤ n : vti[k] := max(vti [k], vt[k]) R2: (2,0,0)

➢ Execute R1
e3
P2
➢ Deliver the message m. (0,0,0) R2 (a): (2,0,0)
R2 (b): (2,1,0)
Deliver
SSWT ZG526 - Distributed Computing 28 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

14
8/9/2024

Vector time example

Assume d=1

(1,0,0) (2,0,0)
e1 e2
P1
(0,0,0)
(2,0,0)
e3 e5
P2
(0,0,0) (2,1,0) (2,2,1)

P3
(0,0,0) e4
(0,0,1)

SSWT ZG526 - Distributed Computing 29 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Vector time example

Assume d=1 R1: update local clock entry

(1,0,0) (2,0,0)
e1 e2
P1

(2,0,0)
e3 e5
P2
(0,0,0) (2,1,0) (2,2,1)

P3
e4
(0,0,1)

SSWT ZG526 - Distributed Computing 30 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

15
8/9/2024

Vector time example

Assume d=1

(1,0,0) (2,0,0)
e1 e2 R2: Max of remote clocks from local
P1 knowledge and message content
(2,0,0) Update local clock entry as per R1

e3 e5
P2
(0,0,0) (2,1,0) (2,2,1)

P3
e4
(0,0,1)

SSWT ZG526 - Distributed Computing 31 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Comparing vector timestamps (1)

• The following relations are defined to compare two (1,0,0) (2,0,0) (3,0,0)
vector timestamps, vh and vk: e1 e2 e3
P1
(2,0,0)

e4 e5
P2
(0,1,0) (2,2,0)

P3
Examples : e6 e7 e8
(1,0,1) = (1,0,1) (0,0,1) (0,0,2) (0,0,3)
(0,1,1) < (1,2,1)
(1,0,0) || (0,0,2) Now we can confidently say with full
knowledge : e1 || e7

SSWT ZG526 - Distributed Computing 32 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

16
8/9/2024

Comparing vector timestamps (2)

• If the process at which an event occurred is known, the (1,0,0) (2,0,0) (3,0,0)
test to compare two timestamps can be simplified as e1 e2 e3
follows: If eventsx and y respectively occurred at P1

processes pi and pj and are assigned timestamps vh

(2,0,0)

and vk, respectively, then

e4 e5
P2
(0,1,0) (2,2,0)

P3
e6 e7 e8
(0,0,1) (0,0,2) (0,0,3)
e2 —> e5 because V_P1[P1] <= V_P2[P1]
e1 || e7 because V_P1[P1] > V_P3[P1] and V_P1[P3] < V_P3[P3]
P1’s view P3’s view

SSWT ZG526 - Distributed Computing 33 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Properties (1)
Strong Consistency
• The system of vector clocks is strongly consistent; thus, by examining the vector
timestamp of two events, we can determine if the events are causally related.
• However, Charron-Bost showed that the dimension of vector clocks cannot be less than
n, the total number of processes in the distributed computation, for this property to
hold.

SSWT ZG526 - Distributed Computing 34 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

17
8/9/2024

Properties (2)
Event Counting

(1,0,0) (2,0,0) (3,0,0)

• If d=1 (in rule R1), then the ith component of vector clock
e1 e2 e3
at process pi, vti[i], denotes the number of events that P1
have occurred at pi until that instant.
(2,0,0)
• So, if an event e has timestamp vh, vh[j] denotes the 2+2-1=3
number of events executed by process pj that causally P2
e4 e5
precede e. Clearly, Σvh[j] − 1 represents the total (0,1,0) (2,2,0)
number of events that causally precede e in the
distributed computation.
P3
e6 e7 e8
(0,0,1) (0,0,2) (0,0,3)

SSWT ZG526 - Distributed Computing 35 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Efficient implementation of vector clock

• If the number of processes in a distributed computation is large, then vector clocks will require
piggybacking of huge amount of information in messages.
• The message overhead grows linearly with the number of processors in the
system and when there are thousands of processors in the system, the message size becomes huge
even if there are only a few events occurring in few processors.
• For 1000 processors, we need 1000 integer vector with every message
• Charron-Bost showed that if vector clocks have to satisfy the strong consistency property, then in
general vector timestamps must be at least of size n, the total number of
processes.
• However, optimizations are possible and next, and we discuss a technique to implement
vector clocks efficiently.

SSWT ZG526 - Distributed Computing 36 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

18
8/9/2024

Singhal-Kshemkalyani’s differential technique

• Between successive message sends to the same process, only a few entries of the vector
clock at the sender process are likely to change.

Capture only changes in vectors

sent with msgs:

[
{ <change position>, <value },
…
]

SSWT ZG526 - Distributed Computing 37 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Matrix clock
• A matrix clock is a mechanism for capturing chronological and causal relationships in a distributed system.

• Matrix clocks are a generalization of the notion of vector clocks. A matrix clock maintains a vector of the vector clocks for each

communicating host.

• Every time a message is exchanged, the sending host sends not only what it knows about the global state of time, but also the

state of time that it received from other hosts.

• This allows establishing a lower bound on what other hosts know, and is useful in applications such as checkpointing and

garbage collection.

SSWT ZG526 - Distributed Computing 38 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

19
8/9/2024

Fowler–Zwaenepoel’s direct-dependency technique

• Fowler–Zwaenepoel direct dependency technique reduces the size of messages by transmitting only a scalar value in

the messages.

• No vector clocks are maintained on-the-fly.

• Instead, a process only maintains information regarding direct dependencies on other processes.

• A vector time for an event, which represents transitive dependencies on other processes, is constructed off-line from

a recursive search of the direct dependency information at processes.

SSWT ZG526 - Distributed Computing 39 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Fowler–Zwaenepoel’s example

SSWT ZG526 - Distributed Computing 40 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

20
8/9/2024

Why do we need time synchronisation

» Distributed system nodes have clock drift
» But we need to know the real time in some cases
» Time of day on a node
» Time difference between 2 nodes
» Time sensitive (stricter than causal ordering) writes and reads to database
replicas
» Improve performance by replacing communication with local computation if
all nodes are time synchronised
» Correlate time-sensitive data from different nodes in sensor networks

SSWT ZG526 - Distributed Computing 41 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Definitions
» Time: Cp(t) is time of Clock at process p. Cp(t)=t for
perfect clock.
» Frequency: C’p(t) is the rate at which clock progresses
» Offset: Cp(t) - t is offset from real-time t.
» Skew: C’p(t) - C’q(t) is difference in frequency between
clocks of processes p and q
» If skews are bounded by r then dC/dt is allowed to
diverge in the range 1-r to 1+r (clock accuracy)
» Drift (rate): C’’p(t) is the rate of change of frequency at
process p

SSWT ZG526 - Distributed Computing 42 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

21
8/9/2024

NTP
» A hierarchy of time servers on a spanning tree
» Root level primary (stratum 1) synchronises with UTC with microseconds offset from attached stratum 0
devices
» Second level (stratum 2) are backup servers synchronised with stratum 1 servers
» Max level is stratum 15
» At the lowest level are the clients which can be configured with NTP server(s) to use for running a
synchronisation algorithm
» The synchronisation algorithm uses request-reply NTP messages and offset-delay
estimation technique to estimate the round-trip delay and hence the clock offset between 2
servers.
» Offset - 10s of millisecond over internet, <1 millisec in LAN but asymmetric routes and congestion can
cause problems in measurement

SSWT ZG526 - Distributed Computing 43 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Offset-delay estimation
T2 T3 » From the perspective of A :
B
» T1, T2, T3, T4 are available
T1 T1, T2, T3 » Offset = [ (T2-T1) + (T3-T4) ] / 2
» Round trip delay = (T4-T1) - (T3-T2)
A » Continuously send messages to estimate offset and
T1 T4
correct local clock of A
asymmetric routes,
congestion can cause » Choose the offset that has the minimum delay (typically
estimation challenges last 8 samples)
» CA(t) = CB(t) + Offset

SSWT ZG526 - Distributed Computing 44 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

22
8/9/2024

Physical or Logical clocks ?

» Normally we use physical clocks when events density is low compared to event
execution time and accuracy of ordering is not critical
» In distributed systems, event rate is orders of magnitude higher and execution time
is orders of magnitude lower and may mandate perfect order of delivery
» Will need accurate physical clock synchronisation to causally order events
» Hence logical clocks are the mostly used
» Important to understand application use case and pick physical or logical clock
solutions

SSWT ZG526 - Distributed Computing 45 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Case study: MongoDB vs Cassandra

» Popular NoSQL large scale distributed

systems for Big Data storage using data
Distributed client application
replication and partitions DB Driver
» A “write” of a data item has to update
multiple replicas Write Read
request request
» A “read” of the data item is consistent if
the “latest” value is made available to any
Replica Replica Replica
process in the system
» So how do you order reads and writes ?

SSWT ZG526 - Distributed Computing 46 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

23
8/9/2024

MongoDB
» Consistency is a design priority
» All “writes” and “reads” happen through
primary replica. Hence, any “read” should
return the latest consistent value.
» Reads and writes are ordered using logical
clock implementation *
» Optionally consistency can be loosened by
reading from replicas

* MongoDB: ACM SIGMOD’19 session:

https://av.tib.eu/media/43077
SSWT ZG526 - Distributed Computing 47 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Case study: Cassandra

» Scale is a design priority with eventual consistency as a design choice .. so no
ordering guarantees
» Reads can go to any replica to allow the system to scale
» Reads can potentially read older values for some time
» e.g. recent posts / comments not appearing on a social network page
» Over time (eventually), read has to return the latest value (“last write wins”)
» Physical clock synchronisation is recommended between replicas to make the sure
replicas converge within a determined time period

** Cassandra: https://www.datastax.com/blog/why-cassandra-doesnt-need-vector-clocks
SSWT ZG526 - Distributed Computing 48 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

24
8/9/2024

BITS Pilani
Pilani|Dubai|Goa|Hyderabad

Contact Session – 3
M3 - Global state and snapshot recording
References : T1 (Chap.4)

Last session exercise - Lamport clock

SSWT ZG526 - Distributed Computing 2 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

1
8/9/2024

Last session exercise - Vector clock

SSWT ZG526 - Distributed Computing 3 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

» Why record global state
» Consistent and in-consistent global states
» Issues in recording state
» Algorithm for FIFO channels - Chandy Lamport
» Properties of Chandy-Lamport algorithm
» Optimizations on Chandy-Lamport - 2 more algorithm sketches
» Algorithm for non-FIFO channels - Lai Yang
» Necessary and sufficient conditions to record global state

SSWT ZG526 - Distributed Computing 4 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

2
8/9/2024

Global state recording

• Recording the global state of a distributed system on-the-fly is

an important problem.
• Why is it non-trivial?

100 200
A B

A, B, and C exchange money between themselves.

The global state needs to capture exact status of the transfers.
Total money in the system cannot change.
C

100

SSWT ZG526 - Distributed Computing 5 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Global state recording

• Recording the global state of a distributed system on-the-fly is

an important problem.
• Why is it non-trivial?

Total
100 200 ——-
A B 100 Then,
200 T1: A transfers 50 to B
100 T2: B transfer 100 to C
——-
400
C

100

SSWT ZG526 - Distributed Computing 6 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

3
8/9/2024

Global state recording

• Recording the global state of a distributed system on-the-fly is

an important problem.
• Why is it non-trivial?

Total
50 100 ——-

]
100 50 200 50
A B 50 in transit
100 Have to record these correctly
100 in transit
100 100
——-
C
400 Total money in the system cannot change

100

SSWT ZG526 - Distributed Computing 7 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

More examples

What is the total number of files in a system when files are moved
around across machines ?
What is the total space left on the system across storage nodes ?

SSWT ZG526 - Distributed Computing 8 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

4
8/9/2024

Uses of Global State Recording

• Recording a “consistent” state of the global computation
– checkpointing in fault tolerance (rollback, recovery)
– when processor fails and computation on that processor needs to be
restarted
– debugging programs

• Detecting “stable” properties in a distributed system via snapshots. A

property is “stable” if, once it holds in a state, it holds in all subsequent
states.

– termination detection
– deadlock detection
termination deadlock

SSWT ZG526 - Distributed Computing 9 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

SSWT ZG526 - Distributed Computing 10 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

5
8/9/2024

Global state of a distributed system

• Collection of local states of processes / nodes and channels

• Notation —

• To be meaningful, states of all components of the distributed

system must be recorded at the same instant
• Possible only when all local clocks are perfectly synchronised or if
there is a global system clock that can be instantaneously read by
all process.
• Impossible to achieve in a real system
• Hence logical clock is a necessity for global state capture

SS ZG 526: Distributed Computing 11

How do we create a global state

SS ZG 526: Distributed Computing 12

6
8/9/2024

Strongly consistent global state

PAST FUTURE
LS1

No need to draw a vertical line /

LS2
cut or even a straight line since
process real times are not
LS3 synchronised

LS4

GS1 = { LS1, LS2, LS3, LS4 }

cut of a space-time diagram

SS ZG 526: Distributed Computing 13

Consistent global state

PAST FUTURE

M LS5

LS6

LS7
LS8

GS2 = { LS5, LS6, LS7, LS8} + M

SS ZG 526: Distributed Computing 14

7
8/9/2024

Inconsistent global state

PAST FUTURE
LS9

LS10
M
LS11

LS12

GS3 = { LS9, LS10, LS11, LS12} ??

But can’t include event M with send in the FUTURE and receive in the
PAST. Violates Causal Ordering.

SS ZG 526: Distributed Computing 15

Topics for today

» Consistent and in-consistent global states (revision)
» Why record global state
» Issues in recording state
» Algorithm for FIFO channels - Chandy Lamport
» Properties of Chandy-Lamport algorithm
» Optimizations on Chandy-Lamport - 2 more algorithm sketches
» Algorithm for non-FIFO channels - Lai Yang
» Necessary and sufficient conditions to record global state

SSWT ZG526 - Distributed Computing 16 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

8
8/9/2024

Issues to address in recording a global state (1)

Issue 1: How to distinguish between the messages to be
recorded in the snapshot* from those not to be recorded?
Solution: P1 snapshot taken

• Any message that is sent by a process before recording its P1

snapshot, must be recorded in the global snapshot.
m1 m2
• Any message that is sent by a process after recording its
snapshot, must not be recorded in the global snapshot. P2

m1 included in global state

m2 excluded in global state

* snapshot is a recording or image of global state

SSWT ZG526 - Distributed Computing 17 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Issues to address in recording a global state (2)

Issue 2: How to determine the instant when a process P1 snapshot taken

takes its snapshot?
Solution: P1

A process pj must record its snapshot before m1 m2

processing a message mij that was sent by process pi P2

after recording its snapshot. P2 snapshot taken
before this time

If P2 takes snapshot here then m2 receive is in

global snapshot but not m2 send —> inconsistent
SSWT ZG526 - Distributed Computing 18 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

9
8/9/2024

Difficulties due to Non-determinism

• Deterministic Computation
• At any point in computation there is at most one event that can happen
next.

• Non-Deterministic Computation
• At any point in computation there can be more than one event that can
happen next.

SSWT ZG526 - Distributed Computing 19 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Example

A B

m
A B

A B

Deterministic A B Non-deterministic
n
- serial - multiple solutions possible
- synchronous - one shown with the dotted arrows
A B

Easy to capture snapshots Snapshot algorithms need to handle non-determinism

SSWT ZG526 - Distributed Computing 20 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

10
8/9/2024

Topics for today

» Why record global state
» Consistent and in-consistent global states (revision)
» Issues in recording state
» Algorithm for FIFO channels - Chandy Lamport
» Properties of Chandy-Lamport algorithm
» Optimizations on Chandy-Lamport
» Algorithm for non-FIFO channels - Lai Yang
» Applications
» Necessary and sufficient conditions to record global state

SSWT ZG526 - Distributed Computing 21 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

• Snapshot recording algorithm discussed next is based on the principle that

inconsistent global states are never recorded

SSWT ZG526 - Distributed Computing 22 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

11
8/9/2024

Snapshot algorithms for FIFO channels

Chandy-Lamport algorithm
• The Chandy-Lamport algorithm uses a control message, called a marker whose role in a FIFO system is to separate messages in the
channels.
• A marker separates the messages in the channel into those to be included in the snapshot from those not to be recorded in the
snapshot.
• Include a message on channel or post-receipt.
• Markers serve as a barrier for state recording - past vs future

Markers are trying to

create a “cut”

Where? Depends when the

marker reaches Pi

Think of a ripple in water and a

specific wave (marker) hitting
islands (processes)

SSWT ZG526 - Distributed Computing 23 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Chandy-Lamport (2)
• Start: The algorithm can be initiated by any process by executing the “Marker Sending Rule”
by which
• (a) it records its local state and
• (b) sends a marker on each outgoing channel before sending any more messages.

marker
state S1
M1
P1 P2

P4 P3

SSWT ZG526 - Distributed Computing 24 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

12
8/9/2024

Chandy-Lamport (3)
• Propagation: A process executes the “Marker Receiving Rule” on receiving a marker on incoming
channel C.
• (a) If the process has not yet recorded its local state, it records the state of C as empty and
executes the “Marker Sending Rule” to record its local state.
• (b) If process has recorded state already, it records messages received on C after last
recording and before this marker on C.
S1 M1 S2
P1 P2
M2

M1
P4 P3

SSWT ZG526 - Distributed Computing 25 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Chandy-Lamport (4)

• Termination: The algorithm terminates after each process has received a marker
on all of its incoming channels.
• Dissemination: All the local snapshots get disseminated to all other processes and
all the processes can determine the global state. A strongly connected graph helps
even if topology is arbitrary.

P1 P2
Disseminate and forward local snapshots
on outgoing channels so that everyone can
construct a global snapshot.
P4 P3

SSWT ZG526 - Distributed Computing 26 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

13
8/9/2024

State Recording Example (1)

FIFO is a must
50 50 100 M 50 M
100 200 c1 150
c1 c1 100 50
50
A B A B A c2
B
c2 c2

c4 c3 c4 c3
c4 c3
100
C
‘A’ initiates
recording of C C
global state
100 200
200

‘A’ takes ‘B’ receives

Assume C has received 100 to avoid snapshot and marker and takes
multiple snapshot initiation in this example sends marker snapshot

SSWT ZG526 - Distributed Computing 27 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

State Recording Example (2)

150 c1 150
c1 150 50 c1 50
50
A B A B A c2
B
c2 c2
M M
c3 c4 c3
c4 c3 M c4
M
C C C
200 200
200

‘C’ records state ‘A’ records state of

‘B’ sends markers incoming channels
and sends marker
as empty.

Finally, A, B, C exchange local states including incoming channel states

SSWT ZG526 - Distributed Computing 28 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

14
8/9/2024

Properties of the recorded global state

• Chandy-Lamport algorithm shows an important property -

• The recorded global state may not correspond to any of the

global states that occurred during the actual computation.
• This happens because a process can change its state S1 S2
asynchronously before the markers it sent are received by other
sites and other sites record their states.
• But the system could have passed through the recorded St
global states in some equivalent executions.
Recorded state may not be
actually seen in this execution

SSWT ZG526 - Distributed Computing 29 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Example

Actual physical states in a computation :

A recorded state (down, , up, M’) is never physically reached.

ref: https://homepage.cs.uiowa.edu/~ghosh/10-16-03.pdf
SSWT ZG526 - Distributed Computing 30 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

15
8/9/2024

How useful is an unseen snapshot

• A recorded state is a feasible state that is reachable from the initial configuration.
• The final state is always reachable from the recorded state.
• The recorded state may not be visited / seen in an actual execution.
• These states are useful
• for debugging because it is possible for the system to reach the state in some
execution instance
• for checking stable predicates
• for checkpointing

SSWT ZG526 - Distributed Computing 31 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

» Why record global state
» Consistent and in-consistent global states
» Issues in recording state
» Algorithm for FIFO channels - Chandy Lamport
» Properties of Chandy-Lamport algorithm
» Optimizations on Chandy-Lamport
» Algorithm for non-FIFO channels - Lai Yang
» Applications
» Necessary and sufficient conditions to record global state

SSWT ZG526 - Distributed Computing 32 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

16
8/9/2024

Improvements of Chandy-Lamport

• Optimizations 1
• when multiple snapshots are triggered
• leading to inefficiency and redundant load
3 2
• to disseminate local states more efficiently
• flooding is too expensive
• build spanning tree during marker propagation -
works in undirected graphs
4 5

spanning tree edge

SSWT ZG526 - Distributed Computing 33 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Optimization algorithm 1 by Spezialetti-Kearns

• Provides 2 optimisations
• avoid taking redundant snapshots when there are multiple initiators
using “regions” or territories for each initiator
• leads to simpler dissemination of local snapshots
• Assumes undirected edges

SSWT ZG526 - Distributed Computing 34 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

17
8/9/2024

Spezialetti-Kearns outline - taking local snapshots

• Marker carries the ID of the initiator process

• Each process including initiator stores the ID of the initiator in
variable “master”
• On receiving a marker, a process who has no master will set master A B
• A region is all those processes that have the same master M: A
• On receiving a marker, if a process already has a master, it will set
M: B
a variable “id_border_set” with received ID in marker and not act
on or forward the marker C
• This sets up a border process between 2 regions M: B master: B
id_border_set: A

SSWT ZG526 - Distributed Computing 35 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Spezialetti-Kearns outline - disseminating snapshots

• When a process executes a marker sending rule by exchange
recording a local snapshot, it records the marker region info
sender as the parent
• This sets up a spanning tree A B
• A process sends its local snapshot and id_border_set
variable to its parent and it finally reaches the snapshot(c)
master of the region
• Initiators / masters can exchange region information C id_border_set: A

parent: B

Think of this as country boundaries getting dynamically formed to include

cities in their region and the capital cities initiating and exchanging
information about their country.
SSWT ZG526 - Distributed Computing 36 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

18
8/9/2024

Optimization algorithm 2 by Venkatesan

• Use case: Applications that need periodic snapshots for
checkpointing
1
• Naive solution: repeated invocations of Chandy-Lamport
• But not efficient m
3 2
• Venkatesan proposed incremental snapshot to save messages
because only few messages are sent on some channels between
2 checkpoints
m
• Assumes bi-directional FIFO links, single initiator, a fixed
spanning tree for sending control messages 4 5

checkpoint i checkpoint i+1

Think of a distributed database doing periodic checkpoints
across nodes so that it can recover from failures

SSWT ZG526 - Distributed Computing 37 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

New “control messages” - not just 1 marker

• Assign version number to snapshots

• All control messages carry the number
• init_snap: sent by initiator (spanning tree root) to children asking to start
snapshot
• snap_complete: sent by child to parent (not every neighbor) after taking
snapshot
• regular: like init_snap but sent by internal nodes to neighbors asking them to
take snapshot
• ack: acknowledge receipt of init_snap or regular messages

SSWT ZG526 - Distributed Computing 38 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

19
8/9/2024

Venkatesan’s modified “marker sending rule”

• Chandy-Lamport “marker sending rule”

• take a snapshot and send marker to all outgoing channels before
sending any more message
• Venkatesan’s “marker sending rule”
• take a snapshot and send “regular” (like a marker) message before
sending any other message but to only those channels where there is
activity after last snapshot

SSWT ZG526 - Distributed Computing 39 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Venkatesan’s outline
spanning tree edge

1. Initiator sends init_snap along spanning tree to start iteration for Initiator
one global snapshot version
2. Process receiving init_snap or regular follows modified marker 1
sending rule and waits for an ack
3. When leaf process finishes with local snapshot and all acks, it m
sends a snap_complete to parent. 3 2
4. When a non-leaf process gets all acks and snap_complete from
children, it sends snap_complete to parent.
m
5. Terminates when initiator receives all acks and snap_complete
from children 4 5

non-spanning tree edge

activity since last snapshot

SSWT ZG526 - Distributed Computing 40 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

20
8/9/2024

Venkatesan’s example
Initiator

1 1 sends init_snap and gets back

1 sends init_snap and gets back
ack, snap_complete ack, snap_complete

m
3 2
2 sends regular and gets
3 sends regular and gets back ack
m
back ack

4 5 no new snapshot needed

non-spanning tree edge
spanning tree edge 2 sends regular and gets back ack,
snap_complete

SSWT ZG526 - Distributed Computing 41 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Topics for today

» Why record global state
» Consistent and in-consistent global states
» Issues in recording state
» Algorithm for FIFO channels - Chandy Lamport
» Properties of Chandy-Lamport algorithm
» Optimizations on Chandy-Lamport
Why is it important ?
» Algorithm for non-FIFO channels - Lai Yang Network links can use protocols such as UDP,
» Necessary and sufficient conditions to record globalwith no ordering guarantee
state

SSWT ZG526 - Distributed Computing 42 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

21
8/9/2024

Global state recording for non-FIFO channels

• A marker (like in Chandy-Lamport) cannot be used to delineate

messages because marker cannot be guaranteed to arrive before
event sent after marker.
hold back a message add to another message

• Here, either some degree of inhibition or piggybacking of control

information on computation messages is to be used to capture out-of-
sequence messages.

SSWT ZG526 - Distributed Computing 43 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Lai-Yang algorithm for non-FIFO channels

The Lai-Yang algorithm uses a coloring scheme where processes and messages are colored RED
and WHITE. No special markers.
1 Every process is initially white and turns red while taking a snapshot. The equivalent of the
“Marker Sending Rule” is executed when a process turns red.
2 Every message sent by a white (red) process is colored white (red).

P1 takes snapshot

m3
m1
m2

SSWT ZG526 - Distributed Computing 44 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

22
8/9/2024

Lai-Yang algorithm for non-FIFO channels

3 Thus, a white (red) message is a message that was sent before (after) the sender of that
message recorded its local snapshot.
4 Every white process takes its snapshot at its convenience, but no later than the instant it
receives a red message.

P1 takes snapshot

m3
m1
m2
P2 has to take a snapshot
by this time (remember
Issue 2 rule for consistent
P2 snapshots in slide 16 ?)

SSWT ZG526 - Distributed Computing 45 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Lai-Yang algorithm continued…

5 Every white process records a history of all white messages sent or received by it
along each channel.
e.g. P1 (when white) records that it has sent m1 and m2, P2 (when white) records it
has received m1 and m3.
6 When a process turns red, it sends these histories along with its snapshot to the
initiator process that collects the global snapshot.
7 The initiator process evaluates transit (LSi, LSj) to compute the state of a channel Cij as
given below:
SCij= white messages sent by Pi on Cij − white messages received by Pj on Cij

SSWT ZG526 - Distributed Computing 46 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

23
8/9/2024

Example
Snapshot initiated - P1 turns RED
History : send [m1, m2] + local state of P1

m3 Initiator
m1
m2 Gets both histories in snapshot
Figures out m2 is in transit and m1,
m3 are accounted in local snapshot
P2 of P2

P2 takes snapshot and turns RED

History : received [m1, m3] + local state of P2

SSWT ZG526 - Distributed Computing 47 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Application of snapshots
» Looking for “consistent and stable” state across processes
» Checkpointing - restart a system from recorded global state and log of messages sent
after checkpoint
» Termination - useful in batch processing systems to know when set of distributed tasks
have completed
» Deadlock - similar to termination but jobs are stuck
» Debugging - inspect global state to find issues

SSWT ZG526 - Distributed Computing 48 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

24
8/9/2024

Topics for today

» Why record global state
» Consistent and in-consistent global states
» Issues in recording state
» Algorithm for FIFO channels - Chandy Lamport
» Properties of Chandy-Lamport algorithm
» Optimizations on Chandy-Lamport
» Algorithm for non-FIFO channels - Lai Yang
» Necessary and sufficient conditions to record global state

Why is it important ?
Distributed database nodes can take periodic checkpoints without strict time coordination with other nodes.
Need to pick which of these checkpoints are correct recording of global state across all nodes.

SSWT ZG526 - Distributed Computing 49 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Conditions for consistent checkpoints

» Each process takes a local checkpoint

» A set of checkpoints (1 for each process) need to satisfy some necessary and
sufficient conditions to form a consistent global state

SSWT ZG526 - Distributed Computing 50 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

25
8/9/2024

“No causal order” is a necessary condition

» Necessary condition: 2 checkpoints cannot have a causal order, i.e. they should be
parallel or potentially parallel. E.g. C1,0 and C3,1 have a causal ordering using {
m1, m2} and cannot be part of a consistent set of local checkpoints. But this is not
a sufficient condition. Why ?

SSWT ZG526 - Distributed Computing 51 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Why “no causal order” is not sufficient ?

» C1,1 and C3,2 have no causal order but cannot be part of a consistent global state
» C1,1 cannot pair with C2,2 and C3.2 cannot pair with C2,1 because of causal order

SSWT ZG526 - Distributed Computing 52 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

26
8/9/2024

Zigzag path

» Sufficient condition: 2 checkpoints cannot have a zigzag path connecting them

» zigzag path: Causal ordering + Message can be sent before arrival of previous on same
path, e.g. { m3, m4 }

SSWT ZG526 - Distributed Computing 53 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

Summary
» Why is global snapshot important
» Algorithms for FIFO using markers and non-FIFO using coloring
» Optimized FIFO algorithms
» Property of snapshot in Chandy-Lamport algorithm
» No zig-zag path as a necessary and sufficient condition of consistency among
snapshots
» Study material: Chapter 4 of Text book 1 (T1)

SSWT ZG526 - Distributed Computing 54 BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

27
8/9/2024

SS ZG 526: Distributed Computing

Session 4: Terminology and basic algorithms

Dr. Anindya Neogi

Associate Professor
[email protected]

Reference: digital content slides from Prof. C. Hota, BITS Pilani

Topics for today

» Distributed algorithms : Basic concepts / terms / classifications
» Example distributed graph algorithms and use cases
» Single source shortest path
» Minimum Spanning Tree (MST)
» Leader election
» Maximal Independent Sets (MIS)
» Connected Dominating Set (CDS)
» Synchronizers

1
8/9/2024

Topology abstractions
» Physical
» Actual network layer connections
» Logical
» Defined in context of a particular application. Nodes
are distributed application nodes. Channels can be
given useful properties, e.g. FIFO, non-FIFO. Can define
arbitrary connectivity or neighbourhood relations. E.g. in
P2P overlays.
» Super-imposed
Logical abstractions
» Higher level on logical topology, e.g. spanning tree,
ring, to provide efficient paths for information
dissemination and gathering.

Classifications (1)
» Application vs control executions
» Application execution is the core program logic
» Control algorithms or “protocols” are part of the distributed middleware. Latter may piggyback
on the application execution (e.g. Lai-Yang global snapshot) but doesn’t interfere.
» Centralized vs distributed
» Centralized - e.g. client-server systems.
» Purely distributed yet efficient algorithms are harder. Many distributed algorithms have a
centralised component - e.g. Chandy-Lamport has an initiator that assembles the global
snapshot. Peer-to-Peer systems demand more distributed logic.

2
8/9/2024

Classifications (2)
» Symmetric vs asymmetric
» Symmetric where all processors execute same logic.
» e.g. logical clock algorithms
» Centralized algorithms or cases where nodes perform different functions, e.g.
root and leaf nodes or leaders, are asymmetric.
» Anonymous
» Does not use process identifiers - structurally elegant but harder to build and
sometimes impossible - e.g. anonymous leader election, total ordering in Lamport
clock. Typically process id is used for resolving ties.

Classifications (3)
» Uniform
» Does not use n or number of processes in code and thus allows scalability transparency. E.g. leader election
in a ring only talks to 2 neighbours.
» Adaptive
» When complexity is a function of k < n and not n, where n is number of nodes / processes. e.g. mutual
exclusion algorithms between k contenders
» Deterministic vs Non-deterministic
» Deterministic programs have no non-deterministic receives, i.e. they specify a source always to receive a
message.
» Non-deterministic executions may produce different results each time. Harder to reason and debug.
» Even in an async system deterministic execution will produce same partial order each time but not for non-
deterministic execution.

3
8/9/2024

Classifications (4)
» Execution Inhibition
» Inhibitory protocols freeze/suspend the normal execution of a process till some conditions are met. Can be
local inhibition (e.g. waiting for a local condition) or global (e.g. waiting for message from another process).
» Message ordering or state recording algorithms may use inhibition
» Sync vs Async
» Sync has
1. known upper bound of communication delay,
2. known bounded drift rate of local clock wrt real time,
3. known upper bound on time taken to perform a logical step.
» Async has none of the above satisfied. Typical distributed systems are async but can be made sync with
synchronisers.

Classifications (5)
» Online vs offline
» Online algorithm works on data that is being generated.
» Preferred. E.g. debugging, scheduling etc. to work with latest dynamic data
» Offline needs all data to be available first
» Wait-free
» Resilient to n-1 process failures thus offering high degree of robustness. But very expensive and may
not be possible in many cases.
» Operations of a process complete in a bounded number of steps despite failures of other processes
» Useful for real-time systems
» e.g. lock-less page cache patches in Linux kernel, RethinkDB
» Will refer in session on “consensus protocols”

4
8/9/2024

Classifications (6)
» Failure models
» Important to specify for an algorithm that claims to have some fault tolerance features
» Process failure models with increasing severity
» Fail-stop: Process stops at an instant and others know about it
» Crash: Same as above but others don’t know about it
» Receive/Send or General omission: Intermittently fail to send and/or receive some messages or
by crashing
» Timing failures: For sync systems only, e.g. violates time bound to finish a step
» Byzantine failure with authentication: Random faults but if a faulty process claims to receive a
message from correct process, it can be verified
» Byzantine failure: Same as above but no verification possible
» In addition, Link failure models include all of the above except fail-stop

Topics for today

» Distributed algorithms : Basic concepts / terms
» Example distributed graph algorithms and use cases
» Single source shortest path
Application level routing
» Minimum Spanning Tree (MST) Content distribution networks
» Leader election
Asymmetric distributed algorithms
» Maximal Independent Sets (MIS)
Setting up wireless networks
» Connected Dominating Set (CDS) Intelligent placement of servers / point-of-presence
Backbone networks
» Synchronizers

5
8/9/2024

1. Bellman-Ford: Single source shortest path

incident link for A
source
» From source S find the shortest path to all nodes in the directed
graph with possibly negative weights*
S
» But it is a distributed algorithm … 8 10
» Topology is not known globally
» Each node knows only its neighbors and incident links and E A
weights -4
1
» Only send messages to neighbours about local state 1
2
» Each process knows total number of nodes n, so that it can
terminate computation D B
» Eg. use — distributed routing protocols -1 -2

C
* negative weights ?? think of currency exchange, ISP contracts for routing etc.
SS ZG 526: Distributed Computing 11

Bellman-Ford: Single source shortest path (sync)

Logic on each process i with knowledge of n incident link for A
source
• length = inf <- distance to source
• parent = null round 1:
S
• neighbors = {set of neighbors} 8 10 parent = S
• weights = {weight(i,j), weight (j,i) for all
length = 10
neighbors j of process i }
E A final:
-4 parent = D
• on receive of UPDATE message from neighbor length = 5
1
• if i is source S then length=0 1
2
• for round=1 to n-1 do <- synchronise
• send UPDATE(i, length) to all neighbours round 1 UPDATE:
D B S—>A: 10
Parallel across all Pi

• await UPDATE(j, length) from each

neighbor j
-1 -2
• for each neighbor j do
• if length > length j + weight (j,i) C
then
• length = length j + weight (j,i)
• parent
SS ZG 526: Distributed Computing = j 12

6
8/9/2024