Source : https://en.wikipedia.

org/wiki/Distributed_computing
Distributed computing
From Wikipedia, the free encyclopedia
"Distributed Information Processing" redirects here. For the computer company, see DIP
Research.
"Distributed application" redirects here. For smart contracts, see Decentralized application.
Distributed computing is a field of computer science that studies distributed systems.
A distributed system is a model in which components located on networked
computers communicate and coordinate their actions by passing messages.[1] The components
interact with each other in order to achieve a common goal. Three significant characteristics of
distributed systems are: concurrency of components, lack of a global clock, and independent
failure of components.[1] Examples of distributed systems vary from SOA-based
systems to massively multiplayer online games to peer-to-peer applications.
A computer program that runs in a distributed system is called a distributed program, and
distributed programming is the process of writing such programs. [2] There are many alternatives
for the message passing mechanism, including pure HTTP, RPC-like connectors and message
queues.[citation needed]
A goal and challenge pursued by some computer scientists and practitioners in distributed
systems is location transparency; however, this goal has fallen out of favour in industry, as
distributed systems are different from conventional non-distributed systems, and the differences,
such as network partitions, partial system failures, and partial upgrades, cannot simply be
"papered over" by attempts at "transparency" (see CAP theorem).[citation needed]
Distributed computing also refers to the use of distributed systems to solve computational
problems. In distributed computing, a problem is divided into many tasks, each of which is solved
by one or more computers,[3]which communicate with each other by message passing. [4]

Contents
  [hide] 

 1Introduction
 2Parallel and distributed computing
 3History
 4Architectures
 5Applications
 6Examples
 7Theoretical foundations
o 7.1Models
o 7.2An example
o 7.3Complexity measures
o 7.4Other problems
o 7.5Properties of distributed systems
 8See also
 9Notes
 10References
 11Further reading
 12External links

Introduction[edit]
The word distributed in terms such as "distributed system", "distributed programming", and
"distributed algorithm" originally referred to computer networks where individual computers were
physically distributed within some geographical area. [5] The terms are nowadays used in a much
wider sense, even referring to autonomous processes that run on the same physical computer
and interact with each other by message passing.[4]
While there is no single definition of a distributed system, [6] the following defining properties are
commonly used:

 There are several autonomous computational entities (computers or nodes), each of which

has its own local memory.[7]
 The entities communicate with each other by message passing.[8]
A distributed system may have a common goal, such as solving a large computational problem;
[9]
 the user then perceives the collection of autonomous processors as a unit. Alternatively, each
computer may have its own user with individual needs, and the purpose of the distributed system
is to coordinate the use of shared resources or provide communication services to the users. [10]
Other typical properties of distributed systems include the following:

 The system has to tolerate failures in individual computers.[11]

 The structure of the system (network topology, network latency, number of computers) is not
known in advance, the system may consist of different kinds of computers and network links,
and the system may change during the execution of a distributed program. [12]
 Each computer has only a limited, incomplete view of the system. Each computer may know
only one part of the input.[13]

Parallel and distributed computing[edit]

(a), (b): a distributed system.

(c): a parallel system.

Distributed systems are groups of networked computers, which have the same goal for their
work. The terms "concurrent computing", "parallel computing", and "distributed computing" have
a lot of overlap, and no clear distinction exists between them. [14] The same system may be
characterized both as "parallel" and "distributed"; the processors in a typical distributed system
run concurrently in parallel.[15] Parallel computing may be seen as a particular tightly coupled form
of distributed computing,[16] and distributed computing may be seen as a loosely coupled form of
parallel computing.[6] Nevertheless, it is possible to roughly classify concurrent systems as
"parallel" or "distributed" using the following criteria:

 In parallel computing, all processors may have access to a shared memory to exchange
information between processors.[17]
 In distributed computing, each processor has its own private memory (distributed memory).
Information is exchanged by passing messages between the processors. [18]
The figure on the right illustrates the difference between distributed and parallel systems. Figure
(a) is a schematic view of a typical distributed system; the system is represented as a network
topology in which each node is a computer and each line connecting the nodes is a
communication link. Figure (b) shows the same distributed system in more detail: each computer
has its own local memory, and information can be exchanged only by passing messages from
one node to another by using the available communication links. Figure (c) shows a parallel
system in which each processor has a direct access to a shared memory.
The situation is further complicated by the traditional uses of the terms parallel and
distributed algorithm that do not quite match the above definitions of parallel and
distributed systems (see below for more detailed discussion). Nevertheless, as a rule of thumb,
high-performance parallel computation in a shared-memory multiprocessor uses parallel
algorithms while the coordination of a large-scale distributed system uses distributed algorithms.
[citation needed]

History[edit]
The use of concurrent processes that communicate by message-passing has its roots
in operating system architectures studied in the 1960s.[19] The first widespread distributed
systems were local-area networks such as Ethernet, which was invented in the 1970s.[20]
ARPANET, the predecessor of the Internet, was introduced in the late 1960s, and ARPANET e-
mail was invented in the early 1970s. E-mail became the most successful application of
ARPANET,[21] and it is probably the earliest example of a large-scale distributed application. In
addition to ARPANET, and its successor, the Internet, other early worldwide computer networks
included Usenet and FidoNet from the 1980s, both of which were used to support distributed
discussion systems.[citation needed]
The study of distributed computing became its own branch of computer science in the late 1970s
and early 1980s. The first conference in the field, Symposium on Principles of Distributed
Computing (PODC), dates back to 1982, and its European counterpart International Symposium
on Distributed Computing (DISC) was first held in 1985.[citation needed]

Architectures[edit]
Various hardware and software architectures are used for distributed computing. At a lower level,
it is necessary to interconnect multiple CPUs with some sort of network, regardless of whether
that network is printed onto a circuit board or made up of loosely coupled devices and cables. At
a higher level, it is necessary to interconnect processes running on those CPUs with some sort
of communication system.[citation needed]
Distributed programming typically falls into one of several basic architectures: client–
server, three-tier, n-tier, or peer-to-peer; or categories: loose coupling, or tight coupling.[22]

 Client–server: architectures where smart clients contact the server for data then format and
display it to the users. Input at the client is committed back to the server when it represents a
permanent change.
 Three-tier: architectures that move the client intelligence to a middle tier so that stateless
clients can be used. This simplifies application deployment. Most web applications are three-
tier.
 n-tier: architectures that refer typically to web applications which further forward their
requests to other enterprise services. This type of application is the one most responsible for
the success of application servers.
 Peer-to-peer: architectures where there are no special machines that provide a service or
manage the network resources.[23]:227 Instead all responsibilities are uniformly divided among
all machines, known as peers. Peers can serve both as clients and as servers. [citation needed]
Another basic aspect of distributed computing architecture is the method of communicating and
coordinating work among concurrent processes. Through various message passing protocols,
processes may communicate directly with one another, typically in a master/slave relationship.
Alternatively, a "database-centric" architecture can enable distributed computing to be done
without any form of direct inter-process communication, by utilizing a shared database.[24]

Applications[edit]
Reasons for using distributed systems and distributed computing may include:

1. The very nature of an application may require the use of a communication network that

connects several computers: for example, data produced in one physical location and
required in another location.
2. There are many cases in which the use of a single computer would be possible in
principle, but the use of a distributed system is beneficial for practical reasons. For
example, it may be more cost-efficient to obtain the desired level of performance by
using a cluster of several low-end computers, in comparison with a single high-end
computer. A distributed system can provide more reliability than a non-distributed
system, as there is no single point of failure. Moreover, a distributed system may be
easier to expand and manage than a monolithic uniprocessor system. [25]

Examples[edit]
Examples of distributed systems and applications of distributed computing include the following:
[26]

 telecommunication networks:
 telephone networks and cellular networks,
 computer networks such as the Internet,
 wireless sensor networks,
 routing algorithms;
 network applications:
 World Wide Web and peer-to-peer networks,
 massively multiplayer online games and virtual reality communities,
 distributed databases and distributed database management systems,
 network file systems,
 distributed information processing systems such as banking systems and airline
reservation systems;
 real-time process control:
 aircraft control systems,
 industrial control systems;
 parallel computation:
 scientific computing, including cluster computing and grid computing and
various volunteer computing projects (see the list of distributed computing projects),
 distributed rendering in computer graphics
Theoretical foundations[edit]
Main article: Distributed algorithm
Models[edit]
Many tasks that we would like to automate by using a computer are of question–answer type: we
would like to ask a question and the computer should produce an answer. In theoretical
computer science, such tasks are called computational problems. Formally, a computational
problem consists of instances together with a solution for each instance. Instances are questions
that we can ask, and solutions are desired answers to these questions.
Theoretical computer science seeks to understand which computational problems can be solved
by using a computer (computability theory) and how efficiently (computational complexity theory).
Traditionally, it is said that a problem can be solved by using a computer if we can design
an algorithm that produces a correct solution for any given instance. Such an algorithm can be
implemented as a computer program that runs on a general-purpose computer: the program
reads a problem instance from input, performs some computation, and produces the solution
as output. Formalisms such as random access machines or universal Turing machines can be
used as abstract models of a sequential general-purpose computer executing such an algorithm.
[citation needed]

The field of concurrent and distributed computing studies similar questions in the case of either
multiple computers, or a computer that executes a network of interacting processes: which
computational problems can be solved in such a network and how efficiently? However, it is not
at all obvious what is meant by "solving a problem" in the case of a concurrent or distributed
system: for example, what is the task of the algorithm designer, and what is the concurrent or
distributed equivalent of a sequential general-purpose computer? [citation needed]
The discussion below focuses on the case of multiple computers, although many of the issues
are the same for concurrent processes running on a single computer.
Three viewpoints are commonly used:
Parallel algorithms in shared-memory model

 All processors have access to a shared memory. The algorithm designer chooses the
program executed by each processor.
 One theoretical model is the parallel random access machines (PRAM) that are used.
[27]
 However, the classical PRAM model assumes synchronous access to the shared memory.
 Shared-memory programs can be extended to distributed systems if the underlying operating
system encapsulates the communication between nodes and virtually unifies the memory
across all individual systems.
 A model that is closer to the behavior of real-world multiprocessor machines and takes into
account the use of machine instructions, such as Compare-and-swap (CAS), is that
of asynchronous shared memory. There is a wide body of work on this model, a summary of
which can be found in the literature.[28][29]
Parallel algorithms in message-passing model

 The algorithm designer chooses the structure of the network, as well as the program
executed by each computer.
 Models such as Boolean circuits and sorting networks are used.[30] A Boolean circuit can be
seen as a computer network: each gate is a computer that runs an extremely simple
computer program. Similarly, a sorting network can be seen as a computer network: each
comparator is a computer.
Distributed algorithms in message-passing model

 The algorithm designer only chooses the computer program. All computers run the same
program. The system must work correctly regardless of the structure of the network.
 A commonly used model is a graph with one finite-state machine per node.
In the case of distributed algorithms, computational problems are typically related to graphs.
Often the graph that describes the structure of the computer network is the problem instance.
This is illustrated in the following example. [citation needed]
An example[edit]
Consider the computational problem of finding a coloring of a given graph G. Different fields
might take the following approaches:
Centralized algorithms[citation needed]

 The graph G is encoded as a string, and the string is given as input to a computer. The
computer program finds a coloring of the graph, encodes the coloring as a string, and
outputs the result.
Parallel algorithms

 Again, the graph G is encoded as a string. However, multiple computers can access the
same string in parallel. Each computer might focus on one part of the graph and produce a
coloring for that part.
 The main focus is on high-performance computation that exploits the processing power of
multiple computers in parallel.
Distributed algorithms

 The graph G is the structure of the computer network. There is one computer for each node
of G and one communication link for each edge of G. Initially, each computer only knows
about its immediate neighbors in the graph G; the computers must exchange messages with
each other to discover more about the structure of G. Each computer must produce its own
color as output.
 The main focus is on coordinating the operation of an arbitrary distributed system. [citation needed]
While the field of parallel algorithms has a different focus than the field of distributed algorithms,
there is a lot of interaction between the two fields. For example, the Cole–Vishkin algorithm for
graph coloring[31] was originally presented as a parallel algorithm, but the same technique can
also be used directly as a distributed algorithm.
Moreover, a parallel algorithm can be implemented either in a parallel system (using shared
memory) or in a distributed system (using message passing). [32] The traditional boundary between
parallel and distributed algorithms (choose a suitable network vs. run in any given network) does
not lie in the same place as the boundary between parallel and distributed systems (shared
memory vs. message passing).
Complexity measures[edit]
In parallel algorithms, yet another resource in addition to time and space is the number of
computers. Indeed, often there is a trade-off between the running time and the number of
computers: the problem can be solved faster if there are more computers running in parallel
(see speedup). If a decision problem can be solved in polylogarithmic time by using a polynomial
number of processors, then the problem is said to be in the class NC.[33] The class NC can be
defined equally well by using the PRAM formalism or Boolean circuits—PRAM machines can
simulate Boolean circuits efficiently and vice versa.[34]
In the analysis of distributed algorithms, more attention is usually paid on communication
operations than computational steps. Perhaps the simplest model of distributed computing is a
synchronous system where all nodes operate in a lockstep fashion. During each communication
round, all nodes in parallel (1) receive the latest messages from their neighbours, (2) perform
arbitrary local computation, and (3) send new messages to their neighbors. In such systems, a
central complexity measure is the number of synchronous communication rounds required to
complete the task.[35]
This complexity measure is closely related to the diameter of the network. Let D be the diameter
of the network. On the one hand, any computable problem can be solved trivially in a
synchronous distributed system in approximately 2D communication rounds: simply gather all
information in one location (D rounds), solve the problem, and inform each node about the
solution (D rounds).
On the other hand, if the running time of the algorithm is much smaller than D communication
rounds, then the nodes in the network must produce their output without having the possibility to
obtain information about distant parts of the network. In other words, the nodes must make
globally consistent decisions based on information that is available in their local neighbourhood.
Many distributed algorithms are known with the running time much smaller than D rounds, and
understanding which problems can be solved by such algorithms is one of the central research
questions of the field.[36]
Other commonly used measures are the total number of bits transmitted in the network
(cf. communication complexity).[citation needed]
Other problems[edit]
Traditional computational problems take the perspective that we ask a question, a computer (or a
distributed system) processes the question for a while, and then produces an answer and stops.
However, there are also problems where we do not want the system to ever stop. Examples of
such problems include the dining philosophers problem and other similar mutual
exclusion problems. In these problems, the distributed system is supposed to continuously
coordinate the use of shared resources so that no conflicts or deadlocks occur.
There are also fundamental challenges that are unique to distributed computing. The first
example is challenges that are related to fault-tolerance. Examples of related problems
include consensus problems,[37]Byzantine fault tolerance,[38] and self-stabilisation.[39]
A lot of research is also focused on understanding the asynchronous nature of distributed
systems:

 Synchronizers can be used to run synchronous algorithms in asynchronous systems. [40]

 Logical clocks provide a causal happened-before ordering of events.[41]
 Clock synchronization algorithms provide globally consistent physical time stamps. [42]
Election