Distributed Systems

(3rd Edition)

Chapter 02: Architectures

Version: March 20, 2022
Architectures: Introduction

Architectures
• Distributed Systems are often complex pieces of software
• Those components are by definition dispersed across
multiple machines
• There are different ways on how to view the organization of
DS
• Logical organization of the collection of software
components
• The actual physical realization
• The organization of DS is mostly about the SW components
that constitute the system
• The Software architectures tell us
• how the various software components are to be organized, and
• how they should interact

2 / 36
Outline

• Architectural styles
• Middleware organization
• System architecture
Architectures: Architectural styles

Architectural styles

Basic idea
A style is formulated in terms of
► (replaceable) components with well-defined interfaces
► the way that components are connected to each other
► the data exchanged between components
► how these components and connectors are jointly configured into
a system.

Connector
A mechanism that mediates communication, coordination, or
cooperation among components. Example: facilities for (remote)
procedure call, messaging, or streaming.

4 / 36
Architectures: Architectural styles

Architectural styles

The most important distributed systems architecture styles

are:
► Layered architectures
► Object-based architectures
► Resource-centered architectures
► Event-based architectures

Important note
Most real-world distributed systems combined different styles

5 / 36
Architectures: Architectural styles Layered architectures

Layered architecture

The basic idea of layered style is :

• Components are organized in a layered fashion

where a component at layer Lj can make a
downcall to a component at lower-level layer Li
(with i< j) and generally expects a response

• Only in exceptional cases will an upcall be made

to a higher-level component

6 / 36
Architectures: Architectural styles Layered architectures

Layered architecture

Different layered organizations

Request/Response
downcall One-way call

Layer N Layer N Layer N

Layer N-1 Layer N-1 Layer N-1

Handle
Upcall
Layer N-2
Layer N-2

Layer 2
Layer N-3

Layer 1

(a) (b) (c)

7 / 36
Architectures: Architectural styles Layered architectures

Example: communication protocols

Protocol, service, interface

Party A Party B

Layer N Layer N

Interface Service

Layer N-1 Layer N-1

Protocol

Layered communication protocols 8 / 36

Architectures: Architectural styles Layered architectures

Two-party communication

Server
1 from so ck et import *
2 s = socket(AF_INET, SOCK_STREAM)
3 (conn, a d d r) = s . a c c e p t ( ) # re t u r n s new s o c k e t and a d d r. c l i e n t
4 while True: # f o re v e r
5 d a t a = conn.recv(1024) # re c e i v e data from c l i e n t
6 i f not d a t a : break # s t o p i f c l i e n t stopped
7 conn.send(str(data)+" * ") # re t u r n s e n t data p l u s an " * "
8 co n n .clo se() # c l o s e t h e connection

Client
1 from so ck et import *
2 s = socket(AF_INET, SOCK_STREAM)
3 s.connect((HOST, PORT)) # connect t o s e r v e r (b l o c k u n t i l accepted)
4 s. se n d ( ’He l l o , world’) # send some data
5 d a t a = s.recv(1024) # re c e i v e t h e response
6 print d a t a # p rin t th e re su l t
7 s.close() # c l o s e t h e connection

Layered communication protocols 9 / 36

Architectures: Architectural styles Layered architectures

Application Layering

Traditional three-layered view

► Application-interface layer contains units for interfacing to users or
external applications
► Processing layer contains the functions of an application, i.e.,
without specific data
► Data layer contains the data that a client wants to manipulate
through the application components

Application layering 6 / 36
Architectures: Architectural styles Layered architectures

Application Layering

Traditional three-layered view

► Application-interface layer contains units for interfacing to users or
external applications
► Processing layer contains the functions of an application, i.e.,
without specific data
► Data layer contains the data that a client wants to manipulate
through the application components

Observation
This layering is found in many distributed information systems, using
traditional database technology and accompanying applications.

Application layering 6 / 36
Architectures: Architectural styles Layered architectures

Application Layering

Example: a simple search engine

User-interface
User interface level

HTML page
Keyword expression containing list
HTML
generator Processing
Query Ranked list level
generator of page titles
Ranking
Database queries algorithm

Web page titles

with meta-information
Database Data level
with Web
pages

Application layering 7 / 36
Architectures: Architectural styles Object-based and service-oriented architectures

Object-based style

Essence
Components are objects, connected to each other through procedure
calls. Objects may be placed on different machines; calls can thus
execute across a network.

State
Object Object

Method
Method call
Object

Object
Object
Interface

Encapsulation
Objects are said to encapsulate data and offer methods on that data
without revealing the internal implementation.

8 / 36
Architectures: Architectural styles Resource-based architectures

RESTful architectures
Essence
View a distributed system as a collection of resources, individually
managed by components. Resources may be added, removed,
retrieved, and modified by (remote) applications.
1. Resources are identified through a single naming scheme
2. All services offer the same interface
3. Messages sent to or from a service are fully self-described
4. After executing an operation at a service, that component forgets
everything about the caller

Basic operations
Operation Description
PUT Create a new resource
GET Retrieve the state of a resource in some representation
DELETE Delete a resource
POST Modify a resource by transferring a new state

9 / 36
Architectures: Architectural styles Resource-based architectures

Example: Amazon’s Simple Storage Service

Essence
Objects (i.e., files) are placed into buckets (i.e., directories). Buckets
cannot be placed into buckets. Operations on ObjectName in bucket
Bucket Name require the following identifier:

http://BucketName.s3.amazonaws.com/ObjectName

Typical operations
All operations are carried out by sending HTTP requests:
► Create a bucket/object: PUT, along with the URI
► Listing objects: GET on a bucket name
► Reading an object: GET on a full URI

10 / 36
Architectures: Architectural styles Resource-based architectures

On interfaces
Issue
Many people like RESTful approaches because the interface to a
service is so simple. The catch is that much needs to be done in the
parameter space.

Amazon S3 SOAP interface

Bucket operations Object operations
ListAllMyBuckets PutObjectInline
CreateBucket PutObject
DeleteBucket CopyObject
ListBucket GetObject
GetBucketAccessControlPolicy GetObjectExtended
SetBucketAccessControlPolicy DeleteObject
GetBucketLoggingStatus GetObjectAccessControlPolicy
SetBucketLoggingStatus SetObjectAccessControlPolicy

11 / 36
Architectures: Architectural styles Resource-based architectures

On interfaces

Simplifications
Assume an interface bucket offering an operation create, requiring
an input string such as mybucket, for creating a bucket “mybucket.”

12 / 36
Architectures: Architectural styles Resource-based architectures

On interfaces

Simplifications
Assume an interface bucket offering an operation create, requiring
an input string such as mybucket, for creating a bucket “mybucket.”

SOAP
import bucket
bucket . creat e( "mybucket ")

12 / 36
Architectures: Architectural styles Resource-based architectures

On interfaces

Simplifications
Assume an interface bucket offering an operation create, requiring
an input string such as mybucket, for creating a bucket “mybucket.”

SOAP
import bucket
bucket . creat e( "mybucket ")

RESTful
PUT "
http://mybucket.s3.amazon
sws.com/
"

12 / 36
Architectures: Architectural styles Resource-based architectures

On interfaces

Simplifications
Assume an interface bucket offering an operation create, requiring
an input string such as mybucket, for creating a bucket “mybucket.”

SOAP
import bucket
bucket . creat e( "mybucket ")

RESTful
PUT "
http://mybucket.s3.amazon
sws.com/
"

Conclusions
Are there any to draw?
12 / 36
Architectures: Architectural styles Publish-subscribe architectures

Coordination
Temporal and referential coupling
Temporally Temporally
coupled decoupled
Referentially Direct Mailbox
coupled
Referentially Event- Shared
decoupled based data space

Event-based and Shared data space

Component Component Component Component

Subscribe Notification
Publish Subscribe Data
delivery
delivery
Event bus

Publis h

Component
Shared (persistent) data space
13 / 36
Architectures: Architectural styles Publish-subscribe architectures

Example: Linda tuple space

Three simple operations

► i n ( t ) : remove a tuple matching template t
► r d ( t ) : obtain copy of a tuple matching template t
► o u t ( t ) : add tuple t to the tuple space

More details
► Calling o u t ( t ) twice in a row, leads to storing two copies of tuple
► t ⇒ a tuple space is modeled as a multiset.
Both i n and r d are blocking operations: the caller will be blocked
until a matching tuple is found, or has become available.

14 / 36
Architectures: Architectural styles Publish-subscribe architectures

Example: Linda tuple space

Bob
1 blog = linda.universe._rd(("MicroBlog",linda.TupleSpace))[1]
2
3 b l o g . _ o u t (( "b o b " , "d i st sy s" , "I am studying chap 2 " ) )
4 blog._out(("bob","distsys","The l i n d a example’s p r e t t y si mp le"))
5 blog._out(("bob","gtcn","Cool book!"))

Alice
1 blog = linda.universe._rd(("MicroBlog",linda.TupleSpace))[1]
2
3 b l o g . _ o u t (( "a l i c e " , "g t c n " , "T h i s graph th eo ry s t u f f i s n o t e a s y " ) )
4 b l o g . _ o u t ( ( " a l i c e " , " d i s t s y s " , " I l i k e systems more t h a n g rap h s"))

Chuck
1 blog = linda.universe._rd(("MicroBlog",linda.TupleSpace))[1]
2
3 t 1 = blog._rd(("bob","distsys",str))
4 t 2 = blog._rd(("alice","gtcn",str))
5 t 3 = b l o g . _ rd (( "b o b " , "g t c n " , str ))

15 / 36
Architectures: Middleware organization Wrappers

Using legacy to build middleware

Problem
The interfaces offered by a legacy component are most likely not
suitable for all applications.

Solution
A wrapper or adapter offers an interface acceptable to a client
application. Its functions are transformed into those available at the
component.

16 / 36
Architectures: Middleware organization Wrappers

Organizing wrappers

Two solutions: 1-on-1 or through a broker

Wrapper

Application Broker

Complexity with N applications

► 1-on-1: requires N × (N − 1) = O(N 2 )
wrappers
► broker: requires 2N = O(N) wrappers

17 / 36
Architectures: Middleware organization Interceptors

Developing adaptable middleware

Problem
Middleware contains solutions that are good for most applications ⇒
you may want to adapt its behavior for specific applications.

18 / 36
Architectures: Middleware organization Interceptors

Intercept the usual flow of control

Client application
Intercepted call
B.doit(val)

Application stub

Request-level interceptor Nonintercepted call

invoke(B, &doit, val)

Object middleware

Message-level interceptor

send(B, “doit”, val)

Local OS

To object B

19 / 36
Architectures: System architecture Centralized organizations

Centralized system architectures

Basic Client–Server Model

Characteristics:
► There are processes offering services (servers)
► There are processes that use services (clients)
► Clients and servers can be on different machines
► Clients follow request/reply model with respect to using services
Client Server

Request

Wait Provide service

Reply

Simple client-server architecture 20 / 36

Architectures: System architecture Centralized organizations

Multi-tiered centralized system architectures

Some traditional organizations
► Single-tiered: dumb terminal/mainframe configuration
► Two-tiered: client/single server configuration
► Three-tiered: each layer on separate machine

Traditional two-tiered configurations

Client machine

User interface User interface User interface User interface User interface
Application Application
Application
Database

User interface

Application Application Application

Database Database Database Database Database

Server machine
(a) (b) (c) (d) (e)
Multitiered Architectures 21 / 36
Architectures: System architecture Centralized organizations

Being client and server at the same time

Three-tiered architecture
Client Application Database
server server
Request
operation
R
e
q
Wait for Wait for u
e
reply data s
t
d Return
a data
Return t
reply a

Multitiered Architectures 22 / 36
Architectures: System architecture Decentralized organizations: peer-to-peer systems

Alternative organizations

Vertical distribution
Comes from dividing distributed applications into three logical layers,
and running the components from each layer on a different server
(machine).

Horizontal distribution
A client or server may be physically split up into logically equivalent
parts, but each part is operating on its own share of the complete data
set.

Peer-to-peer architectures
Processes are all equal: the functions that need to be carried out are
represented by every process ⇒ each process will act as a client and
a server at the same time (i.e., acting as a servant).

23 / 36
Architectures: System architecture Decentralized organizations: peer-to-peer systems

Structured P2P
Essence
Make use of a semantic-free index: each data item is uniquely
associated with a key, in turn used as an index. Common practice: use
a hash function
key(data item) = hash(data item’s value).
P2P system now responsible for storing (key,value) pairs.

Simple example: hypercube

0000 0001 1001

1000
0010 0011 1011
1010

0100
0101 1101
1100
0110 0111 1111
1110

Looking up d with key k ∈ {0, 1, 2,..., 24 − 1} means routing request

to node with identifier k .
Structured peer-to-peer systems 24 / 36
Architectures: System architecture Decentralized organizations: peer-to-peer systems

Example: Chord

Principle
► Nodes are logically organized in a ring. Each node has an m-bit
identifier.
► Each data item is hashed to an m-bit key.
► Data item with key k is stored at node with smallest identifier
id ≥ k , called the successor of key k .
► The ring is extended with various shortcut links to other nodes.

Structured peer-to-peer systems 25 / 36

Architectures: System architecture Decentralized organizations: peer-to-peer systems

Example: Chord

31 0 1
30 2
29 3
Actual node Shortcut 4
28
5
27
26 6

Nonexisting 25 7
node

24 8

23 Node responsible for

9
keys {5,6,7,8,9}
22 10

21 11

20 12
19 13
18 14
17 16 15

lookup(3)@9 : 28
Structured peer-to-peer systems →1→4 26 / 36
Architectures: System architecture Decentralized organizations: peer-to-peer systems

Unstructured P2P

Essence
Each node maintains an ad hoc list of neighbors. The resulting overlay
resembles a random graph: an edge (u, v) exists only with a certain
probability P[(u, v )].

Searching
► Flooding: issuing node u passes request for d to all neighbors.
Request is ignored when receiving node had seen it before.
Otherwise, v searches locally for d (recursively). May be limited
by a Time-To-Live: a maximum number of hops.
► Random walk: issuing node u passes request for d to randomly
chosen neighbor, v . If v does not have d , it forwards request to
one of its randomly chosen neighbors, and so on.

Unstructured peer-to-peer systems 27 / 36

Architectures: System architecture Decentralized organizations: peer-to-peer systems

Flooding versus random walk

Model
Assume N nodes and that each data item is replicated across r
randomly chosen nodes.

Random walk
P[k ] probability that item is found after k attempts:
r r
P[k ] = (1 − )k −1 .
N N
S (“search size”) is expected number of nodes that need to be probed:
N N
r r k −1
S= ∑ k · P[k ] =k =∑1 k·
N
(1 −
N
) ≈ N/ r for 1 « r ≤
N.
k=1

Unstructured peer-to-peer systems 28 / 36

Architectures: System architecture Decentralized organizations: peer-to-peer systems

Flooding versus random walk

Flooding
► Flood to d randomly chosen neighbors
► After k steps, some R(k ) = d · (d − 1) k−1 will have been
reached (assuming k is small).
► With fraction r / N nodes having data, if Nr · R(k ) ≥ 1, we will
found the data item.
have

Comparison
► If r / N = 0.001, then S ≈ 1000
► With flooding and d = 10, k = 4, we contact 7290 nodes.
► Random walks are more communication efficient, but might take
longer before they find the result.

Unstructured peer-to-peer systems 29 / 36

Architectures: System architecture Decentralized organizations: peer-to-peer systems

Super-peer networks
Essence
It is sometimes sensible to break the symmetry in pure peer-to-peer
networks:
► When searching in unstructured P2P systems, having index
servers improves performance
► Deciding where to store data can often be done more efficiently
through brokers.

Super peer
Overlay network of super peers

Weak peer

Hierarchically organized peer-to-peer networks 30 / 36

Architectures: System architecture Decentralized organizations: peer-to-peer systems

Skype’s principle operation: A wants to contact B

Both A and B are on the public Internet
► A TCP connection is set up between A and B for control packets.
► The actual call takes place using UDP packets between
negotiated ports.

A operates behind a firewall, while B is on the public

Internet
► A sets up a TCP connection (for control packets) to a
super peer S
► S sets up a TCP connection (for relaying control packets)
to B
► The actual call takes place through UDP and directly
between A
and B

Both A and B operate behind a firewall

► A connects to an online super peer S through TCP
31 / 36
Architectures: System architecture Hybrid Architectures

Edge-server architecture

Essence
Systems deployed on the Internet where servers are placed at the
edge of the network: the boundary between enterprise networks and
the actual Internet.

Client Content provider

ISP
ISP

Core Internet

Edge server

Enterprise network

Edge-server systems 32 / 36
Architectures: System architecture Hybrid Architectures

Collaboration: The BitTorrent case

Principle: search for a file F
► Lookup file at a global directory ⇒ returns a torrent file
► Torrent file contains reference to tracker: a server keeping an
accurate account of active nodes that have (chunks of) F .
► P can join swarm, get a chunk for free, and then trade a copy
of
that chunk for another one with a peer Q also in the swarm.

Client node
K out of N nodes

Lookup(F) Node 1

A BitTorrent List of nodes Node 2

Web page or torrent file
with (chunks of)
search engine for file F
file F
Web server File server Tracker
Node N

Collaborative distributed systems 33 / 36

Architectures: System architecture Hybrid Architectures

BitTorrent under the hood

Some essential details
► A tracker for file F returns the set of its downloading processes:
the current swarm.
► A communicates only with a subset of the swarm: the neighbor
set NA .
► if B ∈ NA then also A ∈ NB .
► Neighbor sets are regularly updated by the tracker

Exchange blocks
► A file is divided into equally sized pieces (typically each being 256
KB)
► Peers exchange blocks of pieces, typically some 16 KB.
► A can upload a block d of piece D, only if it has piece D.
► Neighbor B belongs to the potential set PA of A, if B has a block
that A needs.
► If B ∈ PA and A ∈ PB : A and B are in a position that they can
trade
a block.
Collaborative distributed systems 34 / 36
Architectures: System architecture Hybrid Architectures

BitTorrent phases

Bootstrap phase
A has just received its first piece (through optimistic unchoking: a node
from NA unselfishly provides the blocks of a piece to get a newly
arrived node started).

Trading phase
|PA| > 0: there is (in principle) always a peer with whom A can trade.

Last download phase

|PA| = 0: A is dependent on newly arriving peers in NA in order to get
the last missing pieces. NA can change only through the tracker.

Collaborative distributed systems 35 / 36

Architectures: System architecture Hybrid Architectures

BitTorrent phases

Development of |P| relative to |

N|. 1.0

0.8

0.6
|P|
|N|
0.4
|N| = 5
0.2 |N| = 10
|N| = 40

0.0 0.2 0.4 0.6 0.8 1.0

Fraction pieces downloaded

Collaborative distributed systems 36 / 36