Information Technology

FIT3143 Parallel Computing

Semester 2 2024

Topic 2:
Inter Process Communications in Parallel Computing
Dr Carlo Kopp, MACM, SMIEEE,
AFAIAA
Faculty of Information Technology
© 2024 Monash University
Why Study Inter Process Communications?
 IPC is central to many parallel applications. APIs for IPC sit beneath the
middleware of such systems, and critically impacts function and performance;
 Foundation knowledge: Because the behaviour and performance of IPC and its
APIs determines many critical aspects of parallel system function and
performance, understanding APIs is essential;
 Foundation knowledge: The limitations of API can limit what a parallel
application can or cannot do;
 Practical skills: When coding parallel applications you will have to rely on
middleware written around the underlying APIs and their limitations, or the APIs
themselves;
 Practical skills: The throughput performance of a API can become a ‘live or
die”
problem for a parallel application, making this an important item to understand;
2
Processes
Processes in Operating Systems [Stallings Ch.3 N/E]
 A program in execution
 An instance of a program running on a computer
 The entity that can be assigned to and executed on a processor
 A unit of activity characterized by the execution of a sequence
of instructions, a current state, and an associated set of system
instructions
 A process comprises:
A. Program code (possibly shared)
B. A set of data
C. A number of attributes describing the state of the process

4
Process Identity and State Information [Stallings Ch.3 N/E]
 At a minimum, a process must have the following attributes so it can be
identified and its state known:
A. Identifier
B. State
C. Priority
D. Program counter
E. Memory pointers
F. Context data
G. I/O status information
H. Accounting information
 This data is usually kept in a Process Control Block (PCB)

5
Process Control Block [Stallings Ch.3 N/E]

 Contains the process elements;

 Created and managed by the
operating system;
 Allows support for multiple
processes
 The structure and format of a
PCB is unique to an operating
system type, and usually not
interchangable between
operating systems;

6
Processes in Memory [Stallings Ch.3 N/E]

7
Processes in Main Memory [Stallings Ch.3 N/E]

8
Process States [Stallings Ch.3 N/E]

Spawn or Terminate or
Fork a new Kill a running process
process
Inter Process Communications
[IPC] may or may not cause a
process to block

9
Unix Process States [Stallings Ch.3 N/E]

10
Networks
Network Protocols and Their Limitations
 The most widely used protocols for carrying network traffic
are TCP and UDP, developed during the 1980s;
 The intent, when TCP and UDP were developed, was to
support remote dumb terminal traffic, and file transfers;
 The earliest applications, telnet and File Transfer Protocol,
were thus used to remotely log on to other hosts, and copy
files between hosts;
 TCP and UDP therefore predate modern distributed
computing techniques.
12
Networking Environment in 1980s to 1990s (Early Internet Era)
User-to-Host – 9,600 bits/s – 64 kilobits/s
Voiceband
Modem

ASCII Serial Super Minicomputer Super Minicomputer

Dumb Terminal
Host-to-Host – 0.2-2.048 Megabits/s
 TCP and UDP were developed before the widespread use of Local Area Networks and end user
broadband services
 Voiceband modems carried traffic over analogue phone lines
 TCP and UDP designed for efficient use with very expensive (and low) bandwidth and very
expensive
(and low) CPU performance

13
Networking Environment in 2024
User-to-Host – 0.05-1 Gigabits/s

Optical Fibre
Desktop
Backhaul
Notebook
Network
Mobile Device Host-to-Host –10-100 Gigabits/s
 TCP and UDP are adequate for end user broadband services but underperform in host-
host links in clusters, clouds and grids
 Bandwidth and CPU performance are now comparatively inexpensive, and this puts a
premium on network protocol performance
 Legacy TCP and UDP protocol overheads and error control increase latency in links and
reduce achievable link throughput – this is a known performance problem in data centres

14
OSI and Internet Reference Models
 The OSI Reference Model
 The TCP/IP (IETF/Internet) Reference Model
– The Link Layer
– The Internet Layer
– The Transport Layer
– The Application Layer
 OSI model and protocols versus the IETF reference model
and protocols
 The model used in the Tanenbaum textbook

15
The OSI Reference Model (1 of 2)
 Principles for the seven layers
– Layers created for different abstractions
– Each layer performs well-defined function
– Function of layer chosen with definition of international standard protocols in
mind
– Minimize information flow across interfaces between boundaries
– Number of layers should be optimum
 Three concepts central to the OSI model:
– Services
– Interfaces
– Protocols

16
The OSI Reference Model (2 of 2)

 The OSI model has seven

layers
 Specific abstractions for each
layer – defined functions per
layer
 As message flows down stack
encapsulation is added
 Packets flow across network
 As message flows up stack
encapsulation is removed
Message Travelling Across Network

17
The TCP/IP [IETF] Reference Model (1 of 4)
 The Link Layer
– Lowest layer in the model
– Describes what links must do to meet the needs of this connectionless
internet layer
 The Internet Layer
– Permits hosts to inject packets into any network and have them travel
independently to the destination
– Defines an official packet format and protocol called IP (Internet Protocol)
– Defines a companion protocol called ICMP (Internet Control Message
Protocol) that helps IP function

18
The TCP/IP [IETF] Reference Model (2 of 4)
 The Transport Layer
– The layer above the internet layer in the TCP/IP
model
– Uses two end-to-end transport protocols
 TCP (Transmission Control Protocol)
 UDP (User Datagram Protocol)
 The Application Layer
– Contains all the higher-level protocols

19
The TCP/IP [IETF] Reference Model (3 of 4)

 The TCP/IP
layers loosely
align with the
OSI model.
 OSI model is
more general
and more finely
grained
 TCP/IP model
specific to IETF
protocol suite

20
The TCP/IP [IETF] Reference Model (4 of 4)

1. The relation of IP, TCP, and UDP protocols are illustrated. We will study
these.

21
Encapsulation in Protocols [TCP/IP] - Stallings

22
IPv4 - Stallings

23
IPv6 - Stallings

24
TCP, UDP and IP Stack

Application Application

TCP UDP

IP
Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5 Pearson Education 2012

 This is a simplified model of the TCP/UDP and IP protocol stack;

 Applications interface to TCP and UDP via an API, which provides
access to the middleware that handles the TCP and UDP
connections;
25
TCP, UDP and IP Stack
 High performance in carrying files and other bulk data was not a
consideration when TCP and UDP were developed;
 TCP was developed to provide for “reliable” data transfers, ensuring that
data (in packets) always arrived, and arrived in order, regardless of
performance losses in the network;
 UDP was developed to provide for “unreliable” data transfers, using a
simpler protocol with slightly better performance than TCP, due to lesser
overheads – applications had to manage reliability and in-order
transfers;
 In a distributed computing environment, especially where high volumes
of traffic must be carried, TCP and UDP are often too slow, as they were
never optimised for high performance – problematic in data centres;

26
Inter Process Communications
Inter-Process Communications
 Inter-Process Communications (IPC) are all mechanisms which permit two or
more processes to exchange messages;
 IPC mechanisms may be restricted to communications within a host machine’s
operating system, or may permit communications between processes on
different host machines connected via a network;
 IPC mechanisms restricted to a host operating system include:
A. Shared Memory
B. Message Passing
C. Unix Signals and analogues

28
Inter-Process Communications
1.Shared Memory:
• Concept: Shared memory allows multiple processes to access
the same memory space. This method involves creating a
memory segment that can be mapped into the address space of
each participating process.
• Characteristics: Shared memory imposes no structure on the
data being communicated, allowing processes to read and write
data freely. This results in high performance due to reduced
copying of data between processes.
• Example Usage: For example, two processes in a video
processing application could share a buffer in memory to quickly
access and modify image data without copying it between the
processes.
• Suppose two processes need to work on a large dataset, such as
an image or matrix. Process29A can write an image array into
 Inter-Process Communications
1.Message Passing:
• Concept: Message passing involves sending and receiving
messages between processes. Each message typically has a
predefined structure that must be adhered to.
• Characteristics: This method offers a clear communication
protocol and can handle more complex data types by
encapsulating them within structured messages. However, it can
introduce overhead due to message copying and context
switching.
• Example Usage: In a client-server model, a server might use
message passing to communicate with multiple client processes,
relaying requests and responses in a structured format (e.g.,
JSON or XML).
• In an application using a message-passing system, a process
might send a command with multiple
30 fields, such as a message
header that includes the sender's ID and a payload with specific
Inter-Process Communications
1.Unix Signals and Analogues:
• Concept: Signals serve as notification tools that allow processes
to inform one another about events or changes in state. For
example, a process can send a SIGTERM signal to request
another process to terminate.
• Characteristics: Signals are typically used for handling
asynchronous events and generally carry no data other than the
signal's type. Due to their nature, they impose a structured
communication pattern.
• Example Usage: In a multi-threaded application, a thread might
send a signal to alert another thread to wake up or perform a
specific action when a resource becomes available.
• Example: When a process receives a SIGINT (interrupt signal), it
typically performs a defined action, such as gracefully
terminating. The signal itself31 does not contain any additional
Inter-Process Communications
Summary of Differences
• Shared Memory: No structure enforced on data; processes can store
and retrieve data in any form.
• Signals: Forces discrete message types; only simple notifications
with no additional data payload.
• Message Passing: Requires discrete and structured messages that
must adhere to specific formats.

32
Shared Memory IPC – Limited to Local Host

33
Stream Oriented Inter-Process Communications
 Stream Oriented IPC is especially important since it is the most commonly
used scheme for IPC within operating systems and between networked
hosts;
 A stream connection imposes no implicit structure on the data being sent –
character mode “streams” of data are handled in a FIFO manner, arriving
typically at the destination in the order they were sent;
 The two most widely used stream oriented IPC schemes are the BSD
Socket mechanism, and the SVR4 STREAMS mechanism; Application
protocols treat the stream as a transparent pipe.
 Both are designed around standard software Application Programming
Interfaces (API), although most recent STREAMS implementations also
include an interface to emulate the BSD Socket API;

34
Stream Oriented IPC – Channel Properties
 The channel is “reliable” when the stream connects two processes on a single
host;
 The channel is “reliable” when the stream connects two processes on different
hosts, using TCP protocol; The channel is “unreliable” when the stream
connects two processes on different hosts, using UDP protocol;
 A stream IPC connection usually delivers messages “in order” thus behaving
like
a FIFO or pipe;
 Depending on the type of stream IPC, the API, and its configuration, the
interface may be “blocking” or “non-blocking”, thus determining process
behaviour during message transfers;
 BSD Sockets and SVR4 STREAMs are the most common APIs in use
for
distributed applications; 35
SVR4 STREAMS vs BSD Sockets (Kopp, 1996) [N/E]

36
Stream Oriented IPC – Integration

Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts and Design Edn. 5, © Pearson Education 2012

 The “middleware” provides the API interface for the user program running in a process;
 The middleware interfaces to the protocol stack software, which is usually embedded in
the operating system kernel;

37
Example: Linux Socket Integration [Stallings Ch.17 N/E]

38
Stream Oriented IPC – Performance
 Performance can become a critical challenge when running parallel
applications on distributed platforms like clusters and clouds
 There are two aspects to performance:
A. Latency [Delay] – the time delay between message transmission
by sending process to receipt by receiving process
B. Throughput [Bandwidth/Capacity] – volume of data sent over link
per
unit time (e.g. Gigabits/s or Gigabytes/s)
 While both can be inter-related due to the design of the network and
the IPC scheme in use, in many applications even high bandwidth
networks may have large latency
 Where data dependencies exist, latency becomes a major problem
39
Remote IPC Mechanisms (Across Hosts)
• Remote IPC involves mechanisms that allow processes running
on different machines within a network to communicate.
• This can include techniques like sockets, Remote Procedure
Calls (RPC), or distributed message queues.

40
Remote Procedure Calls
Remote Procedure Call API (RPC)
 Stream oriented network API interfaces provide an unstructured channel for byte
or message oriented data transfers, which carry data – this is the most basic
abstraction possible, involving read() and write() calls;
 The next level of abstraction is that of a remote procedure call in which a
procedure (i.e. function call) may be executed locally on a host, or on a remote
server host;
 The client process will make a request upon a server process, which involves a
procedure identifier (name) and some list of arguments; the server then returns
the results of the call;
 This is a structured API that bounds the transfers to very specific messages –
calls
and returns of values;
 ONC RPC (RFC1831) protocol is the most widely used remote procedure call
API
42
Remote Procedure Call API (ONC RPC)
 Open Network Computing Remote Procedure Call (ONC RPC RFC1831)
protocol was developed by Sun Microsystems and widely used for Network File
System (NFS) remote mounting of disk storage devices;
 Distributed Computing Environment / Remote Procedure Calls (DCE/RPC) was
developed in competition, and later used in the Microsoft DCOM and .NET
Remoting;
 ONC RPC is structured around the use of “stubs”, which are blocks of code that
provide the RPC API interface, and are compiled into an application binary
program;
 When RPC transfers data, such as arguments or return values, the “endianess”
of the data matters; this is managed with the XDR (eXternal Data Representation)
protocol, which ensures data always has the correct “endianess”;

43
Remote Procedure Calls [Stallings Ch.16]

44
Remote Procedure Calls [Stallings Ch.16]

57
ONC RPC Model (Kopp 1994)

46
ONC RPC Model (Kopp 1994)

47
Summary
Summary
 Processes
 Networks
 Inter Process Communications
 BSD Socket API Programming in the C Language
 Remote Procedure Calls

61
Reading Materials
Reading and References
 Ch.1; Ch.3; Ch.4; Coulouris, Dollimore, Kindberg and Blair, Distributed Systems: Concepts
and Design, Edition 5, © Addison-Wesley, 2012.
1. Tanenbaum, Feamster, Wetherall, Computer Networking, 6th Edition, Pearson Education,
Chapter 1
2. Stallings, Data and Computer Communications, International Edition, 10th edition,
Chapter 1
 https://users.monash.edu/~ckopp/SYSTEMS/IPC-Introduction-0595.htm
 https://users.monash.edu/~ckopp/SYSTEMS/RPC-Intro-0796.htm
 https://users.monash.edu/~ckopp/WALNUT/msc.thesis.ck.pdf
 http://www.cs.utexas.edu/~wcook/Drafts/2006/WSvsDO.pdf
 http://www.eg.bucknell.edu/~cs379/DistributedSystems/rmi_tut.html
 https://users.monash.edu/~ckopp/SYSTEMS/Cluster-Practical-1299.html

63