Introduction to Parallel and

Distributed Computing
Prepared By Shafiq Ahmad
 Contents:
• Definition of Parallel and
Distributed Computing
• Historical Evolution
• Applications in Modern
Computing
• Overview of Parallel vs.
Distributed Systems
• Key Differences between
Parallel and Distributed
Systems
• Conclusion and Next Steps
Definition
Parallel and Distributed Computing refers to the use of multiple computing
resources to solve a problem concurrently. Both parallel and distributed
computing aim to increase computational power, enhance efficiency, and
tackle problems that are either too large or too complex for a single
processor.
• Parallel Computing focuses on executing multiple tasks simultaneously on
multiple processors within a single machine, where the tasks often need
to communicate and synchronize with each other.
• Distributed Computing involves multiple independent machines (or
nodes) working together, connected through a network, to achieve a
common goal.
 Importance
• Scalability: Parallel and distributed computing allow systems to scale as workloads
increase. Instead of upgrading to a faster processor, you can add more processors or
machines.
• Efficiency: By dividing large problems into smaller tasks and executing them
simultaneously, you can achieve faster computations and better resource utilization.
• Cost-Effectiveness: Distributed systems can use commodity hardware and
infrastructure like cloud computing to reduce costs while offering immense
computational power.
• Fault Tolerance: Distributed systems can handle failures gracefully by distributing
tasks across multiple machines, allowing the system to continue functioning even if
individual components fail.
• Solving Large-Scale Problems: Both paradigms are crucial in handling
computationally intensive tasks, like climate modeling, genome sequencing, and big
data analysis.
 Historical Evolution
Parallel and distributed computing have evolved significantly over the decades. Here’s a brief
historical overview:

• 1960s: The idea of parallel computing originated with early supercomputers that used
multiple processors to handle different parts of a problem. Early research focused on how
to divide tasks among processors and how those processors should communicate.

• 1970s: Seymour Cray’s Cray-1 supercomputer was one of the first to use vector
processing, a form of parallel computing. Around the same time, Amdahl’s Law was
formulated, stating the limits of parallelism based on the portion of a program that must
be executed sequentially.

• 1980s: Distributed computing gained momentum with the rise of computer networks,
leading to early models of distributed systems. The client-server architecture became
popular for separating service requests from service providers.
 Historical Evolution (cont)
• 1990s: The proliferation of the internet and improvements in network infrastructure led to the
widespread adoption of distributed systems. Message Passing Interface (MPI) and OpenMP standards
emerged for parallel programming, simplifying communication between processors.

• 2000s: Cloud computing emerged as a form of distributed computing, allowing companies and
researchers to rent computing power on-demand from services like Amazon Web Services (AWS) and
Google Cloud. Distributed computing frameworks like Hadoop and MapReduce revolutionized big
data processing by leveraging commodity hardware to run large-scale computations.

• 2010s: Multicore processors became the standard in consumer devices, and GPU-based parallel
computing surged, especially in fields like machine learning and graphics. Distributed databases and
blockchain technologies started gaining prominence for distributed computing.

• 2020s and Beyond: Edge computing and fog computing aim to push computing resources closer to
data sources to reduce latency. Quantum computing also represents a new frontier, where parallelism
is fundamentally different, leveraging the principles of quantum mechanics.
 Applications in Modern Computing
Parallel and distributed computing have many applications in today’s computing landscape.
Below are a few notable areas:
• Cloud Computing: Cloud computing services (like AWS, Azure, and Google Cloud) rely
heavily on distributed computing to provide scalable and reliable infrastructure for
storage, computation, and application deployment. These services allow users to run
large computations over multiple machines without investing in physical hardware.

• Scientific Simulations: Scientific fields like astrophysics, climate modeling, and molecular
biology require enormous computational resources to simulate natural phenomena.
These simulations are often parallelized to run on supercomputers or distributed across
cloud computing platforms.

• Big Data Processing: Companies and organizations generate and collect massive amounts
of data every day. Frameworks like Apache Hadoop and Apache Spark enable parallel
processing of big data, distributing tasks across thousands of machines to perform data
analysis, machine learning, and more.
 Applications in Modern Computing (cont)
• Artificial Intelligence and Machine Learning: Training deep learning models, especially for tasks like
image recognition, natural language processing, and autonomous systems, requires parallel
computing power, often distributed across clusters of GPUs. Frameworks like TensorFlow and PyTorch
leverage this computing power.

• Video Games and Graphics Rendering: Parallel computing, particularly using GPUs, is central to
rendering complex graphics in video games. Large video game environments and real-time physics
simulations require thousands of parallel computations to render frames smoothly.

• Blockchain and Cryptocurrencies: Distributed computing underpins blockchain technology, where

• Healthcare: Distributed and parallel computing have enabled advancements in healthcare, especially
in medical imaging, genomics, and personalized medicine. Large-scale data analysis, often parallelized
across cloud systems, helps identify patterns in diseases and develop more effective treatments.
 Overview of Parallel vs. Distributed Systems
Parallel Computing:
Architecture: In parallel computing, multiple processors or cores within a single
machine work on different parts of a task simultaneously. Typically, these
processors share the same memory space (shared memory model) or have a
distributed memory model (where each processor has its own memory).
Communication: In shared memory systems, communication happens through
reading and writing to the same memory locations. In distributed memory
systems (such as clusters), communication occurs via explicit message passing.
Examples: Multicore processors, GPU programming (CUDA), parallel
supercomputers.
Overview of Parallel vs. Distributed Systems (cont)
Distributed Computing:
Architecture: Distributed computing involves multiple autonomous machines
connected through a network, each with its own memory and resources. Each
machine may perform different tasks or work collaboratively on the same task
by dividing it into smaller sub-tasks.
Communication: Communication in distributed systems happens over a
network, using protocols like TCP/IP. Machines coordinate their work through
message-passing protocols such as MPI, HTTP, or RESTful services.
Examples: Cloud computing platforms, distributed databases, peer-to-peer
systems, Internet of Things (IoT).
Key Differences:
Resource Sharing:
• Parallel: Processors share resources like memory and CPU.
• Distributed: Resources are distributed across multiple independent machines.

Task Execution:
• Parallel: Tasks are executed simultaneously within one machine.
• Distributed: Tasks are executed across different machines that collaborate.

Communication:
• Parallel: Communication is typically faster and happens through shared memory or high-speed
interconnects (e.g., between CPUs).
• Distributed: Communication is slower and happens over a network (e.g., the internet), which
introduces latency.

Fault Tolerance:
• Parallel: Fault tolerance is typically low since everything is executed on the same machine.
• Distributed: Fault tolerance is high, as failure in one machine does not affect others; tasks can be
redistributed.
Conclusion
• Parallel and distributed computing are fundamental to modern computer
systems, enabling faster computation, scalability, and resource efficiency.
As computing challenges grow—particularly with big data, AI, and cloud
services—understanding these paradigms is crucial for developing future
technologies. From personal devices with multicore processors to global
distributed systems like the cloud, both paradigms play a pivotal role in the
evolution of computing.

• Next Steps: In subsequent lectures, we will delve into specific