Lecture 1:

Introduction to Parallel Computing

To accompany the text ``Introduction to Parallel Computing‘’,

Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar
Addison Wesley, 2003.

Page 1 Introduction
Introduction To Parallel
to High Performance Computing
Computing
 Topic Description
 This topic introduces the students:
 Motivating Parallelism
 Scope of Parallel Computing
 Parallel Computing Terminologies

Page 2 Introduction to High Performance Computing

  Motivating Parallelism

Page 3 Introduction to High Performance Computing

 Fundamentals of Parallel Computing –
Von Neumann Architecture

 For over 40 years, virtually all computers have followed

a common machine model known as the von Neumann
computer. Named after the Hungarian mathematician
John von Neumann.

A von Neumann computer uses the stored-program

concept. The CPU executes a stored program that
specifies a sequence of read and write operations on the
memory.

Page 4 Introduction to High Performance Computing

 Basic Design
 Basic design
◦ Memory is used to store both
program and data instructions
◦ Program instructions are coded
data which tell the computer to do
something
◦ Data is simply information to be
used by the program

A central processing unit (CPU) gets

instructions and/or data from memory,
decodes the instructions and then
sequentially performs them.

Page 5 Introduction to High Performance Computing

 What is Serial Computing?
 Traditionally, software has been written for serial computation:
◦ To be run on a single computer having a single Central
Processing Unit (CPU);
◦ A problem is broken into a discrete series of instructions.
◦ Instructions are executed one after another.
◦ Only one instruction may execute at any moment in time.

Page 6 Introduction to High Performance Computing

 What is Parallel Computing?
 In the simplest sense, parallel computing is the simultaneous use of multiple
compute resources to solve a computational problem.
◦ To be run using multiple CPUs
◦ A problem is broken into discrete parts that can be solved concurrently
◦ Each part is further broken down to a series of instructions
 Instructions from each part execute simultaneously on different CPUs

Page 7 Introduction to High Performance Computing

 Parallel Computing: Resources
 The compute resources can include:
◦ A single computer with multiple processors;
◦ A single computer with (multiple) processor(s) and some
specialized computer resources (GPU, FPGA …)
◦ An arbitrary number of computers connected by a network;
◦ A combination of both.

Page 8 Introduction to High Performance Computing

 Parallel Computing:
The computational problem
 Thecomputational problem usually demonstrates
characteristics such as the ability to be:
◦ Broken apart into discrete pieces of work that can be
solved simultaneously;
◦ Execute multiple program instructions at any moment
in time;
◦ Solved in less time with multiple compute resources
than with a single compute resource.

Page 9 Introduction to High Performance Computing

 Why Parallel Computing?
 This is a legitime question! Parallel computing is
complex on any aspect!

 The primary reasons for using parallel computing:

◦ Save time - wall clock time
◦ Solve larger problems
◦ Provide concurrency (do multiple things at the same
time)

Page 10 Introduction to High Performance Computing

 Why Parallel Computing? (cont.)
 Other reasons might include:
◦ Taking advantage of non-local resources - using
available compute resources on a wide area
network, or even the Internet when local compute
resources are scarce.
◦ Cost savings - using multiple "cheap" computing
resources instead of paying for time on a
supercomputer.
◦ Overcoming memory constraints - single
computers have very finite memory resources. For
large problems, using the memories of multiple
computers may overcome this obstacle.

Page 11 Introduction to High Performance Computing

 Limitations of Serial Computing
 Limits to serial computing - both physical and practical reasons pose
significant constraints to simply building ever faster serial computers.
 Transmission speeds - the speed of a serial computer is directly dependent
upon how fast data can move through hardware. Absolute limits are the
speed of light (30 cm/nanosecond) and the transmission limit of copper
wire (9 cm/nanosecond). Increasing speeds necessitate increasing
proximity of processing elements.
 Limits to miniaturization - processor technology is allowing an increasing
number of transistors to be placed on a chip. However, even with
molecular or atomic-level components, a limit will be reached on how
small components can be.
 Economic limitations - it is increasingly expensive to make a single
processor faster. Using a larger number of moderately fast commodity
processors to achieve the same (or better) performance is less expensive.

Page 12 Introduction to High Performance Computing

  Scope of Parallel Computing
Applications

Page 13 Introduction to High Performance Computing

 Parallel Computing: what is it for?
 Parallel computing is an evolution of serial computing that attempts
to emulate what has always been the state of affairs in the natural
world: many complex, interrelated events happening at the same
time, yet within a sequence.
 Some examples:
◦ Planetary and galactic orbits
◦ Weather and ocean patterns
◦ Tectonic plate drift
◦ Rush hour traffic in Paris
◦ Automobile assembly line
◦ Daily operations within a business
◦ Building a shopping mall
◦ Ordering a hamburger at the drive through.

Page 14 Introduction to High Performance Computing

 Parallel Computing: used for?
 Traditionally,parallel computing has been considered to
be "the high end of computing" and has been motivated
by numerical simulations of complex systems and
"Grand Challenge Problems" such as:
◦ weather and climate
◦ chemical and nuclear reactions
◦ biological, human genome
◦ geological, seismic activity
◦ mechanical devices - from prosthetics to spacecraft
◦ electronic circuits
◦ manufacturing processes

Page 15 Introduction to High Performance Computing

 Parallel Computing: used for?
 Today, commercial applications are providing an equal or greater
driving force in the development of faster computers. These
applications require the processing of large amounts of data in
sophisticated ways. Example applications include:
◦ parallel databases, data mining
◦ oil exploration
◦ web search engines, web based business services
◦ computer-aided diagnosis in medicine
◦ management of national and multi-national corporations
◦ advanced graphics and virtual reality, particularly in the entertainment
industry
◦ networked video and multi-media technologies
◦ collaborative work environments
 Ultimately, parallel computing is an attempt to maximize the
infinite but seemingly scarce commodity called time.

Page 16 Introduction to High Performance Computing

 Parallel Computing Applications
• Parallelism finds applications in very diverse application
domains for different motivating reasons.

• These range from improved application performance to

cost considerations.

Page 17 Introduction to High Performance Computing

 (1) Applications in Engineering and Design

• Design of airfoils (optimizing lift, drag, stability),

internal combustion engines (optimizing charge
distribution, burn), high-speed circuits (layouts
for delays and capacitive and inductive effects),
and structures (optimizing structural integrity,
design parameters, cost, etc.).

• Design and simulation of micro- and nano-scale

systems.

• Process optimization, operations research.

Page 18 Introduction to High Performance Computing

 (2) Scientific Applications
• Functional and structural characterization of genes and proteins.

• Advances in computational physics and chemistry have explored new

materials, understanding of chemical pathways, and more efficient
processes.

• Applications in astrophysics have explored the evolution of galaxies,

thermonuclear processes, and the analysis of extremely large datasets
from telescopes.

• Weather modeling, mineral prospecting, flood prediction, etc., are other

important applications.

• Bioinformatics and astrophysics also present some of the most

challenging problems with respect to analyzing extremely large
datasets.

Page 19 Introduction to High Performance Computing

 (3) Commercial Applications

• Some of the largest parallel computers power the

wall street!

• Data mining and analysis for optimizing business

and marketing decisions.

• Large scale servers (mail and web servers) are often

implemented using parallel platforms.

• Applications such as information retrieval and

search are typically powered by large clusters.

Page 20 Introduction to High Performance Computing

 (4) Applications in Computer Systems
• Network intrusion detection, cryptography, multiparty
computations are some of the core users of parallel computing
techniques.

• Embedded systems increasingly rely on distributed control

algorithms.

• A modern automobile consists of tens of processors

communicating to perform complex tasks for optimizing
handling and performance.

• Conventional structured peer-to-peer networks impose overlay

networks and utilize algorithms directly from parallel
computing.

Page 21 Introduction to High Performance Computing

 The future
During the past 10 years, the trends indicated by
ever faster networks, distributed systems, and
multi-processor computer architectures (even at
the desktop level) clearly show that parallelism
is the future of computing.
It will be multi-forms, mixing general purpose
solutions (your PC…) and very speciliazed
solutions as IBM Cells, ClearSpeed, GPGPU
from NVidia …

Page 22 Introduction to High Performance Computing

 Who and What? (1)
 Top500.org provides statistics on parallel computing
users - the charts below are just a sample. Some things
to note:
◦ Sectors may overlap - for example, research may be classified
research. Respondents have to choose between the two.
 "NotSpecified" is by far the largest application -
probably means multiple applications.

Page 23 Introduction to High Performance Computing

 Who and What? (2)

Page 24 Introduction to High Performance Computing

  Parallel Computing
Terminology

Page 25 Introduction to High Performance Computing

 Some General Terminologies
 Task
◦ A logically discrete section of computational work. A task is typically
a program or program-like set of instructions that is executed by a
processor.

 Parallel Task
◦ A task that can be executed by multiple processors safely (yields
correct results)

 Serial Execution
◦ Execution of a program sequentially, one statement at a time. In the
simplest sense, this is what happens on a one processor machine.
However, virtually all parallel tasks will have sections of a parallel
program that must be executed serially.

Page 26 Introduction to High Performance Computing

 General Terminologies (cont.)
 Parallel Execution
◦ Execution of a program by more than one task, with each task being
able to execute the same or different statement at the same moment in
time.

 Shared Memory
◦ From a strictly hardware point of view, describes a computer
architecture where all processors have direct (usually bus based) access
to common physical memory. In a programming sense, it describes a
model where parallel tasks all have the same "picture" of memory and
can directly address and access the same logical memory locations
regardless of where the physical memory actually exists.

 Distributed Memory
◦ In hardware, refers to network based memory access for physical
memory that is not common. As a programming model, tasks can only
logically "see" local machine memory and must use communications to
access memory on other machines where other tasks are executing.

Page 27 Introduction to High Performance Computing

 General Terminologies (cont.)
 Communications
◦ Parallel tasks typically need to exchange data. There are several ways
this can be accomplished, such as through a shared memory bus or over
a network, however the actual event of data exchange is commonly
referred to as communications regardless of the method employed.

 Synchronization
◦ The coordination of parallel tasks in real time, very often associated
with communications. Often implemented by establishing a
synchronization point within an application where a task may not
proceed further until another task(s) reaches the same or logically
equivalent point.
◦ Synchronization usually involves waiting by at least one task, and can
therefore cause a parallel application's wall clock execution time to
increase.

Page 28 Introduction to High Performance Computing

 General Terminologies (cont.)
 Granularity
◦ In parallel computing, granularity is a qualitative measure of the ratio of
computation to communication.
◦ Coarse: relatively large amounts of computational work are done
between communication events
◦ Fine: relatively small amounts of computational work are done between
communication events

 Observed Speedup
◦ Observed speedup of a code which has been parallelized, defined as:
wall-clock time of serial execution
wall-clock time of parallel execution
◦ One of the simplest and most widely used indicators for a parallel
program's performance.

Page 29 Introduction to High Performance Computing

 General Terminologies (cont.)
 Parallel Overhead
◦ The amount of time required to coordinate parallel tasks, as opposed to
doing useful work. Parallel overhead can include factors such as:
 Task start-up time
 Synchronizations
 Data communications
 Software overhead imposed by parallel compilers, libraries, tools, operating
system, etc.
 Task termination time

 Massively Parallel
◦ Refers to the hardware that comprises a given parallel system - having
many processors. The meaning of many keeps increasing, but currently
BG/L pushes this number to 6 digits.

Page 30 Introduction to High Performance Computing

 General Terminologies (cont.)
 Scalability
◦ Refers to a parallel system's (hardware and/or software) ability to
demonstrate a proportionate increase in parallel speedup with the
addition of more processors. Factors that contribute to scalability
include:
 Hardware - particularly memory-cpu bandwidths and network
communications
 Application algorithm
 Parallel overhead related
 Characteristics of your specific application and coding

Page 31 Introduction to High Performance Computing

 Next week:

Lecture2:
Parallel Platforms
(Part 1)

Page 32 Introduction to High Performance Computing