Multiprocessors

A shared-memory multiprocessor (or just multiprocessor

henceforth) is a computer system in which two or more CPUs
share full access to a common RAM. A program running on any
of the CPUs sees a normal (usually paged) virtual address
space. The only unusual property this system has is that the
CPU can write some value into a memory word and then read
the word back and get a different value (because another CPU
has changed it).
 Types of Multiprocessors
There are mainly two types of multiprocessors i.e. symmetric and asymmetric
multiprocessors. Details about them are as follows:
•Symmetric Multiprocessors
In these types of systems, each processor contains a similar copy of the
operating system and they all communicate with each other. All the processors
are in a peer to peer relationship i.e. no master - slave relationship exists
between them.
An example of the symmetric multiprocessing system is the Encore version of
Unix for the Multimax Computer.
•Asymmetric Multiprocessors
In asymmetric systems, each processor is given a predefined task. There is a
master processor that gives instruction to all the other processors. Asymmetric
multiprocessor system contains a master slave relationship.
Asymmetric multiprocessor was the only type of multiprocessor available
before symmetric multiprocessors were created. Now also, this is the cheaper
option.
 Advantages of Multiprocessor Systems
There are multiple advantages to multiprocessor systems. Some of these are:
More reliable Systems In a multiprocessor system, even if one processor fails,
the system will not halt. This ability to continue working despite hardware failure
is known as graceful degradation. For example: If there are 5 processors in a
multiprocessor system and one of them fails, then also 4 processors are still
working. So the system only becomes slower and does not ground to a halt.
Enhanced Throughput If multiple processors are working in tandem, then the
throughput of the system increases i.e. number of processes getting executed per
unit of time increase. If there are N processors then the throughput increases by
an amount just under N.
More Economic Systems Multiprocessor systems are cheaper than single
processor systems in the long run because they share the data storage,
peripheral devices, power supplies etc. If there are multiple processes that share
data, it is better to schedule them on multiprocessor systems with shared data
than have different computer systems with multiple copies of the data.
Disadvantages of Multiprocessor Systems
There are some disadvantages as well to multiprocessor systems. Some of
these are:
Increased Expense Even though multiprocessor systems are cheaper in the
long run than using multiple computer systems, still they are quite expensive.
It is much cheaper to buy a simple single processor system than a
multiprocessor system.
Complicated Operating System Required There are multiple processors in
a multiprocessor system that share peripherals, memory etc. So, it is much
more complicated to schedule processes and impart resources to processes,
than in single processor systems. Hence, a more complex and complicated
operating system is required in multiprocessor systems.
Large Main Memory Required All the processors in the multiprocessor
system share the memory. So a much larger pool of memory is required as
compared to single processor systems.
Parallel Processing 5 Lecture 47

Coupling of Processors

Tightly Coupled System

- Tasks and/or processors communicate in a highly synchronized fashion
- Communicates through a common shared memory
- Shared memory system
Loosely Coupled System
- Tasks or processors do not communicate in a synchronized fashion
- Communicates by message passing packets
- Overhead for data exchange is high
- Distributed memory system
Parallel Processing 6 Lecture 47

Interconnection Structure
The components that form a multiprocessor system are CPUs, IOPs connected
to input-output devices, and a memory unit.
The interconnection between the components can have different physical
configurations, depending on the number of transfer paths that are available
o Between the processors and memory in a shared memory system
o Among the processing elements in a loosely coupled system
There are several physical forms available for establishing an interconnection
network.
o Time-shared common bus
o Multiport memory
o Crossbar switch
Time Shared Common Bus
A common-bus multiprocessor system consists of a number of processors
connected through a common path to a memory unit.
o Only one processor can communicate with the memory or another processor
at any given time.
o As a consequence, the total overall transfer rate within the system is limited by
the speed of the single path
A more economical implementation of a dual bus structure is depicted in Fig.
below.
Part of the local memory may be designed as a cache memory attached to the
CPU.
Multiport Memory
A multiport memory system employs separate buses between each memory module
and each CPU.
The module must have internal control logic to determine which port will have access
to memory at any given time.
Memory access conflicts are resolved by assigning fixed priorities to each memory
port.
Adv.:
o The high transfer rate can be achieved because of the multiple paths.
Disadv.:
o It requires expensive memory control logic and a large number of cables and
connections
Parallel Processing 9 Lecture 47

Cross Bar Switch

MM1 MM2 MM3 MM4

CPU1

CPU2

CPU3

CPU4

fig. shows the functional design of a crossbar switch connected to one memory module.

Consists of a number of crosspoints that are placed at intersections between processor

buses and memory module paths.
The small square in each crosspoint is a switch that determines the path from a processor
to a memory module.
Adv.:
o Supports simultaneous transfers from all memory modules
Disadv.:
o The hardware required to implement the switch can become quite large and complex.
 Multiprocessor architectures

(a) Two or more CPUs and one or more memory modules all use the same bus for
communication. When a CPU wants to read a memory word, it first checks to see if
the bus is busy. If the bus is idle, the CPU puts the address of the word it wants on
the bus, asserts a few control signals, and waits until the memory puts the desired
word on the bus.
(b) Problem to previous approach, If the bus is busy when a CPU wants to read or write
memory, the CPU just waits until the bus becomes idle. The cache can be inside the
CPU chip, next to the CPU chip, on the processor board, or some combination of all
three. Since many reads can now be satisfied out of the local cache, there will be
much less bus traffic, and the system can support more CPUs.
(c) in this each CPU has not only a cache, but also a local, private memory which it
accesses over a dedicated (private) bus. To use this configuration optimally, the
compiler should place all the program text, strings, constants and other read-only data,
stacks, and local variables in the private memories. The shared memory is then only
used for writable shared variables.
 Parallel Systems
• DEFINATION: A system is said to be a
parallel system in which multiple processor
have direct access to shared memory which
forms a common address space.
• Usually tightly-coupled system are referred
to as parallel system. In this systems there is
a single system wide primary memory that
is shared by all the processors.
• Parallel computing is the use of two or more
processors in combination to solve a single
problem.
Parallel System
Applications of Parallel System
• An example of parallel computing would be
two servers that shares the workload of
routing mail, managing connections to an
accounting system or database etc.
• Supercomputers are usually placed in parallel
system architecture.
• Terminals connected to single servers.
 Advantages of Parallel
System
• Provide concurrency (do multiple things at the
same time)
• Taking advantage of non local sources
• Cost savings
• Save time and money
• Overcoming memory constraints
• Obtain faster result
Disadvantages of Parallel System
• Primary disadvantage is the lack of scalability
between memory and CPUs.
• Programmer responsibility for synchronization
constructs that ensure “correct” access of global
memory.
• It becomes increasingly difficult and expensive to
design and produce shared memory machines
with ever increasing number of processors.
• Reliability and fault tolerance.
 Why Parallel
Computing ?
• Traditionally, software has been written for serial
computation:
– To be run on a single computer having a single
CPU;
– A problem is broken into a discrete series of
instructions.
– Instructions are executed one after another.
– Only one instruction may execute at any
moment of time.
Serial Problem…
 Parallel Computing…
• Parallel computing is a form of computation in which
many calculations are carried out simultaneously.
• In the simplest sense, it is the simultaneous use of
multiple compute resources to solve a computational
problem:
– To be run using multiple CPUs
– A problem is broken into discrete parts that can be
solved concurrently
– Each part is further broken down to a series of
instructions
– Instructions from each part execute simultaneously
Parallel Problem
 Why use Parallel Computing?

• Save time and/or money

• Solve large problems: e.g. – Web search
engines/ databases processing millions of
transactions per second.
• Provide concurrency;
• Use of non-local resources;
 Why use Parallel
Computing?...
• Limits to serial computing:
– Transmission speeds
– Limits to miniaturization
– Economic limitations
• Current computer architecture are increasingly
relying upon hardware level parallelism to improve
performance:
– Multiple execution units
– Pipelined instructions
– Multi- core
 Parallel Computer
Architecture
Parallel Computer Architecture is the method of organizing all
the resources to maximize the performance and the
programmability within the limits given by technology and
the cost at any instance of time. It adds a new dimension in
the development of computer system by using more and
more number of processors.
 Parallel Computer
Architecture…
• Memory is used to store both program
instructions and data
– Program instructions are coded data which tell the
computer to do something
– Data is simply information to be used by the program
• Control unit fetches instructions/data from
memory, decodes the instructions and then
sequentially coordinates operations to
accomplish the programmed task.
 Parallel Computer
Architecture…
• Arithmetic unit That performs basic
arithmetic operations
• Input/Output is the interface to the human
operator
 Parallel Computer
Architecture…
Flynn’s classical
Taxonomy:
 Single Instruction, Single Data
(SISD):
• A serial (non parallel) computer
Single Instruction: load A

load B
• Only one instruction stream is being acted onT
C=A+B
I
by the CPU during anyone clock cycle. M
Store C
E
Single Data : A=B*2
• Only one data stream is being used as input
Store A

during any one clock cycle.

• Deterministic execution.
 Single Instruction, Multiple
Data (SIMD):
prev prev prev
instruct instruct instruct
load A(1) load A(2) load A(n) T
I
load B(1) load B(2) load B(n)
M
C(1)=A(1)*B C(2)=A(2)*B C(n)=A(n)*B E
(1) (2) (n)
store C(1) store C(2) store C(n)

next instruct next instruct next instruct

P1 P2 Pn
 Single Instruction, Multiple
Data (SIMD):
A type of parallel computer
Single instruction:
• All processing units execute the same instruction at
any given clock cycle
Multiple data:
• Each processing unit can operate on a different data
element.
• Best suited for specialize problems characterized by
a high degree of regularity, such as graphics/image
processing.
Multiple Instruction, Single
Data (MISD):
prev prev prev
instruct instruct instruct
load A(1) load A(1) load A(1) T
I
C(1)=A(1) C(2)=A(1) C(n)=A(1) M
*1 *2 *n E
store C(1) store C(2) store C(n)

next next next

instruct instruct instruct
P1 P2 Pn
 Multiple Instruction, Single
Data (MISD):
• A single data stream is fed into multiple
processing units.
• Each processing unit operates on the data
independently via independent instruction
streams.
• Some conceivable usage might be:
o Multiple frequency filters operating on a single
signal stream
 Multiple Instruction, Multiple
Data (MIMD):
prev prev prev
instruct instruct instruct
load A(1) call func D Do 10 i=1,N T
I
load B(1) x=y*z Alpha=
M
w**3
C(1)=A(1)*B sum= x*2 E
(1) Zeta=C(i)
Call sub
store C(1) 1(i,j) 10 continue

next instruct next instruct next instruct

P1 P2 Pn
 Multiple Instruction, Multiple
Data (MIMD):
• Multiple Instruction: Every processor may be
executing a different instruction stream.
• Multiple Data: Every processor may be working
with a different data stream.
• Execution can be synchronous or asynchronous,
deterministic or non- deterministic.
 Micro-architecture
An ISA describes the design of a Computer in terms
of the basic operations it must support. The ISA is
not concerned with the implementation specific
details of a computer. It is only concerned with the
set or collection of basic operations the computer
must support. For example the AMD Athlon and the
Core 2 Duo processors have entirely different
implementations but they support more or less the
same set of basic operations as defined in the x86
Instruction Set.
Micro architectural level lies just below the ISA level
and hence is concerned with the implementation of
the basic operations to be supported by the
Computer as defined by the ISA. Therefore we can
say that the AMD Athlon and Core 2 Duo processors
are based on the same ISA but have different
microarchitectures with different performance and
efficiencies.
 Graphics Processing Unit
•Like a motherboard, a graphics card is a printed circuit board that houses a
processor and RAM. It also has an input/output system (BIOS) chip, which stores
the card's settings and performs diagnostics on the memory, input and output at
startup.
•A graphics card's processor, called a graphics processing unit (GPU), is similar to
a computer's CPU. A GPU, however, is designed specifically for performing the
complex mathematical and geometric calculations that are necessary for
graphics rendering. Some of the fastest GPUs have more transistors than the
average CPU. A GPU produces a lot of heat, so it is usually located under a heat
sink or a fan.
•A GPU however is more dedicated in function. It takes that same function that a
CPU was processing and completes it all at once.
•The specified function requested of a GPU enters the GPU’s hundred’s of cores,
and processes all at a single point in time, where it handles each process parallel
to the next.
•The GPU has become the most powerful processing unit of the system,
especially since organizations are beginning to rely more on the processing
power of a GPU rather than a CPU for the single fact that it can process more at
a single point in time faster than a CPU.
There are two different types of GPUs:
•Integrated GPUs are located on a PC’s CPUand share memory with the CPU
processor.
•Discrete GPUs live on their own card and have their own video memory
(VRAM), so that the PC doesn’t have to use its RAM for graphics.