IT4272E-COMPUTER SYSTEMS

Chapter 7:
Multicores, Multiprocessors, and Clusters

[with materials from Computer Organization and Design, 4th Edition,

Patterson & Hennessy, © 2008, MK]

1
 §7.1 Introduction
7.1. Introduction

❑ Goal: connecting multiple computers

to get higher performance
l Multiprocessors
l Scalability, availability, power efficiency Job 1 Job 2

❑ Job-level (process-level) parallelism

l High throughput for independent jobs Process

❑ Parallel processing program Processor 1 & 2

l Single program run on multiple processors

❑ Multicore microprocessors
l Chips with multiple processors (cores)
2
Types of Parallelism

Time
Time

Data-Level Parallelism (DLP)

Pipelining

Time
Time

Thread-Level Parallelism (TLP) Instruction-Level Parallelism (ILP)

3
Hardware and Software

Hardware Software

• Serial: e.g., Pentium 4 • Sequential: e.g., matrix

multiplication
• Parallel: e.g., quad-core • Concurrent: e.g.,
Xeon e5345 operating system

❑ Sequential/concurrent software can run on

serial/parallel hardware
l Challenge: making effective use of parallel hardware
Quard
Core

4
What We’ve Already Covered
❑ §2.11: Parallelism and Instructions
l Synchronization

❑ §3.6: Parallelism and Computer Arithmetic

l Associativity

❑ §4.10: Parallelism and Advanced Instruction-Level

Parallelism
❑ §5.8: Parallelism and Memory Hierarchies
l Cache Coherence

❑ §6.9: Parallelism and I/O:

l Redundant Arrays of Inexpensive Disks

5
 §7.2 The Difficulty of Creating Parallel Processing Programs
7.2. The Difficulty of Creating Parallel Processing Programs

❑ Parallel software is the problem

❑ Need to get significant performance improvement
l Otherwise, just use a faster uniprocessor, since it’s easier!

❑ Difficulties
l Partitioning
l Coordination
l Communications overhead

6
Amdahl’s Law
❑ Sequential part can limit speedup
❑ Example: 100 processors, 90× speedup?
l Told = Tparallelizable + Tsequential
l Tnew = Tparallelizable/100 + Tsequential
1
l Speedup = = 90
(1− Fparallelizable ) + Fparallelizable /100
l Solving: Fparallelizable = 0.999

❑ Need sequential part to be 0.1% of original time

7
Scaling Example
❑ Workload: sum of 10 scalars, and 10 × 10 matrix
sum
l Speed up from 10 to 100 processors
❑ Single processor: Time = (10 + 100) × tadd
❑ 10 processors
l Time = 10 × tadd + 100/10 × tadd = 20 × tadd
l Speedup = 110/20 = 5.5 (5.5/10 = 55% of potential)
❑ 100 processors
l Time = 10 × tadd + 100/100 × tadd = 11 × tadd
l Speedup = 110/11 = 10 (10/100 = 10% of potential)
❑ Assumes load can be balanced across
processors 8
Scaling Example (cont)

❑ What if matrix size is 100 × 100?

❑ Single processor: Time = (10 + 10000) × tadd
❑ 10 processors
l Time = 10 × tadd + 10000/10 × tadd = 1010 × tadd
l Speedup = 10010/1010 = 9.9 (99% of potential)
❑ 100 processors
l Time = 10 × tadd + 10000/100 × tadd = 110 × tadd
l Speedup = 10010/110 = 91 (91% of potential)

❑ Assuming load balanced

9
Strong vs Weak Scaling

❑ Strong scaling: problem size fixed

l As in example

❑ Weakscaling: problem size proportional to

number of processors
Processor
Workload
l 10 processors, 10 × 10 matrix Nums

- Time = 20 × tadd
l 100 processors ↑, 32 × 32 matrix ↑
- Time = 10 × tadd + 1000/100 × tadd = 20 × tadd
l Constant performance in this example

10
 §7.3 Shared Memory Multiprocessors
7.3. Shared Memory Multiprocessors

❑ SMP: shared memory multiprocessor

l Hardware provides single physical
address space for all processors
l Synchronize shared variables using locks
l Usually adapted in general purpose CPU’s in laptops and
desktops

❑ Memory access time: UMA vs NUMA

11
Shared Memory Arch: UMA

❑ access time to a memory location is independent of which

processor makes the request, or which memory chip
contains the transferred data.
❑ Used for a few processors.

Intel's FSB based UMA Arch

P P P
C C C

Bus contention

M M

12
Shared Memory Arch: NUMA

❑ access time depends on the memory location relative to a

processor.
❑ Used for dozens, hundreds of processors
❑ Processors use the same memory address space.
Distributed Shared Memory, DSM
❑ Intel QPI completes with AMD HyperTransport, not bus.
Quick Path Interconnect Arch
Embedded
Mem P P P
Controller
C C C
direct
M M M

Bus

13
Shared Memory Arch: NUMA

❑ Eg. the memory manager of programming languages also need to be

NUMA aware. Java is NUMA aware.
❑ Eg. Oracle 11g explicitly enabled for NUMA support
❑ Eg. Windows XP SP2, Server 2003, Vista supported NUMA

P P P P P P
C C C C C C

M M M M M M

Bus

NUMA node SMP NUMA node

14
Example:Sun Fire V210 / V240 Mainboard

15
Example:Dell PowerEdge R720

16
Example: Sum Reduction

❑ Sum 100,000 numbers on 100 processor UMA

l Each processor has ID: 0 ≤ Pn ≤ 99
l Partition 1000 numbers per processor
l Initial summation on each processor
100,000 numbers

sum[P0] = 0;
sum[Pn] = 0;
for (i=1000*P0;i<1000*(P0+1);i++)
sum[P0] = sum[P0] for (i=1000*Pn;i<1000*(Pn+1); i++)
+ A[i];
sum[Pn] = sum[Pn] + A[i];

❑ Now need to add these partial sums

l Reduction: divide and conquer
l Half the processors add pairs, then quarter, …
l Need to synchronize between reduction steps 17
Example: Sum Reduction

0 1 2 3

(half is odd) 0 1 2 3 4 5 6

half = 100;
repeat
synch();
if (half%2 != 0 && Pn == 0)
sum[0] = sum[0] + sum[half-1];
/* Conditional sum needed when half is odd;
Processor0 gets missing element */
half = half/2; /* dividing line on who sums */
if (Pn < half) sum[Pn] = sum[Pn] + sum[Pn+half]; 18
 §7.4 Clusters and Other Message-Passing Multiprocessors
7.4. Clusters and Other Message-Passing Multiprocessors

❑ Each processor has private physical address space

❑ Hardware sends/receives messages between processors

19
Loosely Coupled Clusters

❑ Network of independent computers

l Each has private memory and OS
l Connected using I/O system
- E.g., Ethernet/switch, Internet

❑ Suitable for applications with independent tasks

l Web servers, databases, simulations, …

❑ High availability, scalable, affordable

❑ Problems
l Administration cost (prefer virtual machines)
l Low interconnect bandwidth
- c.f. processor/memory bandwidth on an SMP
20
Sum Reduction (Again)
❑ Sum 100,000 on 100 processors
❑ First distribute 1000 numbers to each
l The do partial sums

1,000 numbers

sum = 0;
❑ Reduction for (i=0;i<1000;i++)
sum = sum + AN[i];
l Half the processors send, other half receive and add
l The quarter send, quarter receive and add, …

21
Sum Reduction (Again)

❑ Given send() and receive() operations

limit = 100; half = 100; /* 100 processors */
repeat
half = (half+1)/2; /*send vs. receive dividing line*/
if (Pn >= half && Pn < limit)
send(Pn - half, sum);
if (Pn < (limit/2))
sum = sum + receive();
limit = half; /* upper limit of senders */
until (half == 1); /* exit with final sum */

half =7 0 1 2 3 4 5 6
l Send/receive also provide synchronization
l Assumes send/receive take similar time to addition

22
Message Passing Systems
❑ONC RPC, CORBA, Java RMI, DCOM, SOAP, .NET
Remoting, CTOS, QNX Neutrino RTOS, OpenBinder, D-
Bus, Unison RTOS
❑ Message passing systems have been called "shared
nothing" systems (each participant is a black box).
❑Message passing is a type of communication between
processes or objects in computer science
❑ Opensource: Beowulf, Microwulf

Microwulf Beowulf A sending-message cluster

23
 §7.5 Hardware Multithreading
7.5. Hardware Multithreading

❑ Performing multiple threads of execution in

parallel
l Replicate registers, PC, etc.
l Fast switching between threads
❑ Three designs:
l Coarse-grain multithread, CMT
l Fine-grain multithread, FMT
l Simultaneous multithread, SMT

Coarse: Thiết kế tồi, kém, đơn giản

Fine: thiết kế tốt
Simultaneous: Đồng thời

26
 Coarse-grain multithreading

❑ Only switch on long stall (e.g., L2-cache miss)

❑ Simplifieshardware, but doesn’t hide short stalls
(eg, data hazards)
❑ Thread scheduling policy
l Designate a “preferred” thread (e.g., thread A)
l Switch to thread B on thread A L2 miss
l Switch back to A when A L2 miss returns

❑ Sacrificesvery little single thread performance

(of one thread)
❑ Example: IBM Northstar/Pulsar
27
Coarse-grain multithreading

Same
thread

28
Fine-grain multithread

❑ Switch threads after each cycle (round-robin), L2 miss or

no.
❑ Interleave instruction execution
❑ If one thread stalls, others are executed
❑ Sacrifices significant single thread performance
❑ Need a lot of threads

❑ Not popular today

l Many threads ! many register files

❑ Extreme example: Denelcor HEP

l So many threads (100+),
it didn’t even need caches
29
Fine-grain multithread

Thread A Thread B Thread C Thread D

30
Simultaneous Multithreading
❑ In multiple-issue dynamically scheduled processor
l Schedule instructions from multiple threads
l Instructions from independent threads execute when function
units are available
l Within threads, dependencies handled by scheduling and
register renaming

❑ Example: Intel Pentium-4 HT

l Two threads: duplicated registers, shared function units and
caches

34
Simultaneous Multi-Threading

“permit different threads to occupy the same pipeline stage

at the same time”

❑ This makes most sense with superscalar issue

Inst Cache Data Cache

Decode+Registers
Inst Issue Logic

PCA
Fetch Logic

Fetch

Write Logic
Mem Logic
Logic
PCB

35
Multithreading Example

37
Future of Multithreading
❑ Will it survive? In what form?
❑ Power considerations  simplified microarchitectures
l Simpler forms of multithreading

❑ Tolerating cache-miss latency

l Thread switch may be most effective

❑ Multiple simple cores might share resources more

effectively

38
 §7.6 SISD, MIMD, SIMD, SPMD, and Vector
7.6. Instruction and Data Streams

❑ An alternate classification

Data Streams
Single Multiple
Instruction Single SISD: SIMD: SSE
Streams Intel Pentium 4 instructions of x86
Multiple MISD: MIMD:
No examples today Intel Xeon e5345

◼ SPMD: Single Program Multiple Data

◼ A parallel program on a MIMD computer
◼ Conditional code for different processors

39
Single Instruction, Single Data

❑ Single Instruction: Only one

instruction stream is being acted on
by the CPU during any one clock
cycle
❑ Single Data: Only one data stream is
being used as input during any one
clock cycle
❑ Deterministic execution
❑ Examples: older generation
mainframes, minicomputers and
workstations; most modern day PCs.
40
Single Instruction, Single Data

UNIVAC1 IBM 360 CRAY 1

CDC 7600 PDP 1 Laptop

41
Multi Instruction, Multi Data

◼ Multiple Instruction: Every

processor may be executing a
different instruction stream
◼ Multiple Data: Every processor
may be working with a different
data stream
❑ Execution can be synchronous or asynchronous, deterministic or
non-deterministic
❑ Currently, the most common type of parallel computer - most
modern supercomputers fall into this category.
❑ Examples: most current supercomputers, networked parallel
computer clusters and "grids", multi-processor SMP computers,
multi-core PCs.
❑ Note: many MIMD architectures also include SIMD execution sub-
components
42
Multi Instruction, Multi Data

HP/Compaq
IBM POWER5 Intel IA32
Alphaserver

AMD Opteron Cray XT3 IBM BG/L

43
Single Instruction, Multiple Data

❑ Operate elementwise on vectors of data

l E.g., MMX and SSE instructions in x86
- Multiple data elements in 128-bit wide registers

❑ Allprocessors execute the same instruction at

the same time
l Each with different data address, etc.

❑ Reduced instruction
control hardware
❑ Works best for highly
data-parallel applications, high degree of
regularity, such as graphics/image processing 44
Single Instruction, Multiple Data

❑ Synchronous (lockstep) and deterministic execution

❑ Two varieties: Processor Arrays and Vector Pipelines
❑ Most modern computers, particularly those with
graphics processor units (GPUs) employ SIMD
instructions and execution units.

45
Single Instruction, Multiple Data

ILLIAC IV MasPar

CRAY X-MP Cray Y-MP

Thinking Machines
GPU
CM-2

46
Vector Processors

❑ Highly pipelined function units

❑ Stream data from/to vector registers to units
l Data collected from memory into registers
l Results stored from registers to memory

❑ Example: Vector extension to MIPS

l 32 × 64-element registers (64-bit elements)
l Vector instructions
-lv, sv: load/store vector
-addv.d: add vectors of double
-addvs.d: add scalar to each element of vector of double

❑ Significantly reduces instruction-fetch bandwidth

47
 Example: DAXPY (Y = a × X + Y)
❑ Conventional MIPS code
l.d $f0,a($sp) ;load scalar a
addiu r4,$s0,#512 ;upper bound of what to load
loop: l.d $f2,0($s0) ;load x(i)
mul.d $f2,$f2,$f0 ;a × x(i)
l.d $f4,0($s1) ;load y(i)
add.d $f4,$f4,$f2 ;a × x(i) + y(i)
s.d $f4,0($s1) ;store into y(i)
addiu $s0,$s0,#8 ;increment index to x
addiu $s1,$s1,#8 ;increment index to y
subu $t0,r4,$s0 ;compute bound
bne $t0,$zero,loop ;check if done
❑ Vector MIPS code
l.d $f0,a($sp) ;load scalar a
lv $v1,0($s0) ;load vector x
mulvs.d $v2,$v1,$f0 ;vector-scalar multiply
lv $v3,0($s1) ;load vector y
addv.d $v4,$v2,$v3 ;add y to product
sv $v4,0($s1) ;store the result

48
Vector vs. Scalar
❑ Vector architectures and compilers
l Simplify data-parallel programming
l Explicit statement of absence of loop-carried dependences
- Reduced checking in hardware
l Regular access patterns benefit from interleaved and burst
memory
l Avoid control hazards by avoiding loops

❑ More general than ad-hoc media extensions (such as

MMX, SSE)
l Better match with compiler technology

49
 §7.7 Introduction to Graphics Processing Units
7.7. History of GPUs

Early video cards

• Frame buffer memory with address generation for
video output

3D graphics processing
• Originally high-end computers (e.g., SGI)
• Moore’s Law  lower cost, higher density
• 3D graphics cards for PCs and game consoles

Graphics Processing Units

• Processors oriented to 3D graphics tasks
• Vertex/pixel processing, shading, texture mapping,
rasterization

50
Graphics in the System

51
GPU Architectures
❑ Processing is highly data-parallel
l GPUs are highly multithreaded
l Use thread switching to hide memory latency
- Less reliance on multi-level caches
l Graphics memory is wide and high-bandwidth
❑ Trend toward general purpose GPUs
l Heterogeneous CPU/GPU systems
l CPU for sequential code, GPU for parallel code
❑ Programming languages/APIs
l DirectX, OpenGL
l C for Graphics (Cg), High Level Shader Language
(HLSL)
Heterogeneous: không đồng nhất
l Compute Unified Device Architecture (CUDA) 52
Example: NVIDIA Tesla

Streaming
Multiprocessor

8 × Streaming
processors

53
Example: NVIDIA Tesla
❑ Streaming Processors
l Single-precision FP and integer units
l Each SP is fine-grained multithreaded

❑ Warp: group of 32 threads

l Executed in parallel,
SIMD style
- 8 SPs
× 4 clock cycles
l Hardware contexts
for 24 warps
- Registers, PCs, …

54
 §7.8 Introduction to Multiprocessor Network Topologies
7.8. Interconnection Networks

❑ Network topologies
l Arrangements of processors, switches, and links

Bus Ring

N-cube (N = 3)
2D Mesh
Fully connected

56
Network Characteristics
❑ Performance
l Latency per message (unloaded network)
l Throughput
- Link bandwidth
- Total network bandwidth
- Bisection bandwidth
l Congestion delays (depending on traffic)

❑ Cost
❑ Power
❑ Routability in silicon

58
 §7.9 Multiprocessor Benchmarks
7.9. Parallel Benchmarks

❑ Linpack: matrix linear algebra

❑ SPECrate: parallel run of SPEC CPU programs
l Job-level parallelism

❑ SPLASH:Stanford Parallel Applications for

Shared Memory
l Mix of kernels and applications, strong scaling

❑ NAS (NASA Advanced Supercomputing) suite

l computational fluid dynamics kernels

❑ PARSEC (Princeton Application Repository for

Shared Memory Computers) suite
l Multithreaded applications using Pthreads and 59
 §7.10 Roofline: A Simple Performance Model
7.10. Modeling Performance
❑ Assume performance metric of interest is achievable
GFLOPs/sec
l Measured using computational kernels from Berkeley Design
Patterns

❑ Arithmetic intensity of a kernel

l FLOPs per byte of memory accessed

❑ For a given computer, determine

l Peak GFLOPS (from data sheet)
l Peak memory bytes/sec (using Stream benchmark)

61
Roofline Diagram

Attainable GPLOPs/sec
= Max ( Peak Memory BW × Arithmetic Intensity, Peak FP Performance )

62
Comparing Systems

❑ Example: Opteron X2 vs. Opteron X4

l 2-core vs. 4-core, 2× FP performance/core, 2.2GHz
vs. 2.3GHz
l Same memory system

◼ To get higher performance

on X4 than X2
◼ Need high arithmetic intensity
◼ Or working set must fit in X4’s
2MB L-3 cache

63
Optimizing Performance

❑ Optimize FP performance
l Balance adds & multiplies
l Improve superscalar ILP and use of
SIMD instructions

❑ Optimize memory usage

l Software prefetch
- Avoid load stalls
l Memory affinity
- Avoid non-local data accesses

64
Optimizing Performance

❑ Choice of optimization depends on arithmetic intensity of

code

◼ Arithmetic intensity is
not always fixed
◼ May scale with
problem size
◼ Caching reduces
memory accesses
◼ Increases arithmetic
intensity
65
 §7.11 Real Stuff: Benchmarking Four Multicores …
7.11. Four Example Systems

2 × quad-core
Intel Xeon e5345
(Clovertown)

2 × quad-core
AMD Opteron X4 2356
(Barcelona)

66
Four Example Systems

2 × oct-core
Sun UltraSPARC
T2 5140 (Niagara 2)

2 × oct-core
IBM Cell QS20

67
And Their Rooflines

❑Kernels
l SpMV (left)
l LBHMD (right)

❑Some optimizations
change arithmetic
intensity
❑x86 systems have
higher peak GFLOPs
l But harder to achieve,
given memory
bandwidth

68
Performance on SpMV

❑ Sparse matrix/vector multiply

l Irregular memory accesses, memory bound

❑ Arithmetic intensity
l 0.166 before memory optimization, 0.25 after
◼ Xeon vs. Opteron
◼ Similar peak FLOPS
◼ Xeon limited by shared FSBs
and chipset
◼ UltraSPARC/Cell vs. x86
◼ 20 – 30 vs. 75 peak GFLOPs
◼ More cores and memory
bandwidth

69
Performance on LBMHD

❑ Fluid dynamics: structured grid over time steps

l Each point: 75 FP read/write, 1300 FP ops

❑ Arithmetic intensity
l 0.70 before optimization, 1.07 after
◼ Opteron vs. UltraSPARC
◼ More powerful cores, not
limited by memory bandwidth
◼ Xeon vs. others
◼ Still suffers from memory
bottlenecks

70
Achieving Performance

❑ Compare naïve vs. optimized code

l If naïve code performs well, it’s easier to write high performance
code for the system

System Kernel Naïve Optimized Naïve as % of

GFLOPs/sec GFLOPs/sec optimized
Intel Xeon SpMV 1.0 1.5 64%
LBMHD 4.6 5.6 82%
AMD SpMV 1.4 3.6 38%
Opteron X4 LBMHD 7.1 14.1 50%
Sun UltraSPARC SpMV 3.5 4.1 86%
T2 LBMHD 9.7 10.5 93%
IBM Cell QS20 SpMV Naïve code 6.4 0%
LBMHD not feasible 16.7 0%

71
 §7.12 Fallacies and Pitfalls
7.12. Fallacies
❑ Amdahl’s Law doesn’t apply to parallel computers
l Since we can achieve linear speedup
l But only on applications with weak scaling

❑ Peak performance tracks observed performance

l Marketers like this approach!
l But compare Xeon with others in example
l Need to be aware of bottlenecks

73
Pitfalls
❑ Not developing the software to take account of a
multiprocessor architecture
l Example: using a single lock for a shared composite resource
- Serializes accesses, even if they could be done in parallel
- Use finer-granularity locking

74
 §7.13 Concluding Remarks
7.13. Concluding Remarks

❑ Goal:higher performance by using multiple

processors
❑ Difficulties
l Developing parallel software
l Devising appropriate architectures

❑ Many reasons for optimism

l Changing software and application environment
l Chip-level multiprocessors with lower latency, higher
bandwidth interconnect

❑ An ongoing challenge for computer architects!

75