GPU Computing CIS-543

Lecture 07: CUDA Execution Model

Dr. Muhammad Abid,

DCIS, PIEAS

GPU Computing, PIEAS

 CUDA Execution Model

What's an Execution Model?

an execution model provides an operational view
of how instructions are executed on a specific
computing architecture.
CUDA execution model
exposes an abstract view of the GPU parallel
architecture, allowing you to reason about thread
concurrency.
Why CUDA execution model?
provides insights that are useful for writing
efficient code in terms of both instruction
throughput and memory accesses.

GPU Computing, PIEAS

 Aside: GPU Computing Platforms

NVIDIAs GPU computing platform is enabled

on the following product families:
Tegra: for mobile and embedded devices
GeForce: for Consumer graphics
Quadro: for professional visualization
Tesla: for datacenters

GPU Computing, PIEAS

 GPU Architecture Overview

The GPU architecture is built around a

scalable array of Streaming Multiprocessors
(SM).
GPU hardware parallelism is achieved
through the replication of SMs.

GPU Computing, PIEAS

 GPU Architecture Overview: Fermi SM

CUDA Cores
Shared
Memory/L1
Cache
Register File
Load/Store
Units
Special
Function Units
Warp
Scheduler
GPU Computing, PIEAS
 GPU Architecture Overview

CUDA employs a Single Instruction Multiple

Thread (SIMT) architecture to manage and
execute threads in groups of 32 called warps.
All threads in a warp execute the same
instruction at the same time.
Each thread has its own instruction address
counter and register state, and carries out
the current instruction on its own data.
Each SM partitions the thread blocks
assigned to it into 32-thread warps that it
then schedules for execution on available
hardware resources.
GPU Computing, PIEAS
 Threadblock Scheduling

A thread block is scheduled on only one SM.

Once a thread block is scheduled on an SM,
it remains there until execution completes.
An SM can hold more than one thread block
at the same time.

GPU Computing, PIEAS

 Threadblock Scheduling

GPU Computing, PIEAS

 Warps Scheduling

While warps within a thread block may be

scheduled in any order, the number of active
warps is limited by SM resources.
When a warp idles for any reason (for
example, waiting for values to be read from
device memory), the SM is free to schedule
another available warp from any thread block
that is resident on the same SM.
Switching between concurrent warps has
little overhead because hardware resources
are partitioned among all threads and blocks
on an SM, so the state of the newly
scheduled
GPU Computing, PIEAS warp is already stored on the SM.
 Warps Scheduling

Cores are
grouped into
32-core
group

GPU Computing, PIEAS

 Warps Scheduling

Cores are grouped into 16-core group

GPU Computing, PIEAS Insts belong to different warps
 Shared Memory and Registers

Shared memory and registers are precious

resources in an SM.
Shared memory is partitioned among thread
blocks resident on the SM and registers are
partitioned among threads.
Threads in a thread block can cooperate and
communicate with each other through these
resources.
While all threads in a thread block run
logically in parallel, not all threads can
execute physically at the same time.

GPU Computing, PIEAS

 Aside: SM: The heart of the GPU arch.

The Streaming Multiprocessor (SM) is the

heart of the GPU architecture.
Registers and shared memory are scarce
resources in the SM. CUDA partitions these
resources among all threads resident on an
SM. Therefore, these limited resources
impose a strict restriction on the number of
active warps in an SM, which corresponds to
the amount of parallelism possible in an SM.

GPU Computing, PIEAS

 The Fermi Architecture

First complete GPU computing architecture

to deliver the features required for the most
demanding HPC applications.
Fermi has been widely adopted for
accelerating production workloads.

GPU Computing, PIEAS

 The Fermi Architecture

graphics-
specific
components
largely
omitted.
512 CUDA
cores
Each SM
represented
by a vertical
rectangular CUDA Core, Register file, Warp scheduler & dispatching unit, S

strip
GPU Computing, PIEAS
 The Fermi Architecture

Each CUDA core has a fully pipelined integer

arithmetic logic unit (ALU) and a floating-
point unit (FPU) that executes one integer or
floating-point instruction per clock cycle.
The CUDA cores are organized into 16
streaming multiprocessors (SM), each with
32 CUDA cores.
Fermi has six 64-bit GDDR5 DRAM memory
interfaces supporting up to a total of 6 GB of
global on-board memory, a key compute
resource for many applications.

GPU Computing, PIEAS

 The Fermi Architecture

A host interface connects the GPU to the

CPU via the PCI Express bus.
The GigaThread engine is a global scheduler
that distributes thread blocks to the SM warp
schedulers.
Each multiprocessor has 16 load/store units
allowing source and destination addresses to
be calculated for 16 threads (a half-warp) per
clock cycle.
Special function units (SFUs) execute
intrinsic instructions such as sine, cosine,
square root, , and interpolation.
GPU Computing, PIEAS
 The Fermi Architecture

Each SFU can execute one intrinsic

instruction per thread per clock cycle.
Each SM features two warp schedulers and
two instruction dispatch units.
Shared/ L1 cache : 48kB or 16kB
L2$ : 768KB
Register File: 32k * 32-bit
Concurrent kernels execution: 16 Kernels

GPU Computing, PIEAS

 The Kepler Architecture

released in the fall of 2012, is a fast and

highly efficient, high-performance computing
architecture.

GPU Computing, PIEAS

 Kepler K20X chip block diagram

GPU Computing, PIEAS

 Kepler Architecture Innovations

Enhanced SMs
Dynamic Parallelism
Hyper-Q

GPU Computing, PIEAS

 Kepler Architecture Innovations: Enhanced SMs

GPU Computing, PIEAS

 Kepler Architecture Innovations:
Enhanced SMs
192 single-precision CUDA cores
64 double-precision units,
32 special function units (SFU), and
32 load/store units (LD/ST)
Register file size: 64k
four warp schedulers and eight instruction
dispatchers
The Kepler K20X:
can schedule 64 warps per SM for a total of 2,048
threads
1 TFlop of peak double-precision computing
power
GPU Computing, PIEAS
 Kepler Architecture Innovations:
Dynamic Parallelism
allows the GPU to dynamically launch new
kernels
makes it easier for you to create and
optimize recursive and data-dependent
execution patterns.

GPU Computing, PIEAS

 Kepler Architecture Innovations:
Dynamic Parallelism

GPU Computing, PIEAS

 Kepler Architecture Innovations:
Hyper-Q
adds more simultaneous hardware
connections between the CPU and GPU,
enabling CPU cores to simultaneously run more
tasks on the GPU.
increased GPU utilization
reduced CPU idle time
Fermi GPUs rely on a single hardware work
queue to pass tasks from the CPU to the
GPU, which could cause a single task to
block all other tasks behind it in the queue
from making progress. Kepler Hyper-Q
removes this limitation.
GPU Computing, PIEAS
 Kepler Architecture Innovations:
Hyper-Q

GPU Computing, PIEAS

 Kepler Architecture Innovations:
Hyper-Q
Hyper-Q enables more concurrency on the
GPU, maximizing GPU utilization and
increasing overall performance.
Lesson: Use multiple streams in your
application

GPU Computing, PIEAS

 Compute Capability & Arch. Features

GPU Computing, PIEAS

 Threadblock and Warps

Warps are the basic unit of execution in an

SM.
When you launch a grid of thread blocks, the
thread blocks in the grid are distributed
among SMs.
Once a thread block is scheduled to an SM,
threads in the thread block are further
partitioned into warps.

GPU Computing, PIEAS

 Threadblock and Warps

A warp consists of 32 consecutive threads

and all threads in a warp are executed in
Single Instruction Multiple Thread (SIMT)
fashion; that is, all threads execute the same
instruction, and each thread carries out that
operation on its own private data.

GPU Computing, PIEAS

 Threadblock and Warps

Thread blocks can be configured to be one-, two-, or

three-dimensional. However, from the hardware
perspective, all threads are arranged one-
dimensionally.
Each thread has a unique ID in a block.
For a one-dimensional thread block, the unique
thread ID is stored in the CUDA built-in variable
threadIdx.x.
Threads with consecutive values for threadIdx.x are
grouped into warps.

GPU Computing, PIEAS

 Threadblock and Warps

The logical layout of a two or three-

dimensional thread block can be converted
into its one-dimensional physical layout by
using:
threadIdx.y * blockDim.x + threadIdx.x.
threadIdx.z * blockDim.y * blockDim.x +
threadIdx.y * blockDim.x + threadIdx.x

GPU Computing, PIEAS

 Threadblock and Warps

The number of warps for a thread block can

be determined as follows:
WarpsPerBlock = ceil ( ThreadsPerBlock /

warpSize )
A warp is never split between different thread
blocks.
If thread block size is not an even multiple of
warp size, some threads are left inactive
( still consume SM resources, such as
registers.)
GPU Computing, PIEAS
 Threadblock and Warps

Lesson: Keep threadblock size multitple of

warp size:
This doesn't waste GPU resources
Definitely likely to improve performance
Run the vector addition using <<<N,1>>>
N equals number of elements in a vector
GPU Computing, PIEAS
 Aside Threadblock: Logical View vs
Hardware View
From the logical perspective, a thread block
is a collection of threads organized in a 1D,
2D, or 3D layout.
From the hardware perspective, threads in a
thread block are organized in a 1D layout,
and each set of 32 consecutive threads
forms a warp.

GPU Computing, PIEAS

 Warp Divergence

Occurs when threads in a warp follow

different execution paths.
if (cond) { ... } else { ... }
assume this code in a kernel
Also assume, for some threads cond is true while
for others cond is false. This situation creates
warp divergence
If threads of a warp diverge, the warp serially
executes each branch path, disabling threads
that do not take that path.
Warp divergence can cause significantly
degraded performance.
GPU Computing, PIEAS
 Warp Divergence

GPU Computing, PIEAS

 Warp Divergence

Lesson:
Try to avoid different execution paths within the
same warp.
Keep in mind that the assignment of threads to
warp in a thread block is deterministic. Therefore,
it may be possible (though not trivial, depending
on the algorithm) to partition data in such a way
as to ensure all threads in the same warp take the
same control path in an application.

GPU Computing, PIEAS

 Warp Execution Context

Warp: basic unit of execution in GPUs

Main resources include:
Program counters
Registers
Shared memory
Eacp warp's execution context maintained
on-chip during the entire lifetime of the warp.
Therefore, switching from one execution
context to another has little overhead (1 or 2
cycles).

GPU Computing, PIEAS

 Warp Execution Context

Each SM has a set of 32-bit registers stored

in a register file that are partitioned among
threads, and a fixed amount of shared
memory that is partitioned among thread
blocks.
The number of thread blocks and warps that
can simultaneously reside on an SM for a
given kernel depends on 1) the number of
registers, 2) amount of shared memory and
3) execution configuration

GPU Computing, PIEAS

 Warp Execution Context

GPU Computing, PIEAS

 Warp Execution Context

GPU Computing, PIEAS

 Warp Execution Context

Resource availability generally limits the

number of resident thread blocks per SM.
If there are insufficient registers or shared
memory on each SM to process at least one
block, the kernel launch will fail.

GPU Computing, PIEAS

 Warp Execution State

Active thread block when compute

resources, such as registers and shared
memory, have been allocated to it. The warps
it contains are called active warps.
Active warps can be further classified into the
following three types:
Selected warp: actively executing now
Stalled warp: not ready for execution
Eligible warp: ready for execution but not currently
executing

GPU Computing, PIEAS

 Warp Execution State

The warp schedulers on an SM select active

warps on every cycle and dispatch them to
execution units.
In order to maximize GPU utilization, you
need to maximize the number of active
warps.

GPU Computing, PIEAS

 Latency Hiding/ Full Utilization

An SM relies on thread-level parallelism to

maximize utilization of its functional units.
Utilization is therefore directly linked to the
number of resident warps.
Full compute resource utilization is achieved
when all warp schedulers have an eligible
warp at instruction issue time.
This ensures that the latency of each instruction
can be hidden by issuing other instructions in
other resident warps.

GPU Computing, PIEAS

 Latency Hiding/ Full Utilization

GPU Computing, PIEAS

 Latency Hiding/ Full Utilization

When considering instruction latency,

instructions can be classified into two basic
types:
Arithmetic instructions
Memory instructions

GPU Computing, PIEAS

 Latency Hiding/ Full Utilization

Estimating the number of active warps

required to hide latency.
Number of Required Warps = Inst latency
Throughput (in terms of warp)
Fine-grained multithreading in each CUDA core
Single thread: assume inst2 depends on inst1 and
inst1 latency is 20 cycles then to hide inst1
latency this thread needs 20 instructions b/w inst1
and inst2. The same basic principal applies to
GPUs except that instructions come from different
threads, i.e. fine-grained multithreading.

GPU Computing, PIEAS

 Arith. Inst Latency Hiding/ Full
Utilization
For arithmetic operations, the required
parallelism can be expressed as the number
of operations required to hide arithmetic
latency. The arithmetic operation used as an
example here is a 32-bit floating-point
multiply-add (a + b c), expressed as the
number of operations per clock cycle per SM.
The throughput varies for different arithmetic
instructions.

GPU Computing, PIEAS

 Arith. Inst Latency Hiding/ Full
Utilization
To keep SM fully utilized for MAD inst:
Fermi needs 640 MAD operations
Kepler needs 3,840 MAD operations
These independent operations come from
640 or 3,840 threads or fewer threads but
with more indep operations per thread

GPU Computing, PIEAS

 Arith. Inst Latency Hiding/ Full
Utilization
Lesson: Two ways to maximize SM
arithmetic resources:
Create many threads more warps
Write Fine-grained threads, i.e. theads perform few ops
Fewer threads but more independent operations
per thread, i.e. loop unrolling
Write course-grained threads
Best: combine both: large number of threads
with many indep operations

GPU Computing, PIEAS

 Mem. Inst Latency Hiding/ Full
Utilization
For memory operations, the required
parallelism is expressed as the number of
bytes per cycle required to hide memory
latency.
144 GB/Sec 1.566 GHz 92 Bytes/Cycle
74 or 77KB for the whole device

GPU Computing, PIEAS

 Mem. Inst Latency Hiding/ Full
Utilization
Connecting these values to warp or thread
counts depends on the application.
Suppose each thread moves one float of data (4
bytes) from global memory to the SM for
computation, we would require 18,500 threads or
579 warps to hide all memory latency on Fermi
GPUs:
74 KB 4 bytes/thread 18,500 threads
18,500 threads 32 threads/warp 579 warps
The Fermi architecture has 16 SMs. Therefore, we
require 579 warps 16 SMs = 36 warps per SM to hide
all memory latency.
If each thread performed more than one
independent 4-byte load, fewer threads would be
required
GPU Computing, PIEASto hide the memory latency.
 Mem. Inst Latency Hiding/ Full
Utilization
Lesson: Same conclusion as for "Arithmetic
Inst Latency Hiding".
Two ways to maximize available device
bandwidth:
Create many threads more warps
Write Fine-grained threads, i.e. theads perform few
memory operations
Fewer threads but more independent memory
operations per thread, i.e. loop unrolling
Write course-grained threads with lots of indep memory
operations
Best: combine both: large number of threads
with many indep operations
GPU Computing, PIEAS
 Aside

Latency hiding depends on the number of

active warps per SM,
determined by the execution configuration and
resource constraints (registers and shared
memory usage in a kernel).
Choosing an optimal execution configuration is a
matter of striking a balance between latency
hiding and resource utilization.

GPU Computing, PIEAS

 Occupancy

Ideally, you want to have enough warps to

keep the cores of the device occupied.
Occupancy is the ratio of active warps to
maximum number of warps, per SM.
Occupancy = active warps / maximum warps
Calculating maximum warps per SM:
Get val of maxThreadsPerMultiProcessor
member of cudaDeviceProp structure and divide
by 32.
CUDA Occupancy calculator : assists in
choosing thread block size based on shared
memory and per-thread reg requirements.
GPU Computing, PIEAS
 Using Occupancy Cal

Specify compute capability of the GPU

Next enter the following parameters (which
determine No. of active warps per SM):
Threads per block (execution configuration)
Registers per thread (resource usage)
Shared memory per block (resource usage)
The registers per thread and shared memory
per block:
Use --ptxas-options=-v or -Xptxas -v flag with
nvcc

GPU Computing, PIEAS

 Using Occupancy Cal: optimal values
for reg/thread and shared memory
compute capability = 3.5
Max. 32-bit regs = 65536 = 216
Register allocation unit size = 256 = 28
Total reg allocation units = 216 / 28 = 28
Tot. reg allocation units per warp = 28 / 26 = 4
Tot. regs per thread = 22 * 28 = 210 / 25 = 25
If reg/thread <= 32, amount of parallelism is never
limited by this factor
Configured shared mem = 49152 bytes
Per block shared memory = 49152 / 16 = 3072
bytes, so max no. of block/SM can be scheduled
If shared mem per block <= 3072B, amount of
parallelism is never limited by shared memory
GPU Computing, PIEAS
 Using Occupancy Cal: optimal values
for reg/thread and shared memory
As long as kernel is using regs <= 32 per
thread and shared memory per block <=
3072 bytes, amount of parallelism is not
limited by either reg/ thread or shared
memory/block
Only Execution Configuration can limit amount of
parallelism
Keep the number of threads per block a multiple of warp
size (32).
Avoid small block sizes: Start with at least 128 or 256
threads per block.
CUDA Occupancy calculator :

GPU Computing, PIEAS

 Occupancy Cal

Occupancy Cal gives no info about:

Arrangement of threads in a thread block
Thread block size should be a multiple of warp
size, i.e. 32
Performance
Real-time usage of SM resources
Memory bandwidth utilization
Static occupancy calculation
Use achieved_occupancy metric to cal average
active warps per cycle per SM
Achieved occupancy = average active warps per cycle / tot
warps per SM

GPU Computing, PIEAS

 Occupancy: Controlling Register
Count
Controlling registers per thread:
Use maxrregcount=NUM flag with nvcc, tells the
compiler to not use more than NUM registers per
thread.
Optimal NUM val: obtained from occupancy
calculator or from prevous slide

GPU Computing, PIEAS

 Aside: Grid and Threadblock size

Keep the number of threads per block a

multiple of warp size (32).
Avoid small block sizes: Start with at least
128 or 256 threads per block.
Adjust block size up or down according to
kernel resource requirements.
Keep the number of blocks much greater
than the number of SMs to expose sufficient
parallelism to your device.
Conduct experiments to discover the best
execution configuration and resource usage.
GPU Computing, PIEAS
 Grid & Threadblock heuristics

A must have have skill for CUDA C

Programmer
Using occupancy cal we found the optimal
threadblock size. However, we learnt no info
on how to arrange these threads in a
threadblock
Run prog with different combination of
threadblock sizes and measure exec. time.
2D threadblock in matrix addition: 32 X 32, 512 X
2, 1024 X 1, 1 X 1024

GPU Computing, PIEAS

 Backup Slides just for your information

GPU Computing, PIEAS

 Latency Hiding

Suppose the average

latency for an instruction in
your kernel is 5 cycles.
To keep a throughput of 6
warps executed per cycle,
you will need at least 30
warps in-flight.

GPU Computing, PIEAS