Task decomposition and mapping

Alexandre David

Introduction to Parallel Computing

Overview
Introduction to parallel algorithms
Decomposition techniques
Task interactions
Load balancing

Introduction to Parallel Computing

Introduction
Parallel algorithms have the added dimension of
concurrency.
Typical tasks:
Identify concurrent works.
Map them to processors.
Distribute inputs, outputs, and other data.
Manage shared resources.
Synchronize the processors.

Introduction to Parallel Computing

There are other courses specifically on concurrency. We wont treat the

problems proper to concurrency such as deadlocks, livelocks, theory on
semaphores and synchronization. However, we will use them, and when
needed, apply techniques to avoid problems like deadlocks.

Decomposing problems
Decomposition into concurrent tasks.
No unique solution.
Different sizes.
Decomposition illustrated as a directed graph:
Nodes = tasks.
Edges = dependency.

Task dependency graph

Introduction to Parallel Computing

Many solutions are often possible but few will yield good performance and be
scalable. We have to consider the computational and storage resources
needed to solve the problems.
Size of the tasks in the sense of the amount of work to do. Can be more, less,
or unknown. Unknown in the case of a search algorithm is common.
Dependency: All the results from incoming edges are required for the tasks at
the current node.
We will not consider tools for automatic decomposition. They work fairly well
only for highly structured programs or options of programs.

Task dependency graph?

Vector

Example: Matrix * Vector

Matrix
N tasks, 1 task/row:

Introduction to Parallel Computing

Example: database query processing

MODEL = ``CIVIC'' AND YEAR = 2001 AND
(COLOR = ``GREEN'' OR COLOR = ``WHITE)

Introduction to Parallel Computing

The question is: How to decompose this into concurrent tasks? Different tasks
may generate intermediate results that will be used by other tasks.

A solution

Meas
ure o
Nb. o f concurr
f pro
ency?
c
Optim essors?
al?

Introduction to Parallel Computing

How much concurrency do we have here? How many processors to use? Is it

optimal?

Another Solution

Bet
ter
/wo
rse
?

Introduction to Parallel Computing

Is it better or worse? Why?

Granularity
Number and size of tasks.
Fine-grained: many small tasks.
Coarse-grained: few large tasks.
Related: degree of concurrency.
(Nb. of tasks executable in parallel).
Maximal degree of concurrency.
Average degree of concurrency.

Introduction to Parallel Computing

Previous matrix*vector fine-grained.

Database example coarse grained.
Degree of concurrency: Number of tasks that can be executed in parallel.
Average degree of concurrency is a more useful measure.
Assume that the tasks in the previous database examples have the same
granularity. Whats their average degrees of concurrency? 7/3=2.33 and
7/4=1.75.
Common sense: Increasing the granularity of decomposition and utilizing the
resulting concurrency to perform more tasks in parallel increases performance.
However, there is a limit to granularity due to the nature of the problem itself.

Vector

Coarser Matrix * Vector

Matrix
N tasks, 3 task/row:

Introduction to Parallel Computing

10

10

Granularity
Average degree of concurrency if we take into account
varying amount of work?
Critical path = longest directed path between any start &
finish nodes.
Critical path length = sum of the weights of nodes along
this path.
Average degree of concurrency = total amount of work /
critical path length.

Introduction to Parallel Computing

11

Weights on nodes denote the amount of work to be done on these nodes.

Longest path shortest time needed to execute in parallel.

11

Database example
Critical path (3).
Critical path length = 27.
Av. deg. of concurrency = 63/27.

Critical path (4).

Critical path length = 34.
Av. deg. of conc. = 64/34.

2.33
Introduction to Parallel Computing

1.88
12

12

 Maximum degree of
concurrency.
Critical path length.
Maximum possible speedup.
Minimum number of
processes to reach this
speedup.
Maximum speedup if we
limit the processes to 2,4,
and 8.

Exercise
(a)

(b)

(c)

(d)
Introduction to Parallel Computing

13

13

Interaction between tasks

Tasks often share data.
Task interaction graph:
Nodes = tasks.
Edges = interaction.
Optional weights.
Task dependency graph is a sub-graph of the task
interaction graph.

Introduction to Parallel Computing

18

Another important factor is interaction between tasks on different processors.

Share data implies synchronization protocols (mutual exclusion, etc) to ensure
consistency.
Edges generally undirected. When directed edges are used, they show the
direction of the flow of data (and the flow is unidirectional).
Dependency between tasks implies interaction between them.

18

Processes and mapping

Tasks run on processors.
Process: processing agent executing the tasks. Not
exactly like in your OS course.
Mapping = assignment of tasks to processes.
API exposes processes and binding to processors not
always controlled.
Scheduling of threads is not controlled.
What makes a good mapping?

Introduction to Parallel Computing

19

Here we are not talking directly on the mapping to processors. A processor

can execute two processes.
Good mapping:
Maximize concurrency by mapping independent tasks to different processes.
Minimize interaction by mapping interacting tasks on the same process.
Can be conflicting, good trade-off is the key to performance.
Decomposition determines degree of concurrency.
Mapping determines how much concurrency is utilized and how efficiently.

19

Mapping example

Introduction to Parallel Computing

20

Notice that the mapping keeps one process from the previous stage because
of dependency: We can avoid interaction by keeping the same process.

20

Processes vs. processors

Processes = logical computing agent.
Processor = hardware computational unit.
In general 1-1 correspondence but this model gives
better abstraction.
Useful for hardware supporting multiple programming
paradigms.

Now remains the question:

How do you decompose?
Introduction to Parallel Computing

21

Example of hybrid hardware: cluster of MP machines. Each node has shared

memory and communicates with other nodes via MPI.
1. Decompose and map to processes for MPI.
2. Decompose again but suitable for shared memory.

21

Decomposition techniques
Recursive decomposition.
Divide-and-conquer.
Data decomposition.
Large data structure.
Exploratory decomposition.
Search algorithms.
Speculative decomposition.
Dependent choices in computations.

Introduction to Parallel Computing

22

22

Recursive decomposition
Problem solvable by divide-and-conquer:
Decompose into sub-problems.
Do it recursively.

Combine the sub-solutions.

Do it recursively.

Concurrency: The sub-problems are solved in parallel.

Introduction to Parallel Computing

23

Small problem is to start and finish: with one process only.

23

Quicksort example

<5
<3

<9
<7

<10
<11

Introduction to Parallel Computing

24

Recall on the quicksort algorithm:

Choose a pivot.
Partition the array.
Recursive call.
Combine result: nothing to do.

24

Minimal number
4 9 1 7 8 11 2 12

Introduction to Parallel Computing

25

25

Data decomposition
2 steps:
Partition the data.
Induce partition into tasks.
How to partition data?
Partition output data:
Independent sub-outputs.
Partition input data:
Local computations, followed by combination.
1-D, 2-D, 3-D block decomposition.

Introduction to Parallel Computing

26

Partitioning of input data is a bit similar to divide-and-conquer.

26

Matrix multiplication by block

Introduction to Parallel Computing

27

We can partition further for the tasks. Notice the dependency between tasks.
What is the task dependency graph?

27

Intermediate data partitioning

Linear combination
of the intermediate
results.
Introduction to Parallel Computing

28

28

Owner-compute rule
Process assigned to some data
is responsible for all computations associated with it.
Input data decomposition:
All computations done on the (partitioned) input data
are done by the process.
Output data decomposition:
All computations for the (partitioned) output data are
done by the process.

Introduction to Parallel Computing

29

Important rule, very useful, in particular stresses locality.

29

Exploratory decomposition
Model-checker example
model
(syntax)

states
(semantics)

Introduction to Parallel Computing

30

Suitable for search algorithms. Partition the search space into smaller parts
and search in parallel. We search the solution by a tree search technique.

30

Performance anomalies
Work depends on the order of the search!

Introduction to Parallel Computing

31

31

Speculative decomposition
Dependencies between tasks are not known a-priori.
How to identify independent tasks?
Conservative approach: identify tasks that are
guaranteed to be independent.
Optimistic approach: schedule tasks even if we are
not sure may roll-back later.

Introduction to Parallel Computing

32

Not possible to identify independent tasks in advance. Conservative

approaches may yield limited concurrency. Optimistic approach = speculative.
Optimistic approach is similar to branch prediction algorithms in processors.

32

So far
Decomposition techniques.
Identify tasks.
Analyze with task dependency & interaction graphs.
Map tasks to processes.
Now properties of tasks that affect a good mapping.
Task generation, size of tasks, and size of data.

Introduction to Parallel Computing

33

33

Task generation
Static task generation.
Tasks are known beforehand.
Apply to well-structured problems.
Dynamic task generation.
Tasks generated on-the-fly.
Tasks & task dependency graph not available
beforehand.

Introduction to Parallel Computing

34

The well-structured problem can typically be decomposed using data or

recursive decomposition techniques.
Dynamic tasks generation: Exploratory or speculative decomposition
techniques are generally used, but not always. Example: quicksort.

34

Task sizes
Relative amount of time for completion.
Uniform same size for all tasks.
Matrix multiplication.

Non-uniform.
Optimization & search problems.

Introduction to Parallel Computing

35

Typically the size of non-uniform tasks is difficult to evaluate beforehand.

35

Size of data associated with tasks

Important because of locality reasons.
Different types of data with different sizes
Input/output/intermediate data.
Size of context cheap or expensive communication with
other tasks.

Introduction to Parallel Computing

36

36

Characteristics of task interactions

Static interactions.
Tasks and interactions known beforehand.
And interaction at pre-determined times.
Dynamic interactions.
Timing of interaction unknown.
Or set of tasks not known in advance.
Regular interactions.
The interaction graph follows a pattern.
Irregular interactions.
No pattern.
Introduction to Parallel Computing

37

Static vs. dynamic.

Static or dynamic interaction pattern.
Dynamic harder to code, more difficult for MPI.

37

Example: Image Dithering

Introduction to Parallel Computing

38

The color of each pixel is determined as the weighted average of its original
value and the values of the neighboring pixels. Decompose into regions, 1
task/region. Pattern is a 2-D mesh. Regular pattern.

38

Characteristics of task interactions

Data sharing interactions:
Read-only interactions.
Read only data associated with other tasks.

Read-write interactions.
Read & modify data of other tasks.

Introduction to Parallel Computing

39

Read-only vs. read-write.

Read-only example: matrix multiplication (share input). Read-write example:
15-puzzle with shared priority list of states to be explored; Priority given by
some heuristic to evaluate the distance to the goal.

39

Characteristics of task interactions

One-way interactions.
Only one task initiates and completes the
communication without interrupting the other one.
Two-way interactions.
Producer consumer model.

Introduction to Parallel Computing

40

One-way vs. two-way.

One-way more difficult with MPI since MPI has an explicit send & receive set
of calls. Conversion one-way to two-way with polling or another thread waiting
for communication.

40

Mapping techniques for load balancing

Map tasks onto processes.
Goal: minimize overheads.
Communication.
Idling.
Uneven load distribution may cause idling.
Constraints from task dependency wait for other
tasks.

Introduction to Parallel Computing

41

Minimizing communication may contradict minimizing idling. Put tasks that

communicate with each other on the same process but may unbalance the
load -> distribute them but increase communication.
Load balancing is not enough to minimize idling.

41

Example

Introduction to Parallel Computing

42

Global balancing OK but due to task dependency P4 is idling.

42

Mapping techniques
Static mapping.
NP-complete problem for non-uniform tasks.
Large data compared to computation.
Dynamic mapping.
Dynamically generated tasks.
Task size unknown.

Introduction to Parallel Computing

43

Even static mapping may be difficult: The problem of obtaining an optimal

mapping is an NP-complete problem for non-uniform tasks. In practice simple
heuristics provide good mappings.
Cost of moving data may out-weight the advantages of dynamic mapping.
In shared address space dynamic mapping may work well even with large
data, but be careful with the underlying architecture (NUMA/UMA) because
data may be moved physically.

43

Schemes for static mapping

Mappings based on data partitioning.
Mappings based on task graph partitioning.
Hybrid mappings.

Introduction to Parallel Computing

44

44

Array distribution scheme

Combine with owner computes rule to partition into subtasks.

1-D block distribution scheme.

Introduction to Parallel Computing

45

Data partitioning mapping.

Mapping data = mapping tasks.
Simple block-distribution.

45

Block distribution cont.

Generalize to higher dimensions: 4x4, 2x8.

Introduction to Parallel Computing

46

46

Example: Matrix*Matrix

Partition output of C=A*B.

Each entry needs the same amount of computation.
Blocks on 1 or 2 dimensions.
Different data sharing patterns.
Higher dimensional distributions
means we can use more processes.
sometimes reduces interaction.

Introduction to Parallel Computing

47

In the case of matrix n*n multiplication, 1-D -> n processes at most, 2-D n2
processes at most.

47

Introduction to Parallel Computing

48

O(n2/sqrt(p)) vs. O(n2) shared data.

48

Imbalance problem
If the amount of computation associated with data varies
a lot then block decomposition leads to imbalances.
Example: LU factorization (or Gaussian elimination).

Computations

Introduction to Parallel Computing

49

Exercise on LU-decomposition.

49

LU factorization
Non singular square matrix A (invertible).
A = L*U.
Useful for solving linear equations.

Introduction to Parallel Computing

50

50

LU factorization

In practice we work on A.

N steps

Introduction to Parallel Computing

51

51

LU algorithm
Proc LU(A)
begin
U[k,k]
for k := 1 to n-1 do
for j := k+1 to n do
Normalize L
A[j,k] := A[j,k]/A[k,k]
U[k,j] := A[k,j]/L[k,k]
endfor
L[j,k]
for j := k+1 to n do
A
for i := k+1 to n do
A[i,j] := A[i,j] A[i,k]*A[k,j]
U
endfor
endfor
L
L[i,k] U[k,j]
endfor
Introduction to Parallel Computing
end
52

52

Decomposition

Exercise:
Task dependency graph?
Mapping to 3 & 4 processes?

Introduction to Parallel Computing

53

Load imbalance for individual tasks. Load imbalance from dependencies.

53

Cyclic and block-cyclic distributions

Idea:
Partition an array into many more blocks than
available processes.
Assign partitions (tasks) to processes in a round-robin
manner.
each process gets several non adjacent blocks.

Introduction to Parallel Computing

55

55

Block-Cyclic Distributions

a) Partition 16x16 into 2*4 groups of 2 rows.

p groups of n/p rows.
b) Partition 16x16 into square blocks of size
4*4 distributed on 2*2 processes.
2p groups of n/2p squares.
Introduction to Parallel Computing

56

Reduce the amount of idling because all processes have a sampling of tasks
from all parts of the matrix.
But lack of locality may result in performance penalties + leads to high degree
of interaction. Good value for to find a compromise.

56

Randomized distributions

Irregular distribution with regular mapping!

Not good.
Introduction to Parallel Computing

57

57

1-D randomized distribution

Permutation

Introduction to Parallel Computing

58

58

2-D randomized distribution

2-D block random distribution.

Block mapping.
Introduction to Parallel Computing

59

59

Graph partitioning
For sparse data structures and data dependent
interaction patterns.
Numerical simulations. Discretize the problem and
represent it as a mesh.
Sparse matrix: assign equal number of nodes to
processes & minimize interaction.
Example: simulation of dispersion of a water contaminant
in Lake Superior.

Introduction to Parallel Computing

60

60

Discretization

Introduction to Parallel Computing

61

61

Partitioning Lake Superior

Random partitioning.

Partitioning with minimum

edge cut.
Finding an exact optimal partitioning
is an NP-complete problem.
Introduction to Parallel Computing

62

Minimum edge cut from a graph point of view. Keep locality of data with
processes to minimize interaction.

62

Mappings based on task partitioning

Partition the task dependency graph.
Good when static task dependency graph with known
task sizes.

Mapping on 8
processes.

Introduction to Parallel Computing

63

Determining an optimal mapping is NP-complete. Good heuristics for

structured graphs.
Binary tree task dependency graph: occurs in recursive decompositions as
seen before. The mapping minimizes interaction. There is idling but it is
inherent to the task dependency graph, we do not add more.
This example good on a hypercube. See why?

63

Hierarchical mappings
Combine several mapping techniques in a structured
(hierarchical) way.
Task mapping of a binary tree (quicksort) does not use
all processors.
Mapping based on task dependency graph (hierarchy)
& block.

Introduction to Parallel Computing

64

64

Schemes for dynamic mapping

Centralized Schemes.
Master manages pool of tasks.
Slaves obtain work.
Limited scalability.
Distributed Schemes.
Processes exchange tasks to balance work.
Not simple, many issues.

Introduction to Parallel Computing

65

Centralized schemes are easy to implement but present an obvious bottleneck

(the master).
Self-scheduling: slaves pick up work to do whenever they are idle.
Bottleneck: tasks of size M, it takes t to assign work to a slave at most M/t
processes can be kept busy.
Chunk-scheduling: a way to reduce bottlenecks by getting a group of tasks.
Problem for load imbalances.
Distributed schemes more difficult to implement.
How do you choose sender & receiver? i.e. if A is overloaded, which process
gets something?
Initiate transfer by sender or receiver? i.e. A overloaded sends work or B idle
requests work?
How much work to transfer?
When to transfer?
Answers are application specific.

65

Minimizing interaction overheads

Maximize data locality.
Minimize volume of data-exchange.
Minimize frequency of interactions.
Minimize contention and hot spots.
Share a link, same memory block, etc
Re-design original algorithm to change the interaction
pattern.

Introduction to Parallel Computing

66

Minimize volume of exchange maximize temporal locality. Use higher

dimensional distributions, like in the matrix multiplication example. We can
store intermediate results and update global results less often.
Minimize frequency of interactions maximize spatial locality.
Related to the previously seen cost model for communications.
Changing the interaction pattern: For the matrix multiplication example, the
sum is commutative so we can re-order the operations modulo sqrt(p) to
remove contention.

66

Minimizing interaction overheads

Overlapping computations with interactions to reduce
idling.
Initiate interactions in advance.
Non-blocking communications.
Multi-threading.
Replicating data or computation.
Group communication instead of point to point.
Overlapping interactions.

Introduction to Parallel Computing

67

Replication is useful when the cost of interaction is greater than replicating the
computation. Replicating data is like caching, good for read-only accesses.
Processing power is cheap, memory access is expensive also apply at larger
scale with communicating processes.
Collective communication such as broadcast. However, depending on the
communication pattern, a custom collective communication may be better.

67