Task decomposition and mapping
Alexandre David
Introduction to Parallel Computing
Overview
Introduction to parallel algorithms
Decomposition techniques
Task interactions
Load balancing
Introduction to Parallel Computing
Introduction
Parallel algorithms have the added dimension of
concurrency.
Typical tasks:
Identify concurrent works.
Map them to processors.
Distribute inputs, outputs, and other data.
Manage shared resources.
Synchronize the processors.
Introduction to Parallel Computing
There are other courses specifically on concurrency. We wont treat the
problems proper to concurrency such as deadlocks, livelocks, theory on
semaphores and synchronization. However, we will use them, and when
needed, apply techniques to avoid problems like deadlocks.
Decomposing problems
Decomposition into concurrent tasks.
No unique solution.
Different sizes.
Decomposition illustrated as a directed graph:
Nodes = tasks.
Edges = dependency.
Task dependency graph
Introduction to Parallel Computing
Many solutions are often possible but few will yield good performance and be
scalable. We have to consider the computational and storage resources
needed to solve the problems.
Size of the tasks in the sense of the amount of work to do. Can be more, less,
or unknown. Unknown in the case of a search algorithm is common.
Dependency: All the results from incoming edges are required for the tasks at
the current node.
We will not consider tools for automatic decomposition. They work fairly well
only for highly structured programs or options of programs.
Task dependency graph?
Vector
Example: Matrix * Vector
Matrix
N tasks, 1 task/row:
Introduction to Parallel Computing
Example: database query processing
MODEL = ``CIVIC'' AND YEAR = 2001 AND
(COLOR = ``GREEN'' OR COLOR = ``WHITE)
Introduction to Parallel Computing
The question is: How to decompose this into concurrent tasks? Different tasks
may generate intermediate results that will be used by other tasks.
A solution
Meas
ure o
Nb. o f concurr
f pro
ency?
c
Optim essors?
al?
Introduction to Parallel Computing
How much concurrency do we have here? How many processors to use? Is it
optimal?
Another Solution
Bet
ter
/wo
rse
?
Introduction to Parallel Computing
Is it better or worse? Why?
Granularity
Number and size of tasks.
Fine-grained: many small tasks.
Coarse-grained: few large tasks.
Related: degree of concurrency.
(Nb. of tasks executable in parallel).
Maximal degree of concurrency.
Average degree of concurrency.
Introduction to Parallel Computing
Previous matrix*vector fine-grained.
Database example coarse grained.
Degree of concurrency: Number of tasks that can be executed in parallel.
Average degree of concurrency is a more useful measure.
Assume that the tasks in the previous database examples have the same
granularity. Whats their average degrees of concurrency? 7/3=2.33 and
7/4=1.75.
Common sense: Increasing the granularity of decomposition and utilizing the
resulting concurrency to perform more tasks in parallel increases performance.
However, there is a limit to granularity due to the nature of the problem itself.
Vector
Coarser Matrix * Vector
Matrix
N tasks, 3 task/row:
Introduction to Parallel Computing
10
10
Granularity
Average degree of concurrency if we take into account
varying amount of work?
Critical path = longest directed path between any start &
finish nodes.
Critical path length = sum of the weights of nodes along
this path.
Average degree of concurrency = total amount of work /
critical path length.
Introduction to Parallel Computing
11
Weights on nodes denote the amount of work to be done on these nodes.
Longest path shortest time needed to execute in parallel.
11
Database example
Critical path (3).
Critical path length = 27.
Av. deg. of concurrency = 63/27.
Critical path (4).
Critical path length = 34.
Av. deg. of conc. = 64/34.
2.33
Introduction to Parallel Computing
1.88
12
12
Maximum degree of
concurrency.
Critical path length.
Maximum possible speedup.
Minimum number of
processes to reach this
speedup.
Maximum speedup if we
limit the processes to 2,4,
and 8.
Exercise
(a)
(b)
(c)
(d)
Introduction to Parallel Computing
13
13
Interaction between tasks
Tasks often share data.
Task interaction graph:
Nodes = tasks.
Edges = interaction.
Optional weights.
Task dependency graph is a sub-graph of the task
interaction graph.
Introduction to Parallel Computing
18
Another important factor is interaction between tasks on different processors.
Share data implies synchronization protocols (mutual exclusion, etc) to ensure
consistency.
Edges generally undirected. When directed edges are used, they show the
direction of the flow of data (and the flow is unidirectional).
Dependency between tasks implies interaction between them.
18
Processes and mapping
Tasks run on processors.
Process: processing agent executing the tasks. Not
exactly like in your OS course.
Mapping = assignment of tasks to processes.
API exposes processes and binding to processors not
always controlled.
Scheduling of threads is not controlled.
What makes a good mapping?
Introduction to Parallel Computing
19
Here we are not talking directly on the mapping to processors. A processor
can execute two processes.
Good mapping:
Maximize concurrency by mapping independent tasks to different processes.
Minimize interaction by mapping interacting tasks on the same process.
Can be conflicting, good trade-off is the key to performance.
Decomposition determines degree of concurrency.
Mapping determines how much concurrency is utilized and how efficiently.
19
Mapping example
Introduction to Parallel Computing
20
Notice that the mapping keeps one process from the previous stage because
of dependency: We can avoid interaction by keeping the same process.
20
Processes vs. processors
Processes = logical computing agent.
Processor = hardware computational unit.
In general 1-1 correspondence but this model gives
better abstraction.
Useful for hardware supporting multiple programming
paradigms.
Now remains the question:
How do you decompose?
Introduction to Parallel Computing
21
Example of hybrid hardware: cluster of MP machines. Each node has shared
memory and communicates with other nodes via MPI.
1. Decompose and map to processes for MPI.
2. Decompose again but suitable for shared memory.
21
Decomposition techniques
Recursive decomposition.
Divide-and-conquer.
Data decomposition.
Large data structure.
Exploratory decomposition.
Search algorithms.
Speculative decomposition.
Dependent choices in computations.
Introduction to Parallel Computing
22
22
Recursive decomposition
Problem solvable by divide-and-conquer:
Decompose into sub-problems.
Do it recursively.
Combine the sub-solutions.
Do it recursively.
Concurrency: The sub-problems are solved in parallel.
Introduction to Parallel Computing
23
Small problem is to start and finish: with one process only.
23
Quicksort example
<5
<3
<9
<7
<10
<11
Introduction to Parallel Computing
24
Recall on the quicksort algorithm:
Choose a pivot.
Partition the array.
Recursive call.
Combine result: nothing to do.
24
Minimal number
4 9 1 7 8 11 2 12
Introduction to Parallel Computing
25
25
Data decomposition
2 steps:
Partition the data.
Induce partition into tasks.
How to partition data?
Partition output data:
Independent sub-outputs.
Partition input data:
Local computations, followed by combination.
1-D, 2-D, 3-D block decomposition.
Introduction to Parallel Computing
26
Partitioning of input data is a bit similar to divide-and-conquer.
26
Matrix multiplication by block
Introduction to Parallel Computing
27
We can partition further for the tasks. Notice the dependency between tasks.
What is the task dependency graph?
27
Intermediate data partitioning
Linear combination
of the intermediate
results.
Introduction to Parallel Computing
28
28
Owner-compute rule
Process assigned to some data
is responsible for all computations associated with it.
Input data decomposition:
All computations done on the (partitioned) input data
are done by the process.
Output data decomposition:
All computations for the (partitioned) output data are
done by the process.
Introduction to Parallel Computing
29
Important rule, very useful, in particular stresses locality.
29
Exploratory decomposition
Model-checker example
model
(syntax)
states
(semantics)
Introduction to Parallel Computing
30
Suitable for search algorithms. Partition the search space into smaller parts
and search in parallel. We search the solution by a tree search technique.
30
Performance anomalies
Work depends on the order of the search!
Introduction to Parallel Computing
31
31
Speculative decomposition
Dependencies between tasks are not known a-priori.
How to identify independent tasks?
Conservative approach: identify tasks that are
guaranteed to be independent.
Optimistic approach: schedule tasks even if we are
not sure may roll-back later.
Introduction to Parallel Computing
32
Not possible to identify independent tasks in advance. Conservative
approaches may yield limited concurrency. Optimistic approach = speculative.
Optimistic approach is similar to branch prediction algorithms in processors.
32
So far
Decomposition techniques.
Identify tasks.
Analyze with task dependency & interaction graphs.
Map tasks to processes.
Now properties of tasks that affect a good mapping.
Task generation, size of tasks, and size of data.
Introduction to Parallel Computing
33
33
Task generation
Static task generation.
Tasks are known beforehand.
Apply to well-structured problems.
Dynamic task generation.
Tasks generated on-the-fly.
Tasks & task dependency graph not available
beforehand.
Introduction to Parallel Computing
34
The well-structured problem can typically be decomposed using data or
recursive decomposition techniques.
Dynamic tasks generation: Exploratory or speculative decomposition
techniques are generally used, but not always. Example: quicksort.
34
Task sizes
Relative amount of time for completion.
Uniform same size for all tasks.
Matrix multiplication.
Non-uniform.
Optimization & search problems.
Introduction to Parallel Computing
35
Typically the size of non-uniform tasks is difficult to evaluate beforehand.
35
Size of data associated with tasks
Important because of locality reasons.
Different types of data with different sizes
Input/output/intermediate data.
Size of context cheap or expensive communication with
other tasks.
Introduction to Parallel Computing
36
36
Characteristics of task interactions
Static interactions.
Tasks and interactions known beforehand.
And interaction at pre-determined times.
Dynamic interactions.
Timing of interaction unknown.
Or set of tasks not known in advance.
Regular interactions.
The interaction graph follows a pattern.
Irregular interactions.
No pattern.
Introduction to Parallel Computing
37
Static vs. dynamic.
Static or dynamic interaction pattern.
Dynamic harder to code, more difficult for MPI.
37
Example: Image Dithering
Introduction to Parallel Computing
38
The color of each pixel is determined as the weighted average of its original
value and the values of the neighboring pixels. Decompose into regions, 1
task/region. Pattern is a 2-D mesh. Regular pattern.
38
Characteristics of task interactions
Data sharing interactions:
Read-only interactions.
Read only data associated with other tasks.
Read-write interactions.
Read & modify data of other tasks.
Introduction to Parallel Computing
39
Read-only vs. read-write.
Read-only example: matrix multiplication (share input). Read-write example:
15-puzzle with shared priority list of states to be explored; Priority given by
some heuristic to evaluate the distance to the goal.
39
Characteristics of task interactions
One-way interactions.
Only one task initiates and completes the
communication without interrupting the other one.
Two-way interactions.
Producer consumer model.
Introduction to Parallel Computing
40
One-way vs. two-way.
One-way more difficult with MPI since MPI has an explicit send & receive set
of calls. Conversion one-way to two-way with polling or another thread waiting
for communication.
40
Mapping techniques for load balancing
Map tasks onto processes.
Goal: minimize overheads.
Communication.
Idling.
Uneven load distribution may cause idling.
Constraints from task dependency wait for other
tasks.
Introduction to Parallel Computing
41
Minimizing communication may contradict minimizing idling. Put tasks that
communicate with each other on the same process but may unbalance the
load -> distribute them but increase communication.
Load balancing is not enough to minimize idling.
41
Example
Introduction to Parallel Computing
42
Global balancing OK but due to task dependency P4 is idling.
42
Mapping techniques
Static mapping.
NP-complete problem for non-uniform tasks.
Large data compared to computation.
Dynamic mapping.
Dynamically generated tasks.
Task size unknown.
Introduction to Parallel Computing
43
Even static mapping may be difficult: The problem of obtaining an optimal
mapping is an NP-complete problem for non-uniform tasks. In practice simple
heuristics provide good mappings.
Cost of moving data may out-weight the advantages of dynamic mapping.
In shared address space dynamic mapping may work well even with large
data, but be careful with the underlying architecture (NUMA/UMA) because
data may be moved physically.
43
Schemes for static mapping
Mappings based on data partitioning.
Mappings based on task graph partitioning.
Hybrid mappings.
Introduction to Parallel Computing
44
44
Array distribution scheme
Combine with owner computes rule to partition into subtasks.
1-D block distribution scheme.
Introduction to Parallel Computing
45
Data partitioning mapping.
Mapping data = mapping tasks.
Simple block-distribution.
45
Block distribution cont.
Generalize to higher dimensions: 4x4, 2x8.
Introduction to Parallel Computing
46
46
Example: Matrix*Matrix
Partition output of C=A*B.
Each entry needs the same amount of computation.
Blocks on 1 or 2 dimensions.
Different data sharing patterns.
Higher dimensional distributions
means we can use more processes.
sometimes reduces interaction.
Introduction to Parallel Computing
47
In the case of matrix n*n multiplication, 1-D -> n processes at most, 2-D n2
processes at most.
47
Introduction to Parallel Computing
48
O(n2/sqrt(p)) vs. O(n2) shared data.
48
Imbalance problem
If the amount of computation associated with data varies
a lot then block decomposition leads to imbalances.
Example: LU factorization (or Gaussian elimination).
Computations
Introduction to Parallel Computing
49
Exercise on LU-decomposition.
49
LU factorization
Non singular square matrix A (invertible).
A = L*U.
Useful for solving linear equations.
Introduction to Parallel Computing
50
50
LU factorization
In practice we work on A.
N steps
Introduction to Parallel Computing
51
51
LU algorithm
Proc LU(A)
begin
U[k,k]
for k := 1 to n-1 do
for j := k+1 to n do
Normalize L
A[j,k] := A[j,k]/A[k,k]
U[k,j] := A[k,j]/L[k,k]
endfor
L[j,k]
for j := k+1 to n do
A
for i := k+1 to n do
A[i,j] := A[i,j] A[i,k]*A[k,j]
U
endfor
endfor
L
L[i,k] U[k,j]
endfor
Introduction to Parallel Computing
end
52
52
Decomposition
Exercise:
Task dependency graph?
Mapping to 3 & 4 processes?
Introduction to Parallel Computing
53
Load imbalance for individual tasks. Load imbalance from dependencies.
53
Cyclic and block-cyclic distributions
Idea:
Partition an array into many more blocks than
available processes.
Assign partitions (tasks) to processes in a round-robin
manner.
each process gets several non adjacent blocks.
Introduction to Parallel Computing
55
55
Block-Cyclic Distributions
a) Partition 16x16 into 2*4 groups of 2 rows.
p groups of n/p rows.
b) Partition 16x16 into square blocks of size
4*4 distributed on 2*2 processes.
2p groups of n/2p squares.
Introduction to Parallel Computing
56
Reduce the amount of idling because all processes have a sampling of tasks
from all parts of the matrix.
But lack of locality may result in performance penalties + leads to high degree
of interaction. Good value for to find a compromise.
56
Randomized distributions
Irregular distribution with regular mapping!
Not good.
Introduction to Parallel Computing
57
57
1-D randomized distribution
Permutation
Introduction to Parallel Computing
58
58
2-D randomized distribution
2-D block random distribution.
Block mapping.
Introduction to Parallel Computing
59
59
Graph partitioning
For sparse data structures and data dependent
interaction patterns.
Numerical simulations. Discretize the problem and
represent it as a mesh.
Sparse matrix: assign equal number of nodes to
processes & minimize interaction.
Example: simulation of dispersion of a water contaminant
in Lake Superior.
Introduction to Parallel Computing
60
60
Discretization
Introduction to Parallel Computing
61
61
Partitioning Lake Superior
Random partitioning.
Partitioning with minimum
edge cut.
Finding an exact optimal partitioning
is an NP-complete problem.
Introduction to Parallel Computing
62
Minimum edge cut from a graph point of view. Keep locality of data with
processes to minimize interaction.
62
Mappings based on task partitioning
Partition the task dependency graph.
Good when static task dependency graph with known
task sizes.
Mapping on 8
processes.
Introduction to Parallel Computing
63
Determining an optimal mapping is NP-complete. Good heuristics for
structured graphs.
Binary tree task dependency graph: occurs in recursive decompositions as
seen before. The mapping minimizes interaction. There is idling but it is
inherent to the task dependency graph, we do not add more.
This example good on a hypercube. See why?
63
Hierarchical mappings
Combine several mapping techniques in a structured
(hierarchical) way.
Task mapping of a binary tree (quicksort) does not use
all processors.
Mapping based on task dependency graph (hierarchy)
& block.
Introduction to Parallel Computing
64
64
Schemes for dynamic mapping
Centralized Schemes.
Master manages pool of tasks.
Slaves obtain work.
Limited scalability.
Distributed Schemes.
Processes exchange tasks to balance work.
Not simple, many issues.
Introduction to Parallel Computing
65
Centralized schemes are easy to implement but present an obvious bottleneck
(the master).
Self-scheduling: slaves pick up work to do whenever they are idle.
Bottleneck: tasks of size M, it takes t to assign work to a slave at most M/t
processes can be kept busy.
Chunk-scheduling: a way to reduce bottlenecks by getting a group of tasks.
Problem for load imbalances.
Distributed schemes more difficult to implement.
How do you choose sender & receiver? i.e. if A is overloaded, which process
gets something?
Initiate transfer by sender or receiver? i.e. A overloaded sends work or B idle
requests work?
How much work to transfer?
When to transfer?
Answers are application specific.
65
Minimizing interaction overheads
Maximize data locality.
Minimize volume of data-exchange.
Minimize frequency of interactions.
Minimize contention and hot spots.
Share a link, same memory block, etc
Re-design original algorithm to change the interaction
pattern.
Introduction to Parallel Computing
66
Minimize volume of exchange maximize temporal locality. Use higher
dimensional distributions, like in the matrix multiplication example. We can
store intermediate results and update global results less often.
Minimize frequency of interactions maximize spatial locality.
Related to the previously seen cost model for communications.
Changing the interaction pattern: For the matrix multiplication example, the
sum is commutative so we can re-order the operations modulo sqrt(p) to
remove contention.
66
Minimizing interaction overheads
Overlapping computations with interactions to reduce
idling.
Initiate interactions in advance.
Non-blocking communications.
Multi-threading.
Replicating data or computation.
Group communication instead of point to point.
Overlapping interactions.
Introduction to Parallel Computing
67
Replication is useful when the cost of interaction is greater than replicating the
computation. Replicating data is like caching, good for read-only accesses.
Processing power is cheap, memory access is expensive also apply at larger
scale with communicating processes.
Collective communication such as broadcast. However, depending on the
communication pattern, a custom collective communication may be better.
67
