Data and Work Partitioning
Parallel & Distributed Computing
Decomposition (Partitioning)
• One of the fundamental steps that we need to undertake to solve a
• problem in parallel is to split the computations to be performed into a
set of tasks for concurrent execution defined by the task-dependency
graph.
• Decomposition Techniques are broadly classified as:
Decomposition (Partitioning)
cont…
1. Recursive decomposition
2. Data-decomposition
3. Exploratory decomposition
4. Speculative decomposition.
Decomposition (Partitioning)
cont…
• The recursive and data decomposition techniques are relatively
• general purpose as they can be used to decompose a wide variety of
problems.
• On the other hand, speculative and exploratory decomposition
techniques are more of a special purpose nature because they apply
to specific classes of problems.
Decomposition (Partitioning)
cont… Recursive
Decomposition
• Recursive decomposition is a method for inducing concurrency
in
problems that can be solved using the divide-and-conquer strategy.
• In this technique, a problem is solved by first dividing it into a set of
independent subproblems.
Decomposition (Partitioning)
cont… Recursive
Decomposition
• Each one of these subproblems is solved by recursively applying
a
similar division into smaller subproblems followed by a combination
of their results.
• The divide-and-conquer strategy results in natural concurrency,
as different subproblems can be solved concurrently.
Decomposition (Partitioning)
cont… Recursive Decomposition
(Quicksort)
• Consider the sorting a sequence A of n elements using
quicksort
algorithm.
• Quicksort is a divide and conquer algorithm that starts by selecting a
pivot element x and then partitions the sequence A into
two subsequences A0 and A1 such that all the elements in A0 are
smaller than x and all the elements in A1 are greater than or
Decomposition (Partitioning)
cont… Recursive Decomposition
(Quicksort)
• This partitioning step forms the divide step of the algorithm.
• Each one of the subsequences A0 and A1 is sorted by
recursively calling quicksort.
• Each one of these recursive calls further partitions the sequences.
• This is illustrated in following Figure for a sequence of 12 numbers.
Decomposition (Partitioning)
cont… Recursive Decomposition
(Quicksort)
Decomposition (Partitioning)
cont… Recursive Decomposition
(Quicksort)
• The recursion terminates when each subsequence contains only
a
single element.
• Then the results will be combined to form a sorted list.
Decomposition (Partitioning)
cont… Data Decomposition
• Data decomposition is a powerful and commonly used method
for
• deriving concurrency in algorithms that operate on large
data structures.
• In this method, the decomposition of computations is done in
two steps:
Decomposition (Partitioning)
cont… Data Decomposition
1. In the first step, the data on which the computations are performed
is partitioned, and
2. In the second step, this data partitioning is used to
induce a partitioning of the computations into tasks.
Decomposition (Partitioning)
cont… Data Decomposition
• The operations that these tasks perform on different data partitions
are usually similar.
• The partitioning of data can be performed in many possible ways.
• But we are going to discuss matrix-multiplication.
Decomposition (Partitioning)
cont… Data Decomposition (Matrix
Multiplication)
• To multiply a matrix by another matrix we need to do "dot product"
of rows and columns ... what does that mean?
• Let see with example: 1st Row X 1st Column:
• (1, 2, 3) • (7, 9, 11) = 1×7 + 2×9 + 3×11 = 58
Decomposition (Partitioning)
cont… Data Decomposition (Matrix
Multiplication)
• (1, 2, 3) • (7, 9, 11) = 1×7 + 2×9 + 3×11 = 58
• (1, 2, 3) • (8, 10, 12) = 1×8 + 2×10 + 3×12=
64
Decomposition (Partitioning)
cont… Data Decomposition (Matrix
Multiplication)
• 1st Row X 1st Column: (1, 2, 3) • (7, 9, 11) = 1×7 + 2×9 + 3×11 = 58
• 1st row X 2nd column: (1, 2, 3) • (8, 10, 12) = 1×8 + 2×10 + 3×12= 64
• 2nd row X 1st column: (4, 5, 6) • (7, 9, 11) = 4×7 + 5×9 + 6×11= 139
• 2nd row X 2nd column: (4, 5, 6) • (8, 10, 12) = 4×8 + 5×10 + 6×12= 154
Decomposition Techniques cont…
Exploratory Decomposition
• Exploratory decomposition is used to decompose problems whose
underlying computations correspond to a searching of a solution
from search space.
• In exploratory decomposition, we partition the search space into
smaller parts, and search each one of these parts concurrently, until
the desired solutions are found.
Decomposition Techniques cont…
Speculative Decomposition
• Speculative decomposition is used when a program may take one of
many possible computationally significant branches depending on
the output of other computations.
• In this situation, while one task is performing the computation
whose output is used in deciding the next computation, other
tasks can concurrently start the computations.
Decomposition Techniques cont…
Speculative Decomposition
• This scenario is similar to evaluating one or more of the branches of
a switch statement in C in parallel before the input for the switch
is available.
