Parallel Computing
Outlines - Introduction
Cost versus Performance
What is Parallel Computing?
The Scope of Parallel Computing
Issues in Parallel Computing
1
Cost versus Performance
Cost versus Performance curve and its evolution over the decades.
2000s
1990s
1980s
1970s
1960s
Performance
Cost
3
What is Parallel Computing?
Example: Library and workers to distribute books
1) Dividing a task among workers by assigning them a
set of books is an instance of task partitioning.
2) Passing books to each other is an example of
communication between subtasks.
2
The Scope of Parallel Computing
Applications such as weather prediction and
pollution monitoring.
Satellites collect billions of bits per seconds
of data relating to pollution level and the
thickness of ozone layer.
Example: Weather Modeling & Forecasting
Assumptions:
1) Modeling of weather over an area of 3000 x 3000 miles.
2) Area is being modeled up to a height of 11 miles.
3) 3000 x 3000 x 11 cubic mile domain is partitioned into
segments of size 0.1 x 0.1 x 0.1 cubic miles which is
approximately 1011 different segments.
4) Modeling the weather over a two-day period and the
parameters need to be computed once every half hour.
3000 x 3000 x 11 = 99000000 cubic miles
99000000 / (0.1 x 0.1 x 0.1) = 99000000000 ≈ 1011 Segments
6
3
Assumptions: Weather … Continues
5) The computation of parameters inside a segment
uses initial values and values from neighboring
segments.
Assume that this computation takes 100 instructions,
then a single updating of parameters in entire domain
requires
1011 segments x 100 instructions = 1013 instructions
Since this has to be done approximately 100 times
(two days every half hour, that is 96), then total
number of operations (instructions) is 1015.
24 Hours x 2 x 2 days = 96 ≈ 102 times
7
1013 instructions x 102 times= 1015 instructions
Example: Weather Modeling … (Continues)
On serial supercomputer capable of performing
one billion instructions per second, this weather
modeling would take approximately 280 hours
that is: 1000000000 instructions per second = 109 instruction per second
1015 instructions / 109 = 10(15-9) = 106 seconds
106 seconds / 60 second = 16,666.7 minutes
16,666.7 minutes / 60 minute = 280 hours
280 hour / 24 hour = 11.67 days
4
Example: Weather Modeling … (Continues)
Taking 280 hours (11.67 days) to predict weather
for next 48 hours (2 days) is unreasonable.
Parallel processing makes it possible to predict
weather not only faster but also more accurate.
Example: Weather Modeling … (Continues)
If we have parallel computer with 1000 workstation class
processors, then we can partition 1011 segments of
domain among these processors.
Each processor computes parameters for 108 segments
1011 segments / 103 processors = 108 segments for each processor
Assuming that the computing power of this computer is
100 million instructions per second, the problem can be
solved in less than 3 hours:
100000000 instructions per second = 108 is the power of each processor
108 segments x 100 (instructions per segment) = 108 x 102 =10 (8+2) = 1010
instructions for each processor
1010 instructions x 100 (2 days) = 10(10+2) = 1012 instructions
1012 instructions / 108 instruction per second = 10(12-8) = 104 seconds
104 seconds / 3600 (60 second x 60 minute) sec. ≈ 2.7 hours.
So, the whole process takes 2.7 hours because processors are working in parallel
10
10
5
Issues in Parallel Computing
1) Design of Parallel Computers:
Large number of processors.
Supporting fast communication.
Supporting data sharing.
2) Design of Efficient Algorithms:
Issues in designing parallel algorithms are very
different from those in designing their sequential
computers
Partition: decompose task into several parallel tasks
Load Balancing: distribute load on processors as evenly as possible
Communication: How processors can communicate efficiently to
perform the whole parallel task
11
11
Issues in Parallel … (Continues)
3) Methods for Evaluating Parallel Algorithms:
Given a parallel computer and a parallel algorithm
running on it, we need to evaluate the performance of
the resulting system.
Performance analysis allows us to answer questions
such as:
a) How fast can a problem be solved using parallel processing?
b) How efficient are the processors used?
12
12
6
Issues in Parallel … (Continues)
4) Parallel algorithms are implemented on parallel
computers using a programming language.
Examples: Pthreads, High Performance Fortran (HPF)
5) Parallel Programming Tools:
To facilitate programming of parallel computers.
Examples: MPI and PVM (using Fortran, C/C++, C#,
and Java).
13
13
Issues in Parallel … (Continues)
6) Portable Parallel Programs
In a sense that the parallel program can be executed
under different operating systems and different
architectures
7) Automatic Programming of Parallel Computers:
Parallel compilers are expected to allow us to program a
parallel computer like a serial computer.
14
14
