Parallel and Distributed Computing (CS-3216)

Slides 07 (Massage Passing Paradigm - MPI)

Department of CS, GCU, Lahore

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 1 / 44
Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 2 / 44
 Principles of Message Passing Programming

Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 3 / 44
 Principles of Message Passing Programming

Basics

The logical view of a machine supporting the message-passing

paradigm consists of p processes, each with its own exclusive address
space.
Each data element must belong to one of the partitions of the space;
hence, data must be explicitly partitioned and placed.
All interactions (read-only or read/write) require cooperation of two
processes – the process that has the data and the process that wants
to access the data.
These two constraints, while onerous, make underlying costs very
explicit to the programmer.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 4 / 44
 Principles of Message Passing Programming

Basics

The logical view of a machine supporting the message-passing

paradigm consists of p processes, each with its own exclusive address
space.
Each data element must belong to one of the partitions of the space;
hence, data must be explicitly partitioned and placed.
All interactions (read-only or read/write) require cooperation of two
processes – the process that has the data and the process that wants
to access the data.
These two constraints, while onerous, make underlying costs very
explicit to the programmer.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 4 / 44
 Principles of Message Passing Programming

Basics

Message-passing programs are often written using the asynchronous

or loosely synchronous paradigms.
In the asynchronous paradigm, all concurrent tasks execute
asynchronously.
In the loosely synchronous model, tasks or subsets of tasks
synchronize to perform interactions. Between these interactions, tasks
execute completely asynchronously.
Most message-passing programs are written using the single program
multiple data (SPMD) model.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 5 / 44
 Principles of Message Passing Programming

Basics

Message-passing programs are often written using the asynchronous

or loosely synchronous paradigms.
In the asynchronous paradigm, all concurrent tasks execute
asynchronously.
In the loosely synchronous model, tasks or subsets of tasks
synchronize to perform interactions. Between these interactions, tasks
execute completely asynchronously.
Most message-passing programs are written using the single program
multiple data (SPMD) model.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 5 / 44
 Principles of Message Passing Programming

Basics

MPI defines a standard library for message-passing that can be used

to develop portable message-passing programs.
The MPI standard defines both the syntax as well as the semantics of
a core set of library routines.
It is possible to write fully-functional message-passing programs by
using only the six routines.
Various implementations of MPI standard are available which includes
MPICH, OpenMPI (C, C++, Fortran)
MPI4PY (Python)
OpenMPI, MpjExpress (Java)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 6 / 44
 Principles of Message Passing Programming

Basics

MPI defines a standard library for message-passing that can be used

to develop portable message-passing programs.
The MPI standard defines both the syntax as well as the semantics of
a core set of library routines.
It is possible to write fully-functional message-passing programs by
using only the six routines.
Various implementations of MPI standard are available which includes
MPICH, OpenMPI (C, C++, Fortran)
MPI4PY (Python)
OpenMPI, MpjExpress (Java)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 6 / 44
 Basic Routines of MPI

Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 7 / 44
 Basic Routines of MPI

Basic Routines of MPI

The minimal set of MPI routines.

MPI_Init Initializes MPI.

MPI_Finalize Terminates MPI.
MPI_Comm_size Determines the number of processes.
MPI_Comm_rank Determines the label of the calling process.
MPI_Send Sends a message.
MPI_Recv Receives a message.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 8 / 44
 Basic Routines of MPI

MPI Basics
Starting and Terminating the MPI Library

MPI_Init is called prior to any calls to other MPI routines. Its

purpose is to initialize the MPI environment.
MPI_Finalize is called at the end of the computation, and it
performs various clean-up tasks to terminate the MPI environment.
The prototypes of these two functions are:
int MPI_Init(int *argc, char ***argv)
int MPI_Finalize()
MPI_Init also strips off any MPI related command-line arguments.
All MPI routines, data-types, and constants are prefixed by “MPI_”.
The return code for successful completion is MPI_SUCCESS.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 9 / 44
 Basic Routines of MPI

MPI Basics
Starting and Terminating the MPI Library

MPI_Init is called prior to any calls to other MPI routines. Its

purpose is to initialize the MPI environment.
MPI_Finalize is called at the end of the computation, and it
performs various clean-up tasks to terminate the MPI environment.
The prototypes of these two functions are:
int MPI_Init(int *argc, char ***argv)
int MPI_Finalize()
MPI_Init also strips off any MPI related command-line arguments.
All MPI routines, data-types, and constants are prefixed by “MPI_”.
The return code for successful completion is MPI_SUCCESS.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 9 / 44
 Basic Routines of MPI

MPI Basics
Starting and Terminating the MPI Library

MPI_Init is called prior to any calls to other MPI routines. Its

purpose is to initialize the MPI environment.
MPI_Finalize is called at the end of the computation, and it
performs various clean-up tasks to terminate the MPI environment.
The prototypes of these two functions are:
int MPI_Init(int *argc, char ***argv)
int MPI_Finalize()
MPI_Init also strips off any MPI related command-line arguments.
All MPI routines, data-types, and constants are prefixed by “MPI_”.
The return code for successful completion is MPI_SUCCESS.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 9 / 44
 Basic Routines of MPI

MPI Basics
Communicators

A communicator defines a communication domain – a set of processes

that are allowed to communicate with each other.
Information about communication domains is stored in variables of
type MPI_Comm.
Communicators are used as arguments to all message transfer MPI
routines.
A process can belong to many different (possibly overlapping)
communication domains.
MPI defines a default communicator called MPI_COMM_WORLD which
includes all the processes.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 10 / 44
 Basic Routines of MPI

MPI Basics
Querying Information

The MPI_Comm_size and MPI_Comm_rank functions are used to

determine the number of processes and the label of the calling
process, respectively.
The calling sequences of these routines are as follows:
int MPI_Comm_size(MPI_Comm comm, int *size)
int MPI_Comm_rank(MPI_Comm comm, int *rank)
The rank of a process is an integer that ranges from zero up to the
size of the communicator minus one.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 11 / 44
 Basic Routines of MPI

MPI Basics
Our First MPI Program

#include <mpi.h>
#include <iostream>
using namespace std;

int main(int argc, char *argv[])

{
int npes, myrank;

MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &npes);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
cout << "From process" << myrank << " out of " << npes <<
", Hello World!\n";
MPI_Finalize();
return 0;
}

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 12 / 44
 The Building Blocks: Send and Receive Operations

Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 13 / 44
 The Building Blocks: Send and Receive Operations

Sending and Receiving the Messages

The prototypes of these operations are as follows:

send(void *sendbuf, int nelems, int dest)
receive(void *recvbuf, int nelems, int source)
Consider the following code segments:
P0 P1

a = 100; receive(&a, 1, 0)
send(&a, 1, 1); printf("%d\n", a);
a = 0;
The semantics of the send operation require that the value received
by process P1 must be 100 as opposed to 0.
This motivates the design of the send and receive protocols.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 14 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Non-Buffered Blocking Message Passing Operations

A simple method for forcing send/receive semantics is for the send

operation to return only when it is safe to do so.
In the non-buffered blocking send, the operation does not return until
the matching receive has been encountered at the receiving process.
Idling and deadlocks are major issues with non-buffered blocking
sends.
In buffered blocking sends, the sender simply copies the data into the
designated buffer and returns after the copy operation has been
completed. The data is copied at a buffer at the receiving end as well.
Buffering alleviates idling at the expense of copying overheads.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 15 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Non-Buffered Blocking Message Passing Operations

sending receiving sending receiving sending receiving

process process process process process process

send request to send

request to send request to send

okay to send receive send okay to send receive send receive
okay to send

data data data

(a) Sender comes first; (b) Sender and receiver come (c) Receiver comes first;
idling at sender at about the same time; idling at receiver
idling minimized

Figure: Handshake for a blocking non-buffered send/receive operation. It is easy to see that in
cases where sender and receiver do not reach communication point at similar times, there can
be considerable idling overheads.
Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 16 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Buffered Blocking Message Passing Operations

A simple solution to the idling and deadlocking problem outlined

above is to rely on buffers at the sending and receiving ends.
The sender simply copies the data into the designated buffer and
returns after the copy operation has been completed.
The data must be buffered at the receiving end as well.
Buffering trades off idling overhead for buffer copying overhead.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 17 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Buffered Blocking Message Passing Operations

sending receiving sending receiving

process process process process

send send

data Data copied to

data buffer at receiver

receive

receive

Figure: Blocking buffered transfer protocols: (a) in the presence of communication hardware
with buffers at send and receive ends; and (b) in the absence of communication hardware,
sender interrupts receiver and deposits data in buffer at receiver end.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 18 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Buffered Blocking Message Passing Operations

Bounded buffer sizes can have significant impact on performance.

P0 P1

for (i = 0; i < 1000; i++) { for (i = 0; i < 1000; i++) {

produce_data(&a); receive(&a, 1, 0);
send(&a, 1, 1); consume_data(&a);
} }

What if consumer was much slower than producer?

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 19 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Buffered Blocking Message Passing Operations

Deadlocks are still possible with buffering since receive operations block.
P0 P1

receive(&a, 1, 1); receive(&a, 1, 0);

send(&b, 1, 1); send(&b, 1, 0);

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 20 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Non-Blocking Message Passing Operations

The programmer must ensure semantics of the send and receive.

This class of non-blocking protocols returns from the send or receive
operation before it is semantically safe to do so.
Non-blocking operations are generally accompanied by a
check-status operation.
When used correctly, these primitives are capable of overlapping
communication overheads with useful computations.
Message passing libraries typically provide both blocking and
non-blocking primitives.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 21 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Non-Blocking Message Passing Operations

sending receiving sending receiving

process process process process

send request to send send request to send

Unsafe to Unsafe to
update okay to send update okay to send
receive receive
data being data being
sent sent Unsafe to read
data data data being received

(a) Without hardware support (b) With hardware support

Figure: Non-blocking non-buffered send and receive operations (a) in absence of

communication hardware; (b) in presence of communication hardware.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 22 / 44
 The Building Blocks: Send and Receive Operations

Send and Receive Operations

Non-Blocking Message Passing Operations

Blocking Operations Non−Blocking Operations

Sending process
Sending process
returns after initiating
returns after data
DMA transfer to
Buffered has been copied
buffer. This operation
into communication
may not be
buffer
completed on return

Sending process
blocks until
Non−Buffered
matching receive
operation has been
encountered

Send and Receive Programmer must

semantics assured by explicitly ensure
corresponding operation semantics by polling
to verify completion

Figure: Space of possible protocols for send and receive operations.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 23 / 44
 MPI Routines (Cont.)

Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 24 / 44
 MPI Routines (Cont.)

Sending and Receiving Messages

The basic functions for sending and receiving messages in MPI are the
MPI_Send and MPI_Recv, respectively.
The calling sequences of these routines are as follows:
int MPI_Send(void *buf, int count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm)
int MPI_Recv(void *buf, int count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm, MPI_Status *status)
MPI provides equivalent datatypes for all C datatypes. This is done
for portability reasons.
The datatype MPI_BYTE corresponds to a byte (8 bits) and
MPI_PACKED corresponds to a collection of data items that has been
created by packing non-contiguous data.
The message-tag can take values ranging from zero up to the MPI
defined constant MPI_TAG_UB.
Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 25 / 44
 MPI Routines (Cont.)

Sending and Receiving Messages

The basic functions for sending and receiving messages in MPI are the
MPI_Send and MPI_Recv, respectively.
The calling sequences of these routines are as follows:
int MPI_Send(void *buf, int count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm)
int MPI_Recv(void *buf, int count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm, MPI_Status *status)
MPI provides equivalent datatypes for all C datatypes. This is done
for portability reasons.
The datatype MPI_BYTE corresponds to a byte (8 bits) and
MPI_PACKED corresponds to a collection of data items that has been
created by packing non-contiguous data.
The message-tag can take values ranging from zero up to the MPI
defined constant MPI_TAG_UB.
Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 25 / 44
 MPI Routines (Cont.)

MPI Datatypes

MPI Datatype C Datatype

MPI_CHAR signed char
MPI_SHORT signed short int
MPI_INT signed int
MPI_LONG signed long int
MPI_UNSIGNED_CHAR unsigned char
MPI_UNSIGNED_SHORT unsigned short int
MPI_UNSIGNED unsigned int
MPI_UNSIGNED_LONG unsigned long int
MPI_FLOAT float
MPI_DOUBLE double
MPI_LONG_DOUBLE long double
MPI_BYTE
MPI_PACKED

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 26 / 44
 MPI Routines (Cont.)

Sending and Receiving Messages

MPI allows specification of wildcard arguments for both source and

tag.
If source is set to MPI_ANY_SOURCE, then any process of the
communication domain can be the source of the message.
If tag is set to MPI_ANY_TAG, then messages with any tag are
accepted.
On the receive side, the message must be of length equal to or less
than the length field specified.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 27 / 44
 MPI Routines (Cont.)

Sending and Receiving Messages

On the receiving end, the status variable can be used to get

information about the MPI_Recv operation.
The corresponding data structure contains:
typedef struct MPI_Status {
int MPI_SOURCE;
int MPI_TAG;
int MPI_ERROR;
};
The MPI_Get_count function returns the precise count of data items
received.
int MPI_Get_count(MPI_Status *status, MPI_Datatype datatype,
int *count)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 28 / 44
 MPI Routines (Cont.)

Sending and Receiving Messages

Avoiding Deadlocks

Consider:
int a[10], b[10], myrank;
MPI_Status status;
...
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
if (myrank == 0) {
MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD);
MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD);
}
else if (myrank == 1) {
MPI_Recv(b, 10, MPI_INT, 0, 2, MPI_COMM_WORLD);
MPI_Recv(a, 10, MPI_INT, 0, 1, MPI_COMM_WORLD);
}
...

If MPI_Send is blocking, there is a deadlock.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 29 / 44
 MPI Routines (Cont.)

Sending and Receiving Messages

Avoiding Deadlocks

To exchange messages, MPI provides the following function:

int MPI_Sendrecv(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, int dest, int sendtag,
void *recvbuf, int recvcount, MPI_Datatype recvdatatype,
int source, int recvtag, MPI_Comm comm,
MPI_Status *status)

The arguments include arguments to the send and receive functions. If we

wish to use the same buffer for both send and receive, we can use:
int MPI_Sendrecv_replace(void *buf, int count,
MPI_Datatype datatype, int dest, int sendtag,
int source, int recvtag, MPI_Comm comm,
MPI_Status *status)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 30 / 44
 Collective Communication and Computation Operations

Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 31 / 44
 Collective Communication and Computation Operations

Collective Communication and Computation Operations

MPI provides an extensive set of functions for performing common

collective communication operations.
Each of these operations is defined over a group corresponding to the
communicator.
All processors in a communicator must call these operations.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 32 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The barrier synchronization operation is performed in MPI using:

int MPI_Barrier(MPI_Comm comm)
The one-to-all broadcast operation is:
int MPI_Bcast(void *buf, int count, MPI_Datatype datatype,
int source, MPI_Comm comm)
The all-to-one reduction operation is:
int MPI_Reduce(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, int target,
MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 33 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The barrier synchronization operation is performed in MPI using:

int MPI_Barrier(MPI_Comm comm)
The one-to-all broadcast operation is:
int MPI_Bcast(void *buf, int count, MPI_Datatype datatype,
int source, MPI_Comm comm)
The all-to-one reduction operation is:
int MPI_Reduce(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, int target,
MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 33 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The barrier synchronization operation is performed in MPI using:

int MPI_Barrier(MPI_Comm comm)
The one-to-all broadcast operation is:
int MPI_Bcast(void *buf, int count, MPI_Datatype datatype,
int source, MPI_Comm comm)
The all-to-one reduction operation is:
int MPI_Reduce(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, int target,
MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 33 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Predefined Reduction Operations

Operation Meaning Datatypes

MPI_MAX Maximum C integers and floating point
MPI_MIN Minimum C integers and floating point
MPI_SUM Sum C integers and floating point
MPI_PROD Product C integers and floating point
MPI_LAND Logical AND C integers
MPI_BAND Bit-wise AND C integers and byte
MPI_LOR Logical OR C integers
MPI_BOR Bit-wise OR C integers and byte
MPI_LXOR Logical XOR C integers
MPI_BXOR Bit-wise XOR C integers and byte
MPI_MAXLOC max-min value-location Data-pairs
MPI_MINLOC min-min value-location Data-pairs

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 34 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

If the result of the reduction operation is needed by all processes, MPI

provides:
int MPI_Allreduce(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
To compute prefix-sums, MPI provides:
int MPI_Scan(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 35 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

If the result of the reduction operation is needed by all processes, MPI

provides:
int MPI_Allreduce(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
To compute prefix-sums, MPI provides:
int MPI_Scan(void *sendbuf, void *recvbuf, int count,
MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 35 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The gather operation is performed in MPI using:

int MPI_Gather(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, int target, MPI_Comm comm)
MPI also provides the MPI_Allgather function in which the data are
gathered at all the processes.
int MPI_Allgather(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, MPI_Comm comm)
The corresponding scatter operation is:
int MPI_Scatter(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, int source, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 36 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The gather operation is performed in MPI using:

int MPI_Gather(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, int target, MPI_Comm comm)
MPI also provides the MPI_Allgather function in which the data are
gathered at all the processes.
int MPI_Allgather(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, MPI_Comm comm)
The corresponding scatter operation is:
int MPI_Scatter(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, int source, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 36 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The gather operation is performed in MPI using:

int MPI_Gather(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, int target, MPI_Comm comm)
MPI also provides the MPI_Allgather function in which the data are
gathered at all the processes.
int MPI_Allgather(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, MPI_Comm comm)
The corresponding scatter operation is:
int MPI_Scatter(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, int source, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 36 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The all-to-all personalized communication operation is performed by:

int MPI_Alltoall(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, MPI_Comm comm)
Using this core set of collective operations, a number of programs can
be greatly simplified.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 37 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

The all-to-all personalized communication operation is performed by:

int MPI_Alltoall(void *sendbuf, int sendcount,
MPI_Datatype senddatatype, void *recvbuf, int recvcount,
MPI_Datatype recvdatatype, MPI_Comm comm)
Using this core set of collective operations, a number of programs can
be greatly simplified.

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 38 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Vector Variants

MPI also provides versions in which the size of the arrays can be different.
MPI refers to these operations as the vector variants.
The vector variants of the MPI_Gather and MPI_Allgather operations are
provided by the functions MPI_Gatherv and MPI_Allgatherv , respectively.
int MPI_Gatherv(const void *sendbuf, int sendcount,
MPI_Datatype sendtype, void *recvbuf, const int *recvcounts,
const int *displs, MPI_Datatype recvtype, int root,
MPI_Comm comm)
int MPI_Allgatherv(const void *sendbuf, int sendcount,
MPI_Datatype sendtype, void *recvbuf, const int *recvcounts,
const int *displs, MPI_Datatype recvtype, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 39 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Vector Variants

These functions allow a different number of data elements to be sent

by each process by replacing the recvcount parameter with the array
recvcounts .
The amount of data sent by process i is equal to recvcounts[i] .
Note that the size of recvcounts is equal to the size of the
communicator comm.
The array parameter displs , which is also of the same size, is used to
determine where in recvbuf the data sent by each process will be
stored.
In particular, the data sent by process i are stored in recvbuf starting
at location displs[i] .
Note that, as opposed to the non-vector variants, the sendcount
parameter can be different for different processes.
Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 40 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Vector Variants

These functions allow a different number of data elements to be sent

by each process by replacing the recvcount parameter with the array
recvcounts .
The amount of data sent by process i is equal to recvcounts[i] .
Note that the size of recvcounts is equal to the size of the
communicator comm.
The array parameter displs , which is also of the same size, is used to
determine where in recvbuf the data sent by each process will be
stored.
In particular, the data sent by process i are stored in recvbuf starting
at location displs[i] .
Note that, as opposed to the non-vector variants, the sendcount
parameter can be different for different processes.
Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 40 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Vector Variants

These functions allow a different number of data elements to be sent

by each process by replacing the recvcount parameter with the array
recvcounts .
The amount of data sent by process i is equal to recvcounts[i] .
Note that the size of recvcounts is equal to the size of the
communicator comm.
The array parameter displs , which is also of the same size, is used to
determine where in recvbuf the data sent by each process will be
stored.
In particular, the data sent by process i are stored in recvbuf starting
at location displs[i] .
Note that, as opposed to the non-vector variants, the sendcount
parameter can be different for different processes.
Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 40 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Vector Variants

Similarly to the gather operation, MPI provides a vector variant of the

scatter operation, called MPI_Scatterv , that allows different amounts of
data to be sent to different processes.
int MPI_Scatterv(const void *sendbuf, const int *sendcounts,
const int *displs, MPI_Datatype sendtype, void *recvbuf,
int recvcount, MPI_Datatype recvtype,
int root, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 41 / 44
 Collective Communication and Computation Operations

Collective Communication Operations

Vector Variants

MPI also provides a vector variant of the all-to-all personalized

communication operation called MPI_Alltoallv that allows different
amounts of data to be sent to and received from each process
int MPI_Alltoallv(const void *sendbuf, const int *sendcounts,
const int *sdispls, MPI_Datatype sendtype, void *recvbuf,
const int *recvcounts, const int *rdispls,
MPI_Datatype recvtype, MPI_Comm comm)

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 42 / 44
 Reference Books

Agenda

1 Principles of Message Passing Programming

2 Basic Routines of MPI

3 The Building Blocks: Send and Receive Operations

4 MPI Routines (Cont.)

5 Collective Communication and Computation Operations

6 Reference Books

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 43 / 44
 Reference Books

Reference Books

Introduction to Parallel Computing, Second Edition, by Ananth

Grama
Parallel programming in C with MPI and OpenMP by Michael J.
Quinn
An Introduction to Parallel Programming by Peter S. Pacheco
Professional CUDA C Programming by John Cheng

Department of CS, GCU, Lahore Parallel and Distributed Computing November 17, 2022 44 / 44