11/7/17
MPSoC Architectures
OpenMP
Alberto Bosio, Associate Professor – UM
Microelectronic Departement
[email protected] Introduction to OpenMP
l What is OpenMP?
l Open specification for Multi-Processing
l “Standard” API for defining multi-threaded shared-
memory programs
–www.openmp.org – Talks, examples, forums, etc.
l High-level API
l Preprocessor (compiler) directives ( ~ 80% )
l Library Calls ( ~ 19% )
l Environment Variables ( ~ 1% )
1
11/7/17
A Programmer’s View of OpenMP
l OpenMP is a portable, threaded, shared-memory
programming specification with “light” syntax
l Exact behavior depends on OpenMP implementation!
l Requires compiler support (C or Fortran)
l OpenMP will:
l Allow a programmer to separate a program into serial
regions and parallel regions, rather than T concurrently-
executing threads.
l Hide stack management
l Provide synchronization constructs
l OpenMP will not:
l Parallelize (or detect!) dependencies
l Guarantee speedup
l Provide freedom from data races
Outline
l Introduction
l Motivating example
l Parallel Programming is Hard
l OpenMP Programming Model
l Easier than PThreads
l Microbenchmark Performance Comparison
l vs. PThreads
l Discussion
l specOMP
2
11/7/17
Current Parallel Programming
1. Start with a parallel algorithm
2. Implement, keeping in mind:
• Data races
• Synchronization
• Threading Syntax
3. Test & Debug
4. Debug
5. Debug
Motivation – Threading Library
void* SayHello(void *foo) {
printf( "Hello, world!\n" );
return NULL;
}
int main() {
pthread_attr_t attr;
pthread_t threads[16];
int tn;
pthread_attr_init(&attr);
pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM);
for(tn=0; tn<16; tn++) {
pthread_create(&threads[tn], &attr, SayHello, NULL);
}
for(tn=0; tn<16 ; tn++) {
pthread_join(threads[tn], NULL);
}
return 0;
}
3
11/7/17
Motivation
• Thread libraries are hard to use
– P-Threads/Solaris threads have many library calls
for initialization, synchronization, thread creation,
condition variables, etc.
– Programmer must code with multiple threads in
mind
• Synchronization between threads introduces a
new dimension of program correctness
Motivation
l Wouldn’t it be nice to write serial programs and
somehow parallelize them “automatically”?
l OpenMP can parallelize many serial programs with
relatively few annotations that specify parallelism
and independence
l OpenMP is a small API that hides cumbersome
threading calls with simpler directives
4
11/7/17
Better Parallel Programming
1. Start with some algorithm
• Embarrassing parallelism is helpful, but not
necessary
2. Implement serially, ignoring:
• Data Races
• Synchronization
• Threading Syntax
3. Test and Debug
4. Automatically (magically?) parallelize
• Expect linear speedup
Motivation – OpenMP
int main() {
// Do this part in parallel
printf( "Hello, World!\n" );
return 0;
}
5
11/7/17
Motivation – OpenMP
int main() {
omp_set_num_threads(16);
// Do this part in parallel
#pragma omp parallel
{
printf( "Hello, World!\n" );
}
return 0;
}
OpenMP Parallel Programming
1. Start with a parallelizable algorithm
• Embarrassing parallelism is good, loop-level
parallelism is necessary
2. Implement serially, mostly ignoring:
• Data Races
• Synchronization
• Threading Syntax
3. Test and Debug
4. Annotate the code with parallelization (and
synchronization) directives
• Hope for linear speedup
5. Test and Debug
6
11/7/17
Programming Model - Threading
l Serial regions by default,
annotate to create parallel
regions
l Generic parallel regions
Fork
l Parallelized loops
l Sectioned parallel regions
l Thread-like Fork/Join model Join
l Arbitrary number of logical
thread creation/ destruction
events
Programming Model - Threading
int main() {
// serial region
printf(“Hello…”);
// parallel region
Fork
#pragma omp parallel
{
printf(“World”);
}
// serial again Join
Hello…WorldWorldWorldWorld!
printf(“!”);
}
7
11/7/17
Programming Model – Nested
Threading
• Fork/Join can be nested
Fork
– Nesting complication handled
“automagically” at compile-time
Fork
– Independent of the number of
threads actually running Join
Join
Programming Model – Thread
Identification
Master Thread
• Thread with ID=0 0
• Only thread that exists in
sequential regions
Fork
• Depending on implementation,
may have special purpose inside 0 1 2 3 4 5 6 7
parallel regions
• Some special directives affect only
Join
the master thread (like master)
0
8
11/7/17
Example
int main() {
int tid, nthreads;
omp_set_num_threads(16);
// Do this part in parallel
#pragma omp parallel private(nthreads, tid)
{
printf( "Hello, World!\n" );
/* Obtain and print thread id */
tid = omp_get_thread_num();
if (tid == 0)
{
nthreads = omp_get_num_threads();
printf("I'm the master, Number of threads = %d\n", nthreads);
}
return 0;
Programming Model – Data/Control
Parallelism
l Data parallelism
l Threads perform similar
functions, guided by thread Fork
identifier
l Control parallelism
l Threads perform differing
functions
- One thread for I/O, one for
computation, etc… Join
9
11/7/17
Programming model: Summary
Memory Model
l Shared memory communication
l Threads cooperates by accessing shared variables
l The sharing is defined syntactically
l Any variable that is seen by two or more threads is
shared
l Any variable that is seen by one thread only is
private
l Race conditions possible
l Use synchronization to protect from conflicts
l Change how data is stored to minimize the
synchronization
10
11/7/17
Structure
Programming Model – Concurrent
Loops
l OpenMP easily parallelizes loops
l No data dependencies between
iterations!
l Preprocessor calculates loop
bounds for each thread directly
from serial source
#pragma omp parallel for
for( i=0; i < 25; i++ ) {
printf(“Foo”);
}
11
11/7/17
The problem
l Executes the same code as many times as there are
threads
l How many threads do we have? omp_set_num_threads(n)
What is the use of repeating the same work n times in
parallel? Can use omp_thread_num() to distribute the work
between threads.
l D is shared between the threads, i and sum are private
Programming Model – Concurrent
Loops
12
11/7/17
Programming Model – Concurrent
Loops
Programming Model – Concurrent
Loops
l Load balancing
l If all the iterations execute at the same speed, the
processors are used optimally If some iterations are faster
than others, some processors may get idle, reducing the
speedup
l We don't always know the distribution of work, may need to
re-distribute dynamically
l Granularity
l Thread creation and synchronization takes time Assigning
work to threads on per-iteration resolution may take more
time than the execution itself! Need to coalesce the work to
coarse chunks to overcome the threading overhead
l Trade-off between load balancing and granularity!
13
11/7/17
Controlling Granularity
l #pragma omp parallel if (expression)
l Can be used to disable parallelization in some
cases (when the input is determined to be too small
to be beneficially multithreaded)
l #pragma omp num_threads (expression)
l Control the number of threads used for this parallel
region
Programming Model – Loop
Scheduling
• schedule clause determines how loop iterations are
divided among the thread team
–static([chunk]) divides iterations statically between
threads
- Each thread receives [chunk] iterations, rounding as
necessary to account for all iterations
- Default [chunk] is ceil( # iterations / # threads
)
–dynamic([chunk]) allocates [chunk] iterations per
thread, allocating an additional [chunk] iterations when
a thread finishes
- Forms a logical work queue, consisting of all loop iterations
- Default [chunk] is 1
–guided([chunk]) allocates dynamically, but
[chunk] is exponentially reduced with each allocation
14
11/7/17
Programming Model – Loop
Scheduling
Example
l The function TestForPrime (usually) takes little
time But can take long, if the number is a prime
indeed
l Solution: use dynamic, but with chunks
15
11/7/17
Work sharing: Sections
Sections
l The SECTIONS directive is a non-iterative
work-sharing construct. It specifies that the
enclosed section(s) of code are to be divided
among the threads in the team.
l Independent SECTION directives are nested
within a SECTIONS directive.
l Each SECTION is executed once by a thread in the
team. Different sections may be executed by
different threads. It is possible that for a thread to
execute more than one section if it is quick enough
and the implementation permits such.
16
11/7/17
Example
#include <omp.h>
#define N 1000
main ()
{
int i;
float a[N], b[N], c[N], d[N];
/* Some initializations */
for (i=0; i < N; i++) {
a[i] = i * 1.5;
b[i] = i + 22.35;
}
Example
#pragma omp parallel shared(a,b,c,d) private(i)
{
#pragma omp sections
{
#pragma omp section
for (i=0; i < N; i++)
c[i] = a[i] + b[i];
#pragma omp section
for (i=0; i < N; i++)
d[i] = a[i] * b[i];
} /* end of sections */
} /* end of parallel section */
}
17
11/7/17
Data Sharing
l Shared Memory programming model
l Most variables are shared by default
l We can define a variable as private
// Do this part in parallel
#pragma omp parallel private(nthreads, tid)
{
printf( "Hello, World!\n" );
if (tid == 0)
{
….
}
Programming Model – Data Sharing
l Parallel programs often employ
two types of data int bigdata[1024];
l Shared data, visible to all threads,
similarly named
l Private data, visible to a single thread void* foo(void* bar) {
(often stack-allocated)
int tid;
• PThreads:
– Global-scoped variables are shared
– Stack-allocated variables are private #pragma omp parallel \
shared ( bigdata ) \
private ( tid )
• OpenMP: {
– shared variables are shared /* Calc. here */
– private variables are private
}
}
18
11/7/17
Programming Model – Data Sharing
l private:
l A copy of the variable is created for each thread.
l No connection between the original variable and the
private copies
l Can achieve the same using variables inside { }
Int i;
#pragma omp parallel for private(i)
for (i=0; i<n; i++) { ... }
Programming Model – Data Sharing
l Firstprivate:
l Same, but the initial value is copied from the main
copy
l Lastprivate:
l Same, but the last value is copied to the main copy
19
11/7/17
Thread private
l Similar to private, but defined per variable
l Declaration immediately after variable definition.
l Must be visible in all translation units. Persistent
between parallel sections
l Can be initialized from the master's copy with
l #pragma omp copyin
l More efficient than private, but a global variable!
Synchronization
l What should the result be (assuming 2
threads)?
X=0;
#pragma omp parallel
X = X+1;
20
11/7/17
Synchronization
l 2 is the expected answer But can be 1 with
unfortunate interleaving
l OpenMP assumes that the programmer knows
what he is doing
l Regions of code that are marked to run in
parallel are independent If access collisions are
possible, it is the programmer's responsibility to
insert protection
Synchronization
l Many of the existing mechanisms for shared
programming
l OpenMP Synchronization
l Nowait (turn synchronization off!)
l Single/Master execution
l Critical sections, Atomic updates
l Ordered
l Barriers
l Flush (memory subsystem synchronization)
l Reduction (special case)
21
11/7/17
Single/Master
l #pragma omp single
l Only one of the threads will execute the following
block of code
l The rest will wait for it to complete
l Good for non-thread-safe regions of code (such as
I/O)
l Must be used in a parallel region
l Applicable to parallel for sections
Single/Master
l #pragma omp master
l The following block will be executed by the master
thread
l No synchronization involved
l Applicable only to parallel sections
#pragma omp parallel
{
do_preprocessing () ;
#pragma omp single
read_input () ;
#pragma omp master
notify_input_consumed () ;
do_processing () ; }
22
11/7/17
Critical Sections
l #pragma omp critical [name]
l Standard critical section functionality
l Critical sections are global in the program
l Can be used to protect a single resource in different
functions
l Critical sections are identified by the name
l All the unnamed critical sections are mutually
exclusive throughout the program
l All the critical sections having the same name are
mutually exclusive between themselves
Critical Sections
int x=0;
#pragma omp parallel shared(x)
{
#pragma omp critical
x++;
}
23
11/7/17
Ordered
l #pragma omp ordered statement
l Executes the statement in the sequential order
of iterations
l Example:
#pragma omp parallel for ordered
for (j=0; j<N; j++) {
int result = j*j;
#pragma omp ordered
printf ("computation(%d) = %d\n" ,j ,
result ) ;
}
Barrier synchronization
l #pragma omp barrier
l Performs a barrier synchronization between all
the threads in a team at the given point.
l Example:
#pragma omp parallel
{
int result = heavy_computation_part1 ()
;
#pragma omp atomic
sum += result ;
#pragma omp barrier
heavy_computation_part2 (sum) ;
}
24
11/7/17
Explicit Locking
l Can be used to pass lock variables around
(unlike critical sections!)
l Can be used to implement more involved
synchronization constructs
l Functions:
l omp_init_lock(), omp_destroy_lock(),
omp_set_lock(), omp_unset_lock(), omp_test_lock()
The usual semantics
l Use #pragma omp flush to synchronize memory
Consistency Violation?
25
11/7/17
Consistency Violation?
Reduction
for (j=0; j<N; j++) {
sum =
sum+a[j]∗b[j];
}
l How to parallelize this code?
l sum is not private, but accessing it atomically is too
expensive
l Have a private copy of sum in each thread, then
add them up
l Use the reduction clause!
l #pragma omp parallel for reduction(+: sum)
l An operator must be used: +, -, *...
26
11/7/17
Synchronization Overhead
l Lost time waiting for locks
l Prefer to use structures that are as lock-free as
possible!
Summary
l OpenMP is a compiler-based technique to create
concurrent code from (mostly) serial code
l OpenMP can enable (easy) parallelization of loop-
based code
l Lightweight syntactic language extensions
l OpenMP performs comparably to manually-coded
threading
l Scalable
l Portable
l Not a silver bullet for all applications
27
11/7/17
More Information
• www.openmp.org
l OpenMP official site
• www.llnl.gov/computing/tutorials/openMP/
l A handy OpenMP tutorial
• www.nersc.gov/nusers/help/tutorials/openmp/
l Another OpenMP tutorial and reference
Backup Slides
Syntax, etc
28
11/7/17
lOpenMP Syntax
l General syntax for OpenMP directives
#pragma omp directive [clause…] CR
l Directive specifies type of OpenMP operation
l Parallelization
l Synchronization
l Etc.
l Clauses (optional) modify semantics of Directive
lOpenMP Syntax
l PARALLEL syntax
#pragma omp parallel [clause…] CR
structured_block
Ex:
#pragma omp parallel
Output:
Hello! (T=4)
Hello!
{
Hello!
printf(“Hello!\n”);
Hello!
} // implicit barrier
29
11/7/17
l OpenMP Syntax
l DO/for Syntax (DO-Fortran, for-C)
#pragma omp for [clause…] CR
for_loop
Ex:
#pragma omp parallel
{
#pragma omp for private(i) shared(x) \
schedule(static,x/N)
for(i=0;i<x;i++) printf(“Hello!\n”);
} // implicit barrier
Note: Must reside inside a parallel section
l OpenMP Syntax
More on Clauses
• private() – A variable in private list is private
to each thread
• shared() – Variables in shared list are visible
to all threads
l Implies no synchronization, or even consistency!
• schedule() – Determines how iterations will
be divided among threads
–schedule(static, C) – Each thread will be
given C iterations
- Usually T*C = Number of total iterations
–schedule(dynamic) – Each thread will be given
additional iterations as-needed
- Often less efficient than considered static allocation
• nowait – Removes implicit
CS Architecture Seminarbarrier from end of
block
30
11/7/17
OpenMP Syntax
l
l PARALLEL FOR (combines parallel and
for)
#pragma omp parallel for [clause…] CR
for_loop
Ex:
#pragma omp parallel for shared(x)\
private(i)
\
schedule(dynamic)
for(i=0;i<x;i++) {
printf(“Hello!\n”);
lExample: AddMatrix
Files:
(Makefile)
addmatrix.c // omp-
parallelized
matrixmain.c // non-omp
printmatrix.c // non-omp
31
11/7/17
lOpenMP Syntax
l ATOMIC syntax
#pragma omp atomic CR
simple_statement
Ex:
#pragma omp parallel shared(x)
{
#pragma omp atomic
x++;
} // implicit barrier
OpenMP Syntax
• CRITICAL syntax
#pragma omp critical CR
structured_block
Ex:
#pragma omp parallel shared(x)
{
#pragma omp critical
{
// only one thread in here
}
} // implicit barrier
32
11/7/17
l OpenMP Syntax
ATOMIC vs. CRITICAL
l Use ATOMIC for “simple statements”
l Can have lower overhead than CRITICAL if HW
atomics are leveraged (implementation dep.)
l Use CRITICAL for larger expressions
l May involve an unseen implicit lock
l OpenMP Syntax
l MASTER – only Thread 0 executes a block
#pragma omp master CR
structured_block
l SINGLE – onlyomp
#pragma one single
thread executes
CR a block
structured_block
l No implied synchronization
33
11/7/17
lOpenMP Syntax
l BARRIER
#pragma omp barrier CR
l Locks
l Locks are provided through omp.h library calls
–omp_init_lock()
–omp_destroy_lock()
–omp_test_lock()
–omp_set_lock()
–omp_unset_lock()
lOpenMP Syntax
l FLUSH
#pragma omp flush CR
l Guarantees that threads’ views of memory is
consistent
l Why? Recall OpenMP directives…
l Code is generated by directives at compile-time
- Variables are not always declared as volatile
- Using variables from registers instead of memory can
seem like a consistency violation
l Synch. Often has an implicit flush
- ATOMIC, CRITICAL
34
11/7/17
lOpenMP Syntax
l Functions
omp_set_num_threads()
omp_get_num_threads()
omp_get_max_threads()
omp_get_num_procs()
omp_get_thread_num()
omp_set_dynamic()
omp_[init destroy test set
unset]_lock()
Function for the environment
l omp_set_dynamic(int)
l omp_set_num_threads(int)
l omp_get_num_threads()
l omp_get_num_procs()
l omp_get_thread_num()
l omp_set_nested(int)
l omp_in_parallel()
l omp_get_wtime()
35
