CS516: Parallelization of Programs
Overview of Parallel Architectures
Vishwesh Jatala
Assistant Professor
Department of CSE
Indian Institute of Technology Bhilai
[email protected]
2023-24 W
1
Recap: Why Parallel Architectures?
• Moore’s Law: The number of transistors on a IC doubles about every two years
2
Recap: Moore’s Law Effect
3
Processor Architecture RoadMap
4
Course Outline
■ Introduction
■ Overview of Parallel Architectures
■ Performance
■ Parallel Programming
• GPUs and CUDA programming
■ Case studies
■ Extracting Parallelism from Sequential Programs Automatically
5
Flynn’s Taxonomy
• Flynn’s classification of computer architecture
6
SISD: Single Instruction, Single Data
• The von Neumann architecture
• Implements a universal Turing machine
• Conforms to serial algorithmic analysis
From http://arstechnica.com/paedia/c/ cpu/part-1/cpu1-1.html
7
SIMD: Single Instruction, Multiple Data
• Single control stream
• All processors operating in lock step
• Fine-grained parallelism
8
SIMD: Single Instruction, Multiple Data
• Example: GPUs
From http://arstechnica.com/paedia/c/c pu/part-1/cpu1-1.html
9
MIMD: Multiple Instructions, Multiple Data
• Most the machines that are prevalent
• Multi-core, SMP, Clusters, NUMA machines, etc.
10
Rest of the today’s lecture…
• Flynn’s classification of computer architecture
11
Flynn’s Taxonomy
• Flynn’s classification of computer architecture
12
MIMD: Shared Memory Multiprocessors
• Tightly coupled multiprocessors
• Shared global memory address space
• Traditional multiprocessing: symmetric multiprocessing (SMP)
• Existing multi-core processors, multithreaded processors
• Programming model similar to uniprocessors (i.e., multitasking uniprocessor) except
• Operations on shared data require synchronization
13
Interconnection Schemes for SMP
14
SMP Architectures
15
UMA: Uniform Memory Access
• All processors have the same uncontended latency to memory
• Symmetric multiprocessing (SMP) ~ UMA with bus interconnect
16
UMA: Uniform Memory Access
+ Data placement unimportant/less important (easier to optimize code and make use of available
memory space)
- Scaling the system increases all latencies
- Contention could restrict bandwidth and increase latency
17
How to Scale Shared Memory Machines?
• Two general approaches
• Maintain UMA
• Provide a scalable interconnect to memory
• Scaling system increases memory latency
• Interconnect complete processors with local memory
• NUMA (Non-uniform memory access)
• Local memory faster than remote memory
• Still needs a scalable interconnect for accessing remote memory
18
NUMA: Non Uniform Memory Access
• Shared memory as local versus remote memory
+ Low latency to local memory
- Much higher latency to remote memories
+ Bandwidth to local memory may be higher
- Performance very sensitive to data placement
19
MIMD: Message Passing Architectures
• Loosely coupled multiprocessors
• No shared global memory address space
• Multicomputer network
• Network-based multiprocessors
• Usually programmed via message passing
• Explicit calls (send, receive) for communication
20
MIMD: Message Passing Architectures
21
Historical Evolution: 1960s & 70s
• Early MPs
• Mainframes
• Small number of processors
• crossbar interconnect
• UMA
22
Historical Evolution: 1980s
• Bus-Based MPs
• enabler: processor-on-a-board
• economical scaling
• precursor of today’s SMPs
• UMA
23
Historical Evolution: Late 80s, mid 90s
• Large Scale MPs (Massively Parallel
Processors)
• multi-dimensional interconnects
• each node a computer (proc + cache
+ memory)
• NUMA
• still used for “supercomputing”
24
Flynn’s Taxonomy
• Flynn’s classification of computer architecture
25
SIMD: Single Instruction, Multiple Data
• Example: GPUs
From http://arstechnica.com/paedia/c/c pu/part-1/cpu1-1.html
26
Data Parallel Programming Model
• Programming Model
• Operations are performed on each element of a large (regular) data
structure (array, vector, matrix)
• Simple example (A, B and C are vectors)
C = (A * B)
• The operations can be executed in sequential or parallel steps
• Language supports array assignment
27
On Sequential Hardwares
28
On Data Parallel Hardwares
29
Data Parallel Architectures
• Early architectures directly mirrored programming model
• Single control processor (broadcast each instruction to an array/grid of
processing elements)
• Examples: Connection Machine, MPP (Massively Parallel Processor)
30
Data Parallel Architectures
• Later data parallel architectures
• Higher integration → SIMD units on chip along with caches
• More generic → multiple cooperating multiprocessors (GPUs)
• Specialized hardware support for global synchronization
31
SIMD: Graphics Processing Units
• The early GPU designs
• Specialized for graphics processing only
• Exhibit SIMD execution
• Less programmable
• NVIDIA GeForce 256
• In 2007, fully programmable GPUs
• CUDA released
32
Single-core CPU vs Multi-core vs GPU
33
Single-core CPU vs Multi-core vs GPU
34
NVIDIA V100 GPU
https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf
35
Specifications
36
CPUs vs GPUs
Chip to chip comparison of peak memory bandwidth in GB/s and peak double precision
gigaflops for GPUs and CPUs since 2008.
https://www.nextplatform.com/2019/07/10/a-decade-of-accelerated-computing-augurs-well-for-gpus
37
GPU Applications
38
Specifications
39
Multi-GPU Systems
https://www.azken.com/images/dgx1_images/dgx1-system-architecture-whitepaper1.pdf
40
Summary
• Parallel architectures are inevitable
• Different architectures are evolved
• Flynn’s taxonomy:
• SISD
• MISD
• MIMD
• SIMD
41
References
• David Culler, Jaswinder Pal Singh, and Anoop Gupta. 1998. Parallel Computer
Architecture: A Hardware/Software Approach. Morgan Kaufmann Publishers Inc.,
San Francisco, CA, USA
• https://safari.ethz.ch/architecture/fall2020/doku.php?id=schedule
• https://www.cse.iitd.ac.in/~soham/COL380/page.html
• https://s3.wp.wsu.edu/uploads/sites/1122/2017/05/6-9-2017-slides-vFinal.pptx
• https://ebhor.com/full-form-of-cpu/
• Miscellaneous resources on internet
42
Thank You
43
