Chapter 3

Parallel Programming Models

Abstraction• Machine Level– Looks at hardware, OS, buffers

• Architectural models– Looks at interconnection network, memory

organization, synchronous & asynchronous• Computational Model– Cost models, algorithm complexity, RAM vs. PRAM

• Programming Model– Uses programming language description of process

Control Flows

• Process– Address spaces differ - Distributed

• Thread– Shares address spaces – Shared Memory

• Created statically (like MPI-1) or dynamically during run time (MPI-2 allows this as well as pthreads).

Parallelization of a Program

• Decomposition of the computations– Can be done at many levels (ex. Pipelining).– Divide into tasks and identify dependencies

between tasks.– Can be done statically (at compile time) or

dynamically (at run time)– Number of tasks places an upper bound on the

parallelism that can be used– Granularity: the computation time of a task

Assignment of Tasks• The number of processes or threads does not

need to be the same as the number of processors• Load Balancing: each process/thread having the

same amount of work (computation, memory access, communication)

• Have a tasks that use the same memory execute on the same thread (good cache use)

• Scheduling: assignment of tasks to threads/processes

Assignment to Processors

• 1-1: map a process/thread to a unique processor

• many to 1: map several processes to a single processor. (Load balancing issues)

• OS or programmer done

Scheduling

• Precedence constraints– Dependencies between tasks

• Capacity constraints– A fixed number of processors

• Want to meet constraints and finish in minimum time

Levels of Parallelism

• Instruction Level• Data Level• Loop Level• Functional level

Instruction Level• Executing multiple instructions in parallel.

May have problems with dependencies– Flow dependency – if next instruction needs a

value computed by previous instruction– Anti-Dependency – if an instruction uses a value

from register or memory when the next instruction stores a value into that place (cannot reverse the order of instructions

– Output dependency – 2 instructions store into same location

Data Level

• Same process applied to different elements of a large data structure

• If these are independent, the can distribute the data among the processors

• One single control flow• SIMD

Loop Level

• If there are no dependencies between the iterations of a loop, then each iteration can be done independently, in parallel

• Similar to data parallelism

Functional Level

• Look at the parts of a program and determine which parts can be done independently.

• Use a dependency graph to find the dependencies/independencies

• Static or Dynamic assignment of tasks to processors– Dynamic would use a task pool

Explicit/Implicit Parallelism Expression

• Language dependent• Some languages hide the parallelism in the

language• For some languages, you must explicitly state

the parallelism

Parallelizing Compilers

• Takes a program in a sequential language and generates parallel code– Must analyze the dependencies and not violate them– Should provide good load balancing (difficult)– Minimize communication

• Functional Programming Languages– Express computations as evaluation of a function

with no side effects– Allows for parallel evaluation

More explicit/implicit

• Explicit parallelism/implicit distribution– The language explicitly states the parallelism in

the algorithm, but allows the system to assign the tasks to processors.

• Explicit assignment to processors – do not have to worry about communications

• Explicit Communication and Synchronization– MPI – additionally must explicitly state

communications and synchronization points

Parallel Programming Patterns• Process/Thread Creation• Fork-Join• Parbegin-Parend• SPMD, SIMD• Master-Slave (worker)• Client-Server• Pipelining• Task Pools• Producer-Consumer

Process/Thread Creation

• Static or Dynamic• Threads, traditionally dynamic• Processes, traditionally static, but dynamic has

become recently available

Fork-Join

• An existing thread can create a number of child processes with a fork.

• The child threads work in parallel.• Join waits for all the forked processes to

terminate.• Spawn/exit is similar

Parbegin-Parend

• Also called cobegin-coend• Each statement (blocks/function calls) in the

cobegin-coend block are to be executed in parallel.

• Statements after coend are not executed until all the parallel statement are complete.

SPMD – SIMD

• Single Program, Multiple Data vs. Single Instruction, Multiple Data

• Both use a number of threads/processes which apply the same program to different data

• SIMD executes the statements synchronously on different data

• SPMD executes the statements asynchronously

Master-Slave

• One thread/process that controls all the others

• If dynamic thread/process creation, the master is the one that usually does it.

• Master would “assign” the work to the workers and the workers would send the results to the master

Client-Server

• Multiple clients connected to a server that responds to requests

• Server could be satisfying requests in parallel (multiple requests being done in parallel or if the request is involved, a parallel solution to the request)

• The client would also do some work with response from server.

• Very good model for heterogeneous systems

Pipelining

• Output of one thread is the input to another thread

• A special type of functional decomposition• Another case where heterogeneous systems

would be useful

Task Pools

• Keep a collection of tasks to be done and the data to do it upon

• Thread/process can generate tasks to be added to the pool as well as obtaining a task when it is done with the current task

Producer Consumer

• Producer threads create data used as input by the consumer threads

• Data is stored in a common buffer that is accessed by producers and consumers

• Producer cannot add if buffer is full• Consumer cannot remove if buffer is empty

Array Data Distributions• 1-D– Blockwise• Each process gets ceil(n/p) elements of A, except for the

last process which gets n-(p-1)*ceil(n/p) elements• Alternatively, the first n%p processes get ceil(n/p)

elements while the rest get floor(n/p) elements.

– Cyclic• Process p gets data k*p+i (k=0..ceil(n/))

– Block cyclic• Distribute blocks of size b to processes in a cyclic

manner

2-D Array distribution

• Blockwise distribution rows or columns• Cyclic distribution of rows or columns• Blockwise-cyclic distribution of rows or

columns

Checkerboard

• Take an array of size n x m• Overlay a grid of size g x f – g<=n– f<=m– More easily seen if n is a multiple of g and m is a

multiple of f• Blockwise Checkerboard– Assign each n/g x m/f submatrix to a processor

Cyclic Checkerboard

• Take each item in a n/g x m/f submatrix and assign it in a cyclic manner.

• Block-Cyclic checkerboard– Take each n/g x m/f submatrix and assign all the

data in the submatrix to a processor in a cyclic fashion

Information Exchange

• Shared Variables– Used in shared memory– When thread T1 wants to share information with

thread T2, then T1 writes the information into a variable that is shared with T2

– Must avoid 2 or more processes reading or writing to the same variable at the same time (race condition)

– Leads to non-Deterministic behavior.

Critical Sections

• Sections of code where there may be concurrent accesses to shared variables

• Must make these sections mutually exclusive– Only one process can be executing this section at any

one time• Lock mechanism is used to keep sections mutually

exclusive– Process checks to see if section is “open”– If it is, then “lock” it and execute (unlock when done)– If not, wait until unlocked

Communication Operations• Single Transfer – Pi sends a message to Pj

• Single Broadcast – one process sends the same data to all other processes

• Single accumulation – Many values operated on to make a single value that is placed in root

• Gather – Each process provides a block of data to a common single process

• Scatter – root process sends a separate block of a large data structure to every other process

More Communications

• Multi-Broadcast – Every process sends data to every other process so every process has all the data that was spread out among the processes

• Multi-Accumulate – accumulate, but every process gets the result

• Total Exchange-Each process provides p-data blocks. The ith data block is sent to pi. Each processor receives the blocks and builds the structure with the data in i order.

Applications

• Parallel Matrix-Vector Product– Ab=c where A is n x m and b, c are m– Want A to be in contiguous memory• A single array, not an array of arrays

– Have blocks of rows with allof b calculate a block of c• Used if A is stored row-wise

– Have blocks of columns with a block of b compute columns that need to be summed.• Used if A is stored column-wise

Processes and Threads

• Process – a program in execution– Includes code, program data on stack or heap,

values of registers, PC.– Assigned to processor or core for execution– If there are more processes than resources

(processors or memory) for all, execute in a round-robin time-shared method

– Context switch – changing from one process to another executing on processor.

Fork

• The Unix fork command– Creates a new process– Makes a copy of the program – Copy starts at statement after the fork– NOT shared memory model – Distributed memory

model– Can take a while to execute

Threads

• Share a single address space• Best with physically shared memory• Easier to get started than a process – no copy

of code space• Two types– Kernel threads – managed by the OS– User threads – managed by a thread library

Thread Execution

• If user threads are executed by a thread library/scheduler, (no OS support for threads) then all the threads are part of one process that is scheduled by the OS– Only one thread executed at a time even if there are multiple

processors• If OS has thread management, then threads can be

scheduled by OS and multiple threads can execute concurrently

• Or, Thread scheduler can map user threads to kernel threads (several user threads may map to one kernel thread)

Thread States

• Newly generated• Executable• Running• Waiting• Finished• Threads transition from state to state based

on events (start, interrupt, end, block, unblock, assign-to-processor)

Synchronization

• Locks– A process “locks” a shared variable at the

beginning of a critical section• Lock allows process to proceed if shared variable is

unlocked• Process blocked if variable is locked until variable is

unlocked• Locking is an “atomic” process.

Semaphore

• Usually a binary type but can be integer• wait(s)– Waits until the value of s is 1 (or greater)– When it is, decreases s by 1 and continues

• signal(s)– Increments s by 1

Barrier Synchronization

• A way to have every process wait until every process is at a certain point

• Assures the state of every process before certain code is executed

Condition Synchronization

• A thread is blocked until a given condition is established– If condition is not true, then put into blocked state– When condition true, moved from blocked to

ready (not necessarily directly to a processor)– Since other processes may be executing, by the

time this process gets to a processor, the condition may no longer be true• So, must check condition after condition satisfied

Efficient Thread Programs

• Proper number of threads– Consider degree of parallelism in application– Number of processors– Size of shared cache

• Avoid synchronization as much as possible– Make critical section as small as possible

• Watch for deadlock conditions

Memory Access

• Must consider writing values to shared memory that is held in local caches

• False sharing– Consider 2 processes writing to different memory locations– SHOULD not be an issue since not shared by two cache

memories– HOWEVER, if the memory locations are close to each

other, they may be in the same cache line and actually have the different locations both be in the different caches