parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · parallel...
TRANSCRIPT
![Page 1: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/1.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel design patterns
Marco Danelutto
Dept. of Computer Science, University of Pisa, Italy
May 2011, Pisa
![Page 2: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/2.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Contents
1 Introduction
2 Parallel design patterns
3 Finding concurrency design space
4 Algorithm Structure design space
5 Supporting structures design space
6 Implementation mechanisms design space
7 Conclusions
![Page 3: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/3.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel computing
The problem
Solve a problem using nw processing resources
Obtaining a (close to) nw speedup with respect to timespent sequentially
s(n) =Tseq
Tpar (n)(speedup)
ε(n) =Tid
Tpar (n)=
Tseq
n × Tpar (n)(efficiency)
s(n) =1
f + 1−fn
(Amdhal law)
![Page 4: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/4.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel computing
The problem
Solve a problem using nw processing resources
Obtaining a (close to) nw speedup with respect to timespent sequentially
s(n) =Tseq
Tpar (n)(speedup)
ε(n) =Tid
Tpar (n)=
Tseq
n × Tpar (n)(efficiency)
s(n) =1
f + 1−fn
(Amdhal law)
![Page 5: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/5.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
The problems
Find potentially concurrent activities
alternative decompositions
with possibly radically differences
Parallelism exploitation
program activities (threads, processes)
program interactions (communications, sycnhronizations)
→ overhead
![Page 6: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/6.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
The problems
Find potentially concurrent activities
alternative decompositions
with possibly radically differences
Parallelism exploitation
program activities (threads, processes)
program interactions (communications, sycnhronizations)
→ overhead
![Page 7: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/7.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Structured parallel programming
Algorithmic skeletons
Cole 1988 → common, parametric, reusable parallelismexploitation patternlanguages & libraries since ’90 (P3L, Skil, eSkel, ASSIST,Muesli, SkeTo, Mallba, Muskel, Skipper, BS, ...)high level parallel abstractions (parallel programmingcommunity)
hiding most of the technicalities related to parallelismexploitationdirectly exposed to application programmers
Parallel design patterns
Massingill, Mattson, Sanders 2000 → “Patterns for parallelprogramming” book (2006) (software engineering community)design patterns a la Gamma book
name, problem, solution, use cases, etc.
concurrency, algorithms, implementation, mechanisms
![Page 8: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/8.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Structured parallel programming
Algorithmic skeletons
Cole 1988 → common, parametric, reusable parallelismexploitation patternlanguages & libraries since ’90 (P3L, Skil, eSkel, ASSIST,Muesli, SkeTo, Mallba, Muskel, Skipper, BS, ...)high level parallel abstractions (parallel programmingcommunity)
hiding most of the technicalities related to parallelismexploitationdirectly exposed to application programmers
Parallel design patterns
Massingill, Mattson, Sanders 2000 → “Patterns for parallelprogramming” book (2006) (software engineering community)design patterns a la Gamma book
name, problem, solution, use cases, etc.
concurrency, algorithms, implementation, mechanisms
![Page 9: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/9.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Structured parallel programming
Algorithmic skeletons
Cole 1988 → common, parametric, reusable parallelismexploitation patternlanguages & libraries since ’90 (P3L, Skil, eSkel, ASSIST,Muesli, SkeTo, Mallba, Muskel, Skipper, BS, ...)high level parallel abstractions (parallel programmingcommunity)
hiding most of the technicalities related to parallelismexploitationdirectly exposed to application programmers
Parallel design patterns
Massingill, Mattson, Sanders 2000 → “Patterns for parallelprogramming” book (2006) (software engineering community)design patterns a la Gamma book
name, problem, solution, use cases, etc.
concurrency, algorithms, implementation, mechanisms
![Page 10: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/10.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Concept evolution
Parallelism
parallelism exploitation patterns shared among applications
separation of concerns:
system programmers → efficient implementation of parallelpatternsapplication programmers → application specific details
New architectures
Heterogeneous in Hw & Sw
Multicore NUMA, cache coherent architectures
Further non functional concerns
security, fault tolerance, power management, . . .
![Page 11: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/11.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Concept evolution
Parallelism
parallelism exploitation patterns shared among applications
separation of concerns:
system programmers → efficient implementation of parallelpatternsapplication programmers → application specific details
New architectures
Heterogeneous in Hw & Sw
Multicore NUMA, cache coherent architectures
Further non functional concerns
security, fault tolerance, power management, . . .
![Page 12: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/12.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Concept evolution
Parallelism
parallelism exploitation patterns shared among applications
separation of concerns:
system programmers → efficient implementation of parallelpatternsapplication programmers → application specific details
New architectures
Heterogeneous in Hw & Sw
Multicore NUMA, cache coherent architectures
Further non functional concerns
security, fault tolerance, power management, . . .
![Page 13: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/13.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel design patterns
Researchers active since beginning of the century
S. MacDonald, J. Anvik, S. Bromling, J. Schaeffer, D. Szafron, andK. Tan. 2002. From patterns to frameworks to parallel programs.Parallel Comput. 28, 12 (December 2002), 1663-1683.
Berna L. Massingill, Timothy G. Mattson, Beverly A. Sanders: APattern Language for Parallel Application Programs (Research Note).Euro-Par 2000, LNCS, pp. 678-681, 2000
Berna L. Massingill, Timothy G. Mattson , Beverly A. Sanders,Parallel programming with a pattern language, Springer Verlag, Int.J. STTT 3:1-18, 2001
Berna L. Massingill, Timothy G. Mattson , Beverly A. Sanders,Patterns for Finding Concurrency for Parallel Application Programs,(pre-book)
Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill,Patterns for parallel programming, Addison Wesley, PearsonEducation, 2005
![Page 14: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/14.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel design patterns
![Page 15: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/15.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
The pattern framework
Four patter classes:
1 Finding concurrency
2 Algorithm structure
3 Supporting structure
4 Implementation mechanisms
These are “design spaces”
different concerns
different “kind of programmers” involved
upper layers → application programmerslower layers → system programmers
![Page 16: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/16.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Finding concurrency design space
Three main blocks
1 Decomposition
→ Decomposition of problems into pieces that can becomputed concurrently
2 Dependency analysis
→ support task grouping and dependency analysis
3 Design evaluation
→ aimed at supporting evaluation of alternatives
Used in an iterative process:design → evaluate → redesign → . . .
![Page 17: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/17.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Finding concurrency design space
![Page 18: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/18.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Decomposition patterns
Task decomposition
How can a problem be decomposed into tasks that can executeconcurrently ?
Data decomposition
How can a problem’s date be decomposed into units that canbe operated on relatively independently?
Forces:
Flexibility
Efficiency
Simplicity
![Page 19: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/19.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Dependency analysis patterns
Group tasks
How can the tasks make up a problem be grouped to simplifythe job of managing dependencies?
Order tasks
Given a way of decomposing a problem into tasks and a way ofcollecting these tasks into logically related groups, how mustthese groups of tasks be ordered to satisfy constrains amongtasks?
Data sharing
Given a data and task decomposition for a problem, how isdata shared among the tasks?
![Page 20: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/20.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Design evaluation pattern
Design evaluation pattern
Is the decomposition and dependency analysis so far good tomove on to the next design space, or should the design berevisited?
Forces:
Suitability for the target platform (PE available, sharingsupport, coordination of PE activities, overheads)
Design quality (flexibility, efficiency, simplicity)
Preparation for the next phase of the design (regularity ofthe solution, synchronous/asynchronous interactions, taskgrouping)
![Page 21: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/21.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Algorithm structure
Three main blocks
1 Organize by task
→ when execution by tasks is the best organizing principle
2 Organize by data decomposition
→ when main source of parallelisms is data
3 Organize by flow analysis
→ flow of data imposing ordering on (groups of) tasks
![Page 22: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/22.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Algorithm structure
![Page 23: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/23.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Organize by task
Task parallelism
When the problem is best decomposed into a collection oftasks that can execute concurrently, how can this concurrencybe exploited efficiently?
→ dependency analysis, scheduling, ...
Divide & conquer
Suppose the problem is formulated using the sequentialdivide&conquer strategy. How can the potential concurrencybe exploited?
→ dependency analysis, communication costs, ...
![Page 24: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/24.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Organize by data decomposition
Geometric decomposition
How can an algorithm be organized around a data structurethat has been decomposed into concurrently updatable“chuncks”?
Recursive data
Suppose the problem involves an operation on a recursive datastructure (such as a list, tree or graph) that appears to requiresequential processing. How can operations on these datastructures be performed in parallel?
![Page 25: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/25.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Organize by flow of data
Pipeline
Suppose that the overall computation involves performing acalculation on many sets of data, where the calculation can beviewed in terms of data flowing through a sequence of stages.How can potential concurrency be exploited ?
Event based coordination
Suppose th application can be decomposed into groups ofsemi-independent tasks interacting in an irregular fashion. Theinteraction is determined by the flow of data between themwhich implies ordering constraints between the tasks. How canthese tasks and their interaction be implemented so they canexecute concurrently?
![Page 26: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/26.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Supporting structures
Two main blocks:
1 Program structures
→ approaches for structuring source code
2 Data structures
→ data dependency management
Forces:
Clarity of abstraction
Scalability
Efficiency
Maintainability
Environmental affinity
Sequential equivalence
![Page 27: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/27.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Supporting structures
![Page 28: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/28.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Program structures
SPMD (Single Program Multiple Data)
The interactions between the various UEs cause most of theproblems when writing correct and efficient parallel programs.How can programmers structure their parallel programs tomake these interactions more manageable and easier tointegrate with the core computations?
Master/worker
How should a program be organized when the design isdominated by the need to dynamically balance the work on aset of tasks among the UEs?
![Page 29: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/29.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Program structures (2)
Loop parallelism
Given a serial program whose runtime is dominated by a set ofcomputationally intensive loops, how can it be translated into aparallel program?
Fork/join
In some programs the number of concurrent tasks varies as theprogram executes, and the way these tasks are related preventsthe use of simple control structures such as parallel loops. Howcan a parallel program be constructed around such complicatedsets of dynamic tasks?
![Page 30: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/30.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Data Structures
Shared data
How toes one explicitly manage shared data inside a set ofconcurrent tasks?
Shared queue
How can concurrenty-executing UEs safely share a queue datastructure?
Distributed array
Arrays often need to be partitioned between multiple UEs. Howcan we do this so the resulting program is both readable andefficient?
![Page 31: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/31.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Implementation mechanisms
Directly related to the target architecture:
1 to provide mechanisms suitable to create a set ofconcurrent activities (UE Units of Execution)
→ threads, processes (creation, destruction)
2 to support interactions among the UEs
→ locks, mutexes, semaphores, memory fences, barriers,monitors, ...
3 to support data exchange among the UEs
→ communication channels, queues, shared memory,collective operations (broadcasts, multicast, barrier,reduce) ...
![Page 32: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/32.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Implementation mechanisms
![Page 33: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/33.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Design patterns vs. algorithmic skeletons
Sw engineering vs. HPC community
SwEng Focus on efficiency of the programming process
HPC Focus on performance(and programmer productivity)
Then:
Design patterns “recipes” to be implemented in order toget a working program
Algorithmic skeletons predefined program constructs(language constructs, classes, library entries)implementing parallel patterns
![Page 34: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/34.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Design patterns vs. algorithmic skeletons
Sw engineering vs. HPC community
SwEng Focus on efficiency of the programming process
HPC Focus on performance(and programmer productivity)
Then:
Design patterns “recipes” to be implemented in order toget a working program
Algorithmic skeletons predefined program constructs(language constructs, classes, library entries)implementing parallel patterns
![Page 35: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/35.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Typical skeleton based programdevelopment
1 Figure out which skeleton (composition) models yourproblem
2 Instantiate skeletons
functional parameters (e.g. code, data types) & nonfunctional ones (e.g parallelism degree)
3 Fine tune program performance
parameter sweeping, bottleneck analysis or skeletonrestructuring
→ no parallel debugging, correctness guaranteed
→ no possibility to use parallel patterns not supported bythe skeleton set
![Page 36: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/36.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Typical algorithmic skeleton frameworks
Muesli (H. Kuchen, Munster Univ. D)
C++ class library
stream parallel skeletons
Pipeline, Farm, Branch&Bound, Divide&Conquer (Atomic,Filter, Final, Initial)
data parallel skeletons
DistributedXXX (XXX ∈ { Array, Matrix, SparseMatrix})+ fold, map, scan, zip
target architecture: C++ (MPI + OpenMP)
all communication, synchronization, shared accessproblems solved in the skeleton implementation
program declaration separated from execution
![Page 37: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/37.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel design patterns in perspective
Principles
Architecting parallelsoftware with designpatterns, not just parallelprogramming languages
Split productivity andefficiency layers, not just asingle general-purpose layer
Generating code withsearch-based autotuners,not compilers
Synthesis with sketching
Verification and testing, notone or the other
...
![Page 38: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/38.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Parallel design patterns in perspective (2)
TBB (Thread Building Block library by Intel,2005)
C++ library
currently version 3.0 (since late 2010)
base building blocks for parallel programming with thread
parallel loop, reduce, pipelinetasksparallel containersmutexes
since 3.0 → TBB Design patterns
Agglomeration, Elementwise, Odd-even communication,Wavefront, Reduction, Divide&Conquer, GUI thread,Non-preemptive priorities, Local serializer, Fenced datatransfer, Lazy initialization, Reference counting,Compare-and-swap loop.
![Page 39: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/39.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Sample TBB pattern: D&C
![Page 40: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/40.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Sample TBB pattern: D&C
Example
![Page 41: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/41.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Sample APPL
![Page 42: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/42.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Sample APPL
![Page 43: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/43.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Sample APPL
![Page 44: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/44.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Conclusions
Become an expert in parallel computing
→ by studying and applying parallel design patterns !!!
You will need it
also to program iPhone applications!
![Page 45: Parallel design patterns - unipi.itdidawiki.di.unipi.it/lib/exe/fetch.php/magistrale... · Parallel design patterns Massingill, Mattson, Sanders 2000 !\Patterns for parallel programming"](https://reader034.vdocuments.site/reader034/viewer/2022050508/5f9915f8a65ad97dbe38981e/html5/thumbnails/45.jpg)
Parallel designpatterns
M. Danelutto
Introduction
Parallel designpatterns
Findingconcurrencydesign space
AlgorithmStructuredesign space
Supportingstructuresdesign space
Implementationmechanismsdesign space
Conclusions
Conclusions
Become an expert in parallel computing
→ by studying and applying parallel design patterns !!!
You will need it
also to program iPhone applications!