ics 556 parallel algorithms ebrahim malalla [email protected] office: bldg 22, room 124-8
TRANSCRIPT
![Page 1: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/1.jpg)
ICS 556ICS 556Parallel AlgorithmsParallel Algorithms
Ebrahim MalallaEbrahim Malalla
[email protected]@kfupm.edu.sa
Office:Office: Bldg 22, Room 124-8 Bldg 22, Room 124-8
![Page 2: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/2.jpg)
Sequential ComputersSequential Computers
"…sequential computers are approaching a "…sequential computers are approaching a fundamental physical limit on their potential fundamental physical limit on their potential power. Such a limit is the speed of light…"power. Such a limit is the speed of light…"
![Page 3: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/3.jpg)
``o'' are uniprocessors, ``+''denotes modestly parallel computers with 4--16 processors, ``x'' denotes massively parallel computers with 100s or 1000s of processors
![Page 4: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/4.jpg)
![Page 5: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/5.jpg)
ParallelismParallelism
Is and can be done easily in real life Is and can be done easily in real life (nature)(nature)
E.g., ants and pees colony.E.g., ants and pees colony.
So why not use it to solve human’s So why not use it to solve human’s problems!problems!
![Page 6: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/6.jpg)
Parallel ComputersParallel Computers
Supercomputers (high performance Supercomputers (high performance computers) are needed tocomputers) are needed to
Speedup sequential algorithmsSpeedup sequential algorithmsSolve large complex problems that are Solve large complex problems that are
almost not practically solvable.almost not practically solvable.
![Page 7: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/7.jpg)
Applications of ParallelismApplications of Parallelism Weather forecastingWeather forecasting Simulations (DNA, biomedical, flight control)Simulations (DNA, biomedical, flight control) Scientific ComputationsScientific Computations
Fourier transform, waveletsFourier transform, wavelets Applied linear algebra, matrix factorization and Applied linear algebra, matrix factorization and
multiplicationmultiplication Image and signal processingImage and signal processing Numerical analysisNumerical analysis PDEsPDEs Huge DatabasesHuge Databases Real-time systems with large live stream inputsReal-time systems with large live stream inputs
![Page 8: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/8.jpg)
![Page 9: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/9.jpg)
![Page 10: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/10.jpg)
Unfortunately .. Unfortunately ..
Not every algorithm can be parallelized Not every algorithm can be parallelized
““One woman can have a child in 9 monthsOne woman can have a child in 9 months
But 9 women can’t have a child in 1 month!”But 9 women can’t have a child in 1 month!”
![Page 11: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/11.jpg)
Courses in Parallelism Courses in Parallelism
Parallel ArchitectureParallel Architecture Parallel ProgrammingParallel Programming Parallel AlgorithmsParallel Algorithms Parallel Algorithms for scientific computationsParallel Algorithms for scientific computations Concurrent ProgrammingConcurrent Programming Distributed systemsDistributed systems Distributed AlgorithmsDistributed Algorithms Cluster ComputingCluster Computing
![Page 12: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/12.jpg)
Parallel ArchitectureParallel Architecture
Focuses on the design of hardwaresFocuses on the design of hardwaresE.g., processors, memory, E.g., processors, memory,
communications and communications and interconnections, ...etcinterconnections, ...etc
![Page 13: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/13.jpg)
Parallel ProgrammingParallel Programming Code parallel programs using parallel programming Code parallel programs using parallel programming
languages, e.g., PVM (Parallel Virtual Machine), MPI languages, e.g., PVM (Parallel Virtual Machine), MPI (Massage Passing Interface), OpenMP(Massage Passing Interface), OpenMP
Study: Parallel machine and programming models, Study: Parallel machine and programming models, Programming techniques (partitioning, pipelined Programming techniques (partitioning, pipelined computations, synchronous computations, load balancing computations, synchronous computations, load balancing and termination detection) and termination detection)
Textbooks: Textbooks: Parallel Computing, by Grama, Gupta, Karypis, Kumar, Addison-Parallel Computing, by Grama, Gupta, Karypis, Kumar, Addison-
Wesley, 2003. Wesley, 2003. Parallel Programming, by Wilkinson and Allen, Pearson-Prentice Parallel Programming, by Wilkinson and Allen, Pearson-Prentice
HallHall Fundamentals of Parallel Processing by Jordan and Alaghband, Fundamentals of Parallel Processing by Jordan and Alaghband,
Prentice Hall Prentice Hall
![Page 14: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/14.jpg)
Parallel AlgorithmsParallel Algorithms
Focuses on the design and analysis of Focuses on the design and analysis of parallel algorithmsparallel algorithms
Given a problemGiven a problempartition it into sub-problems, partition it into sub-problems, Distribute these sub-problems to the Distribute these sub-problems to the
processorsprocessorsSolve the communication problemSolve the communication problemCombine the solutionCombine the solution
![Page 15: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/15.jpg)
![Page 16: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/16.jpg)
Parallel Computation ModelsParallel Computation Models
Parallel algorithms are designed to run on Parallel algorithms are designed to run on a specific parallel computera specific parallel computer
But there many different practical models But there many different practical models of them (unlike the sequential computer)of them (unlike the sequential computer)
![Page 17: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/17.jpg)
Shared Memory ModelShared Memory Model
![Page 18: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/18.jpg)
Interconnection Network ModelInterconnection Network Model
Network structure could be array, tree, hypercube, or mesh based design
![Page 19: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/19.jpg)
Combinatorial Circuits ModelCombinatorial Circuits Model
![Page 20: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/20.jpg)
Other optionsOther options
Synchronous or asynchronous Synchronous or asynchronous computationcomputation
SIMD MIDMSIMD MIDM ........
![Page 21: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/21.jpg)
Massively Parallel Massively Parallel ComputerComputer
![Page 22: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/22.jpg)
Top 10 Supercomputers (2007)Top 10 Supercomputers (2007)
ManufacturManufacturer er
Computer Computer TFLOPS TFLOPS CountrCountry y
1 1 IBM IBM BlueGene/L eServer Blue BlueGene/L eServer Blue Gene Gene
280.6 280.6 USA USA
2 2 Sandia/Cray Sandia/Cray Red Storm Cray XT3 Red Storm Cray XT3 101.4 101.4 USA USA
3 3 IBM IBM BGW eServer Blue Gene BGW eServer Blue Gene 91.29 91.29 USA USA
4 4 IBM IBM ASC Purple eServer pSeries ASC Purple eServer pSeries p575 p575
75.76 75.76 USA USA
5 5 IBM IBM MareNostrum JS21 Cluster, MareNostrum JS21 Cluster, Myrinet Myrinet
62.63 62.63 Spain Spain
6 6 Dell Dell Thunderbird PowerEdge Thunderbird PowerEdge 1850, IB 1850, IB
53.00 53.00 USA USA
Google Map of TOP 100 HPCs
![Page 23: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/23.jpg)
![Page 24: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/24.jpg)
Intel Promised Teraflops chipIntel Promised Teraflops chip
Intel has introducedIntel has introduced
a prototype for a prototype for
A supercomputerA supercomputer
Chip capable of Chip capable of
teraflops Performance teraflops Performance
While While consuming consuming
Very little power. Very little power. It has 80 cores and consumes only 62 watts.It has 80 cores and consumes only 62 watts.
![Page 25: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/25.jpg)
Approaches for studyingApproaches for studying Parallel Algorithms Parallel Algorithms
1.1. Model Approach:Model Approach: divide the subject divide the subject according to the parallel computation according to the parallel computation models and for each study different models and for each study different parallel algorithms design techniques parallel algorithms design techniques and consider different application and consider different application problemsproblems
![Page 26: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/26.jpg)
Approaches for studyingApproaches for studying Parallel Algorithms Parallel Algorithms
2.2. Problem Approach:Problem Approach: Divide the subject Divide the subject according to the type of problems that can be according to the type of problems that can be solved by parallel algorithms likesolved by parallel algorithms like
Sorting and searchingSorting and searching Scientific computationScientific computation Graph problemsGraph problems Computational geometryComputational geometry Simulations, graphics and visionSimulations, graphics and vision Biomedical modelingBiomedical modeling
For each type we study design techniques for For each type we study design techniques for models of computations that are best suited.models of computations that are best suited.
![Page 27: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/27.jpg)
Approaches for studyingApproaches for studying Parallel Algorithms Parallel Algorithms
3.3. Models and Methods Approach: Models and Methods Approach: mix of mix of the above two approaches.the above two approaches.
Models:Models: Combinatorial circuits, Combinatorial circuits, interconnected networks, shared interconnected networks, shared memorymemory
Design Methods:Design Methods: prefix computation, list prefix computation, list ranking, divide and conquer, split and ranking, divide and conquer, split and plan, .... etcplan, .... etc
![Page 28: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/28.jpg)
TextbookTextbook
Parallel ComputationParallel Computation
By Selim AklBy Selim Akl
Supplementary Supplementary
Lecture notes, Lecture notes,
current researchcurrent research
paperspapers
![Page 29: ICS 556 Parallel Algorithms Ebrahim Malalla malalla@kfupm.edu.sa Office: Bldg 22, Room 124-8](https://reader035.vdocuments.site/reader035/viewer/2022062712/56649cb15503460f9497632e/html5/thumbnails/29.jpg)
AssessmentAssessment
4-5 assignments4-5 assignments2 exams2 examsPresentation and report based on Presentation and report based on
research papersresearch papers