http:// // ngs computation services: apis and
TRANSCRIPT
![Page 1: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/1.jpg)
http://www.ngs.ac.ukhttp://www.grid-support.ac.uk
http://www.eu-egee.org/http://www.pparc.ac.uk/http://www.nesc.ac.uk/
NGS computation services: APIs and Parallel Jobs
Richard [email protected]
NGS Induction – Rutherford Appleton Laboratory, 2nd / 3rd November 2005
![Page 2: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/2.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
3
Overview
• The C and Java API’s to the low-level tools• Using multiple processors
![Page 3: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/3.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
4
Job submission so far
GLOBUS, etc.
Command-line interfaces
User’s Interface to the grid
GLOBUS, etc.
![Page 4: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/4.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
5
Application-specific tools
GLOBUS, etc.
Command-line interfaces
Application Specificand / orHigher generic tools
e.g. BRIDGES
User’s Interface to the grid
![Page 5: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/5.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
6
GLOBUS, etc.
Application-specific tools
Command-line interfaces
User’s Interface to the grid
API’s:
•Java
•C
•…Application Specificand / orHigher generic tools
e.g. BRIDGES
![Page 6: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/6.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
7
Available API’s
• C http://www.globus.org/developer/api-reference.html
• “Community Grid” CoG http://www.cogkit.org/– Java, Python, Matlab
– (very limited functionality on Windows – no GSI)
![Page 7: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/7.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
8
Non-communicating
Processes
UI
Internet
Head processors of clusters
Worker processors of clusters
Globus_job_submit
Processes run without any communication between them
![Page 8: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/8.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
9
Communicating Processes
UI
Internet
Head processors of clusters
Worker processors of clusters
Globus_job_submit
Processes send messages to each other – Must run on same cluster
![Page 9: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/9.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
10
Communicating Processes
UI
Internet
Head processors of clusters
Worker processors of clusters
MPI
Processes send messages to each other – Must run on same cluster
![Page 10: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/10.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
11
Modes of Parallelism
– Communicating processes: on NGS, you run one globus-job-submit command – but need to code and build program so it is parallelised - MPI
– Communicating processes: on NGS, multiple executables run from a script on the UI
The NGS nodes open these routes to you – but you have to do a bit of work! (Grid is not magic!...)
![Page 11: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/11.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
12
MPI notes• How could the task be split into sub-tasks?
– By functions that could run in parallel??!– By sending different subsets of data to different processes?
More usual ! Overheads of scatter and gather• Need to design and code carefully: be alert to
– sequential parts of your program (if half your runtime is sequential, speedup will never be more than 2)
– how load can be balanced (64 processes with 65 tasks will achieve no speedup over 33 processes)
– Deadlock! • MPI functions are usually invoked from C, Fortran programs,
but also Java• Several example patterns are given in the practical.
Many MPI tutorials are on the Web!
![Page 12: Http:// // NGS computation services: APIs and](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515ed43550346d46f8b5237/html5/thumbnails/12.jpg)
NGS Induction, RAL 2nd/3rd Nov 2005 – Computation Services – Richard Hopkins
13
Practical
1. C API Example
2. Java API usage
3. Concurrent processing
4. MPI example
• Follow link from http://homepages.nesc.ac.uk/~gcw/NGS/