![Page 1: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/1.jpg)
1
P-GRADE Portal: P-GRADE Portal: a workflow-oriented generic a workflow-oriented generic
application development portalapplication development portal
Peter KacsukMTA SZTAKI, Hungary
Univ. of Westminster, UK
![Page 2: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/2.jpg)
2
User concerns of Grid User concerns of Grid systems systems
• How to cope with the variety of Grid systems? (How to move from LCG-2 to gLite?)
• How to develop/create new Grid applications?– To use workflows (DAG)– To use MPI codes
• How to execute Grid applications in a fault-tolerant way?• How to observe (and debug) the application execution in the Grid?• How to tackle performance issues?• How to port legacy applications
– to Grid systems– between Grid systems?
• How to interoperate among Grids, how to execute Grid applications over several Grids in a transparent way? (see Earth Science slides)
• The goal of this talk is to show how a portal, like P-GRADE, can solve all these problems at a high abstraction level
![Page 3: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/3.jpg)
3
Properties of the Properties of the P-GRADE PortalP-GRADE Portal
• General purpose, graphical, workflow-oriented Grid portal
• Supports the development and execution of workflow-based Grid applications
• Components of the workflows can be– Sequential jobs– Parallel jobs (MPI, PVM)– Legacy code (GEMLCA) services
• Enables the exploitation of two levels of parallelism• Solves the interoperability of Grids at the workflow
level
![Page 4: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/4.jpg)
4
Two levels of parallelism by the Two levels of parallelism by the P-GRADE workflowP-GRADE workflow
• Semantics of the workflow enables two levels of parallelism:
The job can be a parallel program
– Parallel execution inside a workflow node– Parallel execution among workflow nodes
Multiple jobs can run parallel
• The P-GRADE Portal workflow concept enables the efficient parallelization of complex problems
![Page 5: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/5.jpg)
5
EGEE Grid
UK NGS
P-GRADE-Portal
London Rome
Athens
Solving Grid interoperability by Solving Grid interoperability by P-GRADE PortalP-GRADE Portal
Different jobs can be executed in different grids
![Page 6: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/6.jpg)
6
Properties of Properties of P-GRADE Grid PortalP-GRADE Grid Portal
• Grid services supported by the portal:– MyProxy – proxy credential management– GridFTP – file transfer– GT2/GT3/GT4 GRAM – job execution– Mercury – parallel job monitoring– PROVE – workflow & job execution visualization– BDII and MDS – obtain information about resources– LCG-2 broker – resource selection– GEMLCA – invoke legacy codes
• GridSphere based – Easy to expand with new portlets– Easy to tailor to end-user needs
• Support for grid interoperability at workflow level
![Page 7: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/7.jpg)
7
Proxy management
P-GRADE portal in a P-GRADE portal in a nutshellnutshell
Definition of Grid resources
Workflow creation
Job mapping to Grid resources
Workflow management and
execution visualization
![Page 8: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/8.jpg)
8
User concerns of Grid User concerns of Grid systemssystems
• How to cope with the variety of Grid systems?• How to develop/create new Grid applications?• How to execute Grid applications in a fault-tolerant
way?• How to observe the application execution in the Grid?• How to tackle performance issues?• How to port legacy applications
– to Grid systems– between Grid systems?
• How to execute Grid applications over several Grids in a transparent way?
![Page 9: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/9.jpg)
9
Workflow creation at the Workflow creation at the client machineclient machine
Certificate server
Portalserver Remote
Clusters to be
controlled
WF EDITOR (save / upload)
WF EDITOR (open)
CERTIFICATE (upload)
CERTIFICATE (download)
WORKFLOW MANAGER (submit)
Workflow (result)
WORKFLOW MANAGER (output)
![Page 10: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/10.jpg)
10
Workflow Workflow EditorEditor: Grid aware workflow : Grid aware workflow mappingmapping
![Page 11: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/11.jpg)
11
Visualization of monitoring Visualization of monitoring system information system information
![Page 12: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/12.jpg)
12
Non-Grid aware (abstract) Non-Grid aware (abstract) workflow mappingworkflow mapping
Can be used if the selected Grid (eg. LCG-2) has a broker:
1. Describe the requirements of the job
2. Select a Grid with broker
3. The workflow manager will contact the broker to find the best resource for your job
![Page 13: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/13.jpg)
13
Workflow Workflow EditorEditor extension with JDL extension with JDL
Select an LCG-2 based Grid!
Don’t select any resource!
Define job requirements with the “Job Description
Language”!
![Page 14: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/14.jpg)
14
User concerns of Grid systemsUser concerns of Grid systems
• How to cope with the variety of Grid systems?• How to develop new Grid applications?• How to execute Grid applications in a fault-tolerant
way?• How to observe the application execution in the Grid?• How to tackle performance issues?• How to port legacy applications
– to Grid systems– between Grid systems?
• How to execute Grid applications over several Grids in a transparent way?
![Page 15: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/15.jpg)
15
Workflow execution Workflow execution support by the portalsupport by the portal
Certificate server
Portalserver Remote
Clusters to be
controlled
EDITOR (save|upload)
EDITOR (open)
CERTIFICATE (upload)
CERTIFICATE (download)
WORKFLOW MANAGER (submit)
Workflow (result)
WORKFLOW MANAGER (output)
![Page 16: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/16.jpg)
16White/Red/Green color means the job is initialised/running/finished
Workflow Manager PortletWorkflow Manager Portlet
- Displays the list of jobs and their status
- The current status of the jobs are represented by colors
- Provides access to their logs and outputs, and visualizes them
![Page 17: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/17.jpg)
17
Rescuing a failed workflowRescuing a failed workflow
A job failed during workflow execution
Read the error log to know why
Attach the workflow editor
![Page 18: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/18.jpg)
18
Rescuing a failed workflowRescuing a failed workflow
Modify the failed job or download a new proxy for it.
Don’t touch the finished jobs!
The execution can continue where it was
stopped!
![Page 19: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/19.jpg)
19
User concerns of Grid systemsUser concerns of Grid systems
• How to cope with the variety of Grid systems?• How to develop new Grid applications?• How to execute Grid applications in a fault-tolerant
way?• How to observe the application execution in the Grid?• How to tackle performance issues?• How to port legacy applications
– to Grid systems– between Grid systems?
• How to execute Grid applications over several Grids in a transparent way?
![Page 20: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/20.jpg)
20
On-Line MonitoringOn-Line Monitoring
![Page 21: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/21.jpg)
21
Job execution Job execution visualization visualization
![Page 22: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/22.jpg)
22
User concerns of Grid systemsUser concerns of Grid systems
• How to cope with the variety of Grid systems?• How to develop new Grid applications?• How to execute Grid applications in a fault-tolerant
way?• How to observe the application execution in the Grid?• How to tackle performance issues?• How to port legacy applications
– to Grid systems– between Grid systems?
• How to execute Grid applications over several Grids in a transparent way?
![Page 23: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/23.jpg)
23
GEMLCA legacy code GEMLCA legacy code architecture objectivesarchitecture objectives
• To deploy legacy code applications as Grid services without reengineering the original code and minimal user effort
• To create Grid workflows where components can also be legacy code applications
• To make these functions available from a Grid Portal
GEMLCA
GEMLCA &
P-GRADE Portal
Integration
![Page 24: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/24.jpg)
24
Combine legacy codeswith new codesinside the same workflow!
Code Code invocationinvocation
Code Code invocationinvocation
Code Code invocationinvocation
Job Job submissionsubmission
Job Job submissionsubmission
Combining legacy and Combining legacy and non-legacy componentsnon-legacy components
Use successfully executed new code as legacy code
![Page 25: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/25.jpg)
25
Legacy code registration pageLegacy code registration page
![Page 26: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/26.jpg)
26
GEMLCA client in a nutshell: GEMLCA client in a nutshell: Traffic Simulation WorkflowTraffic Simulation Workflow
Workflow creation
Definition of legacy code service properties
Monitoring and execution visualization
![Page 27: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/27.jpg)
27
References of P-GRADE portalReferences of P-GRADE portal
• Official portal of– SEE-GRID infrastructure
– HUNGRID infrastructure
• P-GRADE portal is available as service for:– VOCE– UK National Grid Service
• Under preparation for– Croatian Grid– EGRID (Italy)– GridIreland
![Page 28: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/28.jpg)
28
How to access P-GRADE Portal? How to access P-GRADE Portal?
• If you are interested in using P-GRADE Portal:– Take a look at www.lpds.sztaki.hu/pgportal– If you are a user, get an account for one of its
production installations:• HUNGrid portal – SZTAKI• VOCE portal – SZTAKI• SEEGRID portal – SZTAKI• UK NGS portal – University of Westminster
– If you are the administrator of a VO or Grid, ask SZTAKI to install P-GRADE Portal for your VO or Grid:
• EGRID VO portal – ICTP• Croatian Grid portal – Boskovic Institute
![Page 29: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/29.jpg)
29
New, planned features New, planned features
• Parameter study support at– Job level– Workflow level
• New types of parallelism (collaboration with Johan Montagnat):– Pipeline parallelism (e.g. Planck VO and biomed community need it)– multi-thread parallelism
• Collaborative portal (collaboration with Univ. of Reading)• Automatic testing of Grid services and resources from the
portal (collaboration with Univ. of Westminster)– Intelligent brokering– Intelligent error messages– Automatic handling of error situations
• New application-oriented portlets (collaboration with E-Grid and Croatian Grid)
• Interactive workflow development and debugging support
![Page 30: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/30.jpg)
30
Final conclusions Final conclusions
• Users should access any Grids transparently by Grid portals• Every Grid should be accessed via different portals in order to provide a
choice for the users– In the case of EGEE:
• Genius Portal• P-GRADE Portal
– In the case of UK NGS:• Daresbury Portal• P-GRADE Portal
• P-GRADE Portal provides the following principles:– Learn once, use everywhere– Develop once, execute anywhere
• We are ready to collaborate with any team and support any EGEE application by the portal, and extend it with the special needs of the application
www.lpds.sztaki.hu/pgportal/
![Page 31: 1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK](https://reader035.vdocuments.site/reader035/viewer/2022070404/56649f355503460f94c52957/html5/thumbnails/31.jpg)
31
Grid-Enabling Legacy Applications and Grid-Enabling Legacy Applications and Supporting End Users WorkshopSupporting End Users Workshop
within the framework of thewithin the framework of the15th IEEE International Symposium on 15th IEEE International Symposium on
High Performance Distributed High Performance Distributed ComputingComputingHPDC’15HPDC’15
Paris, FranceParis, FranceJune 19-23, 2006June 19-23, 2006
• IMPORTANT DATES
• Paper Abstract Submission: February 06, 2006
• Full Papers Submission: February 13, 2006
• Notice of Acceptance: March 27, 2006
• Final Manuscript Due: April 24, 2006
• Workshop: June 20, 2006