![Page 1: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/1.jpg)
1
MTA SZTAKI
www.portal.p-grade.hu
Application development Application development on EGEE with P-GRADE Portal on EGEE with P-GRADE Portal
Gergely SiposGergely Sipos
[email protected]@sztaki.hu
![Page 2: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/2.jpg)
2
ContentsContents
• P-GRADE Portal in a nutshell
• Workflow development with the Portal
• Workflow execution with the Portal
• Scaling up to a parametric workflow
![Page 3: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/3.jpg)
4
P-GRADE Portal in a nutshellP-GRADE Portal in a nutshell
• General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow-based Grid applications – a tool for Grid orchestration
• Open source with GPL license: http://sourceforge.net/projects/pgportal/ • Extends GridSphere
– Easy to expand with new portlets (e.g. application-specific portlets)– Easy to tailor to end-user needs
• Grid services supported by the portal:
Service EGEE grids Globus grids
Job execution Computing Element GRAM
File storage Storage Element GridFTP server
Certificate management MyProxy
Information system BDII MDS-2, MDS-4
Brokering Workload Management System GTbroker
Job monitoring Mercury
Workflow & job visualization PROVE
Solves Grid interoperability problem at the workflow level
![Page 4: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/4.jpg)
5
What is a P-GRADE Portal workflow?
• a directed acyclic graph where– Nodes represent jobs (batch
programs to be executed on a computing element)
– Ports represent input/output files the jobs expect/produce
– Arcs represent file transfer operations
• semantics of the workflow:– A job can be executed if all
of its input files are available
![Page 5: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/5.jpg)
6
Two levels of parallelism by a workflow
• The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems
• Semantics of the workflow enables two levels of parallelism:
The job can be a parallel program
– Parallel execution inside a workflow node– Parallel execution among workflow nodes
Multiple jobs can run parallel
![Page 6: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/6.jpg)
7
25 x
10 x25 x 5 x
Forecasting dangerous weather situations (storms, fog, etc.), crucial task in the protection of life and property
Processed information:surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models
Requirements:•Execution time < 10 min•High resolution (1km)
Ultra-short range weather forecast Ultra-short range weather forecast (Hungarian Meteorology Service)(Hungarian Meteorology Service)
![Page 7: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/7.jpg)
8
The typical user scenarioThe typical user scenarioPart 1 - development phasePart 1 - development phase
Certificate servers
Portalserver
Gridservices
START EDITOR
OPEN & EDIT or DEVELOP WORKFLOW
SAVE WORKFLOW
![Page 8: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/8.jpg)
9
Certificate servers
Portalserver
Gridservices
TRANSFER FILES, SUBMIT JOBS
DOWNLOAD (SMALL)
RESULTS
DOWNLOAD (SMALL)
RESULTS
The typical user scenarioThe typical user scenarioPart 2 - execution phasePart 2 - execution phase
VISUALIZE JOBS and
WORKFLOW PROGRESS
MONITOR JOBS
DOWNLOAD PROXY CERTIFICATES
![Page 9: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/9.jpg)
10
The typical user scenario The typical user scenario Development phase:Development phase:
Certificate servers
Portalserver
Gridservices
START EDITOR
OPEN & EDIT or DEVELOP or IMPORT
WORKFLOW
SAVE WORKFLOW
![Page 10: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/10.jpg)
11
Workflow developmentWorkflow developmentOpening the workflow editorOpening the workflow editor
The editor is a Java Webstart application
download and installation is only one click!
![Page 11: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/11.jpg)
12
Workflow Workflow EditorEditorDefining the graphDefining the graph
• The aim is to define a DAG of batch jobs:
1. Drag & drop components:jobs and ports
2. Define their properties
3. Connect ports by channels (no cycles, no loops, no conditions)
![Page 12: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/12.jpg)
13
Workflow Workflow EditorEditorProperties of a jobProperties of a job
Properties of a job:• Binary executable• Type of executable• Number of required
processors• Command line parameters• The resource to be used
for the execution:• Grid/VO• (Computing element)
![Page 13: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/13.jpg)
14
Direct resource selection:Direct resource selection:Which computing element to use? Which computing element to use?
The information system portlet
queries BDII and GIIS servers
I still don’t know which resource to
use!
![Page 14: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/14.jpg)
15
Automatic resource selectionAutomatic resource selection
1. Select a broker Grid/VO for the job(e.g. GILDA_LCG2_broker)
2. (Describe the ranks & requirements of the job in JDL)
3. The portal will use the broker to find the best resource for the job!
![Page 15: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/15.jpg)
16
Workflow Workflow EditorEditorDefining broker jobsDefining broker jobs
Select a Grid with broker!(*_BROKER)
Ignore the resource field!
If default JDL is not sufficient use the built-in JDL editor!
![Page 16: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/16.jpg)
17
Workflow Workflow EditorEditorBuilt-in JDL editorBuilt-in JDL editor
JDL look at the gLite Users’ manual!
![Page 17: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/17.jpg)
18
Workflow Workflow EditorEditorDefining input-output filesDefining input-output files
File propertiesType: input: the job reads output: the job generatesFile type: local: comes from my desktop remote: comes from an SEFile: location of the fileInternal file name: Executable reads the file in this name – fopen(“file.in”, …)File storage type (output files only): Permanent: final result Volatile: only data channel
![Page 18: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/18.jpg)
19
• Client side location:result.dat
• LFC logical file name lfn:/grid/gilda/sipos/11-04_-_result.dat
Local fileLocal file
Remote fileRemote file
How to refer to an I/O file?How to refer to an I/O file?
• Client side location:c:\experiments\11-04.dat
• LFC logical file name lfn:/grid/gilda/sipos/11-04.dat
Input file Output file
![Page 19: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/19.jpg)
20
Local vs. remote filesLocal vs. remote files
Portalserver
Gridservices
Computing elements
Storage elements
REMOTE INPUTFILES
REMOTE OUTPUT
FILES
LOCAL INPUT FILES
& EXECUTABLES
LOCAL OUTPUT
FILES
LOCAL INPUT FILES
& EXECUTABLES
LOCAL OUTPUT
FILES
Only the permanent
files!
Your jobs can access storage files directly too!
![Page 20: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/20.jpg)
23
Workflow Workflow EditorEditorSaving the workflowSaving the workflow
Workflow is defined!
Let’s execute it!
![Page 21: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/21.jpg)
24
1. Download proxies2. Submit workflow3. Observe workflow progress4. If some error occurs correct the graph5. Download result
Main steps
Executing workflows Executing workflows with the P-GRADE with the P-GRADE
PortalPortal
![Page 22: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/22.jpg)
25
The typical user scenarioThe typical user scenarioExecution phase – step 1:Execution phase – step 1:
Certificate servers
Portalserver
Gridservices
DOWNLOAD PROXY CERTIFICATES
![Page 23: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/23.jpg)
26
MyProxy interaction in P-GRADE: MyProxy interaction in P-GRADE: Certificate ManagerCertificate Manager
Certificates portletCertificates portlet
• To start your session on the Grid you must create a proxy certificate on the portal server
• “Certificates” portlet:
• to upload a proxy into MyProxy servers
• to download a proxy from MyProxy into the portal server
![Page 24: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/24.jpg)
27
Certificate ManagerCertificate ManagerDownloading a proxyDownloading a proxy
1. MyProxy server access details:• Hostname• Port number• User name (from upload)• Password (from upload)
2. Proxy parameters:• Lifetime• Comment
3. Grid association
![Page 25: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/25.jpg)
29
Certificate ManagerCertificate Managerassociating the proxy with a gridassociating the proxy with a grid
This operation displays the details of the certificate and the list of available Grids (defined by portal administrator)
![Page 26: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/26.jpg)
30
Certificate ManagerCertificate Managerbrowsing proxiesbrowsing proxies
Multiple proxies can be available on the portal server at the same time!
SEE-GRID CEs and SEsHUNGRID CEs and SEs
![Page 27: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/27.jpg)
31
Certificate servers
Portalserver
Gridservices
TRANSFER FILES, SUBMIT JOBS
The typical user scenarioThe typical user scenarioExecution phase - step 2: Execution phase - step 2:
![Page 28: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/28.jpg)
32
Workflow ManagementWorkflow Management(workflow portlet)(workflow portlet)
• The portlet presents the status, size and output of the available workflow in the “Workflow” list
• It has a Quota manager to control the users’ storage space on the server• The portlet also contains the “Abort”, “Attach”, “Details”, “Delete” and
“Delete all” buttons to handle execution of workflows• The “Attach” button opens the workflow in the Workflow Editor• The “Details” button gives an overview about the jobs of the workflow
![Page 29: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/29.jpg)
33White/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 30: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/30.jpg)
34White/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 31: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/31.jpg)
35White/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 32: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/32.jpg)
36White/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 33: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/33.jpg)
37
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
White/Red/Green color means the job is initialised/running/finished
![Page 34: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/34.jpg)
38
Certificate servers
Portalserver
Gridservices
The typical user scenarioThe typical user scenarioExecution phase – step 3:Execution phase – step 3:
VISUALIZE JOBS and
WORKFLOW PROGRESS
MONITOR JOBS
![Page 35: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/35.jpg)
39
- The portal monitors and visualizes workflow progress
On-Line Monitoring both at theOn-Line Monitoring both at the workflow and job levels workflow and job levels (workflow portlet)(workflow portlet)
![Page 36: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/36.jpg)
40
Rescuing a failed workflow 1.Rescuing a failed workflow 1.
A job failed during workflow execution
Read the error log to know why
![Page 37: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/37.jpg)
41
Rescuing a failed workflow 2.Rescuing a failed workflow 2.
Map the failed job onto a different
CE or download a new proxy for it.
Don’t touch the finished jobs!
The execution can continue
from the point of failure!
![Page 38: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/38.jpg)
42
Rescuing a failed workflow 3.Rescuing a failed workflow 3.
![Page 39: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/39.jpg)
43
Rescuing a failed workflow 4.Rescuing a failed workflow 4.
Resume workflow by the Rescue
button
![Page 40: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/40.jpg)
44
Logs provided for each jobLogs provided for each job
![Page 41: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/41.jpg)
45
Analysis of the logAnalysis of the log• 2008.01.09 09:32:19 - Proxy with VOMS extensions created for VO "voce"
with accounting group "".• 2008.01.09 09:32:19 - Job submission in progress...• 2008.01.09 09:32:23 - Job has been submitted successfully!• 2008.01.09 09:32:23 - Job identifier is:• "https://skurut1.cesnet.cz:9000/mD_8VzPhm8AmIToTJKtigg"• 2008.01.09 09:32:26 - EGEE job's status has changed to "Waiting" (host is ).• 2008.01.09 09:33:00 - EGEE job's status has changed to "Ready" (host is ce1-
egee.srce.hr).• 2008.01.09 09:35:46 - EGEE job's status has changed to "Waiting" (host is
egee-ce.grid.niif.hu).• 2008.01.09 09:36:19 - EGEE job's status has changed to "Ready" (host is
ce.cyf-kr.edu.pl).• 2008.01.09 09:36:53 - EGEE job's status has changed to "Waiting" (host is• ce.cyf-kr.edu.pl).• 2008.01.09 09:37:26 - EGEE job's status has changed to "Done" (host is• egee-ce.grid.niif.hu).• 2008.01.09 09:37:26 - Job found to be finished. Checking again if this is• really the case.• 2008.01.09 09:38:03 - EGEE job's status has changed to "Ready" (host is• egee-ce1.gup.uni-linz.ac.at).
![Page 42: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/42.jpg)
46
Fault-tolerance by P-GRADE portalFault-tolerance by P-GRADE portal
• 09:33: the broker assigned the job to a site: ce1-egee.srce.hr• 09:35: The broker moved the job to another site: egee-ce.grid.niif.hu• 09:36: Again the broker moved the job to another site: ce.cyf-kr.edu.pl• 09:37: The broker indicated that the job is Done, but .• 09:38: ... It turned out that the job was not finished (Done - Failed
status), only• it was moved to another site: egee-ce1.gup.uni-linz.ac.at• 09:39: Again the broker moved the job to another site: ares02.cyf-
kr.edu.pl• 09:39: Again the broker moved the job to another site: ce.cyf-kr.edu.pl• 09:40: After trying 10 different sites the VOCE broker gave it up and
aborted the job (the Shallow RetryCount was set for 10):
• 2008.01.09 09:40:16 - The job has been aborted!
![Page 43: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/43.jpg)
47
Fault-tolerance by P-GRADE portalFault-tolerance by P-GRADE portal
• Our fault-tolerant portal did not give it up:• 2008.01.09 09:40:16 - The job can be submitted again (try 1 out of 3,• excluding host(s): ce.cyf-kr.edu.pl)• 2008.01.09 09:40:17 - Proxy with VOMS extensions created for VO
"voce" with• accounting group "".• 2008.01.09 09:40:17 - Job submission in progress...• 2008.01.09 09:40:27 - Job has been submitted successfully!• 2008.01.09 09:40:27 - Job identifier is:• "https://skurut1.cesnet.cz:9000/o22BTVqQsvwzj2wn5KP8_A"• 2008.01.09 09:40:30 - EGEE job's status has changed to "Waiting" (host
is ).• 2008.01.09 09:41:04 - EGEE job's status has changed to "Ready" (host is• eszakigrid66.inf.elte.hu).
![Page 44: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/44.jpg)
48
• 2008.01.09 09:41:37 - EGEE job's status has changed to "Scheduled" (host is• eszakigrid66.inf.elte.hu).• 2008.01.09 09:44:57 - EGEE job's status has changed to "Done" (host is• eszakigrid66.inf.elte.hu).• 2008.01.09 09:44:57 - Job found to be finished. Checking again if this is• really the case.• 2008.01.09 09:45:34 - EGEE job's status has changed to "Waiting" (host is• eszakigrid66.inf.elte.hu).• 2008.01.09 10:06:06 - The job's status hasn't changed for 20 minutes,• resubmitting...
• It is a quite frequently occurring problem in EGEE-like grids that the broker leaves jobs stuck in CEs. queues.) In such case the portal automatically kills the job on this site and resubmits it to the broker.
• 2008.01.09 10:06:06 - Proxy with VOMS extensions created for VO "voce" with accounting group "".
• 2008.01.09 10:06:06 - Job submission in progress...• 2008.01.09 10:06:12 - Job has been submitted successfully!
• 10:10: The job successfully finished with exit code 0 on site: ce.ui.savba.sk
Fault-tolerance by P-GRADE portalFault-tolerance by P-GRADE portal
![Page 45: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/45.jpg)
49
Certificate servers
Portalserver
Gridservices
DOWNLOAD (SMALL)
RESULTS
DOWNLOAD(SMALL)
RESULTS
The typical user scenarioThe typical user scenarioExecution phase – step 5Execution phase – step 5
![Page 46: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/46.jpg)
50
Downloading the results…Downloading the results…
![Page 47: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/47.jpg)
51
File management portletFile management portlet
![Page 48: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/48.jpg)
52
Scaling up a workflow to a Scaling up a workflow to a parameter study with P-GRADEparameter study with P-GRADE
Complete workflow
Files in an LFC directory (e.g. /grid/gilda/sipos/myoutputs)
Files in an LFC directory (e.g. /grid/gilda/sipos/myinputs)
![Page 49: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/49.jpg)
53
Turning a WF into a parameter studyTurning a WF into a parameter study
By turning at least one of the open input ports into a “PS Input port” the WF is turned into a Parameter Study
![Page 50: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/50.jpg)
55
PS Input Port of Simple PSPS Input Port of Simple PS
Remote file Directory instead of
FILE reference
Do not use the prefix lfn:
if the directory isEGEE Grid file
catalogue
![Page 51: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/51.jpg)
56
Simple PS Activity 2: placement of resultSimple PS Activity 2: placement of result
Properties of the VO File Catalog
Menu item PS Properties can be called within the Workflow menu
The Output directory will contain the set of individual compressed files.Each compressed file contains the outputs of an element Workflow have been elaborated over an item of the PS Input Set
One SE of the chosen VO
![Page 52: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/52.jpg)
57
Executing PS workflowsExecuting PS workflows
PS Details for parameter sweep
workflows applications
![Page 53: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/53.jpg)
58
Workflow Manager List Workflow Manager List PS Details viewPS Details view showing eWF-s showing eWF-s
New, middle level list to render the details of a PS Workflow
Statistics shows the progress of the elaboration of the whole PS
The eWorkflow buffer list shows the state of the Workflows being processed.
![Page 54: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/54.jpg)
59
DetailsDetails view of the view of the eWFeWF Ax_EQU_B_voce_PS.6Ax_EQU_B_voce_PS.6
Job level details of an eWorkflow
See, that the button Attach is missing as there is not to
much importance to access the WE until the eWorkflow list
is exhausted
![Page 55: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/55.jpg)
60
Advanced PS WFs in Advanced PS WFs in P-GRADE PortalP-GRADE Portal
![Page 56: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/56.jpg)
61
Advanced parameter studiesAdvanced parameter studies
Generator component(s)
Initial input data
Generate orcut input into smaller pieces
Collector component(s)
Aggregate result
Files in an LFC directory (e.g. /grid/gilda/sipos/myinputs)
Files in an LFC directory (e.g. /grid/gilda/sipos/myoutputs)
Complete workflow
![Page 57: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/57.jpg)
62
Concept of advanced parameter Concept of advanced parameter study workflowsstudy workflows
GEN
SEQ
COLL
SEQSEQSEQ
Parameter study part
Collector part evaluates and
integrates the results
Generator part generates the
input parameter space
![Page 58: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/58.jpg)
63
Parameter generatorParameter generator
Generator can be attached to any parameter input port
Generator can be• Auto generator: to generate text files• Custom generator: to generate any content
Generated files are moved into SE by the portal
![Page 59: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/59.jpg)
64
Definition Window of Auto Generator JobDefinition Window of Auto Generator Job
User defines the template of the text file
User puts keys into the template
User defines values for the keys• Integer number• Real number• Custom set• …By clicking on
a key the definition
window for this key is opened.
![Page 60: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/60.jpg)
65
(Auto) Generator Attribute Editor for SE (Auto) Generator Attribute Editor for SE definitiondefinition
Attribute Editor defines the properties of remote files created by the Generator:
1. Storage Element must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator
![Page 61: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/61.jpg)
66
Detailed view of a PS workflowDetailed view of a PS workflow
Workflow instances
Overall statistics of workflow instances
Collector job(s)
Generator job(s)
![Page 62: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/62.jpg)
67
• Share your workflows or results with other researchers!• Migrate your application from one portal into another!
Additional featuresAdditional features
• Workflows and traces can be exported from the portal server onto your client machine
• Workflows and traces can be imported into the Portal
![Page 63: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/63.jpg)
68
Workflow/trace export/importWorkflow/trace export/import
To export a workflow from the portal onto your machine
To delete every unnecessary files of the workflow
To delete trace/output of the workflow
(if any)
To delete trace/output of the workflow
(if any)
![Page 64: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/64.jpg)
70
Hands-onHands-on
![Page 65: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/65.jpg)
72
![Page 66: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/66.jpg)
73
Multi-VO service portalMulti-VO service portal
![Page 67: 1 MTA SZTAKI pgportal@lpds.sztaki.hu Application development on EGEE with P-GRADE Portal Gergely Sipos sipos@sztaki.hu](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d155503460f949eb578/html5/thumbnails/67.jpg)
75
Thank you!Thank you!
www.lpds.sztaki.hu/[email protected]
Learn once, use everywhereDevelop once, execute anywhere