SHIWA andCoarse-grained Workflow

InteroperabilityGabor Terstyanszky, University of Westminster

Summer SchoolBudapest

02-06 July 2012

SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2

Workflow Interoperability Challenge

SSP Objectives and ServicesObjectives:• To create an environment which enables seamless execution of

workflows of different workflow systems through workflow interoperability

Services:• To support the whole workflow lifecycle: editing, uploading, browsing

downloading and executing workflows • To provide coarse- and fine-grained workflow interoperability solution • To offer Distributed Computing Infrastructure interoperability solution• To provide desktop computer and portal interfaces to manage

workflowsKey actors:• researchers workflow engine developers

workflow developers

3

4

Workflow description abstract workflow•plain text which describes the inputs and outputs and explains what the workflow doesWorkflow implementations concrete workflow• It contains all implementation files or references to these files (via eg. URLs) and also holds other

data/metadata necessary to run the workflow on its associated workflow engine.Workflow configurations• It contains pre-defined input files and parameters or references to these files and parameters and other

data/metadata of a workflow.Workflow engines• It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references

to them

SHIWA Data Structure

5

WF Submission client

Submission Service

Workflow Engine B

Workflow of Workflow Engine B

DCI

Workflow Engine A

Coarse-Grained Interoperability: Single Non-native Workflow

6

J2 WF4

J1

J3

Submission client

Submission Service

Workflow Engine B

Workflow of Workflow Engine B

Workflows of Workflow Engine A

Coarse-Grained Interoperability: Meta-Workflow

DCI

• native workflows: J1, J2, J3• non-native workflows: WF4 - black boxes which are managed as legacy code applications

Workflow Engine A

7

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCI

Kepler WEGWES WE

Taverna WE

Triana WE

MOTEUR WE

ASKALON WE

PGRADE WE

SHIWA Portal

GEMLCA admin

SHIWA Science Gateway

WE1 WEp

GEMLCA Repository

WF1 WFm

SHIWA Repository

WF1 WFn

WS-PGRADE Workflow

engine

WS-PGRADE Workflow

editor

GEMLCA Service

GEMLCA with GIB

SHIWA Proxy Server

Proxy Server

Galaxy WE

Pegasus WE

ProActive WE

Workflow Engines

SSP: CGI Infrastructure

SHIWA Science Gateway Resourcesnative WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.4.1WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployedsubmitter GEMLCA with GIB WEs to ARC, gLite, Globus Unicore DCIsproxy management SHIWA Proxy Server

88

SHIWA Portal: Abstract Workflow

Jobs

Ports

9

SHIWA Portal: Configuring Workflow

10

SHIWA Portal: Executing Workflow

11

SHIWA Repository: Browse View

domain

search

12

inputsoutputs

data sets

implementation

SHIWA Repository: Browse View

1313

Browse Workflows

Browse Implementations

Search

Advanced Search

Sort

SHIWA Repository: Table View

1414

SHIWA Repository: Execution Support

Execution data

15

Taverna workflow

SHIWA Portal: Legacy Code Explorer

16

SHIWA Portal: Repository Portlet

Workflow Repository

17

CGI Developer Scenario: Specifying WE

18

CGI Developer Scenario: Specifying WF through portal

19

CGI Developer Scenario: Specifying WF through desktop

2020

CGI User Scenario: Native WE

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1 WEp

GEMLCA Repository

WE + WF

WF1 WFm

GEMLCA with GIB

WF listWS-PGRADE

Workflow engine

WS-PGRADE Workflow

editor

user

SHIWA Proxy Server

Proxy Server

step 1search WF

step 2 edit WF

Kepler WEGWES WE

Taverna WE

Triana WE

MOTEUR WE

ASKALON WE

PGRADE WE

Galaxy WE

Pegasus WE

ProActive WE

Workflow Engines

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCI

step 7 run WFstep 6

retrieve proxy

step 5retrieve WF

step 4submit WF

step 3retrieve WF data

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1 WEp

GEMLCA Repository

WF1 WFm

GEMLCA with GIBWS-PGRADE

Workflow Engine

WS-PGRADE Workflow

Editor WE + WF

user

GEMLCA Client

MOTEURWorkflow

Engine

MOTEUR Workflow

editor

GEMLCA UI

SHIWA Proxy Server

Proxy Server

CGI User Scenario: Non-native WE

step 1 search WF

step 2 edit WF

step 3 submit WFstep 6

retrieve proxy

step 5 retrieve WF

step 7 run WF

Kepler WEGWES WE

Taverna WE

Triana WE

MOTEUR WE

ASKALON WE

PGRADE WE

Galaxy WE

Pegasus WE

ProActive WE

Workflow Engines

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCI

step 4 forward WF

22

SHIWA Repository

SHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1 WEp

GEMLCA Repository

WF1 WFm

GEMLCA with GIBWS-PGRADE

Workflow Engine

WS-PGRADE Workflow

Editor

SHIWA Proxy Server

Proxy Server

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCI

SHUWA

Desktop

WE

user

step 1 search WF

step 2 retrieve WF

step 3forward

WF bundle

step 4edit WF

step 6submit WF

step 5ask run WF

CGI User Scenario: WF Execution via Desktop

SHIWA RepositorySHIWA Portal

WF1

GEMLCA admin

SHIWA Science Gateway

WFn

WE1 WFp

GEMLCA Repository

WF1 WFm

GEMLCA submitter

WS-PGRADE Workflow

engine

WS-PGRADE Workflow

editor

SSP DCI & FGI Infrastructure

SHIWA Science Gateway Resourcesnative WE ASKALON, MOTEUR, WS-PGRADE

local WEs: Kepler, MOTEUR , Taverna & Triana WEsportal WS-PGRADE v3.4.1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEsGEMLCA

ARC, gLite, Globus and Unicore DCIsproxy server

DCI Bridge

Meta Broker

Proxy Server

JSDL Translator

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCI

Kepler WEGWES WE

Taverna WE

Triana WE

MOTEUR WE

ASKALON WE

PGRADE WE

Galaxy WE

Pegasus WE

ProActive WE

Workflow Engines

2424

SHIWA RepositorySHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1 WFp

GEMLCA Repository

WF1 WFm

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCIWS-PGRADE

Workflow engine

DCI Interoperability Scenario

DCI Bridge

Meta Broker

Proxy Server

JSDL Translator

WS-PGRADE Workflow

editor

step 1 search WF

step 2 edit WF

step 3retrieve WF data

step 4translate WF

step 8submit WF

step 7retrieve

certificate

step 5forward WF

step 6select DCI

2525

SHIWA Simulation Platform:Access

SHIWA Science Gateway (SHIWA Portal + SHIWA Repository)• joint portal and repository account given by the science gateway

administrator

DCI resources• certificate given by the relevant DCI administrator, for example the

VO sysadmin

Access to the SHIWA Portalssp.shiwa-workflow.eu

Access to the SHIWA Repositoryrepo.shiwa-workflow.eu

26

Conclusions

• the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows

• users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities

• workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform

• researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform

• the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources