introduction to unicore - twiki.cern.ch€¦ · tsi 16/09/2010rebecca breufolie 6. when a job is...
TRANSCRIPT
![Page 1: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/1.jpg)
Mit
glie
dd
erH
elm
hol
tz-G
emei
nsc
haf
t
Introduction to UNICORE
16/09/2010 Rebecca Breu
![Page 2: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/2.jpg)
Some Facts About UNICORE
First version has been developed in 1997
UNICORE has been worked on in several German andEuropean projects since
Integrated, complete middleware stack, including user clients
Written in Java, thus many supported operating systems(*nix, Mac, Windows)
Many supported batch systems (LoadLeveler, Torque, SunGrid Engine, . . . )
Web services for communication (WS-RF)
SSL (Secure Sockets Layer) for transport
16/09/2010 Rebecca Breu Folie 2
![Page 3: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/3.jpg)
16/09/2010 Rebecca Breu Folie 3
![Page 4: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/4.jpg)
UNICORE Architecture
Global Registry:Central point of a UNICORE gridKeeps track of all available services
Gateway:”Door to outside world” in firewallmay serve several resources behind one firewall
UNICORE/X:Central point for job processing and managingChecks user certificate with XUUDB
XUUDB (UNICORE user database):Mapping between user certificates, user logins, roles
TSI (Target System Interface):Submits jobs to batch system
Components use SSL connections
16/09/2010 Rebecca Breu Folie 4
![Page 5: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/5.jpg)
The Global Registry
The Global Registry:
Provide clients with information about services
Two kinds: global / local
Global or central registry:
Serves as a ‘Grid’Knows all target systems and workflow servicesServices dynamically register with (one or more) registries
Local registry per service container (e.g. UNICORE/X)
For registering service instances
Full WS-RF Service
16/09/2010 Rebecca Breu Folie 5
![Page 6: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/6.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 7: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/7.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 8: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/8.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 9: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/9.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 10: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/10.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 11: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/11.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 12: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/12.jpg)
When a job is being submitted . . .
Client
Gateway
unicorex
XUUDB
TSI
Client establishes SSL-Connection to Gateway
Client contacts UNICORE/X via Gateway
Client sends signed abstract job to UNICORE/X
UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution
UNICORE/X gets login from XUUDB
UNICORE/X translates abstract job intomachine dependent script
UNICORE/X sends machine dependent script toTSI
16/09/2010 Rebecca Breu Folie 6
![Page 13: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/13.jpg)
The XUUDB
XUUDB:
Maps user certificates to logins on that machine
Assigns roles (user, admin, . . . )
Nr|GcID |Xlogin |Role |Projects |DN
---------------------------------------------------------------------------
1 |OMII_EI |rbreu |user | |CN=Rebecca Breu , OU=JSC , OU=Forschungszentrum Juelich GmbH , O=GridGermany , C=DE
2 |OMII_EI |sandra |user | |EMAILADDRESS=s.bergmann@fz -juelich.de , CN=Sandra Bergmann , OU=ZAM , O=Research Center Juelich , L=Juelich , ST=NRW , C=DE
16/09/2010 Rebecca Breu Folie 7
![Page 14: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/14.jpg)
The TSI
The TSI . . .
forks a process which runs with the user’s ID
creates a temporary directory on the target system (uspace)
changes current working directory to uspace
submits job to local batch system
Input and ouput:
all input needed for job has to be copied into the uspace
all output that is to survive the end of job execution has to becopied elsewhere
Terms used:
File import: File tranfer from somewhere into uspace
File export: File tranfer from uspace to somewhere
16/09/2010 Rebecca Breu Folie 8
![Page 15: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/15.jpg)
The UNICORE/X
Authorises requests using the authorisation service XUUDB
Translates abstract job into concrete job for target system viathe IDB
Provides UNICORE Atomic Services (UAS):
Job Management ServiceStorage Management ServiceFile Transfer ServiceTarget System Sevice. . .
16/09/2010 Rebecca Breu Folie 9
![Page 16: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/16.jpg)
UAS: Target System Factory Service
16/09/2010 Rebecca Breu Folie 10
![Page 17: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/17.jpg)
UAS: Target System Service
Abstract web service interface to target system
List of available applicationsDescription of available resources (e.g. CPUs, RAM)Job submissionLinks to jobs and storages (e.g. user home)
Security & Privacy
Authorisation: Users’ target system instances and jobs protectedby configurable XACML policySecure job submission through message signing
16/09/2010 Rebecca Breu Folie 11
![Page 18: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/18.jpg)
Jobs
Abstract job definitions:
Given in JSDL (Job Submission Description Language)
Job name, description
Resource requirements (RAM, numer of CPUs needed, . . . )
Information about file transfers
An application name and version
Job instances:
Web service interface to submitted jobs
Job status (queued, running, finished, failed, ...)
Exit code of the application
Link to storage that contains input and output files
Have a life time
16/09/2010 Rebecca Breu Folie 12
![Page 19: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/19.jpg)
UAS: Jobs and Storages
16/09/2010 Rebecca Breu Folie 13
![Page 20: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/20.jpg)
UAS: Filetransfers
16/09/2010 Rebecca Breu Folie 14
![Page 21: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/21.jpg)
UAS: File Transfer Protocols
Pluggable mechanism
Both for client-server and server-server transfers
Supported protocols:
Baseline File Transfer (BFT)OGSA ByteIOGridFTPUDT
16/09/2010 Rebecca Breu Folie 15
![Page 22: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/22.jpg)
IDB: Incarnation Database
The IDB is a file with rules for translating abstract jobs intoexecutable scripts. (Belongs to the UNICORE/X configuration.)
<idb:IDBApplication >
<idb:ApplicationName >Bash shell</idb:ApplicationName >
<idb:ApplicationVersion >3.1.16 </idb:ApplicationVersion >
<jsdl:POSIXApplication xmlns:jsdl="http: // schemas.ggf.org/jsdl /2005/11/ jsdl -posix">
<jsdl:Executable >/bin/bash</jsdl:Executable >
<jsdl:Argument >--debugger$DEBUG?</jsdl:Argument >
<jsdl:Argument >-v$VERBOSE?</jsdl:Argument >
<jsdl:Argument >$ARGUMENTS?</jsdl:Argument >
<jsdl:Argument >$SOURCE?</jsdl:Argument >
</jsdl:POSIXApplication >
</idb:IDBApplication >
16/09/2010 Rebecca Breu Folie 16
![Page 23: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/23.jpg)
Define New Applications
To bring new applications into UNICORE:
Define Application in IDB
Access it via UCC or GenericGridBean
For a more customised graphical interface: Write a clientplugin (GridBean)
high-level Java-API
16/09/2010 Rebecca Breu Folie 17
![Page 24: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/24.jpg)
16/09/2010 Rebecca Breu Folie 18
![Page 25: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/25.jpg)
Workflows
Workflow:
Several jobs that depend on each other.
Job excecution after another job has finished
File transfers between jobs
Repeated job execution (loops)
Job execution depending on other job status (if-then-else)
16/09/2010 Rebecca Breu Folie 19
![Page 26: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/26.jpg)
Workflow Services
Client
Workflow Service
Service Orchestrator
unicorexunicorex
Client submits workflow to WorkflowService
Workflow Service creates n normal jobsand passes them to Service Orchestrator
Service Orchestrator submits jobs toUNICORE/X
Service Orchestrator keeps track ofcontstraints (if-conditions, jobs beingexecuted after one another . . . )
16/09/2010 Rebecca Breu Folie 20
![Page 27: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/27.jpg)
Workflow Services
Client
Workflow Service
Service Orchestrator
unicorexunicorex
Client submits workflow to WorkflowService
Workflow Service creates n normal jobsand passes them to Service Orchestrator
Service Orchestrator submits jobs toUNICORE/X
Service Orchestrator keeps track ofcontstraints (if-conditions, jobs beingexecuted after one another . . . )
16/09/2010 Rebecca Breu Folie 20
![Page 28: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/28.jpg)
Workflow Services
Client
Workflow Service
Service Orchestrator
unicorexunicorex
Client submits workflow to WorkflowService
Workflow Service creates n normal jobsand passes them to Service Orchestrator
Service Orchestrator submits jobs toUNICORE/X
Service Orchestrator keeps track ofcontstraints (if-conditions, jobs beingexecuted after one another . . . )
16/09/2010 Rebecca Breu Folie 20
![Page 29: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/29.jpg)
Workflow Services
Client
Workflow Service
Service Orchestrator
unicorexunicorex
Client submits workflow to WorkflowService
Workflow Service creates n normal jobsand passes them to Service Orchestrator
Service Orchestrator submits jobs toUNICORE/X
Service Orchestrator keeps track ofcontstraints (if-conditions, jobs beingexecuted after one another . . . )
16/09/2010 Rebecca Breu Folie 20
![Page 30: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/30.jpg)
UNICORE Clients
Several clients for different purposes:
Rich Client: Eclipse-based graphical client for complex tasks
UCC: Command line client
(Portal Client: Web based client to use with a browser)
16/09/2010 Rebecca Breu Folie 21
![Page 31: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/31.jpg)
Basic Client Configuration
Create a password-protected keystore that holds all keysneededAdd keys to keystore:
User’s private and public keyPublic keys of trusted CAs (must trust signer of involvedgateway keys!)
Add URI of Global Registry
16/09/2010 Rebecca Breu Folie 22
![Page 32: Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection](https://reader034.vdocuments.site/reader034/viewer/2022042418/5f34e4809f673827640acd13/html5/thumbnails/32.jpg)
More information
UNICORE Website: www.unicore.eu/
Documentation: www.unicore.eu/documentation
Wiki: sourceforge.net/apps/mediawiki/unicore/
16/09/2010 Rebecca Breu Folie 23