www.eudat.eu

EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No. 654065

EGI - EUDAT interoperability efforts

DI4R Conference - KrakowMichaela Barth caela@kth.sePeter Gille petergil@kth.se

226/09/16

24 countriesCERN, EMBL-EBI

1 coordinating organization – EGI.euSlide thanks to Matthew Viljoen

326/09/16

Open Solutions

Slide thanks to Matthew Viljoen

426/09/16

Computing and Data Integration

EGI Open Data Platform enhances EGI/EUDAT services through developments in:• FedCloud: EGI AppDB, Cloud

Management Frameworks, hybrid clouds

• Data: federated data -> sharing across VMs, discoverability via DataHub

EGI AAI now based on Open ID ConnectSlide thanks to Matthew Viljoen

EUDAT in a NutShell

5

● EUDAT – offers a complete set of research data services, expertise and technology solutions to all European scientists and researchers. These shared services and storage resources are distributed across 15 European nations and data is stored alongside some of Europe’s most powerful supercomputers.

● One of the main ambitions of EUDAT is to bridge the gap between research infrastructures and e-infrastructures through an active engagement strategy, using the communities that are in the consortium as EUDAT beacons and integrating others through innovative partnerships.

6

supporting multiple research communities as well as individuals, through a geographically distributed, resilient network of 35 European organisations

B2 Service SuiteB2ACCESS

B2HANDLE

8

Joint Access to Data and HPC Services Commercial

stakeholders

Collaboration with Commercial Stakeholders

Joint Access to Data, HTC and Cloud

Computing Resources

Interoperability

Overall WP 7 objectives

9

● Ensure the interoperability of EUDAT with other public and private e-Infrastructures, lowering technical and policy barriers by piloting concrete use cases with user communities

● Provide European researchers and industries with seamless access to data and computing resources for cross-utilization use cases

● Pave the way towards the interoperability of e-Infrastructure tools and services beyond H2020

10

Joint Access to Data and HPC Services Commercial

stakeholders

Collaboration with Commercial Stakeholders

Joint Access to Data, HTC and Cloud

Computing Resources

Interoperability

WP 7 Task 7.2

WP 7

WP7 Task 7.2: Joint Access to Data, HTC and Cloud Computing

Resources• Connects data stored in the CDI to high

throughput and cloud computing resources provided through EGI

• Starting with concrete community pilots• Aiming at a production cross-infrastructure

service (integrating storage resources managed by EUDAT and computing resources available at EGI)

• EGI-EUDAT collaboration started in March 2016.

11

WP7 Task 7.2: Joint Access to Data, HTC and Cloud Computing

ResourcesTwo subtasks:

• 7.2.1 Service Interfaces and Access Policies• with focus on users’ authentication, standards transfer

protocols and tools, execution of computational workflow and harmonization of access policies

• 7.2.2 Community pilots and Coordination of Joint Calls for Proposals• coordinating the implementation of selected community

pilots,• initially intended to start with four community pilots (ICOS,

ELIXIR, EISCAT-3D, BBMRI), now only 2 pilots (ICOS and EPOS)

• organizing a Joint Call for Proposals to expand the pilot activity to further communities

• Main objectives:• provide end-users with a seamless access to an integrated

infrastructure offering both EGI and EUDAT services• pairing data and high-throughput computing resources

together.

12

• Federated services through EGI paired together with existing EUDAT common data services of EUDAT:

• Additionally pooling effect: augmentation of services from existing communities using EGI/EUDAT

Expected Benefits

13

Computation on EGI Federated Cloud and HTC

EUDAT services for transfer, syncing, sharing, staging and preservation of data

• Technical• interoperability (e.g. workflow execution, data

discoverability and provenance), • Authentication, Authorization and Identity (AAI)

Mgmt. • Combination of respective service catalogues

• Policies• access policy• long-term perspective• operational policies

• Operational• Operational tools, technologies, best practices• Security• SLAs

Harmonization on all levels

14

• Transparent access to EGI and EUDAT services:• When a user is authenticated once on EGI and EUDAT,

(s)he should be able to see the EGI and EUDAT services as offered by a unique infrastructure

• Breaking this down into smaller steps:• Allowing users to access EGI and EUDAT web

services with the same credential.• Allowing users to access EGI and EUDAT non web

services with the same credential.• Attributes harmonisation• Enabling EGI services to delegate user’s credential

to EUDAT services and vice versa.• Data privacy issues and policy harmonisation.

EGI/EUDAT AAI Interoperability

15

• Work done so far:• Get an understanding of each other’s AAI layers• breaking the task down into smaller steps• Started draft document• Enabling accounts for some feasibility tests

• Next step:• Complete Roadmap

EGI/EUDAT AAI Interoperability

16

• Work done so far:• EGI and EUDAT selected a set of relevant (= already

collaborating with both infrastructures) user communities

• Started process of getting requirements from user communities and their indication of prioritization of those requirements (“Definition of the universal use case”) → integration activity has been driven by the end users from the start!

• Identified user communities were prominent European Research infrastructure in the field of Earth Science (EPOS and ICOS), Bioinformatics (BBMRI and ELIXIR) and Space Physics (EISCAT-3D).

User community pilots

17

Definition of the universal use case

18

• covers the user needs with respect to the integration of the two infrastructures

• permits a user of either e-infrastructure to instantiate a VM on the EGI Cloud Federation for the execution of a computational job consuming data preserved onto EUDAT resources. The results of such analysis can be staged back to EUDAT storages, and if needed, allocated with Permanent identifiers (PIDs) for future use.

Definition of the universal use case• Demo at EGI Community Forum 2015 by Diego

Scardaci

19

Definition of the universal use case

20

• To implement all the steps of this use case the following integration activities between the two infrastructures has to be fulfilled: (1) harmonisation between the authentication and authorisation model, (2) definition and implementation of the interfaces between the involved EGI and EUDAT services.

• The first phase of the implementation of this use case has been demonstrated at the EGI Community Forum 2015 (Bari, IT). In addition, two pilot use cases (EPOS and ICOS) have been selected to drive the implementation and validate the results.

ICOS Carbon Portal use case

21Slide thanks to Margareta Hellström

Background:ICOS wants

to set up "on demand computing" facilities that allows users to perform calculations based on ICOS observational data.

ICOS Carbon Portal use case status

• Virtual machine with attached block storage instantiated in the EGI Federated Cloud

• Docker container hosting web service and model computations running in the VM with all input data copied to the local storage of the VM

• Data transfer between the VM and B2STAGE instance (at KTH Stockholm) tested using the contextualization script provided by EGI

• Storing of ICOS data tested on the B2SAFE system at KTH

Next steps:• ICOS data storage in B2SAFE (at KTH) and access via B2STAGE

service• Access to common storage for several VMs (e.g. via the EGI

DataHub)• Robot certificates to allow for further automation of the workflow• Load balancing to distribute computations/users requests to

several VMs• Prepare “focused” documentation, with a clear user perspective!22Slide thanks to Margareta Hellström

Challenges encountered so far

When implementing prototypes of use-cases of EPOS and ICOS:

• Scaling up from proof of concept to multiple users

• AAI interoperation• Managing co-existing support systems and

channels• more user-friendly documentation of all services

in the current state (not the final vision) desirable

• steep learning curve for the user communities in using e-infrastructure communities

• 3rd party dependencies• On the plus side: personal contacts highly

appreciated23

Next steps

• Prolong the tasks until March 2018• Deliverable “12 months report”• Continued implementation of use cases

with result evaluation• Joint call for proposals to expand the

pilot activity• Final report including recommendations

for service development and access policy harmonization

24

For more info:

https://b2drop.eudat.euhttps://eudat.eu/services/userdoc/b2drop

https://b2share.eudat.euhttps://eudat.eu/services/userdoc/b2share

https://eudat.eu/services/userdoc/b2safe

https://eudat.eu/services/userdoc/b2stage

http://b2find.eudat.euhttps://eudat.eu/services/userdoc/b2find

http://b2access.eudat.eu

More on EUDAT

26

● EUDAT is a collaborative Pan-European infrastructure and provides also training and consultancy for researchers, research communities, research infrastructures and data centres.

● EUDAT’s vision is to enable European researchers and practitioners from any research discipline to preserve, find, access, and process data in a trusted environment, as part of a Collaborative Data Infrastructure (CDI) conceived as a network of collaborating, cooperating centres, combining the richness of numerous community-specific data repositories with the permanence and persistence of some of Europe’s largest scientific data centres.