20110712.we nmr.utrecht

www.egi.eu EGI-InSPIRE RI-261323

EGI-InSPIRE


EGI user support

&

interaction with the VRCs

Nuno Ferreira

User Community Support Team Officer

@ EGI.eu

WeNMR Devs Workshop, Utrecht, NL

20110712/13 1


• The grid: a primer

• Managing a sustainable infrastructure

• Building sustainable user services

• Summary

Outline


20110712/13 2


What is the grid?

“Coordinated resource sharing and problem solving in dynamic,

multi-institutional virtual organizations”. Foster, I. et al., Int. J. Superc. Appli. (2000)15:3


20110712/13 3

Source: http://www.gridcafe.org

http://www.gridcafe.org/

http://www.gridcafe.org/


Infrastructure (Wikipedia)

Infrastructure is the basic physical and

organisational structures needed for the operation

of a society or enterprise, or the services and

facilities necessary for an economy to function

European Grid Infrastructure (EGI) provides a

service infrastructure that exposes and helps

coordinate a resource infrastructure


20110712/13 4

The Enterprise is the European Research Area





• Summary

Outline


20110712/13 5


EGI Resource Infrastructure


20110712/13 6

Resource Infrastructure

Resource Centres

Resource Centres


Resource Centres

Resource Centres


Resource Centres

Resource Centres

Network

Integrated Resource Infrastructure

Provider

National Grid Initiative

Peer Resource Infrastructure

Provider

Integrated: infrastructure operated by a non-EGI-InSPIRE partner but relying on EGI operational services (MoU) Peer: infrastructure accessible to EGI users, but relying on own operational services

MoUs

EGI.eu


EGI Collaboration

Context & NA2 - EGI-InSPIRE Review

2011 7

NGI: National Grid Initiative EIRO: European Intergovernmental Research Organisation

NGI NGI EIRO NGI EIRO

Virtual Research Community

NGI NGI



EGI.eu


Status and yearly increase


20110712/13 8

Resource Centres • 338 (+6.8%) : Europe, Asia Pacific, North and South America • 96 supporting MPI (+31.5%)

Countries • 51 (57 with integrated RPs) (+18.75%)

Capacity • 240,000 CPU cores • 339,000 with integrated and peer infrastructures (24.9%) • 1.89 Million HEP-SPEC 06* • 102 PB disk, 89 PB tape * HEP-SPEC 06: Computing benchmark based on SPECCPU2006, 10 HEP-SPEC = 4 kSI2k


EGI.eu

• Coordination for European Grid resources

– Established February 8th 2010

– Central policy & services needed to run a grid

– Sustainable small coordinating organisation

• Based in Amsterdam

– Coordinating core (~20 people) in Amsterdam

– Technical services from partners (~20 people)


20110712/13 9

EGI and EGI.eu supported by EGI-InSPIRE project


EGI-InSPIRE project

Integrated Sustainable Pan-European Infrastructure for

Researchers in Europe

A 4 year project with €25M EC contribution

– Project cost €72M

– Total Effort ~€330M

– Effort: 9261PMs

Project Partners (50)

EGI.eu, 38 NGIs, 2 EIROs

Asia Pacific (9 partners) Un-Funded


20110712/13 10

Funded


EGI-InSPIRE objectives

• A sustainable production infrastructure

– Resource providers in Europe and worldwide

– With new technologies as they mature

• Support structured international research

– Sustain current domain specific services

– Attract new user communities (e.g. ESFRI)


20110712/13 11





• Summary

Outline


20110712/13 12


A virtuous service cycle


20110712/13

Used by Researchers

Gathering New

Requirements

New Technology

Assessed

Deployed Infrastructure

Services

13

EGI.eu

National Resource Providers

Current User Communities

New User Communities Technology

Providers


UCB and the VRC model


20110712/13 14

Benefits: 1. Defined communication

channels: named

contact points between

communities & EGI

2. Application integration

3. Policy and procedures:

define interaction

between EGI and users

4. Requirements

gathering: solutions and

influence

5. Dissemination: sharing

of news, breakthroughs,

best practice and

innovation

User Community Board (UCB) with representation from: Life Sciences, WLCG, WeNMR, Astronomy & Astrophysics, Earth Sciences, Computational Chemistry, Arts & Humanities, Hydrometeorology

VO: a group of researchers working collaboratively

EGI Helpdesk

Other Helpdesk

EGI.eu

NGI

Training Events

Trainers

Technical Services

VRC support

units

Other support

units

UCB

User Community

VRC: a grouping of research collaborations working in a common domain


NGI User Support Teams


20110712/13 15

NGI User

Support Teams

Consultancy

Training

Provide access to grid and software services

Porting apps.

Dev. and operate software services

Collecting Feedback

Document

Processes & apps.

Helpdesk

NGI User Support Teams provide:

• Consultancy & training

• Access to grid services

• Application porting

• Developing and operating

software services

• Collecting feedback

• Documentation & helpdesk

NGIs receive:

• Web & wiki resources

• Requirements process

• Applications and services

• VRC coordination support

Support teams are available across 50 partners – they are all different… as are their support skills.


NGI User Support Teams


20110712/13 16

NGI User

Support Teams

Consultancy

Training

Provide access to grid and software services

Porting apps.

Dev. and operate software services

Collecting Feedback

Document

Processes & apps.

Helpdesk

NGI User Support Teams provide:

• Consultancy & training

• Access to grid services

• Application porting

• Developing and operating

software services

• Collecting feedback

• Documentation & helpdesk

NGIs receive:

• Web & wiki resources

• Requirements process

• Applications and services

• VRC coordination support

Support teams are available across 50 partners – they are all different… as are their support skills.


Technical support services


20110712/13 17

Services provided by a few NGIs for the whole community • To support the Virtual Research Communities

• To facilitate the work of NGI support teams

• To improve communication among NGIs, among users and with users

• Bring in expertise and services from Heavy User Communities (HUCs)

Tools and services (current list – will be expanded):

1. EGI Application Database (Marios Chatziangelou, GRNET)

2. Services for VOs (Ignacio Blanquer Espert, UPV; Gonçalo Borges, LIP)

3. Training Marketplace (Claire Devereux, STFC)

4. Requirement Tracker (EGI.eu with help from CESNET)


EGI Application Database


20110712/13 18

Benefits of AppDB: • gives recognition to reusable

applications and tools

• gives recognition to application

developers

Use of AppDB • Register applications/tools

• Reuse of applications/tools

• Map of application landscape

• AppDB gadget has been

instantiated by:

– WeNMR, LSGC, GridPP,

Serbian, UK and Hungarian

NGIs

http://appdb.egi.eu/





Coordination of services for VOs


20110712/13 19

Coordination consists of: • Reviewing available services: management,

monitor & accounting tools (NAGIOS SAM, GANGA, DIANE, CERN dashboard & mini dashboard)

• Consultancy/help for VO managers(*)

• Specification of requirements for improving operational tools to simplify VO management

• Provisioning basic services for new VOs • Developing dashboard infrastructure (*) Further consultancy is provided by NGIs

NGI NGI

VO monitor

NGI NGI

Site resources

https://wiki.egi.eu/wiki/VO_Services






VO Services: NAGIOS SAM


20110712/13 20

Service Status Details for: phys.vo.ibergrid.eu

NAGIOS SAM triggers the executions of probes in grid sites

• Tests to the Storage resources

• Tests to the Computing Resources

• Summary views of successful and unsuccessful services

• Sends emails to the site admins in case of failures

Provision of VO support software for VRCs & VOs (e.g. WeNMR, BioMed)

Benefits: • Sites from a VO are tested (infrastructure tests run probes on NGIs) • Ability to develop and integrate VO’s own probes • A VO can use the central installation from LIP/UPV, or can install its own


VO Services: VO Dashboard


20110712/13 21

http://go.egi.eu/VO_Admin_Dashboard






Training marketplace • Integration of:

– Training event calendar

– Digital library (of training materials)

– Service offering & requesting form

• Purpose of Training Marketplace:

– Register training events

– Use, reuse and share training materials

– Offer training-related services

– Feed training-requirements

– Delegate representative to EGI Training Working

Group

• Results:

– Approximately 60 training events were added

during Project Year 1.


20110712/13 22

http://go.egi.eu/Training_Marketplace






Processing EGI requirements


20110712/13 23

TCB

EGI Requirements

Tracker

VRCs Input Channels

for User requirements

NGIs

HUCs

ESFRIs

EEF

VOs

Developer teams in TNA3.4

UCST (with NGIs)

UCB External

technology providers

EIROs

EGI Helpdesk (GGUS system)

• EMI • IGE • SAGA

USAG

• Permanent storage for requirements

• Complete transparency – For user, NGIs & technology providers

• Requirement submission interface http://go.egi.eu/wrt

http://go.egi.eu/


WeNMR RT Dashboard

http://go.egi.eu/RT.WeNMR


20110712/13 24






PY1 Results: Requirements Tracker

• 110 new requirements: – 48 resolved within EGI-InSPIRE/UCST

– 36 prioritised by UCB of which…

– 20 passed via TCB to Technology Providers

– 26 still being investigated by UCST

• Current issues endorsed by UCB and recognised by TCB: 1. Increased stability and scalability for gLite WMS

2. Better (more verbose and informative) error messages

3. Fixing the known bugs before adding new features

4. Coherency of command line commands, parameters and APIs

5. Better feedback about jobs, automated resubmission of jobs stuck on sites

• Requirements to be refined in community workshops • Data management, Workflow systems & Workload management

• Use to define and build EGI VRC User support platform


20110712/13 25

Proved to be an effective communication channel with EMI and other external Technology Providers

60 requirements also generated from legacy documents from VRCs:

SHARE Roadmap (HealthGrid), ESFRI project reqs., Production Grid Infrastucture (OGF), Request for Enhancements of gLite to support bio-NMR applications, gLite requirements (EGEE), User requirements (EGI-DS), DRIHMS Questionnaire


Community Support Workshops

• Motivation for workshops (from EGI Forums):

– Users indicated need for – collaboration & exploration

– More smaller and focused meetings needed

– Bring specialists together

• Purpose:

– Inform and educate, clarify and specify

• Emerging themes (from requirements):

– portals, visualisation, data management, scalability, workflow,

supporting users, planning training events, developing and

provisioning sustainable apps.


20110712/13 26

http://www.doodle.com/z8xp9hvnas4k64ch

http://www.doodle.com/z8xp9hvnas4k64ch


www.egi.eu/user-support


20110712/13 27


WeNMR as a VRC

WeNMR signed an MoU with EGI 11 April 2011


20110712/13 28

http://www.egi.eu/collaboration/WeNMR.html

http://www.egi.eu/collaboration/WeNMR.html


Summary

• EGI VRC model available and WeNMR VRC created

– WeNMR participating in User Community Board

– WeNMR contributing to Requirements Tracker process

– EGI supporting NGIs who support WeNMR

• Technical Support Services:

User requirements https://rt.egi.eu/rt/Dashboards/1054/WeNMR

AppDB gadget http://wenmr.eu/wenmr/wenmr-grid-statistics

VO services https://grid-monitor03.pd.infn.it:50080/nagios

Training Marketplace http://go.egi.eu/WeNMR-Workshop-20110819

• Community Support Workshops:

– Themes under investigation


20110712/13 29

https://rt.egi.eu/rt/Dashboards/1054/WeNMR

http://wenmr.eu/wenmr/wenmr-grid-statistics





https://grid-monitor03.pd.infn.it:50080/nagios/



http://go.egi.eu/WeNMR-Workshop-20110819





20110712.we nmr.utrecht

Documents

sustainable infrastructure

infrastructure accessible

service infrastructure

people wenmr devs workshop

egi users

coordinated resource

egi user support interaction

european grid resources