“shibbolized irods” (and why it matters) - terena€œshibbolized irods” (and why it matters)...

26
3 rd TERENA Storage Meeting, Dublin, February 12 th -13 th 2009 David Corney, for Jens Jensen, e-Science centre, Rutherford Appleton Lab, UK “Shibbolized iRODS” (and why it matters)

Upload: hakhue

Post on 17-Apr-2018

225 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

3rd TERENA Storage Meeting, Dublin, February 12th -13th 2009

David Corney, for Jens Jensen, e-Science centre, Rutherford Appleton Lab, UK

“Shibbolized iRODS”(and why it matters)

Page 2: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Overview

IntroductionA shared e-infrastructure – current

statusOne area of development: ASPiS

Page 3: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

David Corney CISB07 Nov 19th 2007

3

About STFC...

The Science and Technology Facilities Council (UK)Created on April 1, 2007 (1 of 7 UK research Councils)Responsible for:

– fundamental research in particle physics, nuclear physics, astronomy, space

– major UK facilities for the physical and life sciences• synchrotrons, light sources, lasers, neutrons

– national laboratories at RAL, Daresbury, UKATC– international science projects

• CERN, ESO, ESA, ILL, ESRF…

Over 2000 staff and an annual budget of over £700M

Page 4: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Tier StructureTier 0

Tier 1National centres

Tier 2Regional groups

Institutes

Workstations

Offline farm

Online system

CERN computer centre

RAL,UK

ScotGrid NorthGrid SouthGridLondon

FranceItalyGermanyUSA

Glasgow EdinburghDurhamUseful model for Particle Physics but not necessary for others

Page 5: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Tier-1 HardwareCPU Power (Reconstruction, Simulation, User Analysis etc). 600 systems, 1250 cores, 1500 KSI2K

'Tape' Storage – Long Term retention – write once – read several times a year – 1PB in SL8500 robot + 12 drives

Disk Storage (Frequently Accessed) 138 Servers, 3200 drives, 750TB

Currently about 45 racks – with a further 25 due to arrive for Xmass

Page 6: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

6

Rutherford Appleton Laboratory

Page 7: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

EDNS - European Data Infrastructure for

Neutron and

Science driver - enabling better scienceNeutron diffraction X-ray diffraction NMR

High-qualitystructure refinement

Page 8: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

e-Infrastructure – Access to Multiple Facilities

iCat

SNS - ORNL

ISIS – TS1 + 2

DLS

CLF

ANSTO - Australia

Page 9: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

EDNS - European Data Infrastructure for

Neutron and

Technology Driver – integration and interoperation

Single Infrastructure Single User Experience

CapacityStorage

Publications Repositories

Data Repositories

Software Repositories

Raw Data Catalogue

Data Analysis

Analysed Data Catalogue

Publication Data Catalogue

Publications Catalogue

Raw Data

Data Analysis

Analysed Data

Publication Data

Publications

Facility 1

Raw Data

Data Analysis

Analysed Data

Publication Data

Publications

Facility 2

Raw Data

Data Analysis

Analysed Data

Publication Data

Publications

Facility 3

Different Infrastructures Different User Experiences

Page 10: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Underlying Data Infrastructure

Online Proposal System

User Office System:

User Database

Scheduling

Health and Safety

Proposal Management

Metadata Catalogue

Data Acquisition System

Storage Management

System

DataAccessPortal

Single Sign On Account Creation and Management

ICAT Software Suite, providing the crucial integration of key functions.

Page 11: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

David Corney CISB07 Nov 19th 2007

11

BBSRC Archive systemAll (12) Institutes of the BBSRC

6000 scientists across the UK

50 TB storage capacity (currently)

10 year SLA agreed

Page 12: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

David Corney CISB07 Nov 19th 2007

12

Data Archive/Management Services

High Energy Physics Experiments (CMS, Atlas, LHcb, Alice, H1,...)ISIS (Neutron Muon Source)Diamond Light SourceBritish Atmospheric Data CentreEISCAT (Radar research)National Earth Observation Data CentreBBSRC archiveSolar Physics World Data CentreCICT (Standard IT backups)Central Laser FacilityNational Crystallography Service, University of SouthamptonHartley Library, Southampton UniversityWASP, VIRGO ConsortiumSOLAR-B (Hinode)

Page 13: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Data Policy

• Data Policy (ISIS)– 3 year embargo on data (+1 if requested)– Commercial data is never made public– Instrument Scientists can access all data from their

beamline– Calibration data is public– Any data that involves IPR (e.g. analysed) is private

for perpetuity unless explicitly shared by user

• Automatic Enforcement of policy• A research area

Page 14: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

EDNP

European Data Infrastructure for Neutron and Photon Sources

Combining European Neutron and Synchrotron Facilities

Already a common user community

Across many disciplines– Materials, chemistry, proteomics, pharmaceuticals,

nuclear physics, archaeology …

ESRF

Page 15: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

The ASPiS projectJens Jensen, STFCvia David Corney, STFC

Terena Storage TF, DublinFebruary 2009

Page 16: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

ASPiS: people

• M Hedges, E Liao, T Blanke, CeRCH KCL• A Weise, Reading• A Hasan, Liverpool• J Jensen, R Downing, STFC

Page 17: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

ASPiS

• iRODS as datastore• SSO login via Shibboleth• PERMIS access control policy• Provenance metadata in PASOA• Funded by JISC

Page 18: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

iRODSiRODSPASOAPASOA

Shibservice

Shibservice

PERMISPDP

PERMISPDP

DiskDisk

ApacheApache

User

Page 19: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Shib loginSo what does it do?

• Single password• Password managed by home institution

• S.E.P.

• Home institution provides attrs• ASPiS can use these for access control• And for provenance

Page 20: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Shibboleth loginHomeInst.

HomeInst.

iRODSiRODS

Page 21: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

iRODS

• Rule Engine to manage data workflow• Microservices calling out to ext’l

services• No changes to iRODS itself

• Improves maintenance

Page 22: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Log attrsLog attrs

Access CtrlAccess Ctrl

UpdatemetadataUpdate

metadataPASOAPASOA

PERMISPDP

PERMISPDP

Branch onfile type

Branch onfile type

DocumentmetadataDocumentmetadata

Imagemetadata

Imagemetadata

RuleEngine

iRODS ExampleRule workflow

Page 23: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

UK Access Management Federation(Shibboleth)

UK Access Management Federation(Shibboleth)

STFCiRODSSTFC

iRODS

Reading

iRODSReading

iRODS

King’siRODSKing’siRODS

ASPiSiRODSFederation

Two Federations

Page 24: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Target Users

1. Arts and Humanities2. STFC facilities

– Was Diamond Light Source (no IdP)– Now ISIS Neutron Source

3. SRB users on the National Grid Service

Page 25: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Timescale

Project start01 Apr 2008

Today 31 June2009

Page 26: “Shibbolized iRODS” (and why it matters) - TERENA€œShibbolized iRODS” (and why it matters) Overview ... 50 TB storage capacity ... VIRGO Consortium SOLAR-B (Hinode)

Questions

Thanks for your attention- and to David for giving the presentation

For questions, please contactj dot jensen dot ral at googlemail dot com