Download - SRB and iRODS @ CC-IN2P3
SRB and iRODS @ CC-IN2P3 SRB and iRODS @ CC-IN2P3
Jean-Yves Nief
Pascal Calvat
19/04/23SRB and iRODS @ CC-IN2P3 2
Overview Overview
SRB:– Status.– A couple of examples.– Prospects.
iRODS:– Developments.– Test beds.– Project in production: digital library.
Moving from SRB to iRODS ? JUX (Java Universal eXplorer): GUI in Java for many
protocols like iRODS, SRB.
19/04/23SRB and iRODS @ CC-IN2P3 3
SRB in LyonSRB in Lyon
HEP
BaBar SLAC « mirror » site
CMOS, Calice Data archival
Indra Data distribution and archival
Lattice QCD tens of TB / y
Astroparticle
Antares Tier 0: ~200 TB / y
Auger Tier 0: ~10 TB / y
Edelweiss Tier 0: tens of TB / y
SN Factory One of the online brick: ~GB / d
BiomedicalBioEmergence Tier 0:european project ~ TB/y
Mammography Project with a computing lab
Neuroscience Lyon and Strasbourg hospital
19/04/23SRB and iRODS @ CC-IN2P3 4
SRB and iRODS hardware and SRB and iRODS hardware and software @ CC-IN2P3software @ CC-IN2P3
14 SRB servers (130 TB of disk space):– Sun v480, v20z, v440 (Sparc III) and Thumpers x4500 (AMD
Opteron).– OS: Solaris 9 and Solaris 10.
Almost all the instances on SRB v3.5.0. Interfaced with our MSS: HPSS v5:
– Scripts developed to handle automatic migration/purge of compound resources (disk cache/tape archive).
MCATs: using Oracle 10g. Going to add RHEL4 machines (Linux) for MCATs enabled
servers.
19/04/23SRB and iRODS @ CC-IN2P3 5
Example in HEP: BaBarExample in HEP: BaBar
Data import from SLAC to Lyon. SRB being used since 2004 in production. Fully automated:
– New files created are registered in the SLAC catalog database.– Client application in Lyon: detection of files missing in the Lyon
catalog database + transfer of these files.– Automated error recovery.
Up to 5 TB / day (max. rate observed). Usual rate: 2-3 TB / day (during production periods) 700 TB imported so far (since 2004), 700,000 files. Until end 2008, volume x2.
19/04/23SRB and iRODS @ CC-IN2P3 6
Example in HEP: BaBarExample in HEP: BaBar
HPSS/Lyon
SRB
SRB
SRB
SRBMCAT
(1)
(3)
(2)
SRB
SRBMCAT
HPSS/SLAC
SRB
SLAC zoneSLAC zone CC-IN2P3 zoneCC-IN2P3 zone
Import agentsImport agents
Import agentsImport agents
19/04/23SRB and iRODS @ CC-IN2P3 7
Examples in astrophysics and Examples in astrophysics and astroparticles astroparticles
Underwater: Antares in the pampa: Pierre Auger Observatory
At the top of the mountain: SuperNovae Factory in Hawaii
19/04/23SRB and iRODS @ CC-IN2P3 8
Example in biology: BioEmergenceExample in biology: BioEmergence
European projects involving 5 countries.
Embryogenesis: zebra fish. 2 microscopes now (several in
the future): amount of data could be huged (PB scale).
Data pushed from the microscopes into the SRB.
SRB integrated within their workflow.
CC-IN2P3: core of the system.
19/04/23SRB and iRODS @ CC-IN2P3 9
VirgoVirgo
Interferometer for gravitational waves detection (in production: 60 TB / y).
Need for a reliable data distribution system.
Distribute Ligo data (same experiment in the US) to the european sites: CC-IN2P3 and Bologna.
Have been using bbftp so far.
SRB has replaced bbftp:– Bookkeeping system.– Interface with HPSS.– Handling of Ligo small files.
LIGOLIGO CC-IN2P3CC-IN2P3
CascinaCascina
BolognaBologna
Merge Ligo small filesMerge Ligo small filesduring data importduring data importon the CC-IN2P3 on the CC-IN2P3 server side server side
Push raw data to CC-Push raw data to CC-IN2P3IN2P3
Pull Virgo + Ligo Pull Virgo + Ligo data from Lyondata from Lyon
Visualize data on the Visualize data on the WAN through SRBWAN through SRB
19/04/23SRB and iRODS @ CC-IN2P3 10
SRB @ CC-IN2P3 SRB @ CC-IN2P3
MCATs performance enhancement:– Reindexing made automatically on a weekly basis.
Issues with Oracle performances:– Some oddity in the way Oracle optimized requests.– Request analyzis done on all the MCATs on a daily
basis.
Not completly resolved.
SRB: big consumer of Oracle resources.– Need to separate Oracle instance for the MCATs.
19/04/23SRB and iRODS @ CC-IN2P3 11
SRB prospects @ CC-IN2P3SRB prospects @ CC-IN2P3
More than 10 projects using it:– SRB: critical part of their computing system.– Clients on Linux, Mac OSX, Windows, Solaris, AIX (Blue Gene)
from Europe, USA. Daily traffic can be quite big:
– Hundreds of thousands of connections per day.– Some projects with more than 200,000 connections per day at
peak rate.– Bandwidth peak rate: several Gbits/s.
Will reach 1 PB of data referenced and handled by SRB in 2008.
Need to strengthen the service (add extra MCAT servers).
19/04/23SRB and iRODS @ CC-IN2P3 12
iRODS developments @ CC-IN2P3iRODS developments @ CC-IN2P3
Tests scripts (JY + Thomas):– icommand test script: extensive test of the binary
commands in order to track bugs (both client and server side problems).
To do: include all the new options and new icommands.
– Loading test of the system: launch n test scripts in // on the same machine.
To do: ability to launch process on other servers ? (but too specific to a given site: passwrodless ssh etc…).
19/04/23SRB and iRODS @ CC-IN2P3 13
iRODS developments @ CC-IN2P3iRODS developments @ CC-IN2P3
Host based access control micro-service:– User id, group id.– Hostname or range of IP adresses.– Improved firewall implemented at the iRODS
level.
19/04/23SRB and iRODS @ CC-IN2P3 14
iRODS developments @ CC-IN2P3iRODS developments @ CC-IN2P3
Load balancing micro-services (JY + Jean Aoustet):
Load balancing system:– At a given time, choose the least loaded server to
put/get data or do any other operation.– Have to be used wisely:
• It only makes sense for identical servers on the same site!– Gather metrics on each server (CPU load, network
activity, memory usage, swap, disk occupancy etc…).– Store into iCAT.– Can also be used for monitoring purposes.
19/04/23SRB and iRODS @ CC-IN2P3 15
Load balancing systemLoad balancing system
iRODS iCAT serveriRODS iCAT server
iRODS data serveriRODS data server
iRODS data serveriRODS data server
iRODS data serveriRODS data server
iRODS data serveriRODS data server
1.1. Ask each server for its Ask each server for its metrics: rule engine metrics: rule engine cron task (msi).cron task (msi).
2. Performance script 2. Performance script launched on each launched on each server.server.
Perf scriptPerf script
Perf scriptPerf script
Perf scriptPerf script
Perf scriptPerf script
3. Results sent back to 3. Results sent back to the iCAT.the iCAT.
4. Store metrics into 4. Store metrics into iCAT.iCAT.
DBDB
5. Compute a «quality 5. Compute a «quality factor» for each server factor» for each server stored in an other table: stored in an other table: r.e. cron task (msi).r.e. cron task (msi).
19/04/23SRB and iRODS @ CC-IN2P3 16
Future developmentsFuture developments
HPSS driver: assessment by Thomas Kachelhoffer (HPSS v6) in progress.
Handling MSS resources with the protocol of your choice:– RFIO, pftp etc…– Easy way to plugin a MSS without writing a driver.– Kind of universal driver.
Micro-services for:– Handling accentuated letters in files and directory
names registered into iRODS (eg: Windows world).– Anonymization of DICOM files in biology.
19/04/23SRB and iRODS @ CC-IN2P3 17
iRODS test bedsiRODS test beds
With KEK (see Yoshimi talk): data transfer. LSST:
– Telescope in Chile: first light in 2014.
– Big challenge for data processing.
– USA + Chile for now. France starting to get involved.
– Computing: data management studies (DC 2 and 3).
NCSANCSA
CC-IN2P3CC-IN2P3
SDSCSDSC
(now also Indiana and Chile)(now also Indiana and Chile)
19/04/23SRB and iRODS @ CC-IN2P3 18
iRODS first production instanceiRODS first production instance
Just starting. Digital preservation area:
– Scan old documents and put them into iRODS.– 4-5 users at the moment: transparent access to the documents
from various platforms.– ~ 1-2 TB expected for this year.
Needs:– Propagation of ACLs to newly created collections.– Windows support for the icommands.– GUI interface (standalone or Web).– iRODS fits their needs.
19/04/23SRB and iRODS @ CC-IN2P3 19
Moving from SRB to iRODS ?Moving from SRB to iRODS ?
No plans yet. Will slowly moved the existing SRB projects to the
new system starting in 2010 or a bit earlier. Needs to be confident that all the bits and pieces
are there before the switch:– MSS integration: one of the key thing.
Probably brand new projects could start directly onto iRODS in 2008 such as Digilib.
19/04/23SRB and iRODS @ CC-IN2P3 20
Future plans for iRODSFuture plans for iRODS
Continue developments:– Focus on MSS integration.– Load balancing.
CC-IN2P3 starting to be involved with Arts and Humanities projects:– Fedora and iRODS interface to be studied.
New projects to start directly on iRODS:– 1 more candidate this year.
19/04/23SRB and iRODS @ CC-IN2P3 21
JUX: a Java explorerJUX: a Java explorer
Java Universal eXplorer: developed by Pascal Calvat (CC-IN2P3).
Based on JSaga APIs:– Saga (Simple Apis for Grid Application): OGF specs. Common set
of APIs to interact with middleware (storage, job tasks).– JSaga: implementation of Saga by Sylvain Reynaud, CC-IN2P3.– Plugins developped for various protocols like SRB and iRODS
(using Jargon). JUX can connect to various protocols providing the plugin
is available. Still under development: first version released soon.
19/04/23SRB and iRODS @ CC-IN2P3 22
AcknowledgementAcknowledgement
Thanks to:– Pascal Calvat (JUX), Jean Aoustet (Load Balancing),
Thomas Kachelhoffer (tests + HPSS)..– Wilko Kroeger (SLAC): BaBar.– Yoshimi Iida (transfer tests + iCAT stress tests).
Links: – JSaga (Sylvain Reynaud): http://grid.in2p3.fr/jsaga