the atlas petabyte datastore
DESCRIPTION
The Atlas Petabyte Datastore. A grid enabled, networked data storage system: CrystalGrid Workshop 15 th Sept 2004 David Corney. [email protected]. Data Store Overview. General purpose, multi user, data archive. In use over 20 years. Four major upgrades. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/1.jpg)
The Atlas Petabyte Datastore
A grid enabled, networked data storage system:
CrystalGrid Workshop 15th Sept 2004
David Corney. [email protected]
![Page 2: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/2.jpg)
Data Store Overview
• General purpose, multi user, data archive. • In use over 20 years. Four major upgrades.• Current capacity 1PB – largest (non dedicated) multi user
system in UK academia?• Grid Interfaces:
– SE (Storage Element) – will be SRM compliant– SRB interface (Storage Resource Broker)
![Page 3: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/3.jpg)
STK 9310
![Page 4: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/4.jpg)
Switch_1 Switch_2
RS6000 RS6000RS6000 RS6000
fsc0 fsc1 fsc1fsc0
9940B 9940B 9940B 9940B 9940B 9940B 9940B 9940B
1 2 3 4 5 6 7 8
11 14 11 1415
fsc1fsc0fsc1fsc0
12 13 12 13 15
rmt1 rmt4rmt3rmt2rmt5-8 rmt5-8rmt5-8rmt5-8
A A A A A A A A
STK 9310 “Powder Horn”
Gbit network
1.2TB 1.2TB 1.2TB 1.2TB
![Page 5: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/5.jpg)
flfstk
tapeserv
Farm Server
flfsys(+libflf)
user
flfscan
data transfer (libvtp)
catalogue data
STK tape drive
cellmgr
Catalogue Server (brian)
flfdoexp(+libflf)
flfdoback(+libflf)
datastore (script)
Robot Server (buxton)
ACSLS
API
control info(mount/dismount)
data
data
Tape Robot
flfsys user commands (sysreq)
SE
recycling (+libflf)
read
read
read
Atlas Datastore Architecture
28 Feb 03 - 2 B Strong
SSI
CSI
flfsys farm commands (sysreq)
LMU
flfsys admin commands(sysreq)
administrators
flfaio
flfaio
flfaio
IBM tape drive
flfqryoff(copy of
flfsyscode)
Backupcatalogue
stats
flfsys tapecommands
(sysreq)
servesys
pathtape
long name(sysreq)
short name(sysreq)
frontend
backendPathtape Server (rusty)
(sysreq)
importexport
flfsys import/export commands (sysreq)
libvtp User Node
I/E Server(dylan)
?
Copy BCopy C
ACSLS
cache disk
Copy A
vtp
vtp
user program
tape
(sysreq)
![Page 6: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/6.jpg)
Strategy -
• De-couple user and application from storage media. • Upgrades and media migration occur “behind the scenes”• High resilience - very few Single Point Failures• High reliability high, availability (99.9986% in 2003)• Constant environmental monitoring linked to alarm/call out• Lifetime data integrity checks hardware and software• Fire safe and off-site backups; Tested disaster recovery
procedures; media migration, recycling• Easy to exploit (endless) new technology• Technology watch to monitor future technology path
![Page 7: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/7.jpg)
Robot History
• M860 – 110GB
• STK 4400– 1.2Tbytes
• IBM 3494– 30Tbytes
• STK 9310– 1Pbyte
![Page 8: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/8.jpg)
Hardware upgrade - completed Jun 2003
• STK 9310 “Powderhorn” with 6000 slots (1.2Pbytes)• 4 IBM 3590 B drives now phased out
– 10 Gbyte native– 10 Mbyte/s transfer
• 8 New STK 9940B drives– 200 Gbyte native– 30Mbytes/sec/drive transfer – 240Mbyte/sec theoretical maximum bandwidth
• 4 RS6000 Data servers (+ 4 “others”)• 1Gbit networking (Expected to become 10Gbit by 2005)• Data Migration to new media completed ~ Feb 2004
![Page 9: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/9.jpg)
Users
• Particle Physics Community (LHC: CMS, Atlas, LHcb,….)• ISIS, British Atmospheric Data Centre• EISCAT (Radar research)• National Earth Observation Data Centre• World Data Centre, BITD• Central Laser Facility• Diamond…• National Crystallography Service, Southampton University, • WASP, VIRGO Consortium• Integrative Biology,• Others…
![Page 10: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/10.jpg)
Interfaces
• “Light weight” interfaces:– Client server configuration:
• “tape” command for many platforms• Virtual Tape Protocol (VTP)• Fortran and C callable library
• “Heavy weight” interfaces:– SRB interface– SE interface developed for EDG/GRIDPP/GRIDPP2…
![Page 11: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/11.jpg)
SRB-ADS architecture
SRB MCATDatabase
SRB MCATServer
SRB ADSServer
SRBClient
SRB DiskServer (Local Server)
Atlas Data Store SRB ADS Server
SRB-ISIS server instance
SRB-BADC server instance
SRB-CCLRC server instance
Port 5600
Port 5601
Port 5602
![Page 12: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/12.jpg)
Adding Interfaces for ADS
ADS CentralCatalog Serverflfsys
ADS FarmServer
ADS FarmServer
ADS FarmServer
SRB01Server
SRB02Server
ADS FarmServer
VTP InterfaceSRB Interface
LCGServers
SE Interface
SRBUsers
SEUsers
VTPUsers(tape)
![Page 13: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/13.jpg)
Logical Resource for Containers
ADS-cacheresource
ADS-taperesource
ADS-logical-resource
Sput –c <container-name> <data-file>
Ssyncont
Copies container from cache to tape
![Page 14: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/14.jpg)
The Storage Element (SE) mass storage interface
![Page 15: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/15.jpg)
The Storage Element (SE)• A component of European Data Grid (EDG/EGEE) middleware
developed by CCLRC’s e-Science & PPD departments
• Uniform Grid Interface which enables a standard protocol for mass data transfer across the grid, between the many diverse Mass Storage Systems, including:
– Atlas Petabyte Data Store
– CASTOR
– ENSTOR
– HPSS
– Others…
![Page 16: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/16.jpg)
SE Deployment
CERN – Castor and disk
UAB Barcelona – Castor
RAL – Atlas DataStore and Disk
ESA/ESRIN – disk
CC-IN2P3 – HPSS
INFN / CNAF – disk
FZK Karlsruhe – disk
![Page 17: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/17.jpg)
Questions?
![Page 18: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/18.jpg)
Digital Curation Centre
• Joint collaboration between CCLRC, UKOLN, and Edinburgh and Glasgow Universities.
• Provide advice, support, research and Development into aspects of Digital Curation for the UK HE community
• Funded jointly by JISC and EPSRC - £1m/year for three years initially. Feb 2004- 2007
• Establish collaboration with industrial partners…
![Page 19: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/19.jpg)
Objectives
• Vibrant research programme
– addressing the wider issues of digital curation
• Collaborative Associates Network of Data Organisations
– strong links across existing community of practice
– engagement with curators (individuals & organisations)
• Services
– to evaluate tools, methods, standards and policies
– a repository of tools and technical information
• ‘Virtuous circle’
– expertise, experience & requirement feed into the DCC research programme
![Page 20: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/20.jpg)
Digital Curation Centre - Organisation
Service Operations Group
management & governance
Industryresearch collaborators
standards bodies
users: communities of practice
U. of EdinburghU. of Glasgow
UKOLN(Bath)
NDCC/NeSC focus & physical presence
curation organisations e.g. DPC
JISC & Research Councils
Management BoardAdvisory
Group
Collaborative Associates Network of DataOrganisations
CCLRC
Steering & Policy Committee
Research Co-ordination Committee
![Page 21: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/21.jpg)
CCLRC’s role within the DCC
• Standards watch
• Standards definition and publication
• Tools watch
• Tools selection and certification
• Registry of metadata standards
• Metadata research
![Page 22: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/22.jpg)
DCC role in Certification
• DCC will help to create
– Standards against which to perform audit and certification• OAIS Reference Model and follow-on work
– Processes for accreditation and certification• Work in Digital Repository Certification Task Force
– Organisation(s) to perform accreditation and certification
![Page 23: The Atlas Petabyte Datastore](https://reader033.vdocuments.site/reader033/viewer/2022051115/568148d3550346895db5ee0d/html5/thumbnails/23.jpg)
ADS SRB Interface for CMS
SRBMCATserver
SRBADS
Server
ADS ServerFarm
Oracle Database Cluster
ADSTape Robot
System
ADSPathtape
server
File System
File System
SRBClient
SRBFile
Server
SRBClient
SRBFile
Server
CSF disk serverAt RAL
CMS nodeExternal to RAL
SRBClient,
External