d. duellmann, cern data management at the lhc1 data management at cern’s large hadron collider...
TRANSCRIPT
D. Duellmann, CERN Data Management at the LHC 1
Data Management at CERN’s Data Management at CERN’s Large Hadron Collider (LHC)Large Hadron Collider (LHC)
Dirk DDirk DüüllmannllmannCERN IT/DB, SwitzerlandCERN IT/DB, Switzerland
http://cern.ch/dbhttp://cern.ch/db
http://pool.cern.chhttp://pool.cern.ch
D. Duellmann, CERN Data Management at the LHC 2
Outline
• Short Introduction to CERN & LHC• Data Management Challenges • The LHC Computing Grid (LCG)
• LCG Data Management Components • Object Persistency and the POOL Project• Connecting to the GRID – LCG Replica Location
Service
CERN - The European Organisation for Nuclear ResearchThe European Laboratory for Particle Physics
• Fundamental research in particle physics• Designs, builds & operates large accelerators• Financed by 20 European countries (member
states) + others (US, Canada, Russia, India, ….)~€650M budget - operation + new accelerators 2000 staff + 6000 users (researchers) from all over the
world
• Next Major Research Project - LHC start ~2007 • 4 LHC Experiments, each with
• 2000 physicists, 150 universities, apparatus costing ~€300M, computing ~€250M to setup, ~€60M/year to run
• 10-15 year lifetime
D. Duellmann, CERN Data Management at the LHC 4
airp
ort
Computer Centre Geneva
27km
D. Duellmann, CERN Data Management at the LHC 5
The LHC machineThe LHC machine
Two counter- circulating
proton beams
Collision energy 7+7 TeV
27 Km of magnetswith a field of 8.4 Tesla
Super-fluid Heliumcooled to 1.9°K
The world’s largest superconducting structure
D. Duellmann, CERN Data Management at the LHC 6
online systemmulti-level triggerfilter out backgroundreduce data volume from40TB/s to 500MB/s
level 1 - special hardware
40 MHz (40 TB/sec)level 2 - embedded processorslevel 3 - PCs
75 KHz (75 GB/sec)5 KHz (5 GB/sec)100 Hz(500 MB/sec)data recording &
offline analysis
D. Duellmann, CERN Data Management at the LHC 7
LHC Data Challenges
• 4 large experiments, 10-15 year lifetime
• Data rates: 500MB/s – 1.5GB/s
• Total data volume: 12-14PB / year• Several hundred PB total !
• Analysed by thousands of users world-wide
• Data reduced from “raw data” to “analysis data” in a small number of well-defined steps
interactivephysicsanalysis
batchphysicsanalysis
batchphysicsanalysis
detector
event summary data
rawdata
eventreprocessing
eventreprocessing
eventsimulation
eventsimulation
analysis objects(extracted by physics topic)
Data Handling and Computation for
Physics Analysisevent filter(selection &
reconstruction)
event filter(selection &
reconstruction)
processeddata
les.
rob
ert
son
@ce
rn.c
h
CERN
Estimated DISK Capacity at CERN
0
1000
2000
3000
4000
5000
6000
7000
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
year
Tera
Byt
es
LHC
Other experiments
Estimated Mass Storage at CERN
LHC
Other experiments
0
20
40
60
80
100
120
14019
98
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Year
Pet
aByt
es
Estimated CPU Capacity at CERN
0
1,000
2,000
3,000
4,000
5,000
6,000
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
year
K S
I95
LHC
Other experiments
Moore’s law
Planned capacity evolution at CERN
Mass Storage Disk
CPU
physics group
regional group
les.
rob
ert
son
@ce
rn.c
h
CERNTier2
Lab a
Uni a
Lab c
Uni n
Lab m
Lab b
Uni bUni y
Uni x
Tier3physics
department
Desktop
Germany
Tier 1
USAUK
France
Italy
……….
CERN Tier 1
……….
The LHC Computing
Centre
Multi Tiered Computing Models - Computing GridsMulti Tiered Computing Models - Computing Grids
D. Duellmann, CERN Data Management at the LHC 11
LHC Data ModelsLHC Data Models
• LHC data models are complex!• Typically hundreds (500-1000) of
structure types (classes in OO)• Many relations between them• Different access patterns
• LHC experiments rely on OO technology• OO applications deal with networks of
objects• Pointers (or references) are
used to describe inter object relations
• Need to support this navigational model in our data store
EventEvent
TrackListTrackList
TrackerTracker Calor.Calor.
TrackTrackTrackTrackTrackTrack
TrackTrackTrackTrack
HitListHitList
HitHitHitHitHitHitHitHitHitHit
D. Duellmann, CERN Data Management at the LHC 12
What is POOL?What is POOL?
• POOL is the common persistency framework for physics applications at the LHC • Pool Of persistent Objects for LHC
• Hybrid Store – Object Streaming & Relational Database• Eg ROOT I/O for object streaming
- complex data, simple consistency model (write once)• Eg RDBMS for consistent meta data handling
- simple data, transactional consistency
• Initiated in April 2002 • Ramping up over the last year from 1.5 FTE to ~10 FTE
• Common effort between LHC experiments and the CERN Database group • project scope and architecture and development • => Rapid feedback cycles between project and its users
• First larger data productions starting now!
D. Duellmann, CERN Data Management at the LHC 13
Component ArchitectureComponent Architecture
• POOL (as most other LCG software) is based on a strict component software approach• Components provide technology neutral APIs • Communicate with other components only via abstract
component interfaces
• Goal: Insulate the very large experiment software systems from concrete implementation details and technologies used today
• POOL user code is not dependent on any implementation libraries • No link time dependency on any implementation packages
(e.g. MySQL, Root, Xerces-c..) • Component implementations are loaded at runtime via a plug-
in infrastructure
• POOL framework consists of three major, weakly coupled, domains
D. Duellmann, CERN Data Management at the LHC 14
POOL ComponentsPOOL Components
POOL API
Storage Service FileCatalog Collections
ROOT I/OStorage Svc
XMLCatalog
MySQLCatalog
EDG Replica Location Service
ExplicitCollection
ImplicitCollection
RDBMSStorage Svc
D. Duellmann, CERN Data Management at the LHC 15
POOL Generic Storage HierarchyPOOL Generic Storage Hierarchy
• A application may access databases (eg streaming files) from one or more file catalogs
• Each database is structured into containers of one specific technology (eg ROOT trees or RDBMS Tables)
• POOL provides a “Smart Pointers” type pool::Ref<UserClass>• to transparently load objects from
the back end into a client side cache
• define persistent inter object associations across file or technology boundaries
POOL Context
FileCatalog
Database
Container
Object
D. Duellmann, CERN Data Management at the LHC 16
Data Dictionary & StorageData Dictionary & Storage
Technology dependent
DictionaryGeneration
CIN
T
dic
tio
nar
yC
INT
d
icti
on
ary
I/O I/O
Data I/O
GCC-XMLGCC-XML
LCG dictionary codeLCG dictionary code
Abstract DDL
Abstract DDL
Code GeneratorCode Generator
LC
Gd
icti
on
ary
LC
Gd
icti
on
ary
Gat
eway
Gat
eway
Reflection
Oth
er
Cli
en
tsO
the
r C
lie
nts
C++ Header
C++ Header
D. Duellmann, CERN Data Management at the LHC 17
POOL File CatalogPOOL File Catalog
• Files are referred to inside POOL via a unique and immutable file identifier which is system generated at file creation time• This allows to provide stable inter-file reference
• FileID are implemented as Global Unique Identifier (GUID) • Allows to create consistent sets of files with internal references
- without requiring a central ID allocation service• Catalog fragments created independently can later be merged
without modification to corresponding data file
Logical NamingLogical Naming
Object LookupObject Lookup
FileIDLFN1LFN1 PFN1, technologyPFN1, technology
LFN2LFN2
LFNnLFNn
PFN2, technologyPFN2, technology
PFNn, technologyPFNn, technology
File Identity and File Identity and metadatametadata
D. Duellmann, CERN Data Management at the LHC 18
StorageElement
EDG Replica Location Services - Basic Functionality
Replica ManagerReplica Location
Service
Replica Metadata Catalog
StorageElement
Files have replicas stored at many Grid sites on Storage Elements.
Each file has a unique GUID.Locations corresponding to the GUID are kept in the Replica Location Service.
Users may assign aliases to the GUIDs. These are kept in the Replica Metadata Catalog.
The Replica Manager provides atomicity for file operations, assuring consistency of SE and catalog contents.
jam
es.
case
y@
cern
.ch
D. Duellmann, CERN Data Management at the LHC 19
StorageElement
Interactions with other Grid Middleware Components
Replica ManagerReplica Location
Service
Replica Optimization Service
Replica Metadata Catalog
SEMonitor
Network Monitor
Information Service
Resource Broker
User Interface orWorker Node
StorageElement
Virtual OrganizationMembership Service
Applications and users interface to data through the Replica Manager either directly or through the Resource Broker.
jam
es.
case
y@
cern
.ch
D. Duellmann, CERN Data Management at the LHC 20
RLS Service Goals
• To offer production quality services for LCG 1 to meet the requirements of forthcoming (and current!) data challenges
• e.g. CMS PCP/DC04, ALICE PDC-3, ATLAS DC2, LHCb CDC’04
• To provide distribution kits, scripts and documentation to assist other sites in offering production services
• To leverage the many years’ experience in running such services at CERN and other institutes
• Monitoring, backup & recovery, tuning, capacity planning, …
• To understand experiments’ requirements in how these services should be established, extended and clarify current limitations
• Not targeting small-medium scale DB apps that need to be run and administered locally (to user)
D. Duellmann, CERN Data Management at the LHC 21
Conclusions
• Data Management at LHC remains a significant challenge because of data volume, project lifetime, complexity of S/W and H/W setups.
• The LHC Computing Grid (LCG) approach is based on eg the EDG and GLOBUS Middleware projects and uses a strict component approach for physics application software
• The LCG-POOL project has developed a technology neutral persistency framework which is currently being integrated into the experiment production systems
• In conjunction with POOL a data catalog production service is provided to support several upcoming data productions in the 100 of terabyte area
D. Duellmann, CERN Data Management at the LHC 23
LHC Software Challenges
• Experiment software systems are large and complex• Developed by teams of expert developers• Permanent evolution and improvement for years…
• Analysis is performed by many end user developers• Often participating only for short time • Usually without strong computer science background• Need simple and stable software environment
• Need to manage change over a long project lifetime • Migration to new software, implementation languages• New computing platforms, storage media• New computing paradigms ???
• Data management system needs to be designed such confine the impact of unavoidable change during the project