porting scientific applications on grid: cern experience
DESCRIPTION
Porting Scientific Applications on GRID: CERN Experience. Patricia M é ndez Lorenzo CERN (IT-PSS/ED) Trieste, 10th February 2006 ICTP/INFM-Democritos Workshop on Porting Scientific Applications on Computational GRIDs. Outlook. ◘ This is an introductory presentation - PowerPoint PPT PresentationTRANSCRIPT
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Porting Scientific Applications on GRID: CERN ExperiencePatricia Méndez Lorenzo
CERN (IT-PSS/ED)
Trieste, 10th February 2006
ICTP/INFM-Democritos Workshop on Porting Scientific Applications on Computational GRIDs
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Outlook
◘ This is an introductory presentation
◘ Let’s see what is CERN, what is LCG, the elements, the actors and how to get involved
◘ This afternoon, during the 2nd talk we will see some practical examples of communities getting involved in the GRID
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
CERN
The European Organization for Nuclear ResearchThe European Laboratory for Particle Physics
◘ Fundamental research in particle physics◘ Designs, builds &operates large accelerators
◘ Financed by 20 European countries (member states) + others (US,
Canada, Russia, India, etc)2000 staffs + 6000 users from all over the
world
◘ Next huge challenge: LHC (starts in 2007) experiment: 2000 physicists, 150 universities, with an operation life greater than 10 years
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
LHC- Physics Goals
◘ Higgs particle Key particle in the Standard Model that could explain the elementary particle masses
◘ Search for super-symmetric particles and possible extra dimensionsTheir discovery would be a serious push for Super Symmetric theories or “String Theories” aiming at the unification of the fundamental forces in the Universe
◘ Anti-matter issuesWhy the Universe is made of matter instead of an equal quantity of matter and antimatter
◘ Understand the early Universe (10 -20 – 10-6 seconds)Soup of quarks and gluons stabilized into nucleons and then nuclei and atoms
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
The LHC Experiment
The LHC: Generation of 40 million particle collisions (events) per second
at the center of each for experiments
Reduce by online computers that filter out
a few hundred good events per sec
Recorded on disk and magnetic tape
at 100-1000 MB/sec: 15 PB/year
ALICE, ATLAS, CMS
and LHCb
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
The LHC Computing Environment
LCG (LHC computing Grid) has been developed to build and maintain a storage and analysis infrastructure for the
entire high-energy physics community
◘ LHC is beginning the data taking in summer 2007➸ Enormous volume of data
• Few PB/year at the beginning of the machine operation• Several hundred PB yearly produced for all experiments in 2012
➸ Large amount of processing power ◘ As a solution a LCG world-wide Grid is proposed
➸ Established using a world-wide distributed federal Grid➸ Many components, services, software, etc, to coordinate
◘ Takes place at an unprecedented scale➸ Many institutes, experiments and people working closely together
◘ LCG must be ready at full production capacity, functionality and reliability in less than 1 year!
LCG is an essential part of the chain allowing the physicists to perform their analyses
➸ It has to be a stable, reliable and easy to use service
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
LCG: The LHC Computing Grid
Tier-0 – the accelerator centreData acquisition and initial
Processing of raw dataDistribution of data to the different
Tier’s
Canada – Triumf (Vancouver)France – IN2P3 (Lyon)Germany – Forschunszentrum KarlsruheItaly – CNAF (Bologna)Netherlands – NIKHEF/SARA (Amsterdam)Nordic countries – distributed Tier-1
Spain – PIC (Barcelona)Taiwan – Academia SInica (Taipei)UK – CLRC (Oxford)US – FermiLab (Illinois) – Brookhaven (NY)
Tier-1 – “online” to the data acquisition process high availabilityManaged Mass Storage – grid-enabled data serviceData-heavy analysisNational, regional support
Tier-2 – ~100 centres in ~40 countriesSimulation
End-user analysis – batch and interactive
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
LCG in the World
May 2005140 Grid sites
34 countries
12000 CPUs
8 PetaBytes
30 sites3200 cpus
25 Universities4 National Labs2800 CPUs
Grid3
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
What is the structure we have at CERN?
◘ CERN is the T0
➸ Deployment: We pack and distribute the software to the sites
➸ Development: Development of new projects and also part of the software
➸ Support: Assistance to experiments and sites
EXPERIMENTS
SITES
LCG DEPLOYMENT
ARDA
EIS APPLICATIONS
Contact through ARDA-EIS
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Our Tier1 centers
ALICE ATLAS CMS LHCb
1 GridKa Karlsruhe Germany X X X X
2 CCIN2P3 Lyon France X X X X
3 CNAF Bologna Italy X X X X
4 NIKHEF/SARA Amsterdam Netherlands X X X
5 NDGF Distributed Dk, No, Fi, Se X X
6 PIC Barcelona Spain X X X
7 RAL Didcot UK X X X X
8 TRIUMF Vancouver Canada X
9 BNL Brookhaven X
10 FNAL Batavia X
11 ASCC Taipei X X
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
The Elements of the Middleware
UI UI
CE
RB/BDII
SE
WN
WN
WN
WN
WN
WN
LFC
Connections to UI
Resources Searching
Sent to th
e batc
h
system Distribution to CPUs
Ouputs copied to
Storage Resources
Catalogs getting track of the inputs
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Current status of the Experiments and the GRID
◘ They have to get ready
➸ How to take raw data and how
to store and distribute them
➸ How to produce simulated data
➸ How to analyze data
◘ They are not alone,
EGEE/LCG assist them➸ ARDA Group
➸ Application area Group
Successfully completed 6 months before the data taking
They have to be ready to a high stability level
Ramp up their capacity to twice the nominal data rates expected for the production phase
EXPERIMENTS SITES
SERVICES: EGEE/LCG
THIS IS A FULL CHALLENGE
FOR ALL OF US
THE SUPPORT AMONG THE
3 ACTORS IS FUNDAMENTAL
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
We have a fundamental Challenge in front of us:
Service ChallengeAs we have just seeing experiments and Grid teams are ramping up… but making what exactly?
Service Challenges (SC): Grid part ◘ We provide services to the users (done) ◘ But we have to test them to the necessary level of functionality, reliability and scale ◘ We are preparing, hardening and delivering the production of the LCG environment ◘ Moreover we have to run an environment as realistic as possible
Data Challenges (DC): Experiments part ◘ Experiments test their LCG based production chains and performance of the Grid fabric
➸ Processing data from simulated events➸ Emulating step by step the scale they will have to face during real data taking
◘ From 2005 experiments include SC testing as part of their DC
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Assistance to Experiments
ARDA: A Realization of Distributed Analysis for LHC
◘ Main Goal: ➸ Coordination of activities needed to prototype distributed analysis systems for the LHC➸ It does NOT develop middleware, go one step further, closer to the users
◘ Main Actors:➸ People of each experiment (they know what they need)➸ Middleware development team (they know what they have)➸ Experiment support team: EIS (they can contact both)
◘ New Gridifications:➸ Coordinated by this group (You will go through us, surely)➸ Application of their tools to new communities (We can help you)
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
OK, now you know the project, its elements and its actors
And however you want still to
go inside… ☺Good choice, let’s see
how
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
New Contributions
◘ Several different communities are joining the GRID project◘ It has applications to all these fields needing large amount of computational and space resources
◘ We are direct supporting (outside LHC exp.):
➸ Biomed (medical applications)
➸ Geant4 (simulation toolkit)
➸ UNOSAT, ITU (UNO projects)
◘ And how to put then inside the EGEE/LCG is a HOT TOPIC at this moment
This afternoon
Next part of this 1st talk
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Before proceeding…
◘ There are several points under discussion at this moment about how to admit new Virtual Organizations arriving to the GRID
➸ Last discussion, during the Grid Deployment Board meeting this Wednesday
◘ I going to explain the procedure if the request is associated in somehow with CERN
➸ This is my experience
◘ What to do in the rest of cases is being discussed➸ It should have maximal priority inside the EGEE environment
➸ From my point of view it must be flexibly ➸ I will explain you also my proposal
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
What we are doing now for Geant4
◘ Full Geant4 gridification this afternoon◘ Right now, just see it as a new community…◘ But a “special” community because: ➸ They pretend to run twice per year (Not the whole year)
► To validate their own software
➸ It is used as simulation tool by many Grid VOs► A good validated product, will make experiments life easier
➸ Its software is very well known ► stable, reliable, quite long tested► LCG thought to use this software as a part of the LCG test
➸ The tests inside LCG were asked by people placed at CERN► Geant4 at CERN, LCG support at CERN, easier to gridify and support this community
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Immersion of Geant4 in LCG
◘ As any other new VO, they will need support➸ Provided by EGEE/LCG
◘ Support is fundamental➸ To involve you quickly and safely ➸ Till you gain familiarity it will play the role of software, production and VO manager➸ We dial with the site on your behalf
◘ They will need to gain familiarity with the system and to test it before passing through the whole EGEE/LCG VO procedure➸ You have to learn before taking a decision➸ This will be a situation to face more and more
UNOSAT and more
◘ EGEE/LCG has to face:➸ What to do with light new VOs arriving ➸ The solution should not be “dteam” (deployment team VO)➸The most important point is already setup: YOU HAVE SUPPORT
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Our Challenge
Provide a formal procedure to give them support
◘ It is not the same if the community contacts directly the support at CERN◘ This should not be like that and a common policy should be found
➸ A regional policy delegation could be the solution ◘ Most of all in terms of VO policy a procedure should be fixed◘ Something we still have to clarify
➸ Are the sites free enough to provide resources to any new VO as local users?
► Well, this is what we have done for Geant4 and this does seem the method to follow
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Current Procedure
2) EGAAP
3) NA4/SA14) CICs/ROCs
Init
ial
dep
loym
ent
con
fig
ura
tio
n
Asks for change
Recommended VO candidate
Resource proposal
VO requirements
1) New applications
Geant4 application
Sites
1. Geant4 is quite known
2. It is fully supported
3. It was the 3rd production
4. We had a short time to begin the production
EGEE
ProcedureLC
GP
roce
dure
Wha
t we
did
The solution for us is to begin the support here
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
What we have done
◘ We presented the Geant4 community last GDB and we asked for local support
➸ In my ideal world, once you ask for the entry a support person should be assigned to you to discuss with you your hopes and the viability
➸ Let him to deal with the LCG/Sites Management ◘ At the same time we were following the official procedure to become VO ➸ After testing the product and seeing you want it, this is mandatory
◘ In the case of Geant4 ➸ We asked directly for the support to Geant4 site per site ➸ We got in time in 5 sites ➸ We got the total amount of CPUs required for Geant4 (120CPUs)
➸ LCG Efficiency: 99% ➸ At any moment the Geant4 production was totally followed not to interfere in
a wrong way with the sites► This is fundamental for the sites► Your LCG support is responsible of your production► Many other communities are running and you should not interfere them (and they should not interfere you!)
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Light-weight VO registration
◘ We are considering to have an incubator VO to assist new communities
➸ This VO is standard for all new communities ➸ It allows you to play with the system
➸ During this time your support plays with you ➸ You do not decide the policy of this VO, we do it ➸ You will not make productions inside, you just learn ➸ LCG has already experience, we know how to deal with
◘ Warning: You should have a limited time inside ➸ Then you are a integrated VO ➸ You should go through the registrations steps ➸ Now your support makes exactly that: LCG support no more, no less
► With a strict LCG support no problems should be seen► Experiments happy, sites happy, LCG happy
Trieste, 10th February 2006 Patricia Méndez Lorenzo
Enabling Grids for E-sciencE
INFSO-RI-508833
Final Message
LCG was born with a clear objective
Assist LHC experiments during their real data taking
The project is however extensible and quite attractive for any other community
We are thinking at this moment how to procede with the light-weight VOs
This is foreseen and it is part of the project
SUPPORT is the most important point if you arrive to the GRID... And the infrastructure is already developed