ian bird lcg project leader project status report wlcg lhcc referees’ meeting 21 st september 2009
TRANSCRIPT
Ian BirdLCG Project Leader
Project status report
WLCG LHCC Referees’ meeting21st September 2009
Agenda
Overall status Planning and milestones Follow up actions from STEP’09 Status of Tier 0 Status of Tier 1 sites (seen via milestones etc) Resource requests and installation status
Comments on CRSG report & process
Status of preparations for EGEE EGI transition
Experiment reports NB LHCb external meeting, thus no LHCb representative today
Overall summary
Since July has been fairly quiet Experiments did follow-up tests with some sites where problems had
been seen during STEP’09 exercise In general workloads have been continuous
WLCG service has been running according to the defined procedures Reporting and follow up of problems at
same level
Focus on stability and resolution of problems in preparation for data taking
2009 2010 2011
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb
SU pp running HI
2009 2010 2011
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb
SU PP running HI?
STEP’09May+June
WLCG timeline 2009-2010
2009 Capacity commissioned
2010 Capacity commissioned
Switch to SL5/64bit completed?
Deployment of glexec/SCAS; CREAM; SRM upgrades; SL5 WN
EGEE-III ends EGI ... ??? ??EGI
HEP – SSC
EMI
(SA4)
Resource review
Milestones - 1
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
Pilot Jobs Frameworks
WLCG-08-14 May 2008
Pilot Jobs Frameworks studied and accepted by the Review working groupWorking group proposal complete and accepted by the Experiments.
ALICE ATLAS CMSLHCb
Nov 2007
Tier-2 and VOs Sites Reliability ReportsWLCG-08-09 Jun
2008Weighted Average Reliability of the Tier-2 Federation above 95% for 80% of Sites Weighted according to the sites CPU resources
See separated table of Tier-2 Federations 80% of the Sites above 95% reliability
WLCG-08-11 Apr2009
VO-Specific Tier-1 Sites Reliability Considering each Tier-0 and Tier-1 site
(and by VO?)
Jul 2009
Aug 2009
Sep 2009
Tier2 reliabilities: published monthly40% > 95% reliability; 80% > 90% reliability (federation weighted)
Tier 1: VO-specific reliabilities now published regularly
Milestones - 2
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
SL5 Milestones
WLCG-09-21 Mar 2009
SL5 gcc 4.3 (WN 4.1 binaries)Tested by the ExperimentsExperiments should test whether the MW on SL5 support their grid applications
ALICE ATLAS CMS LHCb
WLCG-09-22 Jul 2009
SL5 Deployed by the Sites (64 bits nodes)Assuming the tests by the Experiments were successful. Otherwise a real gcc 4.3 porting of the WN software is needed.
Tier-1 Sites Procurement - 2009
SLC5: MB issued statement urging sites to migrate asap; clarifying no show stoppers for the experiments
Ideally would like significant resources under SL5 available before data taking.
Many large sites have migrated
Milestones - 3
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
SCAS/glExec Milestones
WLCG-09-17 Jan 2009
SCAS Solutions Available for DeploymentCertification successful and SCAS packaged for deployment
Done in March 2009
WLCG-09-18 Apr 2009
SCAS Verified by the ExperimentsExperiment verify that the SCAS implementation is working (available at CNAF and NL-T1)
ALICEn/a
ATLASCMSn/a ?
LHCb
WLCG-09-19 09-18 + 1
Month
SCAS + glExec Deployed and Configured at the Tier-1 SitesSCAS and glExec ready for the Experiments.
WLCG-09-20 09-18 + 3
Month
SCAS + glExec Deployed and Configured at the Tier-2 SitesSCAS and glExec ready for the Experiments.
Glexec/SCAS available for deployment; deployed at several test sites; general deployment is very slow
Milestones – 4
Accounting Milestones
WLCG-09-02 Apr 2009
Wall-Clock Time Included in the Tier-2 Accounting Reports The APEL Report should include CPU and wall-clock accounting
APEL
WLCG-09-03 Jul 2009
Tier-2 Sites Report Installed Capacity in the Info System Both CPU and Disk Capacity is reported in the agreed GLUE 1.3 format.
% of T2 Sites Reporting
WLCG-09-04a Jul 2009
Sites publishing the User Level Accounting information
WLCG-09-04b Jul 2009
User Level Accounting verified and approved by the Experiments ALICE ATLAS CMS LHCb
~100
Almost all sites report CPU information
Comparison of published data and expected values tbd
Follow up on published storage information to be done
User-level accounting available in APEL; policy now approved; sites asked to enable the publishing of the data
Milestones – 5
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
STEP 2009 - Tier-1 Validation
WLCG-09-23 Jun 2009
Tier-1 Validation by the Experiments ALICE n/a n/a n/a n/a n/a
ATLAS n/a n/a n/a
CMS n/a n/a n/a n/a
LHCb n/a n/a n/a n/a n/a
All Tier 1’s tested by STEP’09
Were 3 with concern: ASGC, NL-T1, FZK
FZK and NL-T1 have since demonstrated satisfactory functionality
ASGC still of concern: meeting with ASGC management in early Oct
Milestones – 6
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
CREAM CE Rollout
WLCG-09-25 Apr 2009
Release of CREAM CE for deployment
WLCG-09-26 May 2009
All European T1 + TRIUMF and CERN at least 1 CE. 5 T2s supporting 1 CE n/a n/a n/a n/a n/a
WLCG-09-27 Jul 2009
2 T2s for each experiment provide 1 CREAM-CE each. ALICE ATLAS CMS LHCb
WLCG-09-28 Sep 2009
50 sites in addition to the ones above
Milestones – 7
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
Metrics and Monitoring Milestones
WLCG-09-08 Nov 2008
Experiments Dataflows clear for the Tier-1 SitesExperiments should present the data flows they expect to reach at the Sites (a la LHCb)
ALICE ATLAS CMSLHCb
Nov 2007
WLCG-09-09 Removed
Tier-1 Sites Define Their MSS MetricsTier-1 Sites specify which metrics are going to be collected to demonstrate the dataflow supported
WLCG-09-10 June 2009
Tier-1 Sites Publish Their MSS Metrics in SLSTier-1 Sites make their current MSS metrics available via SLS
WLCG-09-11 TDB Automatic Alarms (SAM, etc) at the Tier-1 SitesTier-1 Sites should be able to automatically send, receive and handle alarms and problem notifications
WLCG-09-12 TDB Monitoring of the Storage SystemsThe Storage systems used provide monitoring information to Sites and Experiments CASTOR dCache DPM StoRM BestMan
WLCG-09-13 TDB Performance Metrics?User Response, Services DowtimesOperations KPI
Milestones – 8
01-Sep-09 WLCG High Level Milestones
Done (green) Late < 1 month
(orange) Late > 1 month (red)
ID Date Milestone ASGCCC
IN2P3CERN
DE-KIT
INFN CNAF
NDGF PIC RALSARA NIKHE
F
TRIUMF BNL FNAL
CPU Benchmarks/Units Milestones
WLCG-09-14 Dec 2008
CPU New Unit Working Group CompletedAgreement on Benchmarking Methods Conversion Proposal and Test Machines CPU New Benchmarking Unit Working Group
WLCG-09-15 Feb 2009
Sites Pledges in HEPSPEC-06Pledged from the Sites should be converted to the new unit LCG Office
WLCG-09-16 Apr 2009
New Experiments Requirement in HEPSPEC-06Experiments should convert their requirements to the new unit (or by LCG Office) ALICE ATLAS CMS LHCb
WLCG-09-24 May 2009
Sites Benchmark their Capacity in the HEPSPEC-06Resources from the Sites should be converted to the new unit
Site benchmarking in progress
STEP09 follow-up:Tier 0 + Tier 1s
All: MSS metrics Need instant real-time monitoring of
throughput (and per day overview) and to view transfers per experiment (WAN in/out; LAN – WNs)
Tools for site and grid NL-T1:
Communication & SIRs Lack of tape drives (now installed) DMF tuning needed? Unexplained low performance LAN b/w to WN too small Shared SW area problems Repeat tests
ASGC: Castor version? Job scheduling T2 v T1; ATLAS v CMS Low efficiency for CMS reprocessing jobs Repeat tests
FZK: Improve communication to world SAN issues Shared SW area problems SRM server overload dcap access problems Too many lcg-cp overload gridftp Repeat tests
NDGF: No MSS performance monitoring Low throughput Analysis jobs overloaded network No Panda pre-staging What is action to improve?
CNAF: Shared SW area problems
Site visits – planned for FZK + NL-T1
This was the summary immediately after STEP’09
Tier 2s
Shared SW areas affect CPU efficiencies in some sites (also T1) ATLAS efficiencies different between WMS and PANDA
Likely due to different data access methods Data transfer timeouts (what? see analysis summary) Intra-VO fairshares how to manage? ( GDB?) VO communications
Need for VO-independent channel to reach all T2 admins Network infrastructure not good enough for data rates
This was the summary immediately after STEP’09
Data management
Some detailed tuning issues; e.g.: FTS: Timeouts for large files FTS: Increase the number of concurrent jobs to increase bandwidth LFC: Deploy bulk methods (new additions)
dCache: These were discussed in August MB:
Need a clear strategy for Chimera (and risk analysis) Explain “golden” release v what we have (and risk analysis)
Site decisions, but must be based on understanding of risks Other issues:
Data transfer timeouts (dcap/rfio) at high (analysis) load Better data access strategies???
Pre-stage vs copy locally vs Posix-io General investigation of data access methods and problems
Work already done by ATLAS – extended to other experiments Shared software areas:
Site issue – NFS not a good solution Monitoring:
More real time on MSS transfers (Tier 0 and useful at Tier 1s) MSS metrics Improve dashboards – ATLAS vs CMS
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 18
Tier0 Status
• Resource Status– Still on schedule to have balance of 2009
pledge operational in October, but schedule is tight, especially for commissioning of new water-cooled racks.
– 1st pass of tape repack (to 1TB media) complete; now have 29PB of free media space
– Migration to SLC5 continues (50% of resources now on SLC5), but still concerns over general take up.• delay of switch of lxplus alias is a concern.
SLC4
SLC5
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 19
Tier0 Status
• CASTOR– Generally smooth running since STEP
• Deploying 2.1.8.10 across LHC experiments; completes tomorrow with LHCb and can then have improved monitoring of migration rates
• CASTOR 2.1.9.0 deployed on tape servers; this is the base for future tape optimisations (e.g. transfer aggregation)
– Main issue has been service degradations due to inadvertent misuse by individual users• An operations priority for future developments
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 20
CMS
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 21
ATLAS
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 22
LHCb
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 23
ALICE
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Tier0 Status - 24
Tier0 Status
• Computer Centre Infrastructure– Decision on construction of new Computer
Centre in Prévessin suspended pending clearer view of• costs for a container based solution• long-term computing requirements
– Projects now underway to• Reinforce “critical power” infrastructure in B513
to guarantee support for load of ~480kW.• Plan installation of containers in 2010 (for both
“critical” and “physics” loads.
Mass storage
Upgrades for dCache and Castor – to get stable versions for data taking
dCache: 1.9.4 introduces ACLs to ensure file protection Migration to new namespace (Chimera) from pnfs for better scalability
(recommended for large sites) Site decision, but had full discussion of risks/advantages
1.9.5 will be “golden” version for support during 1st year of data taking Castor:
2.1.9 (consolidation version) will be deployed at CERN Encourage RAL, CNAF, ASGC to upgrade to this for better analysis
support
Resolution of open issues from SRM functionality requests Remember: “addendum” of functionality that had bee requested, but put on
hold in order to ensure stable versions See next slides
Missing SRM features by priority
Extremely important Space protection
But at least tape protection is available everywhere File pinning
On CASTOR is almost non-existent But pinning importance is lower with file pre-staging services
Rather important VOMS awareness
Missing from CASTOR
Useful Target space selection
Missing in dCache an StoRM
Nice to have Ls returns all spaces with a copy of the file
Missing in CASTOR (where it makes sense)
Scalability and stability (CMS) Main issues at Tier-2’s It is required that the SRM front-end should guarantee that the activity of a single user
could not disrupt the service
Middleware
WMS: WMS 3.2 available – can submit to CREAM and to the ARC-CE
Compute Element New version of CREAM with many bug fixes – now deployable in parallel
with existing CEs Once the old WMS have been retired, can seriously use CREAM in
production Glexec/SCAS (needed for multi-user pilot jobs)
Now ready ... But deployment take-up by sites is very slow Information system:
Latest version can also handle new schema with improved service discovery
Generally: Continuous process of patching and updating middleware as needed
Resources LHCC report and CRSG report only recently available
2009 resources being installed (deadline October)
2010 resources Use requests as presented in July review/listed in final CRSG report
Except: ALICE where RSG report has wrong numbers – updates provided by ALICE; anticipate updated CRSG report
Message, agreed with CERN management, sent to funding agencies for RRB preparation: experiment requests have been reviewed by both the LHCC and C-RSG bodies who have been
working closely together. The result of their work led to agreement on the experiment requests in the attached table* to be used until data taking and the running conditions beyond 2010 are known. Their results also clearly state that computing resources should not in any way hinder the LHC physics programme.
Consequently at the next CRRB..., the main emphasis will be on the 2010 resource situation. In order to prepare the meeting, each site is asked to:
confirm that 2009 resources pledged are now available in production, or if not explain any difference
confirm 2010 pledges, and for sites supporting more than one experiment, indicate the split between the experiments
Until first data-taking and further news about the running scenario beyond 2010 is known, the CRRB Chairman has agreed that exceptionally we do not require your pledge data beyond 2010 for this meeting.
Resource installations & issues
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
CPU Time Delivered
month (2009)
MS
I2K
-da
ys
.
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
Disk Storage Used
month (2009)
Te
raB
yte
s
.
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
Tape Storage Used
month (2009)
Te
raB
yte
s
.
ASGC NL-T1
NL-T1NL-T1
Comments on review process/outcome
Timeliness of report and overall timing of the process Long procurement cycles are unavoidable and we must have the conclusions earlier Results needed in the Spring for requests for the following year (now in Autumn RRB) The report is essential well in advance of the RRB to allow discussion and feedback
Much effort has been invested by the RSG members and others – in future it might be better to better coordinate between the LHCC and the RSG?
Is having dedicated scrutineers per experiment most efficient? E.g. comparison of CMS and ATLAS was required.
Actually the fundamental questions have not been answered: Trigger rates Amount of MC # of reprocessings
Experiment models and RSG simple models agree within 10% (level of uncertainty any way); future understanding must be based on: Experience with data Real answers to the above basic parameter questions
C-RRB (and C-RSG) modelled on other RRBs; but computing is different and the process needs to reflect that: Combine RSG and LHCC reviews/reviewers? Change timing ...