WLCG Tier-2 site in Prague: a little bit of WLCG Tier-2 site in Prague: a little bit of history, current status and future perspectiveshistory, current status and future perspectives
Dagmar Adamova, Jiri Chudoba, Marek Elias, Lukas Fiala, Tomas Kouba, Milos Lokajicek, Jan Svec
Prague 04 .09. 2014
OutlineOutline Introducing the WLCG Tier-2 site in Prague
A couple of history flashbacks we celebrate the 10th anniversary
Current issues
Summary and Outlook
HEP Computing in Prague: site praguelcg2 HEP Computing in Prague: site praguelcg2 (a.k.a. the farm (a.k.a. the farm GOLIGOLIAS)AS)
• A national computing center for processing data from various HEP experiments– Located in the Institute of Physics (FZU) in Prague – Basic infrastructure already in 2002, but
– OFFICIALLY STARTED IN 2004 10th ANNIVERSARY THIS YEAR
• Certified as a Tier2 center of LHC Computing Grid (praguelcg2) – Collaboration with several Grid projects.
• April 2008, WLCG MoU signed by Czech republic (ALICE+ATLAS).
• Excellent network connectivity: Multiple dedicated 1 – 10 Gb/s connections to collaborating institutions. Connected to LHCONE.
• Provides computing services for ATLAS + ALICE, D0, Solid state physics, Auger, Star ...
• Started in 2002 with:• 32 dual PIII 1.2GHz, 1 GB RAM, 18 GB SCSI HDD, 100 Mb/s Ethernet rack servers …. (29 of these
decommissioned in 2009)• Storage - disk array 1TB: HP server TC4100
History: 2002 -> 2014History: 2002 -> 2014
Current numbersCurrent numbers
5
•1 batch system (torque + maui)•2 main WLCG VOs: ALICE, ATLAS
– FNAL's D0 (dzero) user group– Other VOs: Auger, Star
• ~ 4000 cores published in the Grid• ~ 3 PB on new disk servers (DPM, XRootD, NFS)•Regular yearly upscale of resources on the basis of various financial supports, mainly the academic grants. •The WLCG services include:
– Apel publisher, Argus Authorization service, BDII, several UIs, Alice VOBOX, Cream CEs, Storage Elements
The use of virtualization at the site is quite extensive.
•ALICE disk XRootD Storage Element ALICE::Prague::SE– ~ 1.113 PB of disk space in total– Redirector/client + 3 clients @ FZU, 5 clients @ NPI Rez a distributed storage cluster
Site UsageSite Usage
ATLAS and ALICE – continuous productionother projects – shorter campaigns
ALICEALICE
ATLASATLAS
Some history flashbacks Some history flashbacks (celebrating the 10(celebrating the 10thth anniversary) anniversary)
88
ALICE PDC 2004 ALICE PDC 2004 resource statistics: resource statistics: 14 sites14 sites
ALICE 2014 resource statistics: 74 sites
99
ALICE PDC ALICE PDC resources statistics resources statistics - 2005- 2005
25 sites in operation
Running jobs (8 November 2005)
Farm Min Avg Max
Sum 1160 1651 1771
CCIN2P3 134 210 231
CERN-L 268 286 304
CNAF 255 362 394
FZK 0 531 600
Houston 0 3 14
Münster 2 58 81
Prague 43 61 71
Sejong 2 2 2
Torino 33 41 43
20062006• ALICE vobox set-up
– fixing problems with the vobox proxy (unwanted expirations)
– AliEn services set up– manually changing the RBs used by the JAs successful participation in the ALICE PDC'06: – Prague site delivered ~ 5% of total computing
resources (6 Tier1s, 30 Tier2s)
• Problems with the fair-share of the site local batch system (then PBSPro)
20072007• still problems with functioning of the ALICE vobox-
proxy • during the PDC'07 problems with job submission due
to malfunctions of the default RBs the failover submission configured
• Prague site delivered ~ 2.6% of total computing resources (significant increase of the number of Tier-2s)
• migration to gLite3.1 ALICE vobox on 64-bit SLC4 machine
• upgrade of the local CE serving ALICE to lcg-CE 3
• repeating problems with job submission through RB's in Oct. the site re-configured for the WMS submission
• migration to the Torque batch system on a part of the site: some WNs on 32bit in PBS and some on 64bit in Torque
• installation and tuning of the creamCE • hybrid state:
– ‘glite’ vobox and WNs, 32bit– ‘cream’ vobox submitting JAs directly to creamCE
Torque, 64bit– Dec: ALICE jobs submitted only to the creamCE
20082008 20092009
20102010• creamCE 1.6 / gLite 3.2/sl5 64bit installed in Prague we were the first ALICE Tier-2 where cream1.6 was tested and put in
production• NGI_CZ set in operation
20112011• Start of IPv6 implementation• The site router got an IPv6 address• Routing set-up in special VLANs• ACLs directly implemented in the router• IPv6 address configuration: DHCPv6• Set-up of an IPv6 testbed
20122012• Optimization of the ALICE XRootD storage cluster performance• an extensive tuning of the cluster motivated by a remarkably different
performance of the individual machines:– data was migrated from the machine to be tuned to free disk arrays at another machine of the cluster.– the migration procedure done so that the data was accessible all the time.– the empty machine re-configured.– number of disks in one array reduced.– set-up of disk failure monitoring.– raid controller cache carefully configured.– readahead option set to a multiple of (stripe_unit * stripe_width) of the underlying RAID array.– no partition table used to ensure proper alignment of the file systems: they were created with right
geometry options ("-d su=X, w=YYk“ mmkfs.xfs switches).– mounting performed with the noatime option.
Parameters of one of the optimized XRootD servers before and after tuning
20132013Almost all machines migrated to SL6CVMFS installed on all machinesConnected to LHCONE
praguelcg2 contribution to WLCG Tier-2 praguelcg2 contribution to WLCG Tier-2 ATLAS+ALICE computing resourcesATLAS+ALICE computing resources
• http://accounting.egi.eu/• A long-term slide down due to problems with financial support
Current issues Current issues
Monitoring issuesMonitoring issues• A number of monitoring tools in use: NAGIOS, MUNIN, GANGLIA, MRTG,
NETFLOW, Gstat, MonALISA• Nagios:
– IPv6-only or IPv4-only servers connected to the central Dual stack node via Livestatus– Some checks can be run form IPv4-only or IPv6-only Nagios nodes
• MUNIN2:– current version 2.0.19– IPv6 in testing
• Ganglia:
– problems if the proper gai.conf is not present– gmetad doesn’t bind to IPv6 address on aggregators
• NetFlow:– plan to switch from v5 to v9 to use nfdump + nfsen
• Some new sensors are needed to fully deploy IPv6, some additional work necessary
• MonALISA REPOSITORY:– A simple test version installed, plans for future development
Network monitoring – weathermap
LHCONE link is heavily utilized(capacity 10 Gbps)
Nagios for alerts
Network architecture at FZUNetwork architecture at FZU
Outgoing IPv4 local traffic from DPM servers
Outgoing IPv6 local traffic from DPM servers
IPv6 deploymentIPv6 deployment
•Currently on Dual-stack: dpm headnode, all production disk nodes, all but 2 subclusters of WNs•Over IPv6 goes: dpns between disknodes and headnode, srm between WNs and headnode, actual data transfer via gridftp•IPv6 enabled on the ALICE vobox
Site services managementSite services management
• Since 2008 services management done with CFEngine version 2– cfagent Nagios sensor developed: a python script checking CFEngine logs for
fresh records (error signals if the log is too old)
• CFEngine v2 used for production
• Puppet used for IPv6 testbed
• Migration to the overall Puppet management in progress
NGI_CZNGI_CZ
• Since 2010, NGI_CZ is recognized and in operation: https://wiki.metacentrum.cz/metawiki/NGI_CZ#Farma_golias_aka_praguelcg2
• all the events and relevant information about praguelcg2• 2 sites involved: praguelcg2 and prague_cesnet_lcg2• significant part of the services provided by the praguelcg2 team
Services provided by NGI_CZ for the EGI infrastructure: Accounting (APEL, DGAS, Cesga portal) Resources database (GOC DB) Operations - https://operations-portal.egi.eu/
ROD (Regional Operator on Duty) Top level BDII
VOMS servers Meta VO User support (GGUS/RT) - https://rt4.cesnet.cz/rt/
Middleware versions: UMD 3.0.0, EMI 3.0
Use of external resourcesUse of external resources
• Not much really to choose from• Longer term usage of the cluster ‘skurut’ in Prague: site
prague_cesnet_lcg2, courtesy of CESNET association– a long-time established cooperation
• NGI_CZ provided a single opportunity to use ~ 35 TB disk storage in Pilsen– for testing purposes mostly– dCache manager used– Evaluating the effect of switching/tuning TTreeCache , dCap RA– Not much of help as an extension of home resources
Summary and OutlookSummary and Outlook Prague Tier-2 site was performing as a distinguished member of the WLCG
collaboration for 10 years nowA stable upscale of resourcesHigh-level accessibility, reliable delivery of services, fast response to problems
Into the upcoming years, we will do our best to keep up the reliability and performance level of the services
Crucial is the high-capacity, state-of-the-art network infrastructure provided by CESNET
However, the future LHC runs will require a huge upscale of resources which will be impossible for us to achieve with the expected flat budget
As everybody else these days, we are in a search for external resources: got some help from CESNET but need more. As widely recommended, we very likely will try to collaborate with non-HEP scientific projects to get access to additional resources in the future
A couple of current plots A couple of current plots
RUNNING ALICE JOBS IN PRAGUE in 2013/2014:Average = 996, maximum = 2227Total number of processed jobs: ~ 5 millions
GRID for ALICE in Prague – Monitoring jobs (MonALISA)GRID for ALICE in Prague – Monitoring jobs (MonALISA)
2626
ALICE Disk Storage Elements – 62 endpoints, ~ 34 PBALICE Disk Storage Elements – 62 endpoints, ~ 34 PBPrague scores with the largest Tier-2 storagePrague scores with the largest Tier-2 storage
NETWORK TRAFFIC ON PRAGUE ALICE STORAGE CLUSTER in 2013/2014:
(Total disk space capacity 1.113 PB)
Max total traffic IN/write: 195 MB/sMax total traffic OUT/read: 1.05 GB/s
Total data OUT/read : 5.322 PB
GRID for ALICE in Prague – Monitoring storage (MonALISA)GRID for ALICE in Prague – Monitoring storage (MonALISA)