The Finnish Grid Infrastructure Computing Environment and Tools Wednesday, 21 st of May EGI Community Forum 2014 Helsinki Lus Alves Systems Specialist at CSC IT Center for Science, Ltd. Finland csc.fi CSC - IT Center for Science, Ltd. Private and non-profit company owned by the Ministry of Education and Culture; Provides IT support and resources for academia, research institutes and companies; Part of the Finnish national research structure; Finnish partner on: Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 2 Computing Resources for Science Sisu - Cray XC30 Super Computer [Upgrading] Massive computational challenges > cores, > 23TB memory Theoretical peak performance > 240 Tflop/s Taito - HP-Cluster [Upgrading] Small and medium-sized tasks Theoretical peak performance 180 Tflop/s Hippu - Application server Interactive usage, without job scheduler Post-processing, e.g. vizualization Pouta - Cloud Service [New] Openstack Finnish Grid Infrastructure (FGI) Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 3 About FGI Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 4 In the beginning, we had M-Grid Interest in Grid technology rose in Finland during 2003 A consortium of 7 Universities, HIP and CSC was formed which successfully obtained funding for the FIRST Finnish Computing Grid M-Grid Effort was driven by CSC and Kai Nordlund (HU) M-Grid was operational from 2005 to Sites Theoretical total computing capacity ~ 2.5 TFlops Infrastructure had aged significantly by end 2008 Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 5 Then, FGI is born Second generation M-Grid planned since 2009 Application for funding made in October 2010 FIRI grant approved beginning 2011 Consortium of 9 universities and CSC Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 6 Finnish Grid Infrastructure (FGI) 10 Computing Clusters connected through network and Grid middleware that provide a peak capacity of 154 TFLOPS; Available to any researcher affiliated to a Finnish Research Institution; Operations and coordination by CSC; Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 7 FGI in EGI FGI is the Finnish NGI and EGI sees us as NGI_FI CSC is the Finnish Operations Center Uses the monitoring and service tools provided by EGI Follows EGI procedures for operations Manages the Regional Operational on Duty team Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 8 FGI - also a Federation Sites maintain their own clusters Local use is open at all sites Site administrators are encouraged to collaborate and communicate Attending weekly admin meetings Providing Grid software support for users Becoming part of the FGI community Small team from CSC coordinates general administration and support Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 9 Hardware details Standard node configuration HP SG7 scale out dual 6 core 2.67GHz Xeon X GB memory (min.) Big Memory nodes HP Proliant DL 580 G7 server 1 TB memory GPGPU nodes 2 Nvidia Tesla cards in a standard compute node Disk servers: Total storage capacity of about 1 PB QDR InfiniBand & Gigabit ethernet for interconnect and network. Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 10 Operating system is Scientific Linux 6 Scheduler used is Slurm Hardware distribution Aalto: 112 nodes, 8 GPGPU nodes, two 1TB big memory nodes Lappeenranta: 16 nodes Eastern Finland: 64 nodes Helsinki:49 nodes, 20 GPGPU nodes, one 1 TB big memory node Jyvskyl: 48 nodes, 8 GPGPU nodes Oulu: 30 nodes Tampere (TUT):37 nodes, 8 GPGPU nodes, one 1 TB big memory node Turku:20 nodes bo Akademi: 8 GPGPU nodes CSC:24 nodes (with 96GB memory) Operating System and Scheduler Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 11 Finnish University and Research Network FUNET is an advanced data communications network serving the Finnish research community. It connects about 80 research organizations and over users. Membership in Funet is open to all Finnish university-level academies and public research institutions.Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 12 Grid Middleware FGI uses the ARC middleware Developed by NorduGrid, part of the European Middleware Initiative (EMI) More info:Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 13 Software distribution - Cern VM-FS Central repository for FGIs software Makes it easy to distribute software Modules and Runtime Environments are shared through CVMFS Each Cluster has a Squid proxy that caches most used files More details on Ulf Tigerstedt presentation Managing multidisciplinary software repositories for grid with CernVM-FS here:Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 14 FGI Computing Environment and Tools Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 15 Scientist's User Interface (SUI) More info at:Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 16 ARC xRSL file generator tool on SUI Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 17 arcrunner Grid Job Submission Manager Gridification tool developed and maintained by Kimmo Mattila (CSC) Actively used to run large job sets on FGI i.e. BLAST, InterProScan, Exonerate Selects suitable and available resouces, Submits, Monitors and Fetches jobs outputs arcrunner -xrsl average.xrsl Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 18 Runtime Environments (RTE) Extended Resource Specification Language (xRSL) file example: Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 19 & (executable=runamberMPI.sh) (jobname=amber-test) (stdout=std.out) (stderr=std.err) (gmlog=gridlog_1) (walltime=1h) (memory=200) (disk=1000) (count=6) (runtimeenvironment=ENV/ONENODE) (runtimeenvironment=APPS/CHEM/AMBER-12) (inputfiles= ( "gbin" "gbin" ) ( "md12.x" "md12.x" ) ( "prmtop" "prmtop" ) ) (outputfiles= ( "output.tar" "output.tar" ) ) Available RTEs: AMBER AutoDock BLAST Bowtie and BWA Cufflinks Elmer EMBOSS Exonerate FreeSurfer GPAW Gromacs GSNAP HMMER 3.0 Interproscan5 Matlab Compiler Runtime MISO MrBayes NAMD ORCA R SAMtools SHRiMP TopHat Modules Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 20 Results Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 21 Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 22 Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 23 FGI FGI in the Clouds Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 24 Future: A Grid-Cloud Hybrid Cloud-Enabled FGI Application for funds submitted in April 2014 Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 25 Thank you Questions? Credits and special thanks to: Jura Tarus; Ulf Tigerstedt; Kimmo Mattila; Universities FGI admins; CSCs CE group and Staff; FGIs former members; FGI users. Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 26 More information about and how to use FGI: Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 27 Conclusions Performance comparison Per core performance ~2 x compared to Vuori/Louhi Better interconnects enhance scaling Larger memory Smartest collective communications Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 28 Sisu&Taito vs. FGI vs. Local Cluster Sisu&Taito (Phase 1) FGIMerope Availability Available CPU Intel Sandy Bridge, 2 x 8 cores, 2.6 GHz, Xeon E Intel Xeon, 2 x 6 cores, 2.7 GHZ, X5650 Interconnect Aries / FDR IBQDR IB Cores / RAM/core 2 / 4 GB 16x 256GB/node 2 / 4 / 8 GB4 / 8 GB Tflops 244 / GPU nodes in Phase2886 Disc space 2.4 PB1+ PB100 TB Wednesday, May 21, 2014 EGI CF 2014 Helsinki, Finland 29