HPC in France and EuropeOverview of GENCI and PRACE

Stéphane REQUENA, CTO GENCI

2

Supercomputing - driving Science and Industry through simulation

Franco-British Workshop on Big Data in Science

EnvironmentWeather / ClimatologyPollution / Ozone Hole

Ageing SocietyMedicineBiology

EnergyPlasma Physics

Fuel Cells

Materials / Inf. TechSpintronics

Nano-science

Aerospace

AutomotiveVirtual power plant

Finance Multimedia

06/11/2012

3

HPC is a «key technology»

06/11/2012 Franco-British Workshop on Big Data in Science

Supercomputers : an indispensable tool to solve the most challenging problems via simulations

Access to world class computers : essential to be competitive in science and engineering

Providing competitive HPC services : a continuous endeavor

This has been acknowledged by leading industrial nations

→ Europe : PRACE

→ France : GENCI

A 200M$ effort by 6 agencies

Missions :

To implement a national HPC strategy in France and to provide the 3 national HPC academic centres with supercomputers

To contribute to the creation of the European HPC ecosystem

To promote numerical simulation and HPC in academia and industry

GENCI Grand Equipement National de Calcul Intensif

1 %

49 %

20 %

20 %

10 %

Chart Title

06/11/12 Franco-British Workshop on Big Data in Science

GENCI : powering the 3 national HPC centres

Franco-British Workshop on Big Data in Science

A 80x increase

in 5 years

Coordination and optimization of investments in HPC

Common allocation of HPC hours via call for proposals

CT Scientific Area

1 Environment

2 CFD, reactive & complex flows

3 Bio medical and Health

4 Astrophysics and geophysics

5 Theoretical and plasma physics

6 CS, algorithmic and mathematics

7 Molecular systems and biology

8 Quantum chemistry and molecular simulation

9 Physics, chemistry and materials

10 New and transverse applications

06/11/2012

Resources divided into scientific areas

MontpellierCINES

Bruyères-le-Châtel

Orsay IDRIS

TGCC

5

6

A huge effort for increasing French HPC capacities

Category FeaturesHybrid cluster BULL - 103 Teraflop/s

(+ 192 GPU Teraflop/s SP)Cluster of SMPFat nodes

IBM x3750M4 cluster – 233 Teraflop/s

MPP IBM BG/Q - 836 Teraflop/s

Cluster of SMPThin nodes

SGI Altix ICE - 267 Teraflop/s

Cluster of SMPThin, Hybrid and Fat nodes

BULL Bullx cluster2.0 Petaflops

06/11/2012 Franco-British Workshop on Big Data in Science

PRACE: a European Research Infrastructure (RI) & ESFRI list-item

PRACE RI is in operation since April 2010• PRACE AISBL created with 20 countries, head office in Brussels• Now 25 member countries

PRACE RI is providing services since august 2010• Now 6 Tier0 systems available • 4.3 billions core hours awarded to 159 projects through a single pan-European peer review

process

Funding secured for 2010-2015• 400 Million€ from France, Germany, Spain and Italy, provided as Tier0 services on TCO basis• 130 Million€ additional funding = 70 Million€ from EC FP7 preparatory and implementation projects

+ 60 Million€ from PRACE members : Technical, organizational and legal support for PRACE

• Prepared the creation of the AISBL as a legal entity• Established the PRACE brand• Provided extensive HPC Training • Deployed and evaluated promising architectures• Ported and petascaled applications

1st Council June 9, 2010

PRACE-3IP kick-off in Paris

06/11/2012

06/11/12 Franco-British Workshop on Big Data in Science 7

2012: PRACE is providing nearly 15 PFlop/s...

JUQUEEN: IBM BlueGene/Q at GCS partner FZJ (Forschungszentrum Jülich)

CURIE: Bull Bullx at GENCI partner CEA.

HERMIT: Cray at GCS partner HLRS

(High Performance Computing Center Stuttgart). SuperMUC: IBM

at GCS partner LRZ (Leibniz-Rechenzentrum)

FERMI: IBM BlueGene/Q at CINECA

Mare Nostrum: IBMat BSC

06/11/12 Franco-British Workshop on Big Data in Science8

06/11/2012

9

to face the tempest!The UPSCALE project aims to continue developing our climate modelling capability and goes for even higher global resolution, all the way to 12km, which is not even envisioned for Met Office global weather forecasting before 2015.

PRACE-1IP Kick-off meeting in

PRACE boosts Science

Credits: Prog. Pier Luigi Vidale, Univ. Reading, U.K. - Cray XE6 System Hermit in GCS@HLRS also in NATURE Climate Change July 2012

AWARD : 144M CPU HOURS

9

CURIE : the French PRACE supercomputer

CURIE, France’s commitment to PRACE, is overseen by GENCI Located in and operated by CEA DAM teams A modular and balanced architecture by

Cluster of SMP nodes with fat, thin and hybrid nodes Complementary to other PRACE Tier0 systems Fully available since March 8, 2012

Global peak performance of 2 PFlop/s

> 92 000 Intel cores, 360 TB memory,

15 PB Lustre @ 250 GB/s,120 racks, < 200 m2 - 2,5 MW

50 kms of cables

In honour of Marie Curie

06/11/12 Franco-British Workshop on Big Data in Science

11

Example of recent results on CURIE Understanding the evolution of the Universe (1/2)

Grand challenge conducted by Observatoire de Paris and the DEUS Consortium (http://www.deus-consortium.org)

Goal : perform 3 FULL Universe simulations, from Big Bang to nowdays using 3 different dark energy distributions Influence of the dark matter wrt evolution of the Universe Direcly linked with the 2011 Physics Nobel Prize Data will be used to feed next EU EUCLID telescope

Unprecedented HPC requirements >550 billions particles, 81923 mesh in a 21 h-1 Gpc box RAMSES code (CEA) and a dedicated workflow toolchain 76k cores, >300 TB of main memory Specific memory, MPI and parallel I/O optimisations

Franco-British Workshop on Big Data in Science 06/11/2012

12

Example of recent results on CURIE Understanding the evolution of the Universe (2/2)

WORLWIDE record finished 2 months ago First FULL Universe ΛCDM, LCDM and RPCDM simulations performed on

Curie thin nodes 3 runs for a total of 92 hours elapsed on 76 032 cores, last run lasted 29

hours without any failure WWOOUUAAHHH CURIE is very stable ! A strong need of substained I/O rate on the Lustre scratch fs

We have here a BigData problem A total of 10 PB of full data

(scratch and rough data) generated 4 PB of rough results after simulation

1.2 PB of refined data after post processing for the 3 dark energy simulations need to be made available to worldwide scientists !

Franco-British Workshop on Big Data in Science 06/11/2012

Explosion of computational dataAn another example from climatology

Evolution of the global climate 5th IPCC campaign , French production on a dedicated NEC SX9 : > 1TB/day Strong issues with storage, post processing and archive of data And the future is :

06/11/2012 Franco-British Workshop on Big Data in Science 13

CMIP5 CMIP6 CMIP7Year 2012 2017 2022Power factor 1 30 1000Npp 200 357 647Resolution [km] 100 56 31Number of mesh points [millions] 3,2 18,1 108,4Ensemble size 200 357 647Number of variables 800 1068 1439Interval of 3-dimensional output (hours) 6 4 3Years simulated 90000 120170 161898Storage density 0,00002 0,00002 0,00002Archive size (Pb) (atmosphere) 5,31 143,42 3766,99

One conclusion Data is exploding

Observational/experimental data Particle accelerators and detectors (LHC@CERN) Genome sequencer and personalized medecine Next gen satellites and (radio)telescopes Captors/sensors in weather forecast/climatology or

oil & gas Finance, insurance, …

Computational data Increase on HPC resources (PRACE = 15 PF in 2012) Increase of space and time resolution of models Multi-physics and multi scale simulations Rise of uncertainties quantification, ensemble simulations, …

With problems relative to Size of data (number and size of files), (un)structured data, format, ..) Uncertainties of data and fault tolerance Metadata issue Post processing (20% of time) Dissemination of refined data to worldwide communities during decades

That means :

To deploy PERENE and SUBSTAINABLE Research Infrastructures06/11/2012 Franco-British Workshop on Big Data in Science 14

Another conclusion People is aware about that !

On the hardware and system software side Multi level storage with new I/O devices : mix of flash based memory (SSD, PCM, …) with hard drives Asynchronous I/O and Active I/O (servers embedded into I/O controllers) Next generation of parallel file system (Lustre, GPFS, Xyratec, …) Flops will be “almost free” -> post processing at the same time as computation

A lot of European projects and R&D initiative PRACE implementation projects : data management, remote viz, portals, … EUDAT : data services between computing/data centers and end users communities EESI2 : cartography about Exascale R&D efforts French INRIA Blobseer R&D project, …

But a lot of applications will need to be rewritten/adapted The complete I/O strategy need to be re thinked New methods for data analysis/exploration are needed (MapReduce, Hadoop, NOSQL, …?) Rough data will stay in computing/data center and ONLY refined data will go out Networks bandwidth will need to increase Use of remote visualisation

06/11/2012 Franco-British Workshop on Big Data in Science 15

16

HPC : en route to international synergies

HPC enables scientific discoveries and innovation for both research and industry

We mustn’t follow the trends. We must anticipate them ! To face future societal or industrial challenges To prepare users for future parallel architectures and applications To increase involvement of scientists or engineers in these techniques

Franco-British Workshop on Big Data in Science

Global European HPC ecosystem integration

06/11/2012