Presented by

Scientific Computing Group

Ricky A. Kendall

Scientific Computing

Center for Computational Sciences

2 Kendall_SciComp_0611

Cray Supercomputing

Center for ExcellenceJ. LevesqueT. Darland

L. DeRose4

D.Kiefer4

J. Larkin4

N. Wichmann4

Scientific ComputingR. Kendall

L. Rael

S. Ahern## V. Lynch5

S. Alam5 B. MesserE. Apra5 R. MillsR. Barrett5 G.Ostrouchov5

R. Barretto2 R. SankaranD. Banks3 A. TharringtonJ. Daniel R. ToedteM. Fahey T. WhiteS. Klasky#

J. Kuehn5

#End-to-End Solutions Lead## Viz Task Lead

User Assistanceand Outreach

J. WhiteL. Rael

Y. DingD. FrederickC. FusonC. Halloy3

S. HempflingM. HenleyJ. HinesS. Jones6

B. RenaudB. WhittenL. Williams5

K. Wong3

High-PerformanceComputing Operations

A. BakerL. Gregg*

M. Bast J. Lothian J. Becklehimer4 D. Maxwell##

J. Breazeale6 M. McNamara4

J. Brown6 J. Miller6

A. Enger4 G. Phipps, Jr.6

C. England G. PikeJ. Evanko4 V. Rothe#

M. Griffith S. Shpanskiy V. Hazlewood5 D. VasilT. Jones C. Willis4

C. Leach6 T. Wilson6

D. Londo4 S. White

#Team Lead, Computer Ops##Technical Coordinator

Technology IntegrationS. CanonL. Gregg*

T. BarronS. Carter##

K. MatneyM. MinichS. OralD. SteinertF. WangV. WhiteW. Yu5

## Networking Lead

Site PreparationK. Dempsey

Hardware AcquisitionA. Bland*

Test and AcceptanceDevelopmentS. Canon

CommissioningA. Baker

Project ManagementD. Hudson5

Project R & DA. Geist

Cray Proj. DirectorK. Kafka4

Operations CouncilA. Stewart, Quality AssuranceT. Jones, Cyber SecurityM. Dobbs, Facility Mgmt.W. McCrosky, Finance OfficerW. Besancenez, ProcurementM. Palermo, HR Mgr.K. Johnson, RecruitingR. Toedte, Safety & HealthN. Wright, Org. Mgmt. Specialist

Center for Computational Sciences/

Leadership Computing FacilityA. Bland, Director

L. Gregg, Division Secretary

Deputy Project Director

K. Boudwin

Director for ScienceD. Kothe

Director of OperationsD. Kothe*

Chief Technology OfficerA. Geist

LCF System ArchitectS. Poole

Advisory CommitteeJ. Bernholc S. KarinT. Dunning D. KeyesJ. Dongarra D. ReedK. Droegemeier T. SterlingJ. Hack W. Washington

1Student2Post Graduate3JICS4Cray, Inc.5Matrixed6Subcontract*InterimNew position within ORNLNew hire at ORNL

3 Kendall_SciComp_0611

Scientific computingTasks and deliverables Project partnerships in science, visualization, workflow, and data analytics

Level of integration is dependent upon the project High-level user assistance

Main staff function. Whatever it takes! Algorithmic development at scale

For scientific codes, visualization and workflow Application and library code optimization and scaling

How to use the LCF resources for science User voice within LCF planning exercises

Users are the focus Requirements gathering, documentation

What’s next for our users and our center Exploiting parallel I/O & other technologies in applications

Specific focus for targeted applications Acquiring & deploying viz and end-to-end systems

Infrastructure building Benchmarking and evaluation exercises with Vendor interaction

Learning how to best use current and new LCF resources

4 Kendall_SciComp_0611

Partnership with projects on LCF resources

Porting/TuningLibrary utilizationCompiler flags

OptimizationsChoice of Tools

Code Designand

algorithmic developments

@scale

Shared R&D Staff

Pilot End StationsProjects

Le

vel o

f in

teg

rati

on

5 Kendall_SciComp_0611

High-level user assistance

ProjectProject

Liaison End-to-End

Visualization

Compute

6 Kendall_SciComp_0611

Algorithmic and programming model development at scale Complex Tradeoffs for applications

Usually among storage, computation and communication

Redesign new methods Fixed grid to AMR grids

Programming model changes Message passing to one-sided model transformation

7 Kendall_SciComp_0611

Application and library optimization and scaling Applications for the LCF systems

Catamount issues for Apps Dual Core/Single core settings Compiler flags for optimal performance (Application/Library specific) Modifications for scaling

Libraries Built and kept up-to-date HDF5, PETSc, FFTW, etc.

Application

HDF5

PETSc

0 1024 2048 3072 4096 5120 6144 7168 8192

Number of Cores

0

1024

20483072

4096

5120

6144

7168

8192

Before

After

Ideal

8 Kendall_SciComp_0611

Requirements gathering and documentation Participate in Applications Requirements Council (ARC) Determine requirements for applications on the machine

Algorithmic Programming models Architectural Infrastructure

Develop ties to applications areas not currently at scale on the machine Work with developers to understand application properties

Memory footprint Algorithmic scalability Potential for future resources

Collela’s 7 Dwarfs Anaylsis The 7 algorithm types are scattered broadly among science domains, with no one

particular algorithm being ubiquitous and no one algorithm going unused; Structured grids and dense linear algebra algorithms are the most widely used

algorithms (by over half of the representative codes), hence system attributes such as node peak flops and memory capacity, memory latency, and interconnect latency will be important;

Particle-based and Monte Carlo algorithms, which have similar properties from a system standpoint, are also broadly used, and can tax interconnect latency and in some cases node memory capacity, depending upon implementation and usage.

9 Kendall_SciComp_0611

Collela’s 7 Dwarfs analysisScienceDomain Code Structured

GridUnstructured

Grids FFTDenseLinear

Algebra

SparseLinear

AlgebraParticle Monte

Carlo

Accelerator Physics T3P X X

Astrophysics CHIMERA X X X X

Astrophysics FLASH X X X X X

Biology LAMMPS X X

Chemistry MADNESS X X

Chemistry NWCHM X X

Chemistry OReTran X X X

Climate CAM X X X

Climate POP/CICE X X X

Combustion S3D X

Fusion AORSA X X X

Fusion GTC X X X X

Geophysics PFLOTRAN X X X

Materials Science QMC/DCA X X

Materials Science QBOX X X X

Nanoscience CASINO X X X

Nanoscience LSMS X X

Nuclear Energy NEWTRNX X X X

Nuclear Physics CCSD X

QCD MILC X X

10 Kendall_SciComp_0611

System attribute priority per science domain

System Attribute Climate Astrophysics Fusion Chemistry Combustion Accelerator Physics Biology Materials

Science

Node Peak Flops

Mean Time to Interrupt (MTTI)

WAN Network Bandwidth

Node Memory Capacity

Local Storage Capacity

Archival Storage Capacity

Memory Latency

Interconnect Bandwidth

Disk Latency

Interconnect Bandwidth

Memory Bandwidth

Disk Bandwidth

Highest Medium Lowest

11 Kendall_SciComp_0611

Contact

Ricky A. KendallScientific ComputingCenter for Computational Sciences(865) 576-6905kendallra@ornl.gov

11 Presenter_date