Presented by

The Evolving Environment at NCCS

Vickie Lynch

2

Compute Node Linux

• Jaguar will be down November 1-5 to change from Catamount to Compute Node Linux (CNL).• CNL is a stripped-down version of Linux on the compute nodes. • Try the system running CNL that is accessible to all jaguar users: jaguarcnl.ccs.ornl.gov. • Jaguarcnl contains 2,976 dual core XT3 compute nodes each running CNL.• Eight additional cabinets will be removed from jaguar and added to jaguarcnl increasing it from 32 cabinets to 40 cabinets (7440 cores).

3

Changes for Compute Node Linux

• Currently only the /tmp/work/$USER lustre directory is visible to compute nodes. • There are several important differences between Catamount and CNL, and users are encouraged to view the details of those differences at http://info.nccs.gov/resources/jaguar.• Due to the differences between CNL and Catamount, users should recompile on the CNL system prior to running.• The batch option “-lsize” specifies cores under CNL.• aprun replaces yod as the job launcher under CNL.• Each user is currently allowed only one job in a running state.• The maximum walltime limit is 12 hours.

4

CNL batch example

#PBS -A STF006#PBS -N test#PBS -j oe#PBS -q debug#PBS -l walltime=0:50:00,size=4

cd /tmp/work/vel/cqlptrexport PGPLOT_FONT=/tmp/work/vel/pgplot_xt3/grfont.dataprun -n4 ./cqlp

5

Pathscale compiler

The pathscale compiler is now available on jaguar and jaguarcnl.

Before compiling: module swap PrgEnv-pgi PrgEnv-pathscale

Use ftn –O3 instead of ftn –fast for compiler optimization

Use -Wl,--allow-multiple-definition for the ldr flag if you get too many warnings about multiple definitions

Use a SuperLU library compiled with pathscale: /spin/home/vel/lib/SuperLU_DIST/libsuperlu.a

6

Quad-core Upgrade

• Jaguar shutdown for upgrade: December 4, 2007- ?• During this time users will have access to the remaining 40 XT cabinet transition system (7440 cores).• Jaguar will be upgraded to 250 teraflops. • The dual-core processors will be replaced with quad-core processors.• Memory will be doubled. • This transition will involve significant down time.• Consider using OpenMP across the cores of a processor

7

Petaflop system

• A petaflop system will be installed at the NCCS in 2008.• The LCF Petaflop system will also be quad-core.• It will come with a new interconnect that should enable better scalability than the current XT4. • This new interconnect is called Gemini and it will allow you to effectively perform Co-Array Fortran, UPC and Global Arrays.

8

Summary