6-prgenv shaheen-neser fall 2019 · 2019-09-11 · use them exactly like you would use the original...

Overview on Shaheen Cray Programming Environment and TIPS

Bilel [email protected]

Computational ScientistKAUST Supercomputing Lab

@mnoukhiya @KAUST_HPC

Bilel Hadri 1

Shaheen Programming Environment

Bilel Hadri – Shaheen 101 TIPS on PrgEnv 2

Shaheen 2 Cray XC40 flyer

• Get started with the flyer


https://hpc.kaust.link/Shaheen_flyer

https://hpc.kaust.link/Shaheen_flyer

Shaheen 2 Cray XC40

• Three programming environments are supported • Cray CCE (default), Intel and GNU

• The programming environment modules, “PrgEnv-intel”, “PrgEnv-gnu”, and “PrgEnv-cray”, which include the compilers and matching libraries, provide user friendly programming environments – Just module swap to PrgEnv-intel or PrgEnv-gnu– Compiler wrappers for serial and parallel

• ftn for Fortran code/ cc for C code / CC for C++ code

• Using compiler wrappers provided by Cray, ftn, cc and CC for Fortran, C and C++ respectively, the header and library paths and libraries can be included in the compile/link line automatically.

– Do not purge modules


Use them exactly like you would use the original compiler, e.g. To compile.


Compiler Driver Wrappers

ftn, cc, and CC, are not Cray compilers; they invoke the Intel, GNU, or Cray compilers under the hood, depending on the loaded programming environment module (PrgEnv-xxx)

#to use Cray compilers ftn –O3 myprog.f90 # Fortran cc –O3 myprog.c # for C CC –O3 myC++code.C # for C++

#to use Interl compilersmodule swap PrgEnv-cray PrgEnv-intelftn –O3 myprog.f90 # Fortran cc –O3 myprog.c # for C CC –O3 myC++code.C # for C++

#to use GNU compilersmodule swap PrgEnv-intel PrgEnv-gnuftn –O3 myprog.f90 # Fortran cc –O3 myprog.c # for C CC –O3 myC++code.C # for C++

Compilers• Use ftn, cc, and CC to compile instead of the underlying native compilers (ifort, icc, icpc,

gfortran, gcc, g++..)T

• Compilers wrappers link statically by default – Recommended for performance at scale

• Set environment variable CRAYPE_LINK_TYPE=dynamic to link dynamically– may take a some time to load shared libraries at runtime

• Compiler wrappers do cross compilation– Compiling on login nodes to run on compute nodes – One may run into trouble with GNU automake or cmake. Add the specifier –

host=x86_64-unknown-linux-gnu for the configure tool .


Compilers• Intel - better chance of getting processor specific optimizations• Cray compiler – many new features and optimizations, especially with Fortran;

useful tools like reveal work with Cray compiler only • GNU - widely used by open software

• More information from compilers options on the man page


PrgEnv Description Real Compilers

PrgEnv-cray Cray Compilation Environment crayftn, craycc, crayCCPrgEnv-intel Intel Composer Suite ifort, icc, icpcPrgEnv-gnu GNU Compiler Collection gfortran, gcc, g++

OpenMP

� OpenMP is supported by all of the PrgEnvs.� CCE (PrgEnv-cray) recognizes and interprets OpenMP directives by

default. If you have OpenMP directives in your application but do not want to use them, disable OpenMP recognition with –hnoopm.

PrgEnv Enable OpenMP Disable OpenMPPrgEnv-cray -homp -hnoompPrgEnv-intel -qopenmpPrgEnv-gnu -fopenmp

-qopenmp is the replacement option for -openmp, which is deprecated.

Bilel Hadri – Shaheen 101 TIPS on PrgEnv

Cray Scientific Libraries

• Compiler wrappers takes care of not only the compiler but also libs like BLAS, SCALAPACK, MPI,..

• Cray Scientific Libraries package, LibSci, is a collection of numerical routines optimized for best performance on Cray systems.– LibSci is loaded by default and this is for all programming environment– No user flags or options are required for compiling or linking.– LibSci library collection contains; BLAS, BLACS, LAPACK, ScaLAPACK, IRT, CRAFFT,

CASE, FFT, FFTW2, FFTW3



• FFTW: Cray’s main FFT library is FFTW from MIT with some additional optimizations for Cray hardware

• Cray PETSc (with CASK – Cray Adaptive Sparse Kernels)

• Cray Trilinos (with CASK – Cray Adaptive Sparse Kernels)

• Just need to module load and compile your code

• No need to put the whole path of the libraries



• Cray TPSL (Third Party Scientific Libraries) contains a collection of outside mathematical libraries that can be used with PETSc and Trilinos– The TPSL increase the flexibility of PETSc and Trilinos by providing users with multiple

options for solving problems in dense and sparse linear algebra– The cray-tpsl module is automatically loaded when PETSc or Trilinos is loaded. The

libraries included are MUMPs, SuperLU, SuperLU_dist, ParMetis, Hypre, Sundials, and Scotch.

• Intel MKL: The Intel Math Kernel libraries is an alternative to LibSci– Features tuned performance for Intel CPUs as well– Linking is quite complicated but with Intel compilers (PrgEnv-intel) is usually

straightforward using the Intel Link advisor– http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor


http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor

Over 100s of libraries and packages installed• www.hpc.kaust.edu.sa/app6


Modules available on Shaheen

13Bilel Hadri – Shaheen 101 TIPS on PrgEnv

Find your app/library• module avail

• module –S name

• Get the path: module show name


module show gsl-------------------------------------------------------------------/sw/xc40cle6/modulefiles/gsl/2.4:

module-whatis The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers.prepend-path PATH /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/binprepend-path MANPATH /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/share/mansetenv GSL_DIR /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0setenv GSL_INC -I /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/includesetenv GSL_LIB -L /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/libprepend-path PE_PRODUCT_LIST NICS_GSLsetenv NICS_GSL_INCLUDE_OPTS -I /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/includesetenv NICS_GSL_POST_LINK_OPTS -L /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/libprepend-path PKG_CONFIG_PATH /sw/xc40cle6/gsl/2.4/cle6.05_gnu7.2.0/lib/pkgconfigprepend-path PE_PKGCONFIG_PRODUCTS NICS_GSLprepend-path PE_PKGCONFIG_LIBS gsl------------------------------------------------------------------

How to use ?• Several flavor or R and python

– R/3.5.1, cray-R/3.4.2– cray-python, python/2.7.14, python/3.7.3, python/miniconda,….


#!/bin/bash#SBATCH --job-name=run#SBATCH --output=run%j.out#SBATCH --error=run%j.err#SBATCH --nodes=1#SBATCH --time=00:15:00

module load r/3.5.1srun --hint=nomultithread R --no-save CMD BATCH ./rscript.sh

=====rscript.sh

require(Rcpp)cppFunction('int fibonacci(const int x) {

if (x == 0) return(0);if (x == 1) return(1);return (fibonacci(x - 1)) + fibonacci(x - 2);}'

)fibonacci(15)

Modules snapshot• Avoid editing the .bashrc. Use module snapshot

• Do not purge module

• The basic idea is that you take a snapshot of your environment and save it with a name. You can then restore it whenever you need. You can have many snapshot “files” saved too.

• So, if for example you work with two (or more) applications that require different environment setup, you can have a snapshot for each one of them, and whenever you need to work on one of them, you just restore the corresponding snapshot.

• The usage is:– module snapshot [-f | --force] <filename>– module restore <filename>


My Account• Check your account:

– Type groups to get all the allocation projects

• Check the allocation– sb kxxxx

• Check the usage allocation between users– sb_users kxxxx

• How is it charge? 32 cores x number of nodes x time_of_execution


Project kxxx: Project DescriptionPI: John Doe

Allocations Core hours--------------------------2015-08-26 2000000--------------------------Expiry on 2020-01-01--------------------------Allocated 2000000Shaheen 215563Neser 61--------------------------Balance 1784375--------------------------Project kxxx: Project Description

PI: John Doe

User Name Core hours------------ ---------------------------------------- ----------user1 John Doe 205279user2 Jane Doe 10284------------ ---------------------------------------- ----------Charged 215563------------ ---------------------------------------- ----------

Debugging• Program errors ( crashes, hangs, wrong results)

• How to identify and fix them?– Print

• difficult, tedious and time-consuming– Debuggers tools

• Compile only once, execute, then check variables; visualize and get stats– Many tools are available on Shaheen

• ARM (DDT), Totalview, ATP,STAT, valgrind, gdb4hpc• Compile with –g option and load the module and use the GUI to run step by step your

code


ftn -g -o mycode mycode.f90salloc -N 1 -t 30:00 --partition=debug module load allinea-forgeddt ./mycode

Neser


Neser• Access granted with justified needs• Compiler available:– Cray, Intel and GNU– PrgEnv different than Shaheen

• For serial, – use GNU (gcc/gfortran)– Use Intel ( ifort/icc)

• For parallel code– mpicc, mpif90 are the compiler for C and Fortran with GNU– mpiicc, mpiifort are the compiler for C and Fortran with Intel – ftn, cc with Cray compiler

• More info– https://www.hpc.kaust.edu.sa/neser


https://www.hpc.kaust.edu.sa/neser

Neser• File Systems

– Neser is connected directly to Shaheen Lustre Parallel filesystem• Running Jobs: Two queues partitions:

– default workq ( similar to Shaheen)– tesla for GPU node

• FAQs– To launch jobs inside your job scripts, you can use srun and mpirun. Make

sure you use the same mpirun launcher from the MPI library used. – The compute nodes are exclusive, meaning that even when all the

resources within a node are not utilized by a given job, another job will not have access to these resources.

– The jobs are limited to a maximum of 4 nodes with up to 2 jobs running. The remaining jobs will stay in the queue.

– To access large memory node, specify in the job script #SBATCH --mem=200G or larger


Tips and summary


Best Practices for Performance (1) • Check the system details thoroughly

– Never assume ! ( Login nodes different than compute)

• Choose a compiler and MPI to build your application – All are not same ! Rely on the latest versions

• Start with some basic compiler flags and try additional flags one at a time – Optimization is incremental ! Benchmarking and testing is a must

• Use the built-in/optimized libraries and tools to save time and improve performance – Libraries Tools are your friends !– By doing the different steps of optimizations:– You can achieve huge speedup ( o(10x ) and more ) by using Optimized Mathematics

libraries (MKL)– Optimizing the cache and memory

– To get more memory ( use more nodes and less process/core per node)


Best Practices for Performance (2)

• Don’t Reinvent the wheel

• Test your application at every level to arrive at an optimized code – Check correctness !

• Customize your runtime environment to achieve desired goals – Play with the number of threads, memory and core affinity

• Profile and adjust optimization and runtime environments accordingly– Start with small and short runs

• READ the manual and/or attend the tutorials/workshops !

• Visit https://www.hpc.kaust.edu.sa/trainingBilel Hadri – Shaheen 101 TIPS on PrgEnv 24

KSL CS Team• Shaheen is a shared resource

– Be kind to your neighbor users – Don’t run on login.

• Use adequately your allocation– Check your usage, sb kxxxx ,sb_user kxxxx– Prepare in advance the project proposal

• Security– Don’t share your account with others.

• Need help: send a ticket [email protected]– Help us to help you :D Provide details:

• Which HPC system? • What is the problem? When did it happen? What modules were loaded? How

did you try to fix or work around it? Send the error and job script.• Acknowledge KAUST Supercomputing Lab and HPC resources used in your

papers.


mailto:[email protected]

Shaheen Clinic• When: This afternoon 2:00pm – 4:30pm• Where: Building 1, Level 0, Room: 115 ( seaside)

The event will be followed by the Shaheen Clinic, a “talk to the experts” session when you can have face to face interaction with the trainers and the Shaheen support staff.

• Objective:• One-to-One consultancy with Shaheen support staff to solve your problems on Shaheen

and Neser• Provide suggestions and advise to accelerate your application execution• Discuss various challenges or issues during your job executions• Getting more familiar with Shaheen environment: e.g. job submissions, queues,

architectures…• Collaborative work with your projects• If the 30-min scheduled time is not sufficient to provide solution or unable to address the

issue, a follow-up meeting will be scheduled and/or RT-Ticket will be created• If you have any questions, please contact us at [email protected]

Bilel Hadri – Shaheen Workshop Oct. 2018 26

mailto:[email protected]


THANKS !

[email protected]@mnoukhiya @KAUST_HPC

6-prgenv shaheen-neser fall 2019 · 2019-09-11 · use them exactly like you would use the original...

Documents